Can you Ignore duplicate inodes?

[bootandy]

Firstly good job, pdu is really fast.

If you create a hard symlink like this: ln file hard_link it appears twice in pdu where as it is only taking up one 'space' on disk. Both file and hard_link have the same Inode - Hence I only 'count' the disk space for one of them. Therefore in dust I chose to ignore the duplicate file as removing either 'file' or 'hard_link' wouldn't recover disk space until they were both removed.

I'm curious to see how you would solve the above problem. I have tried and using channels (slow) and tried locks round a shared hashmap (even slower)

I'm in the middle of a dust re-write to see if I can make it faster based on some of your ideas. :-).

You can see me doing the check for duplicate inodes here:
https://github.com/bootandy/dust/blob/master/src/utils/mod.rs#L253

[KSXGitHub]

Firstly, thank you for creating dust, for despite its limitations, its way of representing data inspired me to create this tool.

I am well aware of pdu's limitations.

There are 2 main reasons why I decided not caring about hardlinks:

• There is no distinguishing between a hardlink and a main inode. I could make some arbitrary rules (such as the first inode is the main inode) but the parallel nature of pdu makes testing impossible.
• I haven't found a way to accurately demonstrate the existence of hardlinks other than assume 1 to be real and others to be fake (which is not completely true). Therefore, I decided that it is better to treat all hardlinks as equal.

I'm curious to see how you would solve the above problem

I'm in the middle of a dust re-write to see if I can make it faster based on some of your ideas. :-).

Not caring about hardlinks was a design choice, not a trade-off choice between speed and utility. But if, hypothetically, I want to detect hardlinks, I wouldn't aggregate all inodes into one HashSet because it would require Mutex lock, which would render parallelization utterly pointless. Instead, what I would turn DataTree::data into a Vec of inodes/sizes pair (by using rayon's into_par_iter().map().collect()), then aggregate and filter them later during visualization (which is single-threaded). Of course, this advice would only work with the data model that I chose (DataTree is a hierarchical tree, not a pseudo-iterator).

When you finish your rewrite, can you notify me? I want to update the benchmarks.

[KSXGitHub]

@bootandy 🎉 Congratulation on the new release! It seems that the new implementation improved a lot as it rises from second-last place to second place in my benchmark.