This is an archive of the discontinued LLVM Phabricator instance.

[CachePruning] Clarify the per-directory entry limit on Linux ext4.
ClosedPublic

Authored by MaskRay on Apr 11 2018, 4:49 PM.

Details

Summary

508 root node entries (root_limit)
510 internal node entries (node_limit)

For a filename with 40 bytes, its sizeof(ext4_dir_entry_2) = 48, a linear directory can contain at most floor(4096/48)=85 of them.
The real per-directory entry limit should be 508*510*85 = 22021800
The limit varies with the average length of filenames.

However, the Linux ext4 code does not try rebalancing the htree, so we will not be able to create filenames in a full leaf node. This is demonstrated with the following example, certain filenames cannot be used while others can:

% touch d/0000000000000000000000000000000000816a6f
touch: cannot touch 'd/0000000000000000000000000000000000816a6f': No
space left on device
% touch d/0000000000000000000000000000000000816a70
# succeeded

Diff Detail

Repository
rL LLVM

Event Timeline

MaskRay created this revision.Apr 11 2018, 4:49 PM
pcc added inline comments.Apr 11 2018, 5:33 PM
include/llvm/Support/CachePruning.h
56 ↗(On Diff #142099)

Update this bit.

57 ↗(On Diff #142099)

My first instinct is not to add this much detail, but I guess it might dissuade the next person who comes along to try to "correct" this comment.

MaskRay added inline comments.Apr 11 2018, 7:04 PM
include/llvm/Support/CachePruning.h
56 ↗(On Diff #142099)

Change to what?

57 ↗(On Diff #142099)

Yeah I did not want to write the comment but you asked me to do so..

pcc added inline comments.Apr 11 2018, 7:17 PM
include/llvm/Support/CachePruning.h
56 ↗(On Diff #142099)

The second sentence does not follow from the first because the second is about files in a directory and the first is about total files. I might just say something like "some file systems have a limit on the total number of files, and some also limit the number of files per directory, such as ext4, which has a default limit of about 10M."

57 ↗(On Diff #142099)

If you write something like the above, it should probably be enough.

pcc accepted this revision.Apr 12 2018, 3:25 PM

Close enough.

include/llvm/Support/CachePruning.h
55 ↗(On Diff #142275)

Remove sentence as it is duplicated below.

This revision is now accepted and ready to land.Apr 12 2018, 3:25 PM
MaskRay marked 7 inline comments as done.Apr 12 2018, 3:30 PM
This revision was automatically updated to reflect the committed changes.