This is an archive of the discontinued LLVM Phabricator instance.

Add Cache Pruning support
ClosedPublic

Authored by mehdi_amini on Mar 23 2016, 5:56 PM.

Details

Summary

Incremental LTO will usea cache to store object files.
This patch handles the pruning part of the cache, exposing
a few knobs:

  • Pruning interval: the implementation keeps a "timestamp" file in the directory and will scan it only after a given interval since the last modification of the timestamp file. This is for performance purpose, we don't want to scan continuously the folder.
  • Entry expiration: this is the time after which a file that hasn't been used is remove from the cache.
  • Maximum size: expressed in percentage of the available disk space, it helps to avoid that we blow up the disk space.

Diff Detail

Event Timeline

mehdi_amini retitled this revision from to Add Cache Pruning support.
mehdi_amini updated this object.
mehdi_amini added a reviewer: dexonsmith.
mehdi_amini added a subscriber: llvm-commits.
silvas added a subscriber: silvas.Mar 23 2016, 7:27 PM
silvas added inline comments.
lib/Support/CachePruning.cpp
54

Can you use sys::fs::status? If not, can you add a comment about why not?

bruno added a subscriber: bruno.Mar 24 2016, 9:51 AM

It might be worth looking at how clang implements pruning for old modules. The option in question is -fmodules-prune-interval.

This is copy-pasted-adapted from module pruning. I plan to refactor module pruning to use this at some point.

What about:

sys::fs::file_status Status;
if (sys::fs::status(LHS, Status))
  return false;
auto LastModTime = Status.getLastModificationTime();

This will not help with the access to StatBuf.st_atime.

Not that it should block your progress here, but at some point sys::fs::file_status should be augmented with this info. Can you add a FIXME above the ::stat call and mention that?

include/llvm/Support/CachePruning.h
2

nanage => manage

lib/Support/CachePruning.cpp
2

Director -> Directory

I plan on adding this to sys::fs::file_status indeed, hopefully it is not too hard and this patch can wait for it.

Use llvm::sys:fs::status() instead of stats.

mehdi_amini removed a subscriber: silvas.

Use llvm::sys:fs:disk_space() introduced in D18467

tejohnson added inline comments.
lib/Support/CachePruning.cpp
11

Stale description.

103

When would that happen? Not sure what "nothing interesting there" means or if this is an error?

mehdi_amini added inline comments.Mar 25 2016, 2:08 PM
lib/Support/CachePruning.cpp
103

Just means we can't read this file I guess? (copy pasted from clang module pruning)

silvas added inline comments.Mar 29 2016, 3:34 PM
lib/Support/CachePruning.cpp
23

Do you still need these includes? I can't find anything using them after staring for a while at the code.

mehdi_amini added inline comments.Mar 29 2016, 3:59 PM
lib/Support/CachePruning.cpp
23

Oh yeah those aren't needed anymore after the refactoring in sys::filesystem, will cleanup.

Remove obsolete #include

mehdi_amini marked 5 inline comments as done.

Update header description

Should be ok now

Anything else to fix or can we move on and fix whatever comes up post-commit?

mehdi_amini accepted this revision.Apr 2 2016, 10:38 AM
mehdi_amini added a reviewer: mehdi_amini.
This revision is now accepted and ready to land.Apr 2 2016, 10:38 AM

r265209
I'll address any comment made post-commit on the list.

mehdi_amini closed this revision.Apr 11 2016, 12:55 PM
chapuni added inline comments.
lib/Support/CachePruning.cpp
44

Don't use errno. Fixed in r269565.

Sure! We should not peek errno unless the context that calls C libraries.