[ThinLTO]Expose cache entry expiration time option in llvm-lto and fix a test

[jhenderson]

Summary

Whilst working in the area, I noticed that two cases in a ThinLTO test were passing for the wrong reasons, since rL340374. The tests were supposed to be testing that files were being pruned due to the cache size, but they were in fact being pruned because they were older than the default expiration period of 1 week.

This change fixes the tests by explicitly setting the expiration time to the maximum value. This required the option to be exposed in llvm-lto.

Assigning all files in the cache a similar time, it is possible to see that the newest files are still being kept, and that we aren't passing for the wrong reason again. In the event that the entry expiration were to expire for them, then the test would start failing, because these files would be removed too.

[inglorion]

Thanks! For the case where you don't want the files to expire, can you instead set the expiration age to 0? That should stop the pruner from removing them.

[jhenderson]

In D51992#1235574, @inglorion wrote:

Thanks! For the case where you don't want the files to expire, can you instead set the expiration age to 0? That should stop the pruner from removing them.

Unfortunately, it's not possible to disable expiration age explicitly using llvm-lto. The interface it uses ignores attempts to set any of the cache options to 0, thus making it impossible to disable them completely if they are already set (and the expiration age is set by default). I don't know why the interface is this way, but it was a deliberate decision made some time ago.

[jhenderson]

Ping!

[jhenderson]

Ping!

[rnk]

lgtm. llvm-lto is a testing tool, so it makes sense to expose whatever options the underlying LTO interface exposes.

test/ThinLTO/X86/cache.ll

 ; Only 8B and 76B files should stay after pruning.
 ; RUN: %python -c "with open(r'%t.cache/llvmcache-foo-100k', 'w') as file: file.truncate(102400)"
 ; RUN: touch -t 198002011200 %t.cache/llvmcache-foo-100k
 ; RUN: %python -c "with open(r'%t.cache/llvmcache-foo-16', 'w') as file: file.truncate(16)"
 ; RUN: touch -t 198002021200 %t.cache/llvmcache-foo-16
 ; RUN: %python -c "with open(r'%t.cache/llvmcache-foo-77k', 'w') as file: file.truncate(78848)"
 ; RUN: touch -t 198002031200 %t.cache/llvmcache-foo-77k
 ; RUN: %python -c "with open(r'%t.cache/llvmcache-foo-8', 'w') as file: file.truncate(8)"
+; RUN: touch -t 198002041200 %t.cache/llvmcache-foo-8
 ; RUN: %python -c "with open(r'%t.cache/llvmcache-foo-76', 'w') as file: file.truncate(76)"
-; RUN: llvm-lto -thinlto-action=run -exported-symbol=globalfunc %t2.bc %t.bc -thinlto-cache-dir %t.cache --thinlto-cache-max-size-bytes 78847
+; RUN: touch -t 198002051200 %t.cache/llvmcache-foo-76
+; RUN: llvm-lto -thinlto-action=run -exported-symbol=globalfunc %t2.bc %t.bc -thinlto-cache-dir %t.cache --thinlto-cache-max-size-bytes 78847 --thinlto-cache-entry-expiration 4294967295
 ; RUN: ls %t.cache/llvmcache-foo-8
 ; RUN: ls %t.cache/llvmcache-foo-76
 ; RUN: not ls %t.cache/llvmcache-foo-16
 ; RUN: not ls %t.cache/llvmcache-foo-100k
 ; RUN: not ls %t.cache/llvmcache-foo-77k
 
 ; Verify that specifying max number of files in the cache directory prunes
 ; it to this amount, removing the oldest files first.
 ; RUN: rm -Rf %t.cache && mkdir %t.cache
 ; Create cache files with different sizes.
 ; Only 75B and 76B files should stay after pruning.
 ; RUN: %python -c "print(' ' * 1023)" > %t.cache/llvmcache-foo-1023
 ; RUN: touch -t 198002011200 %t.cache/llvmcache-foo-1023
 ; RUN: %python -c "print(' ' * 15)" > %t.cache/llvmcache-foo-15
 ; RUN: touch -t 198002021200 %t.cache/llvmcache-foo-15
 ; RUN: %python -c "print(' ' * 7)" > %t.cache/llvmcache-foo-7
 ; RUN: touch -t 198002031200 %t.cache/llvmcache-foo-7
 ; RUN: %python -c "print(' ' * 75)" > %t.cache/llvmcache-foo-75
+; RUN: touch -t 198002041200 %t.cache/llvmcache-foo-75
 ; RUN: %python -c "print(' ' * 76)" > %t.cache/llvmcache-foo-76
-; RUN: llvm-lto -thinlto-action=run -exported-symbol=globalfunc %t2.bc %t.bc -thinlto-cache-dir %t.cache --thinlto-cache-max-size-files 4
+; RUN: touch -t 198002051200 %t.cache/llvmcache-foo-76
+; RUN: llvm-lto -thinlto-action=run -exported-symbol=globalfunc %t2.bc %t.bc -thinlto-cache-dir %t.cache --thinlto-cache-max-size-files 4 --thinlto-cache-entry-expiration 4294967295
 ; RUN: ls %t.cache/llvmcache-foo-75
 ; RUN: ls %t.cache/llvmcache-foo-76
 ; RUN: not ls %t.cache/llvmcache-foo-15
 ; RUN: not ls %t.cache/llvmcache-foo-1024
 ; RUN: not ls %t.cache/llvmcache-foo-7
 
 target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-apple-macosx10.11.0"
 
 define void @globalfunc() #0 {
 entry:
   ret void
 }

tools/llvm-lto/llvm-lto.cpp

 static cl::opt<int>
     ThinLTOCacheMaxSizeBytes("thinlto-cache-max-size-bytes",
     cl::desc("Set ThinLTO cache pruning directory maximum size in bytes."));
 
 static cl::opt<int>
     ThinLTOCacheMaxSizeFiles("thinlto-cache-max-size-files", cl::init(1000000),
     cl::desc("Set ThinLTO cache pruning directory maximum number of files."));
 
+static cl::opt<unsigned>
+    ThinLTOCacheEntryExpiration("thinlto-cache-entry-expiration", cl::init(604800) /* 1w */,
+    cl::desc("Set ThinLTO cache entry expiration time."));
+
 static cl::opt<std::string> ThinLTOSaveTempsPrefix(
     "thinlto-save-temps",
     cl::desc("Save ThinLTO temp files using filenames created by adding "
              "suffixes to the given file path prefix."));
 
 static cl::opt<std::string> ThinLTOGeneratedObjectsDir(
     "thinlto-save-objects",
     cl::desc("Save ThinLTO generated object files using filenames created in "

 public:
   ThinLTOCodeGenerator ThinGenerator;
 
   ThinLTOProcessing(const TargetOptions &Options) {
     ThinGenerator.setCodePICModel(getRelocModel());
     ThinGenerator.setTargetOptions(Options);
     ThinGenerator.setCacheDir(ThinLTOCacheDir);
     ThinGenerator.setCachePruningInterval(ThinLTOCachePruningInterval);
+    ThinGenerator.setCacheEntryExpiration(ThinLTOCacheEntryExpiration);
     ThinGenerator.setCacheMaxSizeFiles(ThinLTOCacheMaxSizeFiles);
     ThinGenerator.setCacheMaxSizeBytes(ThinLTOCacheMaxSizeBytes);
     ThinGenerator.setFreestanding(EnableFreestanding);
 
     // Add all the exported symbols to the table of symbols to preserve.
     for (unsigned i = 0; i < ExportedSymbols.size(); ++i)
       ThinGenerator.preserveSymbol(ExportedSymbols[i]);
   }