[ThinLTO][ELF] Add a ThinLTO warning if cache_size_bytes or cache_size_files is too small for the current link job.

[MaggieYi]

Summary

In this patch, a ThinLTO warning is added if cache_size_bytes or cache_size_files is too small for the current link job. The warning recommends the user to consider adjusting --thinlto-cache-policy.

A specific case for ThinLTO cache pruning is that the current build is huge, and the cache wasn't big enough to hold the intermediate object files of that build. So in doing that build, a file would be cached, and later in that same build it would be evicted. This was significantly decreasing the effectiveness of the cache. By giving this warning, the user could identify the required cache size/files and improve ThinLTO link speed.

[tejohnson]

Thanks! Couple suggestions below.

lld/ELF/LTO.cpp

345

Can this handling be incorporated into pruneCache which is called a bit further below? The advantage of that is that it gets invoked by all cache users, which is more than just lld/ELF (i.e. the other lld formats, the old LTO API, and looks like lldb uses this as well). You could maybe just emit the warning the first time a file is pruned due to either of the size limits.

lld/test/ELF/lto/cache-warnings.ll

27

I'm concerned that this test will end up being brittle and start failing as the compiler evolves to change the IR and make the bitcode larger or smaller. I think you can use command substitution via $() in the RUN line to perhaps compute the size of the files and +/- 5 from the total.

[MaskRay]

lld/ELF/LTO.cpp

203

message(

Drop trailing .. Please follow the style of other messages and diagnostics.

Be terse in some wording; suggest the number of created files (number) exceeds the maximum ... (number)

lld/test/ELF/lto/cache-warnings.ll

3

Non-RUN non-CHECK comments use ;; for new tests.

[MaggieYi]

Update the patch following the reviewers comments, which include:

1. Just emit a warning the first time a file is pruned due to either of the size/number limits and add a test for it.
2. Use python and command substitution via $() in the RUN line to compute the size of the files and +/- 5 from the total.
3. Use lld 'warn()' to generate the warning.
4. Use ';;' for Non-RUN non-CHECK comments.

[MaggieYi]

Thanks Teresa and Fangrui for the comments. Sorry for the delay in updating the patch. I have been trying to find a way to get the warning invoked by all cache users.

As mentioned above, I cannot incorporate the handling into pruneCache.

I managed to get the total cache file sizes, which come out from the current link, by modifying the llvm/lib/LTO/LTO.cpp file.

diff --git a/llvm/lib/LTO/LTO.cpp b/llvm/lib/LTO/LTO.cpp
index dc28b681a151..a24f3149bead 100644
--- a/llvm/lib/LTO/LTO.cpp
+++ b/llvm/lib/LTO/LTO.cpp
@@ -74,6 +74,10 @@ cl::opt<bool> EnableLTOInternalization(
     "enable-lto-internalization", cl::init(true), cl::Hidden,
     cl::desc("Enable global value internalization in LTO"));

+/// The required cache size for the current link job. If thinLTOCacheDir is
+/// not specified, it is disabled by default.
+static uint64_t RequireCacheSize = 0;
+
 // Computes a unique hash for the Module considering the current list of
 // export/import and other global analysis results.
 // The hash is produced in \p Key.
@@ -1292,16 +1296,32 @@ public:
       return RunThinBackend(AddStream);

     SmallString<40> Key;
+    uint64_t CachedFileSize = 0;
     // The module may be cached, this helps handling it.
     computeLTOCacheKey(Key, Conf, CombinedIndex, ModuleID, ImportList,
                        ExportList, ResolvedODR, DefinedGlobals, CfiFunctionDefs,
                        CfiFunctionDecls);
-    Expected<AddStreamFn> CacheAddStreamOrErr = Cache(Task, Key);
+    Expected<AddStreamFn> CacheAddStreamOrErr = Cache(Task, Key, &CachedFileSize);
     if (Error Err = CacheAddStreamOrErr.takeError())
       return Err;
     AddStreamFn &CacheAddStream = *CacheAddStreamOrErr;
-    if (CacheAddStream)
-      return RunThinBackend(CacheAddStream);
+    if (CacheAddStream) {
+      auto Result = RunThinBackend(CacheAddStream);
+      // Get the file size of new created cache file.
+      Expected<std::unique_ptr<CachedFileStream>> StreamOrErr =
+          CacheAddStream(Task);
+      if (Error Err = StreamOrErr.takeError())
+        report_fatal_error(std::move(Err));
+      std::unique_ptr<CachedFileStream> &Stream = *StreamOrErr;
+      auto FileName = Stream->ObjectPathName;
+      std::error_code EC = llvm::sys::fs::file_size(FileName, CachedFileSize);
+      if (EC)
+        report_fatal_error(Twine("Unable to obtain file size for a file ") +
+                           FileName + ": " + EC.message() + "\n");
+      RequireCacheSize += CachedFileSize;
+      return Result;
+    }
+    RequireCacheSize += CachedFileSize;

     return Error::success();
   }

I also have to change FileCache by adding a new pointer type parameter CachedFileSize:

diff --git a/llvm/include/llvm/Support/Caching.h b/llvm/include/llvm/Support/Caching.h
index bef23ae757f2..017bfd232afb 100644
--- a/llvm/include/llvm/Support/Caching.h
+++ b/llvm/include/llvm/Support/Caching.h
@@ -52,8 +52,8 @@ using AddStreamFn =
 ///
 /// if (AddStreamFn AddStream = Cache(Task, Key))
 ///   ProduceContent(AddStream);
-using FileCache =
-    std::function<Expected<AddStreamFn>(unsigned Task, StringRef Key)>;
+using FileCache = std::function<Expected<AddStreamFn>(
+    unsigned Task, StringRef Key, uint64_t *CachedFileSize)>;

diff --git a/llvm/lib/Support/Caching.cpp b/llvm/lib/Support/Caching.cpp
index d6902f660e39..6b9f7bd4f1d0 100644
--- a/llvm/lib/Support/Caching.cpp
+++ b/llvm/lib/Support/Caching.cpp
@@ -37,7 +37,8 @@ Expected<FileCache> llvm::localCache(Twine CacheNameRef,
   TempFilePrefixRef.toVector(TempFilePrefix);
   CacheDirectoryPathRef.toVector(CacheDirectoryPath);

-  return [=](unsigned Task, StringRef Key) -> Expected<AddStreamFn> {
+  return [=](unsigned Task, StringRef Key,
+             uint64_t *CachedFileSize = nullptr) -> Expected<AddStreamFn> {
     // This choice of file name allows the cache to be pruned (see pruneCache()
     // in include/llvm/Support/CachePruning.h).
     SmallString<64> EntryPath;
@@ -48,6 +49,12 @@ Expected<FileCache> llvm::localCache(Twine CacheNameRef,
         Twine(EntryPath), sys::fs::OF_UpdateAtime, &ResultPath);
     std::error_code EC;
     if (FDOrErr) {
+      if (CachedFileSize) {
+        sys::fs::file_status Status;
+        EC = sys::fs::status(*FDOrErr, Status);
+        if (!EC)
+          *CachedFileSize = Status.getSize();
+      }
       ErrorOr<std::unique_ptr<MemoryBuffer>> MBOrErr =
           MemoryBuffer::getOpenFile(*FDOrErr, EntryPath,
                                     /*FileSize=*/-1,

If the developers use FileCache but don’t need to get the cached file sizes, they need to pass the nullptr, e.g.

Expected<AddStreamFn> CacheAddStreamOrErr = Cache(Task, UniqueKey, nullptr);

I am not sure whether this method is acceptable?

Could you please let me know your thoughts on this?

Thanks,

lld/ELF/LTO.cpp

203

Thanks, fixed it by using 'warn()' function and modified the message information.

345

Thanks.

1. Just emit a warning the first time a file is pruned and add a test for it.
2. I am not able to put this handling into pruneCache since pruneCache performs pruning for all cache files in the given cache directory. In this patch, we add up the sizes of the cache files that come out of the current link job. The warning is only given if cache_size_bytes or cache_size_files is too small for the total sizes/number of cache files that come out of the current link job. In the pruneCache function, there is not any information about cache files which were used by the current link job. Therefore, I cannot use pruneCache.

lld/test/ELF/lto/cache-warnings.ll

3

Thanks, fixed.

27

Thanks, the test is fixed following your suggestion.

[MaggieYi]

Updating the correct patch.

Changes include:

1. Just emit a warning the first time a file is pruned due to either of the size/number limits and add a test for this.
2. Use python and command substitution via $() in the RUN line to compute the size of the files and +/- 5 from the total.
3. Use lld 'warn()' to generate the warning.
4. Use ';;' for Non-RUN non-CHECK comments.

[MaskRay]

lld/ELF/LTO.cpp

210

lld uses camelCase variable names.

218

Drop trailing period:)

I haven't read through the patch yet.

[MaggieYi]

The changes include:

1. Corrected the cache-warnings.ll test to pass both Linux and Windows.
2. Corrected variable names.
3. Dropped trailing in the message.

[MaggieYi]

lld/ELF/LTO.cpp

210

Thanks, fixed.

218

Thanks, fixed :) .

[MaggieYi]

Gentle ping ..

[MaggieYi]

I think I might be able to incorporate checkCacheCapacity into pruneCache. I am working on this.

[MaggieYi]

Update the patch following the reviewers' comments, which include:

1. Incorporate checkCacheCapacity into pruneCache.
2. Emit two warnings if a file is pruned due to reach both size and number limits. If both cache files' size and number exceed the limits, I think it's better to print separated warnings in order to let the user adjusts cache_size_bytes and cache_size_files at the same time.
3. Use llvm support 'llvm::errs()' to generate the warning.
4. Add support and lit tests for other lld formats: COFF, MachO and Wasm.
5. For COFF format, I removed 'WARN-NOT' check since the warning is printed twice. The reason is that 'LLD_IN_TEST' is set to 2 for COFF. 'LLD_IN_TEST' determines how many times main is run inside each process. Therefore, main is run twice which result in two warnings.

[tejohnson]

Thanks, I like the new version with the checking centralized in pruneCache. A few suggestions below.

lld/test/ELF/lto/cache-warnings.ll

30

I'm not sure invoking lld via python subprocess is ideal for testing. I think this could be done much more simply via shell substitution via $() (you have to REQUIRE shell though). Even if you need python to compute the total file sizes initially you could then do something like

RUN: ld.lld .... --thinlto-cache-policy=prune_interval=0s:cache_size_bytes=$(($(cat size.txt) + 5)) ...

I know I have seen $(cat somefile.txt) used before, so I assume the evaluation described above should also work - can you give it a try to simplify a lot of this?

llvm/include/llvm/Support/CachePruning.h

83

If all clients are passing in files with your patch, probably just go ahead and remove the default parameter.

llvm/lib/Support/CachePruning.cpp

270

Probably using the warning interface is better than errs(). You could probably pass in the ErrorHandler if needed.

Also, @MaskRay had an earlier suggestion about the format of the message that I think would be good to incorporate (including the number of files in the limit and the actual number of files that caused the limit to be exceeded): https://reviews.llvm.org/D135590#inline-1308533

[MaggieYi]

Thanks for your comments.

I have updated the patch following reviewers' comments:

1. Modified the format of the message including the number of files in the limit and the actual number of files that caused the limit to be exceeded.
2. Corrected variable names since LLVM uses 'SamelCase' variable names.
3. Since warnings are moved from 'lld' to 'llvm/support', used llvm 'WithColor::warning()' (instead of lld 'warn()') to generate the warning.
4. Simplified the tests using shell substitution via $().

[MaggieYi]

lld/test/ELF/lto/cache-warnings.ll

30

Thanks, I have simplified the tests following your suggestion.

llvm/include/llvm/Support/CachePruning.h

83

There is an exceptional case. In the 'lldb' project, in the constructor of 'DataFileCache', 'pruneCache' function is called before 'localCache' function, which is different from other cases. In this case, the cache pruning is not considered/caused by the current 'localCache'. The new warnings should not be given in this case. Therefore, I use the default parameter to cover the above situation.

llvm/lib/Support/CachePruning.cpp

270

Thanks, since warnings are moved from 'lld' to 'llvm/support', I have used llvm 'WithColor::warning()' (instead of 'lld::warn()') to generate the warning.

@MaskRay, sorry to miss your earlier suggestion. @tejohnson, thanks for your reminding.
I have updated the warning message following the suggestion.

[tejohnson]

lgtm but please wait a bit to see if @MaskRay has any more comments

[MaskRay]

lld/test/COFF/lto-cache-warnings.ll

2

Does any feature require shell? I feel that every RUN line is normal and can be interpreted by lit, so shell is likely unneeded. But you can check this by running the test on a Windows machine which I assume you have access to:)

7

Nit: -r is more common than -R.

Note, if you do ; RUN: rm -rf %t && mkdir %t && cd %t, you can drop every occurrence %t below and just use an arbitrary word as the directory name, and you can avoid the slightly strange %t.cache reference in %python -c "import os, sys; os.chdir(r\"%t.cache\");

24

Use single quotes to avoid some escapes.

llvm/lib/Support/CachePruning.cpp

263

files.size() is greater than the number of inputs by one

264

llvmcache.timestamp was just mentioned above, so repeating it here seems verbose (and if we decide to rename, there is one comment needing to update).

267

Technically it may not be +1. It's + config->ltoPartitions (see config->ltoPartitions = args::getInteger(args, OPT_lto_partitions, 1); and ParallelCodeGenParallelismLevel). I suggest that you avoid being so specific in this sentence.

269

const size_t

Drop a pair of parentheses below: if (Policy.MaxSizeFiles && ActualNums > Policy.MaxSizeFiles)

[MaggieYi]

@tejohnson and @MaskRay, thanks for reviewing the patch.

I have updated the patch following @MaskRay's suggestions.

[MaggieYi]

@MaskRay, please let me know if there are any further changes required. Thanks.

lld/test/COFF/lto-cache-warnings.ll

2

As mentioned in the Teresa's comment, cache_size_bytes=$(($(cat %t.size.txt)+5)) requires shell. I have checked the test on a Windows machine and it failed in the RUN command which contains cache_size_bytes=$(($(cat %t.size.txt)+5)).

7

Thanks, fixed! For Macho test, I used ;RUN: cd %t, since the .o files exist in the %t folder as well.

24

Fixed, thanks.

llvm/lib/Support/CachePruning.cpp

264

Thanks, I have removed the timestamp file name.

267

Thanks for pointing this out, I have modified the sentence.

269

Fixed, thanks.

[MaskRay]

Hi @MaggieYi, thanks for the updates. You can probably mark comments as done to indicate there is no further action item. Note: if you mark some comments, you can either click "Submit" or upload a new diff (arc diff) to cause all comments to actually be marked as done.

lld/test/COFF/lto-cache-warnings.ll

24

Is fd 2 intended to be redirected to %t.size.txt as well?

[MaggieYi]

Removed '2>&1' from 'RUN: %python'.

lld/test/COFF/lto-cache-warnings.ll

24

Thanks, I have removed '2>&1'.

[MaggieYi]

@MaskRay, thanks again. I have marked comments as done.

lld/COFF/LTO.cpp

if (config->thinLTOIndexOnly) {
     if (!config->ltoObjPath.empty())
       saveBuffer(buf[0], config->ltoObjPath);
     if (indexFile)
       indexFile->close();
     return {};
   }
 
   if (!config->ltoCache.empty())
-    pruneCache(config->ltoCache, config->ltoCachePolicy);
+    pruneCache(config->ltoCache, config->ltoCachePolicy, files);
 
   std::vector<InputFile *> ret;
   for (unsigned i = 0; i != maxTasks; ++i) {
     // Assign unique names to LTO objects. This ensures they have unique names
     // in the PDB if one is produced. The names should look like:
     // - foo.exe.lto.obj
     // - foo.exe.lto.1.obj
     // - ...

lld/ELF/LTO.cpp

BitcodeCompiler::BitcodeCompiler() {
   // Initialize ltoObj.
   lto::ThinBackend backend;
   auto onIndexWrite = [&](StringRef s) { thinIndices.erase(s); };
   if (config->thinLTOIndexOnly) {
     backend = lto::createWriteIndexesThinBackend(
         std::string(config->thinLTOPrefixReplace.first),
         std::string(config->thinLTOPrefixReplace.second),
         config->thinLTOEmitImportsFiles, indexFile.get(), onIndexWrite);
   } else {
     backend = lto::createInProcessThinBackend(
         llvm::heavyweight_hardware_concurrency(config->thinLTOJobs),
         onIndexWrite, config->thinLTOEmitIndexFiles,
         config->thinLTOEmitImportsFiles);
   }
 
   ltoObj = std::make_unique<lto::LTO>(createConfig(), backend,
                                        config->ltoPartitions);
 
   // Initialize usedStartStop.
   if (ctx.bitcodeFiles.empty())
     return;
   for (Symbol *sym : symtab.getSymbols()) {
     if (sym->isPlaceholder())
       continue;
     StringRef s = sym->getName();
     for (StringRef prefix : {"__start_", "__stop_"})
       if (s.startswith(prefix))
         usedStartStop.insert(s.substr(prefix.size()));
   }
 }
 
 BitcodeCompiler::~BitcodeCompiler() = default;
if (!ctx.bitcodeFiles.empty())
     checkError(ltoObj->run(
         [&](size_t task) {
           return std::make_unique<CachedFileStream>(
               std::make_unique<raw_svector_ostream>(buf[task]));
         },
         cache));
 
   // Emit empty index files for non-indexed files but not in single-module mode.
   if (config->thinLTOModulesToCompile.empty()) {
     for (StringRef s : thinIndices) {
       std::string path = getThinLTOOutputFile(s);
       openFile(path + ".thinlto.bc");
       if (config->thinLTOEmitImportsFiles)
         openFile(path + ".imports");
     }
   }
 
   if (config->thinLTOEmitIndexFiles)
     thinLTOCreateEmptyIndexFiles();
 
   if (config->thinLTOIndexOnly) {
     if (!config->ltoObjPath.empty())
       saveBuffer(buf[0], config->ltoObjPath);
 
     // ThinLTO with index only option is required to generate only the index
     // files. After that, we exit from linker and ThinLTO backend runs in a
     // distributed environment.
     if (indexFile)
       indexFile->close();
     return {};
   }
 
   if (!config->thinLTOCacheDir.empty())
-    pruneCache(config->thinLTOCacheDir, config->thinLTOCachePolicy);
+    pruneCache(config->thinLTOCacheDir, config->thinLTOCachePolicy, files);
 
   if (!config->ltoObjPath.empty()) {
     saveBuffer(buf[0], config->ltoObjPath);
     for (unsigned i = 1; i != maxTasks; ++i)
       saveBuffer(buf[i], config->ltoObjPath + Twine(i));
   }
 
   if (config->saveTempsArgs.contains("prelink")) {

lld/MachO/LTO.cpp

std::vector<ObjFile *> BitcodeCompiler::compile() {
   checkError(ltoObj->run(
       [&](size_t task) {
         return std::make_unique<CachedFileStream>(
             std::make_unique<raw_svector_ostream>(buf[task]));
       },
       cache));
 
   if (!config->thinLTOCacheDir.empty())
-    pruneCache(config->thinLTOCacheDir, config->thinLTOCachePolicy);
+    pruneCache(config->thinLTOCacheDir, config->thinLTOCachePolicy, files);
 
   // In ThinLTO mode, Clang passes a temporary directory in -object_path_lto,
   // while the argument is a single file in FullLTO mode.
   bool objPathIsDir = true;
   if (!config->ltoObjPath.empty()) {
     if (std::error_code ec = fs::create_directories(config->ltoObjPath))
       fatal("cannot create LTO object path " + config->ltoObjPath + ": " +
             ec.message());

lld/test/COFF/lto-cache-warnings.ll

+; REQUIRES: x86, shell
+
+; RUN: opt -module-hash -module-summary %s -o %t.o
+; RUN: opt -module-hash -module-summary %p/Inputs/lto-cache.ll -o %t2.o
+
+; RUN: rm -rf %t && mkdir %t
+
+;; Check cache policies of the number of files.
+;; Case 1: A value of 0 disables the number of files based pruning. Therefore, there is no warning.
+; RUN: lld-link /verbose /lldltocache:%t /lldltocachepolicy:prune_interval=0s:cache_size_files=0 /out:%t3 /entry:main %t2.o %t.o 2>&1 | FileCheck --implicit-check-not=warning: %s
+;; Case 2: If the total number of the files created by the current link job is less than the maximum number of files, there is no warning.
+; RUN: lld-link /verbose /lldltocache:%t /lldltocachepolicy:prune_interval=0s:cache_size_files=3 /out:%t3 /entry:main %t2.o %t.o 2>&1 | FileCheck %s --implicit-check-not=warning:
+;; Case 3: If the total number of the files created by the current link job exceeds the maximum number of files, a warning is given.
+; RUN: lld-link /lldltocache:%t /lldltocachepolicy:prune_interval=0s:cache_size_files=1 /out:%t3 /entry:main %t2.o %t.o 2>&1 | FileCheck %s --check-prefix=NUM
+
+;; Check cache policies of the cache size.
+;; Case 1: A value of 0 disables the absolute size-based pruning. Therefore, there is no warning.
+; RUN: lld-link /verbose /lldltocache:%t /lldltocachepolicy:prune_interval=0s:cache_size_bytes=0 /out:%t3 /entry:main %t2.o %t.o 2>&1 | FileCheck %s --implicit-check-not=warning:
+
+;; Get the total size of created cache files.
+; RUN: rm -rf %t && mkdir %t && cd %t
+; RUN: lld-link /lldltocache:%t /lldltocachepolicy:prune_interval=0s:cache_size_bytes=32k /out:%t3 /entry:main %t2.o %t.o 2>&1
+; RUN: %python -c "import os, sys; print(sum(os.path.getsize(filename) for filename in os.listdir('.') if os.path.isfile(filename) and filename.startswith('llvmcache-')))" > %t.size.txt
+
+;; Case 2: If the total size of the cache files created by the current link job is less than the maximum size for the cache directory in bytes, there is no warning.
+; RUN: lld-link /verbose /lldltocache:%t /lldltocachepolicy:prune_interval=0s:cache_size_bytes=$(($(cat %t.size.txt) + 5)) /out:%t3 /entry:main %t2.o %t.o 2>&1 | FileCheck %s --implicit-check-not=warning:
+
+;; Case 3: If the total size of the cache files created by the current link job exceeds the maximum size for the cache directory in bytes, a warning is given.
+; RUN: lld-link /verbose /lldltocache:%t /lldltocachepolicy:prune_interval=0s:cache_size_bytes=$(($(cat %t.size.txt) - 5)) /out:%t3 /entry:main %t2.o %t.o 2>&1 | FileCheck %s --check-prefix=SIZE
+
+;; Check emit two warnings if pruning happens due to reach both the size and number limits.
+; RUN: lld-link /lldltocache:%t /lldltocachepolicy:prune_interval=0s:cache_size_files=1:cache_size_bytes=1 /out:%t3 /entry:main %t2.o %t.o 2>&1 | FileCheck %s --check-prefixes=NUM,SIZE
+
+; NUM: warning: ThinLTO cache pruning happens since the number of{{.*}}--thinlto-cache-policy
+; SIZE: warning: ThinLTO cache pruning happens since the total size of{{.*}}--thinlto-cache-policy
+
+target datalayout = "e-m:w-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-pc-windows-msvc"
+
+define void @globalfunc() #0 {
+entry:
+  ret void
+}

lld/test/ELF/lto/cache-warnings.ll

+; REQUIRES: x86, shell
+
+; RUN: opt -module-hash -module-summary %s -o %t.o
+; RUN: opt -module-hash -module-summary %p/Inputs/cache.ll -o %t2.o
+
+; RUN: rm -rf %t && mkdir %t
+
+;; Check cache policies of the number of files.
+;; Case 1: A value of 0 disables the number of files based pruning. Therefore, there is no warning.
+; RUN: ld.lld --verbose --thinlto-cache-dir=%t --thinlto-cache-policy=prune_interval=0s:cache_size_files=0 %t2.o %t.o -o %t3 2>&1 | FileCheck %s --implicit-check-not=warning:
+;; Case 2: If the total number of the files created by the current link job is less than the maximum number of files, there is no warning.
+; RUN: ld.lld --verbose --thinlto-cache-dir=%t --thinlto-cache-policy=prune_interval=0s:cache_size_files=3 %t2.o %t.o -o %t3 2>&1 | FileCheck %s --implicit-check-not=warning:
+;; Case 3: If the total number of the files created by the current link job exceeds the maximum number of files, a warning is given.
+; RUN: ld.lld --thinlto-cache-dir=%t --thinlto-cache-policy=prune_interval=0s:cache_size_files=1 %t2.o %t.o -o %t3 2>&1 | FileCheck %s --check-prefixes=NUM,WARN
+
+;; Check cache policies of the cache size.
+;; Case 1: A value of 0 disables the absolute size-based pruning. Therefore, there is no warning.
+; RUN: ld.lld --verbose --thinlto-cache-dir=%t --thinlto-cache-policy=prune_interval=0s:cache_size_bytes=0 %t2.o %t.o -o %t3 2>&1 | FileCheck %s --implicit-check-not=warning:
+
+;; Get the total size of created cache files.
+; RUN: rm -rf %t && mkdir %t && cd %t
+; RUN: ld.lld --thinlto-cache-dir=%t --thinlto-cache-policy=prune_interval=0s:cache_size_bytes=32k %t2.o %t.o -o %t3 2>&1
+; RUN: %python -c "import os, sys; print(sum(os.path.getsize(filename) for filename in os.listdir('.') if os.path.isfile(filename) and filename.startswith('llvmcache-')))" > %t.size.txt
+
+;; Case 2: If the total size of the cache files created by the current link job is less than the maximum size for the cache directory in bytes, there is no warning.
+; RUN: ld.lld --verbose --thinlto-cache-dir=%t --thinlto-cache-policy=prune_interval=0s:cache_size_bytes=$(($(cat %t.size.txt) + 5)) %t2.o %t.o -o %t3 2>&1 | FileCheck %s --implicit-check-not=warning:
+
+;; Case 3: If the total size of the cache files created by the current link job exceeds the maximum size for the cache directory in bytes, a warning is given.
+; RUN: ld.lld --verbose --thinlto-cache-dir=%t --thinlto-cache-policy=prune_interval=0s:cache_size_bytes=$(($(cat %t.size.txt) - 5)) %t2.o %t.o -o %t3 2>&1 | FileCheck %s --check-prefixes=SIZE,WARN
+
+;; Check emit two warnings if pruning happens due to reach both the size and number limits.
+; RUN: ld.lld --thinlto-cache-dir=%t --thinlto-cache-policy=prune_interval=0s:cache_size_files=1:cache_size_bytes=1 %t2.o %t.o -o %t3 2>&1 | FileCheck %s --check-prefixes=NUM,SIZE,WARN
+
+; NUM: warning: ThinLTO cache pruning happens since the number of{{.*}}--thinlto-cache-policy
+; SIZE: warning: ThinLTO cache pruning happens since the total size of{{.*}}--thinlto-cache-policy
+; WARN-NOT: warning: ThinLTO cache pruning happens{{.*}}--thinlto-cache-policy
+
+target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+define void @globalfunc() {
+entry:
+  ret void
+}

lld/test/MachO/lto-cache-warnings.ll

+; REQUIRES: x86, shell
+
+; RUN: rm -rf %t; split-file %s %t
+; RUN: opt -module-hash -module-summary %t/foo.ll -o %t/foo.o
+; RUN: opt -module-hash -module-summary %t/bar.ll -o %t/bar.o
+
+;; Check cache policies of the number of files.
+;; Case 1: A value of 0 disables the number of files based pruning. Therefore, there is no warning.
+; RUN: %lld -v -cache_path_lto %t --thinlto-cache-policy=prune_interval=0s:cache_size_files=0 -o %t/test %t/foo.o %t/bar.o 2>&1 | FileCheck %s --implicit-check-not=warning:
+;; Case 2: If the total number of the files created by the current link job is less than the maximum number of files, there is no warning.
+; RUN: %lld -v -cache_path_lto %t --thinlto-cache-policy=prune_interval=0s:cache_size_files=3 %t/foo.o %t/bar.o -o %t/test 2>&1 | FileCheck %s --implicit-check-not=warning:
+;; Case 3: If the total number of the files created by the current link job exceeds the maximum number of files, a warning is given.
+; RUN: %lld -cache_path_lto %t --thinlto-cache-policy=prune_interval=0s:cache_size_files=1 %t/foo.o %t/bar.o -o %t/test 2>&1 | FileCheck %s --check-prefixes=NUM,WARN
+
+;; Check cache policies of the cache size.
+;; Case 1: A value of 0 disables the absolute size-based pruning. Therefore, there is no warning.
+; RUN: %lld -v -cache_path_lto %t --thinlto-cache-policy=prune_interval=0s:cache_size_bytes=0 %t/foo.o %t/bar.o -o %t/test 2>&1 | FileCheck %s --implicit-check-not=warning:
+
+;; Get the total size of created cache files.
+; RUN: cd %t
+; RUN: %lld -cache_path_lto %t --thinlto-cache-policy=prune_interval=0s:cache_size_bytes=32k %t/foo.o %t/bar.o -o %t/test 2>&1
+; RUN: %python -c "import os, sys; print(sum(os.path.getsize(filename) for filename in os.listdir('.') if os.path.isfile(filename) and filename.startswith('llvmcache-')))" > %t.size.txt
+
+;; Case 2: If the total size of the cache files created by the current link job is less than the maximum size for the cache directory in bytes, there is no warning.
+; RUN: %lld -v -cache_path_lto %t --thinlto-cache-policy=prune_interval=0s:cache_size_bytes=$(($(cat %t.size.txt) + 5)) %t/foo.o %t/bar.o -o %t/test 2>&1 | FileCheck %s --implicit-check-not=warning:
+
+;; Case 3: If the total size of the cache files created by the current link job exceeds the maximum size for the cache directory in bytes, a warning is given.
+; RUN: %lld -v -cache_path_lto %t --thinlto-cache-policy=prune_interval=0s:cache_size_bytes=$(($(cat %t.size.txt) - 5)) %t/foo.o %t/bar.o -o %t/test 2>&1 | FileCheck %s --check-prefixes=SIZE,WARN
+
+;; Check emit two warnings if pruning happens due to reach both the size and number limits.
+; RUN: %lld -cache_path_lto %t --thinlto-cache-policy=prune_interval=0s:cache_size_files=1:cache_size_bytes=1 %t/foo.o %t/bar.o -o %t/test 2>&1 | FileCheck %s --check-prefixes=NUM,SIZE,WARN
+
+; NUM: warning: ThinLTO cache pruning happens since the number of{{.*}}--thinlto-cache-policy
+; SIZE: warning: ThinLTO cache pruning happens since the total size of{{.*}}--thinlto-cache-policy
+; WARN-NOT: warning: ThinLTO cache pruning happens{{.*}}--thinlto-cache-policy
+
+;--- foo.ll
+
+target triple = "x86_64-apple-macosx10.15.0"
+target datalayout = "e-m:o-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+
+define void @globalfunc() #0 {
+entry:
+  ret void
+}
+
+
+;--- bar.ll
+
+target triple = "x86_64-apple-macosx10.15.0"
+target datalayout = "e-m:o-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+
+define i32 @main() {
+entry:
+  call void (...) @globalfunc()
+  ret i32 0
+}
+
+declare void @globalfunc(...)

lld/test/wasm/lto/cache-warnings.ll

+; REQUIRES: shell
+
+; RUN: opt -module-hash -module-summary %s -o %t.o
+; RUN: opt -module-hash -module-summary %p/Inputs/cache.ll -o %t2.o
+
+; RUN: rm -rf %t && mkdir %t
+
+;; Check cache policies of the number of files.
+;; Case 1: A value of 0 disables the number of files based pruning. Therefore, there is no warning.
+; RUN: wasm-ld --verbose --thinlto-cache-dir=%t --thinlto-cache-policy=prune_interval=0s:cache_size_files=0 %t2.o %t.o -o %t3 2>&1 | FileCheck %s --implicit-check-not=warning:
+;; Case 2: If the total number of the files created by the current link job is less than the maximum number of files, there is no warning.
+; RUN: wasm-ld --verbose --thinlto-cache-dir=%t --thinlto-cache-policy=prune_interval=0s:cache_size_files=3 %t2.o %t.o -o %t3 2>&1 | FileCheck %s --implicit-check-not=warning:
+;; Case 3: If the total number of the files created by the current link job exceeds the maximum number of files, a warning is given.
+; RUN: wasm-ld --thinlto-cache-dir=%t --thinlto-cache-policy=prune_interval=0s:cache_size_files=1 %t2.o %t.o -o %t3 2>&1 | FileCheck %s --check-prefixes=NUM,WARN
+
+;; Check cache policies of the cache size.
+;; Case 1: A value of 0 disables the absolute size-based pruning. Therefore, there is no warning.
+; RUN: wasm-ld --verbose --thinlto-cache-dir=%t --thinlto-cache-policy=prune_interval=0s:cache_size_bytes=0 %t2.o %t.o -o %t3 2>&1 | FileCheck %s --implicit-check-not=warning:
+
+;; Get the total size of created cache files.
+; RUN: rm -rf %t && mkdir %t && cd %t
+; RUN: wasm-ld --thinlto-cache-dir=%t --thinlto-cache-policy=prune_interval=0s:cache_size_bytes=32k %t2.o %t.o -o %t3 2>&1
+; RUN: %python -c "import os, sys; print(sum(os.path.getsize(filename) for filename in os.listdir('.') if os.path.isfile(filename) and filename.startswith('llvmcache-')))" > %t.size.txt
+
+;; Case 2: If the total size of the cache files created by the current link job is less than the maximum size for the cache directory in bytes, there is no warning.
+; RUN: wasm-ld --verbose --thinlto-cache-dir=%t --thinlto-cache-policy=prune_interval=0s:cache_size_bytes=$(($(cat %t.size.txt) + 5)) %t2.o %t.o -o %t3 2>&1 | FileCheck %s --implicit-check-not=warning:
+
+;; Case 3: If the total size of the cache files created by the current link job exceeds the maximum size for the cache directory in bytes, a warning is given.
+; RUN: wasm-ld --verbose --thinlto-cache-dir=%t --thinlto-cache-policy=prune_interval=0s:cache_size_bytes=$(($(cat %t.size.txt) - 5)) %t2.o %t.o -o %t3 2>&1 | FileCheck %s --check-prefixes=SIZE,WARN
+
+;; Check emit two warnings if pruning happens due to reach both the size and number limits.
+; RUN: wasm-ld --thinlto-cache-dir=%t --thinlto-cache-policy=prune_interval=0s:cache_size_files=1:cache_size_bytes=1 %t2.o %t.o -o %t3 2>&1 | FileCheck %s --check-prefixes=NUM,SIZE,WARN
+
+; NUM: warning: ThinLTO cache pruning happens since the number of{{.*}}--thinlto-cache-policy
+; SIZE: warning: ThinLTO cache pruning happens since the total size of{{.*}}--thinlto-cache-policy
+; WARN-NOT: warning: ThinLTO cache pruning happens{{.*}}--thinlto-cache-policy
+
+target datalayout = "e-m:e-p:32:32-p10:8:8-p20:8:8-i64:64-n32:64-S128"
+target triple = "wasm32-unknown-unknown-wasm"
+
+define void @globalfunc() #0 {
+entry:
+  ret void
+}

lld/wasm/LTO.cpp

std::vector<StringRef> BitcodeCompiler::compile() {
   checkError(ltoObj->run(
       [&](size_t task) {
         return std::make_unique<CachedFileStream>(
             std::make_unique<raw_svector_ostream>(buf[task]));
       },
       cache));
 
   if (!config->thinLTOCacheDir.empty())
-    pruneCache(config->thinLTOCacheDir, config->thinLTOCachePolicy);
+    pruneCache(config->thinLTOCacheDir, config->thinLTOCachePolicy, files);
 
   std::vector<StringRef> ret;
   for (unsigned i = 0; i != maxTasks; ++i) {
     if (buf[i].empty())
       continue;
     if (config->saveTemps) {
       if (i == 0)
         saveBuffer(buf[i], config->outputFile + ".lto.o");

llvm/include/llvm/Support/CachePruning.h

 // various policies.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_SUPPORT_CACHEPRUNING_H
 #define LLVM_SUPPORT_CACHEPRUNING_H
 
 #include "llvm/ADT/Optional.h"
+#include "llvm/Support/MemoryBuffer.h"
 #include <chrono>
 
 namespace llvm {
 
 template <typename T> class Expected;
 class StringRef;
 
 /// Policy for the pruneCache() function. A default constructed

 /// For example: "prune_interval=30s:prune_after=24h:cache_size=50%"
 /// which means a pruning interval of 30 seconds, expiration time of 24 hours
 /// and maximum cache size of 50% of available disk space.
 Expected<CachePruningPolicy> parseCachePruningPolicy(StringRef PolicyStr);
 
 /// Peform pruning using the supplied policy, returns true if pruning
 /// occurred, i.e. if Policy.Interval was expired.
 ///
+/// Check whether cache pruning happens using the supplied policy, adds a
+/// ThinLTO warning if cache_size_bytes or cache_size_files is too small for the
+/// current link job. The warning recommends the user to consider adjusting
+/// --thinlto-cache-policy.
+///
 /// As a safeguard against data loss if the user specifies the wrong directory
 /// as their cache directory, this function will ignore files not matching the
 /// pattern "llvmcache-*".
-bool pruneCache(StringRef Path, CachePruningPolicy Policy);
-
+bool pruneCache(StringRef Path, CachePruningPolicy Policy,
+                const std::vector<std::unique_ptr<MemoryBuffer>> &Files = {});
 } // namespace llvm
 
 #endif

llvm/lib/LTO/ThinLTOCodeGenerator.cpp

for (auto IndexCount : ModulesOrdering) {
           return;
         }
         ProducedBinaryFiles[count] = writeGeneratedObject(
             count, CacheEntryPath, *OutputBuffer);
       }, IndexCount);
     }
   }
 
-  pruneCache(CacheOptions.Path, CacheOptions.Policy);
+  pruneCache(CacheOptions.Path, CacheOptions.Policy, ProducedBinaries);
 
   // If statistics were requested, print them out now.
   if (llvm::AreStatisticsEnabled())
     llvm::PrintStatistics();
   reportAndResetTimings();
 }

llvm/lib/Support/CachePruning.cpp

 
 #include "llvm/Support/CachePruning.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/Errc.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Path.h"
+#include "llvm/Support/WithColor.h"
 #include "llvm/Support/raw_ostream.h"
 
 #define DEBUG_TYPE "cache-pruning"
 
 #include <set>
 #include <system_error>
 
 using namespace llvm;
if (Key == "prune_interval") {
                                      inconvertibleErrorCode());
     }
   }
 
   return Policy;
 }
 
 /// Prune the cache of files that haven't been accessed in a long time.
-bool llvm::pruneCache(StringRef Path, CachePruningPolicy Policy) {
+bool llvm::pruneCache(StringRef Path, CachePruningPolicy Policy,
+                      const std::vector<std::unique_ptr<MemoryBuffer>> &Files) {
   using namespace std::chrono;
 
   if (Path.empty())
     return false;
 
   bool isPathDir;
   if (sys::fs::is_directory(Path, isPathDir))
     return false;
auto RemoveCacheFile = [&]() {
     TotalSize -= FileInfo->Size;
     NumFiles--;
     LLVM_DEBUG(dbgs() << " - Remove " << FileInfo->Path << " (size "
                       << FileInfo->Size << "), new occupancy is " << TotalSize
                       << "%\n");
     ++FileInfo;
   };
 
+  // files.size() is greater the number of inputs  by one. However, a timestamp
+  // file is created and stored in the cache directory if --thinlto-cache-policy
+  // option is used. Therefore, files.size() is used as ActualNums.
+  const size_t ActualNums = Files.size();
+  if (Policy.MaxSizeFiles && ActualNums > Policy.MaxSizeFiles)
+    WithColor::warning()
+        << "ThinLTO cache pruning happens since the number of created files ("
+        << ActualNums << ") exceeds the maximum number of files ("
+        << Policy.MaxSizeFiles
+        << "); consider adjusting --thinlto-cache-policy\n";
+
   // Prune for number of files.
   if (Policy.MaxSizeFiles)
     while (NumFiles > Policy.MaxSizeFiles)
       RemoveCacheFile();
 
   // Prune for size now if needed
   if (Policy.MaxSizePercentageOfAvailableSpace > 0 || Policy.MaxSizeBytes > 0) {
     auto ErrOrSpaceInfo = sys::fs::disk_space(Path);
auto TotalSizeTarget = std::min<uint64_t>(
         AvailableSpace * Policy.MaxSizePercentageOfAvailableSpace / 100ull,
         Policy.MaxSizeBytes);
 
     LLVM_DEBUG(dbgs() << "Occupancy: " << ((100 * TotalSize) / AvailableSpace)
                       << "% target is: "
                       << Policy.MaxSizePercentageOfAvailableSpace << "%, "
                       << Policy.MaxSizeBytes << " bytes\n");
 
+    size_t ActualSizes = 0;
+    for (const auto &File : Files)
+      if (File)
+        ActualSizes += File->getBufferSize();
+
+    if (ActualSizes > TotalSizeTarget)
+      WithColor::warning()
+          << "ThinLTO cache pruning happens since the total size of the cache "
+             "files consumed by the current link job ("
+          << ActualSizes << "  bytes) exceeds maximum cache size ("
+          << TotalSizeTarget
+          << " bytes); consider adjusting --thinlto-cache-policy\n";
+
     // Remove the oldest accessed files first, till we get below the threshold.
     while (TotalSize > TotalSizeTarget && FileInfo != FileInfos.end())
       RemoveCacheFile();
   }
   return true;
 }