[Clang][ThinLTO] Add a cache for compile phase output.

[ychen]

Summary

Currently the link phase has a object file cache whereas the compile phase always
perform optimizations (most likely happen for large source files and O2 or above)
which could potentially waste time optimizing a file that finally hit the object file cache.
For example, with Intel W-2133 and 64GB memory, compile X86ISelLowering.cpp with -flto=thin -O3
takes about 40s (takes about 10s with caching implemented by this patch).
The patch makes sure bitcodes that hit LTO cache also skip IR optimizations.

Add a driver/cc1 flag (-fthinlto-cache-dir, default off) to cache the minimized or regular ThinLTO bitcode file.
The caching is only trigger if the input is large than -fthinlto-cache-min-filesize=. Default minimum is 1024 IR instructions.
Cache pruning (-fthinlto-cache-policy=) shares the implementation with lld --thinlto-cache-policy.

[tejohnson]

I haven't read through the patch in detail yet, but from the description it sounds like a cache is being added for the compile step outputs, e.g. the bitcode output object of the "clang -flto=thin -c" step? Typically the build system takes care of incremental build handling for these explicit outputs, just as it would for a non-LTO clang compile output. Can you clarify the motivation?

[ychen]

In D69327#1719109, @tejohnson wrote:

I haven't read through the patch in detail yet, but from the description it sounds like a cache is being added for the compile step outputs, e.g. the bitcode output object of the "clang -flto=thin -c" step? Typically the build system takes care of incremental build handling for these explicit outputs, just as it would for a non-LTO clang compile output. Can you clarify the motivation?

Hi Teresa, thanks for the feedback. I think the motivation is to provide an option that reliably caches thinLTO compile output regardless of the external setup of compiler cache or the platform is used. I'm not sure if compiler cache such as ccache could save -fthin-link-bitcode output and if there is a ccache equivalence on windows? Even if such tool exists, having linking phase caching managed by toolchain/linker and the compile phase by an external tool feels awkward and fragile. This patch mostly shares the implementation with linke phase caching.

Additionally, this also has a small benefit of trigging a little bit more caching because it is hashing IR instead of preprocessor output or even literal source file.

[steven_wu]

Can you clarify what do you mean by 'waste time optimizing a file that finally hit the object file cache'?

No matter what build system to use, it should figure out during an incremental build that the input wasn't changed and not rerun the clang invocation on the same input. If you are looking to achieve what ccache is doing, I don't think implement in the compiler is a good option. That should really be done on the build system level. This solution is strictly worse than ccache because ccache is able to fetch the optimized bitcode without even running clang frontend and ir-gen.

[tejohnson]

I was just typing up a response similar to @steven_wu 's response... I don't think clang should be in the business of caching the outputs of a prior clang invocation, the build system should and usually does avoid re-executing if the inputs have not changed. Note that this is very different from the caching of objects LTO is doing - because those are not otherwise emitted at all, the cache has to be built in.

[tejohnson]

In D69327#1719295, @tejohnson wrote:

I was just typing up a response similar to @steven_wu 's response... I don't think clang should be in the business of caching the outputs of a prior clang invocation, the build system should and usually does avoid re-executing if the inputs have not changed. Note that this is very different from the caching of objects LTO is doing - because those are not otherwise emitted at all, the cache has to be built in.

And also because even if some of the inputs of the link change, not all backend threads necessarily need to be re-executed and not all native (intermediate) objects will change.

[ychen]

Thanks for the inputs @steven_wu @tejohnson. Totally agree with the points you brought up. One last thing I'm not quite sure is the caching of -fthin-link-bitcode. It is a -cc1 option since it is a kind of implementation of ThinLTO, right? I'm a little hesitant to begin writing up patches to teach build system/caching tool (I could think of at least three for our workload) to recognize this option because of that. If there are any changes to the option, the same thing needs to be done again. Do you have any thoughts on that? Is the option in use for your workload and do you think it is stable enough to have build systems caching for it? (Another option is to produce -fthin-link-bitcode output post compile time which I assume having total build time impact to some degree).

[steven_wu]

In D69327#1719411, @ychen wrote:

Thanks for the inputs @steven_wu @tejohnson. Totally agree with the points you brought up. One last thing I'm not quite sure is the caching of -fthin-link-bitcode. It is a -cc1 option since it is a kind of implementation of ThinLTO, right? I'm a little hesitant to begin writing up patches to teach build system/caching tool (I could think of at least three for our workload) to recognize this option because of that. If there are any changes to the option, the same thing needs to be done again. Do you have any thoughts on that? Is the option in use for your workload and do you think it is stable enough to have build systems caching for it? (Another option is to produce -fthin-link-bitcode output post compile time which I assume having total build time impact to some degree).

-fthin-link-bitcode option is used to run distributed thin link. The format is not stable but it is deterministic for a fixed compiler version. You should be able to cache the thin-link-bitcode and expected it to be used only by the same compiler version.

For any build system that implements caching, it must take compiler version into consideration because different compiler will produce different output. I don't think the rule to cache thin-link-bitcode is any different from any other output during the build.

[ychen]

In D69327#1719419, @steven_wu wrote:

In D69327#1719411, @ychen wrote:

Thanks for the inputs @steven_wu @tejohnson. Totally agree with the points you brought up. One last thing I'm not quite sure is the caching of -fthin-link-bitcode. It is a -cc1 option since it is a kind of implementation of ThinLTO, right? I'm a little hesitant to begin writing up patches to teach build system/caching tool (I could think of at least three for our workload) to recognize this option because of that. If there are any changes to the option, the same thing needs to be done again. Do you have any thoughts on that? Is the option in use for your workload and do you think it is stable enough to have build systems caching for it? (Another option is to produce -fthin-link-bitcode output post compile time which I assume having total build time impact to some degree).

-fthin-link-bitcode option is used to run distributed thin link. The format is not stable but it is deterministic for a fixed compiler version. You should be able to cache the thin-link-bitcode and expected it to be used only by the same compiler version.

For any build system that implements caching, it must take compiler version into consideration because different compiler will produce different output. I don't think the rule to cache thin-link-bitcode is any different from any other output during the build.

I tried ccache, it does not cache -fthin-link-bitcode output.

From this link, it seems ccache only cares about "-o" output.
https://github.com/ccache/ccache/blob/ac9911b47b8a3777d20a3c481f90f877e8f9a81d/src/ccache.cpp#L2616

[ychen]

Sorry for the confusion @steven_wu. By stable I mean the probability that the -fthin-link-bitcode option is replaced with some other thinlink mechanism under the distributed build environment. Since at least for ccache, the compilation output caching depends on the semantics of options ("-o" is assumed to be compilation output). For the case of -fthin-link-bitcode, both -Xclang -fthin-link-bitcode and -o are the output. I'm not familiar with compiler cache tools, but having the caching depends on a cc1 option feels not right since it is not an option of any other compilers, so most caching tools don't recognize it.

[tejohnson]

We rely on the minimized bitcode from that option in our builds, so it won't be going away. We could add it as a driver option, but it doesn't sound like that will solve the ccache issue. I'm not very familiar with that cache support, or if there is a way to express other output files (in our build system we identify all the outputs of a build action). At least with something like make and ninja the compile action should not be done again if none of the inputs change.

[steven_wu]

ccache does not have the support for this, I am just saying that this can be easily implemented in ccache and that would be much better than the proposed solution here.

If we need to add a clang driver flag so build system can better support to detect thin bitcode as output and caching them, we should just add the official driver flag.

[ychen]

Thank you @steven_wu @tejohnson. I created D69406 to promote the flag to the driver.

clang/include/clang/Basic/CodeGenOptions.h

public:
   /// importing.
   std::string ThinLTOIndexFile;
 
   /// Name of a file that can optionally be written with minimized bitcode
   /// to be used as input for the ThinLTO thin link step, which only needs
   /// the summary and module symbol table (and not, e.g. any debug metadata).
   std::string ThinLinkBitcodeFile;
 
+  /// The directory ThinLTO caches compile phase outputs.
+  std::string ThinLTOCacheDir;
+
+  /// Pruning policy for ThinLTO compile phase cache.
+  std::string ThinLTOCachePolicy;
+
   /// Prefix to use for -save-temps output.
   std::string SaveTempsFilePrefix;
 
   /// Name of file passed with -fcuda-include-gpubinary option to forward to
   /// CUDA runtime back-end for incorporating them into host-side object file.
   std::string CudaGpuBinaryFileName;
 
   /// The name of the file to which the backend should save YAML optimization

clang/include/clang/Basic/CodeGenOptions.def

 CODEGENOPT(PrepareForThinLTO , 1, 0) ///< Set when -flto=thin is enabled on the
                                      ///< compile step.
 CODEGENOPT(LTOUnit, 1, 0) ///< Emit IR to support LTO unit features (CFI, whole
                           ///< program vtable opt).
 CODEGENOPT(EnableSplitLTOUnit, 1, 0) ///< Enable LTO unit splitting to support
              /// CFI and traditional whole program
              /// devirtualization that require whole
              /// program IR support.
+CODEGENOPT(ThinLTOCacheMinFilesize, 32, 1024)
+                          ///< Minimal number of instructions of the Module to
+                          ///< be considered for ThinLTO compile phase caching.
 CODEGENOPT(IncrementalLinkerCompatible, 1, 0) ///< Emit an object file which can
                                               ///< be used with an incremental
                                               ///< linker.
 CODEGENOPT(MergeAllConstants , 1, 1) ///< Merge identical constants.
 CODEGENOPT(MergeFunctions    , 1, 0) ///< Set when -fmerge-functions is enabled.
 CODEGENOPT(MSVolatile        , 1, 0) ///< Set when /volatile:ms is enabled.
 CODEGENOPT(NoCommon          , 1, 0) ///< Set when -fno-common or C++ is enabled.
 CODEGENOPT(NoDwarfDirectoryAsm , 1, 0) ///< Set when -fno-dwarf-directory-asm is

clang/include/clang/Driver/Options.td

 def flto_jobs_EQ : Joined<["-"], "flto-jobs=">,
   Flags<[CC1Option]>, Group<f_Group>,
   HelpText<"Controls the backend parallelism of -flto=thin (default "
            "of 0 means the number of threads will be derived from "
            "the number of CPUs detected)">;
 def fthinlto_index_EQ : Joined<["-"], "fthinlto-index=">,
   Flags<[CoreOption, CC1Option]>, Group<f_Group>,
   HelpText<"Perform ThinLTO importing using provided function summary index">;
+def fthinlto_cache_dir_EQ : Joined<["-"], "fthinlto-cache-dir=">,
+  Flags<[CoreOption, CC1Option]>, Group<f_Group>, MetaVarName<"<directory>">,
+  HelpText<"Specify cache directory for ThinLTO compile phase">;
+def fthinlto_cache_min_filesize_EQ : Joined<["-"], "fthinlto-cache-min-filesize=">,
+  Flags<[CoreOption, CC1Option]>, Group<f_Group>, MetaVarName<"<Num of IR instructions> (default 1024)">,
+  HelpText<"Files with size that are below this limit skip ThinLTO compile phase caching">;
+def fthinlto_cache_policy_EQ : Joined<["-"], "fthinlto-cache-policy=">,
+  Flags<[CoreOption, CC1Option]>, Group<f_Group>,
+  HelpText<"Pruning policy for the ThinLTO compile phase cache">;
 def fmacro_backtrace_limit_EQ : Joined<["-"], "fmacro-backtrace-limit=">,
                                 Group<f_Group>, Flags<[DriverOption, CoreOption]>;
 def fmerge_all_constants : Flag<["-"], "fmerge-all-constants">, Group<f_Group>,
   Flags<[CC1Option, CoreOption]>, HelpText<"Allow merging of constants">;
 def fmessage_length_EQ : Joined<["-"], "fmessage-length=">, Group<f_Group>;
 def fms_extensions : Flag<["-"], "fms-extensions">, Group<f_Group>, Flags<[CC1Option, CoreOption]>,
   HelpText<"Accept some non-standard constructs supported by the Microsoft compiler">;
 def fms_compatibility : Flag<["-"], "fms-compatibility">, Group<f_Group>, Flags<[CC1Option, CoreOption]>,

clang/lib/CodeGen/BackendUtil.cpp

 #include "llvm/CodeGen/SchedulerRegistry.h"
 #include "llvm/CodeGen/TargetSubtargetInfo.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/IR/IRPrintingPasses.h"
 #include "llvm/IR/LegacyPassManager.h"
 #include "llvm/IR/Module.h"
 #include "llvm/IR/ModuleSummaryIndex.h"
 #include "llvm/IR/Verifier.h"
+#include "llvm/LTO/Caching.h"
 #include "llvm/LTO/LTOBackend.h"
 #include "llvm/MC/MCAsmInfo.h"
 #include "llvm/MC/SubtargetFeature.h"
 #include "llvm/Passes/PassBuilder.h"
 #include "llvm/Passes/PassPlugin.h"
 #include "llvm/Passes/StandardInstrumentations.h"
 #include "llvm/Support/BuryPointer.h"
+#include "llvm/Support/CachePruning.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/PrettyStackTrace.h"
+#include "llvm/Support/SaveAndRestore.h"
+#include "llvm/Support/SHA1.h"
 #include "llvm/Support/TargetRegistry.h"
 #include "llvm/Support/TimeProfiler.h"
 #include "llvm/Support/Timer.h"
 #include "llvm/Support/raw_ostream.h"
+#include "llvm/Support/VCSRevision.h"
 #include "llvm/Target/TargetMachine.h"
 #include "llvm/Target/TargetOptions.h"
 #include "llvm/Transforms/Coroutines.h"
 #include "llvm/Transforms/IPO.h"
 #include "llvm/Transforms/IPO/AlwaysInliner.h"
 #include "llvm/Transforms/IPO/PassManagerBuilder.h"
 #include "llvm/Transforms/IPO/ThinLTOBitcodeWriter.h"
 #include "llvm/Transforms/InstCombine/InstCombine.h"
class EmitAssemblyHelper {
   const HeaderSearchOptions &HSOpts;
   const CodeGenOptions &CodeGenOpts;
   const clang::TargetOptions &TargetOpts;
   const LangOptions &LangOpts;
   Module *TheModule;
 
   Timer CodeGenerationTime;
 
-  std::unique_ptr<raw_pwrite_stream> OS;
-
   TargetIRAnalysis getTargetIRAnalysis() const {
     if (TM)
       return TM->getTargetIRAnalysis();
 
     return TargetIRAnalysis();
   }
 
   void CreatePasses(legacy::PassManager &MPM, legacy::FunctionPassManager &FPM);
class EmitAssemblyHelper {
   void CreateTargetMachine(bool MustCreateTM);
 
   /// Add passes necessary to emit assembly or LLVM IR.
   ///
   /// \return True on success.
   bool AddEmitPasses(legacy::PassManager &CodeGenPasses, BackendAction Action,
                      raw_pwrite_stream &OS, raw_pwrite_stream *DwoOS);
 
+  using OptimizerFn =
+      std::function<void(Module &, raw_ostream &, raw_ostream *)>;
+  void runOptimizer(bool IsThinLTOCompilePhase, raw_ostream &OS,
+                    raw_ostream *ThinLinkOS, OptimizerFn OptimizeModule);
+
   std::unique_ptr<llvm::ToolOutputFile> openOutputFile(StringRef Path) {
     std::error_code EC;
     auto F = std::make_unique<llvm::ToolOutputFile>(Path, EC,
                                                      llvm::sys::fs::OF_None);
     if (EC) {
       Diags.Report(diag::err_fe_unable_to_open_output) << Path << EC.message();
       F.reset();
     }
if (TM->addPassesToEmitFile(CodeGenPasses, OS, DwoOS, CGFT,
                               /*DisableVerify=*/!CodeGenOpts.VerifyModule)) {
     Diags.Report(diag::err_fe_unable_to_interface_with_target);
     return false;
   }
 
   return true;
 }
 
+/// Computes a unique hash for the Module considering the list of options.
+/// The hash is produced in \p Key.
+/// FIXME: This is being very conservertive by hashing a lot of options.
+///        Could be relaxed if necessary.
+static void computeLTOCacheKey(SmallString<40> &Key, const Module &M,
+                     const CodeGenOptions &CGOpts,
+                     const clang::TargetOptions &TOpts,
+                     const LangOptions &LOpts) {
+  SHA1 Hasher;
+
+  // Start with the compiler revision
+  Hasher.update(LLVM_VERSION_STRING);
+  Hasher.update(LLVM_REVISION);
+
+  auto AddString = [&](StringRef S) {
+    Hasher.update(S);
+    Hasher.update(ArrayRef<uint8_t>{0});
+  };
+  auto AddIntegral = [&](auto I) {
+    constexpr size_t NumBytes = sizeof(I);
+    uint8_t Data[NumBytes];
+    for (uint8_t &D : Data) {
+      D = I;
+      I >>= 8;
+    }
+    Hasher.update(ArrayRef<uint8_t>{Data, NumBytes});
+  };
+  auto AddEnum = [&](auto E) {
+    using UnderTy = typename std::underlying_type<decltype(E)>::type;
+    AddIntegral(static_cast<UnderTy>(E));
+  };
+
+  // Include the hash for the current module.
+  ModuleHash ModHash = {{0}};
+  SmallString<0> BC;
+  raw_svector_ostream BCOS(BC);
+  WriteBitcodeToFile(M, BCOS, CGOpts.EmitLLVMUseLists, nullptr,
+                     /*GenerateHash=*/true, &ModHash);
+  Hasher.update(ArrayRef<uint8_t>((uint8_t *)&ModHash[0], sizeof(ModHash)));
+
+  // Hash CodeGenOptions.
+  {
+    // Not hashing caching related options.
+    CodeGenOptions &CGOptsCopy = const_cast<CodeGenOptions&>(CGOpts);
+    SaveAndRestore<std::string> Save1(CGOptsCopy.ThinLTOCacheDir, "");
+    SaveAndRestore<std::string> Save2(CGOptsCopy.ThinLTOCachePolicy, "");
+    // FIXME: SaveAndRestore does not work for bitfields.
+    const unsigned SavedThinLTOCacheMinFilesize =
+        CGOptsCopy.ThinLTOCacheMinFilesize;
+    CGOptsCopy.ThinLTOCacheMinFilesize = 0;
+
+    uint64_t HashInput = 0;
+    unsigned NumOfBits = 0;
+  #define CODEGENOPT(Name, Bits, Default)                 \
+    if (Bits + NumOfBits > sizeof(HashInput)) {           \
+      AddIntegral(HashInput);                             \
+      HashInput = 0;                                      \
+      NumOfBits = 0;                                      \
+    }                                                     \
+    HashInput &= (CGOpts.Name << NumOfBits);              \
+    NumOfBits += Bits;
+  #define ENUM_CODEGENOPT(Name, Type, Bits, Default)      \
+    AddEnum(CGOpts.get##Name());
+  #include "clang/Basic/CodeGenOptions.def"
+    if (HashInput)
+      AddIntegral(HashInput);
+
+    CGOptsCopy.ThinLTOCacheMinFilesize= SavedThinLTOCacheMinFilesize;
+  }
+
+  AddString(CGOpts.CodeModel);
+  AddString(CGOpts.CoverageDataFile);
+  AddString(CGOpts.CoverageNotesFile);
+  AddString(CGOpts.ProfileFilterFiles);
+  AddString(CGOpts.ProfileExcludeFiles);
+  Hasher.update(ArrayRef<uint8_t>((const uint8_t *)CGOpts.CoverageVersion, 4));
+  AddString(CGOpts.DebugPass);
+  AddString(CGOpts.DebugCompilationDir);
+  AddString(CGOpts.DwarfDebugFlags);
+  AddString(CGOpts.RecordCommandLine);
+  AddString(CGOpts.FloatABI);
+  AddString(CGOpts.FPDenormalMode);
+  AddString(CGOpts.LimitFloatPrecision);
+  AddString(CGOpts.MainFileName);
+  AddEnum(CGOpts.RelocationModel);
+  AddString(CGOpts.ThreadModel);
+  AddString(CGOpts.TrapFuncName);
+  for (const auto &I : CGOpts.DependentLibraries)
+    AddString(I);
+  for (const auto &I : CGOpts.LinkerOptions)
+    AddString(I);
+  AddString(CGOpts.InstrProfileOutput);
+  AddString(CGOpts.SampleProfileFile);
+  AddString(CGOpts.ProfileInstrumentUsePath);
+  AddString(CGOpts.ProfileRemappingFile);
+  AddString(CGOpts.ThinLinkBitcodeFile);
+  AddString(CGOpts.SaveTempsFilePrefix);
+  AddString(CGOpts.CudaGpuBinaryFileName);
+  AddString(CGOpts.OptRecordFile);
+  AddString(CGOpts.OptRecordPasses);
+  AddString(CGOpts.OptRecordFormat);
+  AddString(CGOpts.SymbolPartition);
+  for (const auto &I : CGOpts.RewriteMapFiles)
+    AddString(I);
+  AddIntegral(static_cast<size_t>(CGOpts.SanitizeRecover.Mask.hash_value()));
+  AddIntegral(static_cast<size_t>(CGOpts.SanitizeTrap.Mask.hash_value()));
+  Hasher.update(ArrayRef<uint8_t>(CGOpts.CmdArgs));
+  for (const auto &I : CGOpts.NoBuiltinFuncs)
+    AddString(I);
+  for (const auto &I : CGOpts.Reciprocals)
+    AddString(I);
+  AddString(CGOpts.PreferVectorWidth);
+  AddIntegral(CGOpts.XRayInstrumentationBundle.Mask);
+  for (const auto &I : CGOpts.DefaultFunctionAttrs)
+    AddString(I);
+  for (const auto &I : CGOpts.PassPlugins)
+    AddString(I);
+
+  // Hash clang::TargetOptions
+  AddString(TOpts.Triple);
+  AddString(TOpts.HostTriple);
+  AddString(TOpts.CPU);
+  AddString(TOpts.FPMath);
+  AddString(TOpts.ABI);
+  AddEnum(TOpts.EABIVersion);
+  AddString(TOpts.LinkerVersion);
+  for (const auto &I : TOpts.FeaturesAsWritten)
+    AddString(I);
+  for (const auto &I : TOpts.Features)
+    AddString(I);
+  AddString(TOpts.CodeModel);
+  AddString(TOpts.SDKVersion.getAsString());
+
+  // Hash LangOptions
+  for (const auto &I : LOpts.SanitizerBlacklistFiles)
+    AddString(I);
+  AddEnum(LOpts.getDefaultFPContractMode());
+  AddIntegral(static_cast<uint8_t>(LOpts.SjLjExceptions));
+  AddIntegral(static_cast<uint8_t>(LOpts.SEHExceptions));
+  AddIntegral(static_cast<uint8_t>(LOpts.DWARFExceptions));
+  AddIntegral(static_cast<uint8_t>(LOpts.WasmExceptions));
+  AddIntegral(static_cast<size_t>(LOpts.Sanitize.Mask.hash_value()));
+  AddIntegral(static_cast<uint8_t>(LOpts.ObjCAutoRefCount));
+  AddIntegral(static_cast<uint8_t>(LOpts.Coroutines));
+
+  Key = toHex(Hasher.result());
+}
+
+void EmitAssemblyHelper::runOptimizer(bool IsThinLTOCompilePhase,
+                                      raw_ostream &OS, raw_ostream *ThinLinkOS,
+                                      OptimizerFn OptimizeModule) {
+  auto ModuleIsEligibleForCaching = [&](const Module &M) {
+    unsigned ModuleInstCount = 0;
+    for (const auto &F : M)
+      for (const auto &BB : F) {
+        ModuleInstCount += BB.size();
+        // FIXME: Tune default file size limit.
+        if (ModuleInstCount > CodeGenOpts.ThinLTOCacheMinFilesize)
+          return true;
+      }
+    return false;
+  };
+
+  const bool UseThinLTOCache = IsThinLTOCompilePhase &&
+                               !CodeGenOpts.ThinLTOCacheDir.empty() &&
+                               ModuleIsEligibleForCaching(*TheModule);
+  if (!UseThinLTOCache) {
+    OptimizeModule(*TheModule, OS, ThinLinkOS);
+    return;
+  }
+
+  SmallString<40> Key;
+  computeLTOCacheKey(Key, *TheModule, CodeGenOpts, TargetOpts, LangOpts);
+
+  unsigned CacheDiagID =
+      Diags.getCustomDiagID(DiagnosticsEngine::Error,
+                            "ThinLTO cache error for '%0': '%1'");
+
+  bool ThinLTOCacheHit = false;
+
+  auto ThinLinkOSHook = [&](size_t, std::unique_ptr<MemoryBuffer> mb) {
+    *ThinLinkOS << mb->getBuffer();
+    ThinLTOCacheHit = true;
+  };
+
+  auto OSHook = [&](size_t, std::unique_ptr<MemoryBuffer> mb) {
+    OS << mb->getBuffer();
+    if (!ThinLinkOS) {
+      ThinLTOCacheHit = true;
+      return;
+    }
+
+    if (Expected<lto::NativeObjectCache> Cache = lto::localCache(
+            CodeGenOpts.ThinLTOCacheDir, ThinLinkOSHook, "llvmcache-thinlink-"))
+      (void)(*Cache)(/*Task=*/-1, Key);
+    else
+      Diags.Report(CacheDiagID)
+          << CodeGenOpts.ThinLTOCacheDir << Cache.takeError();
+  };
+
+  if (Expected<lto::NativeObjectCache> BitcodeCache =
+          lto::localCache(CodeGenOpts.ThinLTOCacheDir, OSHook, "llvmcache-bc-"))
+    (void)(*BitcodeCache)(/*Task=*/-1, Key);
+  else
+    Diags.Report(CacheDiagID)
+        << CodeGenOpts.ThinLTOCacheDir << BitcodeCache.takeError();
+
+  if (ThinLTOCacheHit)
+    return;
+
+  // Run optimizer passes if miss ThinLTO cache.
+  buffer_ostream CacheOS(OS);
+  OptimizeModule(*TheModule, CacheOS, ThinLinkOS);
+
+  if (Expected<CachePruningPolicy> ThinLTOCachePolicy =
+      parseCachePruningPolicy(CodeGenOpts.ThinLTOCachePolicy))
+    pruneCache(CodeGenOpts.ThinLTOCacheDir, *ThinLTOCachePolicy);
+  else
+    Diags.Report(CacheDiagID)
+        << CodeGenOpts.ThinLTOCachePolicy << ThinLTOCachePolicy.takeError();
+
+  // Save OS to cache.
+  SmallString<64> EntryPath;
+  sys::path::append(EntryPath, CodeGenOpts.ThinLTOCacheDir,
+                    "llvmcache-bc-" + Key);
+  std::unique_ptr<llvm::ToolOutputFile> CopyOS = openOutputFile(EntryPath);
+  if (!CopyOS)
+    return;
+  CopyOS->os() << CacheOS.str();
+  CopyOS->keep();
+
+  // Save
+  if (ThinLinkOS) {
+    EntryPath.clear();
+    sys::path::append(EntryPath, CodeGenOpts.ThinLTOCacheDir,
+                      "llvmcache-thinlink-" + Key);
+    ThinLinkOS->flush();
+    if (std::error_code EC =
+            sys::fs::copy_file(CodeGenOpts.ThinLinkBitcodeFile, EntryPath)) {
+      unsigned DiagID = Diags.getCustomDiagID(
+          DiagnosticsEngine::Warning, "ThinLTO saving to cache fail: '%0'");
+      Diags.Report(DiagID) << EC.message();
+    }
+  }
+}
+
 void EmitAssemblyHelper::EmitAssembly(BackendAction Action,
                                       std::unique_ptr<raw_pwrite_stream> OS) {
   TimeRegion Region(FrontendTimesIsEnabled ? &CodeGenerationTime : nullptr);
 
   setCommandLineOpts(CodeGenOpts);
 
   bool UsesCodeGen = (Action != Backend_EmitNothing &&
                       Action != Backend_EmitBC &&
PerFunctionPasses.add(
       createTargetTransformInfoWrapperPass(getTargetIRAnalysis()));
 
   CreatePasses(PerModulePasses, PerFunctionPasses);
 
   legacy::PassManager CodeGenPasses;
   CodeGenPasses.add(
       createTargetTransformInfoWrapperPass(getTargetIRAnalysis()));
 
+  bool IsThinLTOCompilePhase = false;
   std::unique_ptr<llvm::ToolOutputFile> ThinLinkOS, DwoOS;
 
   switch (Action) {
   case Backend_EmitNothing:
     break;
 
   case Backend_EmitBC:
     if (CodeGenOpts.PrepareForThinLTO && !CodeGenOpts.DisableLLVMPasses) {
       if (!CodeGenOpts.ThinLinkBitcodeFile.empty()) {
         ThinLinkOS = openOutputFile(CodeGenOpts.ThinLinkBitcodeFile);
         if (!ThinLinkOS)
           return;
       }
       TheModule->addModuleFlag(Module::Error, "EnableSplitLTOUnit",
                                CodeGenOpts.EnableSplitLTOUnit);
-      PerModulePasses.add(createWriteThinLTOBitcodePass(
-          *OS, ThinLinkOS ? &ThinLinkOS->os() : nullptr));
+      IsThinLTOCompilePhase = true;
     } else {
       // Emit a module summary by default for Regular LTO except for ld64
       // targets
       bool EmitLTOSummary =
           (CodeGenOpts.PrepareForLTO &&
            !CodeGenOpts.DisableLLVMPasses &&
            llvm::Triple(TheModule->getTargetTriple()).getVendor() !=
                llvm::Triple::Apple);
default:
     if (!AddEmitPasses(CodeGenPasses, Action, *OS,
                        DwoOS ? &DwoOS->os() : nullptr))
       return;
   }
 
   // Before executing passes, print the final values of the LLVM options.
   cl::PrintOptionValues();
 
+  // Now that we have all of the passes ready, run them.
+  OptimizerFn OptFn = [&](Module &M, raw_ostream &OS, raw_ostream *ThinLinkOS) {
     // Run passes. For now we do all passes at once, but eventually we
     // would like to have the option of streaming code generation.
-
     {
       PrettyStackTraceString CrashInfo("Per-function optimization");
       llvm::TimeTraceScope TimeScope("PerFunctionPasses", StringRef(""));
 
       PerFunctionPasses.doInitialization();
       for (Function &F : *TheModule)
         if (!F.isDeclaration())
           PerFunctionPasses.run(F);
       PerFunctionPasses.doFinalization();
     }
 
     {
       PrettyStackTraceString CrashInfo("Per-module optimization passes");
       llvm::TimeTraceScope TimeScope("PerModulePasses", StringRef(""));
-    PerModulePasses.run(*TheModule);
+      if (IsThinLTOCompilePhase)
+        PerModulePasses.add(createWriteThinLTOBitcodePass(OS, ThinLinkOS));
+      PerModulePasses.run(M);
     }
+  };
+  raw_ostream *TheThinLinkOS = ThinLinkOS ? &ThinLinkOS->os() : nullptr;
+  runOptimizer(IsThinLTOCompilePhase, *OS, TheThinLinkOS, OptFn);
 
   {
     PrettyStackTraceString CrashInfo("Code generation");
     llvm::TimeTraceScope TimeScope("CodeGenPasses", StringRef(""));
     CodeGenPasses.run(*TheModule);
   }
 
   if (ThinLinkOS)
if (CodeGenOpts.OptimizationLevel == 0) {
       addSanitizersAtO0(MPM, TargetTriple, LangOpts, CodeGenOpts);
     }
   }
 
   // FIXME: We still use the legacy pass manager to do code generation. We
   // create that pass manager here and use it as needed below.
   legacy::PassManager CodeGenPasses;
   bool NeedCodeGen = false;
+  bool IsThinLTOCompilePhase = false;
   std::unique_ptr<llvm::ToolOutputFile> ThinLinkOS, DwoOS;
 
   // Append any output we need to the pass manager.
   switch (Action) {
   case Backend_EmitNothing:
     break;
 
   case Backend_EmitBC:
     if (CodeGenOpts.PrepareForThinLTO && !CodeGenOpts.DisableLLVMPasses) {
       if (!CodeGenOpts.ThinLinkBitcodeFile.empty()) {
         ThinLinkOS = openOutputFile(CodeGenOpts.ThinLinkBitcodeFile);
         if (!ThinLinkOS)
           return;
       }
       TheModule->addModuleFlag(Module::Error, "EnableSplitLTOUnit",
                                CodeGenOpts.EnableSplitLTOUnit);
-      MPM.addPass(ThinLTOBitcodeWriterPass(*OS, ThinLinkOS ? &ThinLinkOS->os()
-                                                           : nullptr));
+      IsThinLTOCompilePhase = true;
     } else {
       // Emit a module summary by default for Regular LTO except for ld64
       // targets
       bool EmitLTOSummary =
           (CodeGenOpts.PrepareForLTO &&
            !CodeGenOpts.DisableLLVMPasses &&
            llvm::Triple(TheModule->getTargetTriple()).getVendor() !=
                llvm::Triple::Apple);
if (!AddEmitPasses(CodeGenPasses, Action, *OS,
       return;
     break;
   }
 
   // Before executing passes, print the final values of the LLVM options.
   cl::PrintOptionValues();
 
   // Now that we have all of the passes ready, run them.
-  {
+  OptimizerFn OptFn = [&](Module &M, raw_ostream &OS, raw_ostream *ThinLinkOS) {
     PrettyStackTraceString CrashInfo("Optimizer");
-    MPM.run(*TheModule, MAM);
-  }
+    if (IsThinLTOCompilePhase)
+      MPM.addPass(ThinLTOBitcodeWriterPass(OS, ThinLinkOS));
+    MPM.run(M, MAM);
+  };
+  raw_ostream *TheThinLinkOS = ThinLinkOS ? &ThinLinkOS->os() : nullptr;
+  runOptimizer(IsThinLTOCompilePhase, *OS, TheThinLinkOS, OptFn);
 
   // Now if needed, run the legacy PM for codegen.
   if (NeedCodeGen) {
     PrettyStackTraceString CrashInfo("Code generation");
     CodeGenPasses.run(*TheModule);
   }
 
   if (ThinLinkOS)

clang/lib/Driver/ToolChains/Clang.cpp

void Clang::ConstructJob(Compilation &C, const JobAction &JA,
   }
 
   if (const Arg *A = Args.getLastArg(options::OPT_fthinlto_index_EQ)) {
     if (!types::isLLVMIR(Input.getType()))
       D.Diag(diag::err_drv_arg_requires_bitcode_input) << A->getAsString(Args);
     Args.AddLastArg(CmdArgs, options::OPT_fthinlto_index_EQ);
   }
 
+  if (Args.getLastArg(options::OPT_fthinlto_cache_dir_EQ))
+    Args.AddLastArg(CmdArgs, options::OPT_fthinlto_cache_dir_EQ);
+
+  if (Args.getLastArg(options::OPT_fthinlto_cache_min_filesize_EQ))
+    Args.AddLastArg(CmdArgs, options::OPT_fthinlto_cache_min_filesize_EQ);
+
+  if (Args.getLastArg(options::OPT_fthinlto_cache_policy_EQ))
+    Args.AddLastArg(CmdArgs, options::OPT_fthinlto_cache_policy_EQ);
+
   if (Args.getLastArg(options::OPT_save_temps_EQ))
     Args.AddLastArg(CmdArgs, options::OPT_save_temps_EQ);
 
   // Embed-bitcode option.
   // Only white-listed flags below are allowed to be embedded.
   if (C.getDriver().embedBitcodeInObject() && !C.getDriver().isUsingLTO() &&
       (isa<BackendJobAction>(JA) || isa<AssembleJobAction>(JA))) {
     // Add flags implied by -fembed-bitcode.

clang/lib/Frontend/CompilerInvocation.cpp

static bool ParseCodeGenArgs(CodeGenOptions &Opts, ArgList &Args, InputKind IK,
   if (Arg *A = Args.getLastArg(OPT_save_temps_EQ))
     Opts.SaveTempsFilePrefix =
         llvm::StringSwitch<std::string>(A->getValue())
             .Case("obj", FrontendOpts.OutputFile)
             .Default(llvm::sys::path::filename(FrontendOpts.OutputFile).str());
 
   Opts.ThinLinkBitcodeFile = Args.getLastArgValue(OPT_fthin_link_bitcode_EQ);
 
+  Opts.ThinLTOCacheDir = Args.getLastArgValue(OPT_fthinlto_cache_dir_EQ);
+  Opts.ThinLTOCachePolicy = Args.getLastArgValue(OPT_fthinlto_cache_policy_EQ);
+  Opts.ThinLTOCacheMinFilesize= getLastArgUInt64Value(
+      Args, options::OPT_fthinlto_cache_min_filesize_EQ, 1024);
+
   Opts.MSVolatile = Args.hasArg(OPT_fms_volatile);
 
   Opts.VectorizeLoop = Args.hasArg(OPT_vectorize_loops);
   Opts.VectorizeSLP = Args.hasArg(OPT_vectorize_slp);
 
   Opts.PreferVectorWidth = Args.getLastArgValue(OPT_mprefer_vector_width_EQ);
 
   Opts.MainFileName = Args.getLastArgValue(OPT_main_file_name);

clang/test/CodeGen/thin_link_bitcode.c

 // REQUIRES: x86-registered-target
 //
 // RUN: %clang_cc1 -o %t -flto=thin -fthin-link-bitcode=%t.nodebug -triple x86_64-unknown-linux-gnu -emit-llvm-bc -debug-info-kind=limited %s
 // RUN: llvm-bcanalyzer -dump %t | FileCheck %s
 // RUN: llvm-bcanalyzer -dump %t.nodebug | FileCheck %s --check-prefix=NO_DEBUG
 // RUN: %clang_cc1 -o %t.newpm -flto=thin -fexperimental-new-pass-manager -fthin-link-bitcode=%t.newpm.nodebug -triple x86_64-unknown-linux-gnu -emit-llvm-bc -debug-info-kind=limited  %s
 // RUN: llvm-bcanalyzer -dump %t.newpm | FileCheck %s
 // RUN: llvm-bcanalyzer -dump %t.newpm.nodebug | FileCheck %s --check-prefix=NO_DEBUG
+
+// Test optimized bitcode files caching
+// RUN: rm -Rf %t.cache && mkdir %t.cache
+// RUN: %clang_cc1 -o %t -flto=thin -fthin-link-bitcode=%t.nodebug -fthinlto-cache-dir=%t.cache -fthinlto-cache-min-filesize=1 -triple x86_64-unknown-linux-gnu -emit-llvm-bc -debug-info-kind=limited %s
+// RUN: ls %t.cache | count 3
+// RUN: llvm-bcanalyzer -dump %t.cache/llvmcache-bc-* | FileCheck %s
+// RUN: llvm-bcanalyzer -dump %t.cache/llvmcache-thinlink-* | FileCheck %s --check-prefix=NO_DEBUG
+
+// RUN: rm -Rf %t.cache && mkdir %t.cache
+// RUN: %clang_cc1 -o %t -flto=thin -fthin-link-bitcode=%t.nodebug -fthinlto-cache-dir=%t.cache -fthinlto-cache-min-filesize=100 -triple x86_64-unknown-linux-gnu -emit-llvm-bc -debug-info-kind=limited %s
+// RUN: ls %t.cache | count 0
 int main (void) {
   return 0;
 }
 
 // CHECK: COMPILE_UNIT
 // NO_DEBUG-NOT: COMPILE_UNIT

llvm/include/llvm/LTO/Caching.h

 /// Buffer callbacks must be thread safe.
 using AddBufferFn =
     std::function<void(unsigned Task, std::unique_ptr<MemoryBuffer> MB)>;
 
 /// Create a local file system cache which uses the given cache directory and
 /// file callback. This function also creates the cache directory if it does not
 /// already exist.
 Expected<NativeObjectCache> localCache(StringRef CacheDirectoryPath,
-                                       AddBufferFn AddBuffer);
+                                       AddBufferFn AddBuffer,
+                                       StringRef Prefix = "llvmcache-");
 
 } // namespace lto
 } // namespace llvm
 
 #endif

llvm/lib/LTO/Caching.cpp

 #else
 #include <io.h>
 #endif
 
 using namespace llvm;
 using namespace llvm::lto;
 
 Expected<NativeObjectCache> lto::localCache(StringRef CacheDirectoryPath,
-                                            AddBufferFn AddBuffer) {
+                                            AddBufferFn AddBuffer,
+                                            StringRef Prefix) {
   if (std::error_code EC = sys::fs::create_directories(CacheDirectoryPath))
     return errorCodeToError(EC);
 
   return [=](unsigned Task, StringRef Key) -> AddStreamFn {
     // This choice of file name allows the cache to be pruned (see pruneCache()
     // in include/llvm/Support/CachePruning.h).
     SmallString<64> EntryPath;
-    sys::path::append(EntryPath, CacheDirectoryPath, "llvmcache-" + Key);
+    sys::path::append(EntryPath, CacheDirectoryPath, Prefix + Key);
     // First, see if we have a cache hit.
     SmallString<64> ResultPath;
     Expected<sys::fs::file_t> FDOrErr = sys::fs::openNativeFileForRead(
         Twine(EntryPath), sys::fs::OF_UpdateAtime, &ResultPath);
     std::error_code EC;
     if (FDOrErr) {
       ErrorOr<std::unique_ptr<MemoryBuffer>> MBOrErr =
           MemoryBuffer::getOpenFile(*FDOrErr, EntryPath,