[ThinLTO] Out-of-process CodeGenerator for legacy C API

[steven_wu]

Summary

Add an out-of-process ThinLTOCodeGenerator for legacy C API. Instead of
spawning threads and run codegen inside libLTO, OutOfProcess
CodeGenerator constructs clang invocations and uses "clang -fthinlto-index="
option to generate object files. The goal of this design is to enable
libLTO ThinLTOCodeGenerator to expose more information to the build
system so the build system can manage the system resource better.

The other goals/benefits for this design are:

• No linker change is needed using this model. libLTO is in charge of

constructing the job, waiting for the completion and handing back the results
to linker with the exact same API.

• Although this patch only comes with a proof-of-concept out-of-process

code generator which invokes clang locally, it can be easily extended. It
provides a ThinLTOCodegenManager abstraction, which provides customization
points to adapt to any protocol or system.

• It is currenlty using a cl::opt to switch between different ThinLTO

codegen mode which means user can easily toggle between different codegen
models.

Based on patch by Daniel Grumberg

[steven_wu]

Ping. Is there any comments of implementing C API in this approach?

[tejohnson]

In D55080#1328556, @steven_wu wrote:

Ping. Is there any comments of implementing C API in this approach?

Sorry for not responding earlier. I'll take a closer look this week, but a couple of high level questions below. I also added @pcc for thoughts.

Can you give a description of how this interacts with your distributed build system? I see references to something called XPC, but I don't know what that is. Is this something that is Apple build system specific? AFAICT btw this approach would not work with our build system, although we are using the new LTO API so not affected directly. But I have a concern that this may make it harder to migrate the old to the new LTO API, but hopefully it can be designed so that the two approaches could both be supported in a single LTO API.

Regarding linker changes, the only linker changes required by our distributed build approach is to exit the linker early (after the thin link), and to write out the individual index files (which is actually handled within the new LTO API so not much needs to be done in the linker for that). Presumably you could do that latter part inside the old LTO API as well (as you are doing here in this patch).

[steven_wu]

Thanks for taking a look. This patch is adding the customization points for thinLTO legacy API, which the code generator constructs clang invocations to do code generation. There is no dependency on any build system here and it only has a prove of concept codegen manager which invokes clang directly and collect the result back. You can replace this codegen manager with any protocol that is needed to talk to build system to run clang codegen.
XPC is the way to send information between process on Darwin, which is probably what we are going to use to talk to build system. If interested, I can post a patch which have example how to construct XPC communications, but there isn’t a build system you can use to listen on the other side to run the job yet.
When I say there are no code change for linker, I really mean there is no need to change a single line of code (maybe we need to add an API to select codegen manager in the future). ld64 really has a different approach using C API, which it tries to map the object file output back to the bitcode it gets as input. Terminating and relaunching the linker might has unexpected semantic changes for LTO. In the long run, maybe ld64 needs to design a new set of APIs to use the new C++ APIs but this is out of scope of this patch.

[dang]

In D55080#1328603, @steven_wu wrote:

Thanks for taking a look. This patch is adding the customization points for thinLTO legacy API, which the code generator constructs clang invocations to do code generation. There is no dependency on any build system here and it only has a prove of concept codegen manager which invokes clang directly and collect the result back. You can replace this codegen manager with any protocol that is needed to talk to build system to run clang codegen.
XPC is the way to send information between process on Darwin, which is probably what we are going to use to talk to build system. If interested, I can post a patch which have example how to construct XPC communications, but there isn’t a build system you can use to listen on the other side to run the job yet.
When I say there are no code change for linker, I really mean there is no need to change a single line of code (maybe we need to add an API to select codegen manager in the future). ld64 really has a different approach using C API, which it tries to map the object file output back to the bitcode it gets as input. Terminating and relaunching the linker might has unexpected semantic changes for LTO. In the long run, maybe ld64 needs to design a new set of APIs to use the new C++ APIs but this is out of scope of this patch.

To reiterate @steven_wu 's point the aim of this patch is to hide the mechanics of out-of-process code generation from ld64, because we can't exit the linker early-like with gold. This is because ld64 tries to remap symbol information deduced by looking at the bitcode to whatever it gets back after thinlto code generation. These customisation point are very similar to what a ThinBackendProc is in the new LTO API. In this case LocalProcessCodeGenManager is like a ThinBackendProc, that emits sliced indices and constructs a clang invocation that performs the codegen and then manages communicating the outputs to the linker via a callback.

[tejohnson]

In D55080#1329591, @dang wrote:

In D55080#1328603, @steven_wu wrote:

Thanks for taking a look. This patch is adding the customization points for thinLTO legacy API, which the code generator constructs clang invocations to do code generation. There is no dependency on any build system here and it only has a prove of concept codegen manager which invokes clang directly and collect the result back. You can replace this codegen manager with any protocol that is needed to talk to build system to run clang codegen.
XPC is the way to send information between process on Darwin, which is probably what we are going to use to talk to build system. If interested, I can post a patch which have example how to construct XPC communications, but there isn’t a build system you can use to listen on the other side to run the job yet.
When I say there are no code change for linker, I really mean there is no need to change a single line of code (maybe we need to add an API to select codegen manager in the future). ld64 really has a different approach using C API, which it tries to map the object file output back to the bitcode it gets as input. Terminating and relaunching the linker might has unexpected semantic changes for LTO. In the long run, maybe ld64 needs to design a new set of APIs to use the new C++ APIs but this is out of scope of this patch.

To reiterate @steven_wu 's point the aim of this patch is to hide the mechanics of out-of-process code generation from ld64, because we can't exit the linker early-like with gold. This is because ld64 tries to remap symbol information deduced by looking at the bitcode to whatever it gets back after thinlto code generation. These customisation point are very similar to what a ThinBackendProc is in the new LTO API. In this case LocalProcessCodeGenManager is like a ThinBackendProc, that emits sliced indices and constructs a clang invocation that performs the codegen and then manages communicating the outputs to the linker via a callback.

Thanks for the clarifications. Would it be possible to utilize ThinBackendProc for this instead of a new CodeGenManager class? I.e. make a new derived version that does the index file write and spawns the local processes? The advantage is that it would start converging the implementations. And I think this could aid in refactoring suggested below to avoid duplication. Another advantage is that both LTO API's would have access to all backend implementations (in process, write indexes and exit, write indexes and use local processes, etc).

lib/LTO/ThinLTOOutOfProcessCodeGenerator.cpp

182

There's a huge amount of code duplication between this and the base ThinLTOCodeGenerator::run(). Perhaps ThinLTOCodeGenerator can be refactored to use a CodegenManager, and have an in-process thread version of CodegenManager so that both can use the same base run() method but the customization points would be in the CodegenManager virtual methods. Or even better, refactor to use ThinBackendProc (see comment above)?

[steven_wu]

Thanks for the clarifications. Would it be possible to utilize ThinBackendProc for this instead of a new CodeGenManager class? I.e. make a new derived version that does the index file write and spawns the local processes? The advantage is that it would start converging the implementations. And I think this could aid in refactoring suggested below to avoid duplication. Another advantage is that both LTO API's would have access to all backend implementations (in process, write indexes and exit, write indexes and use local processes, etc).

It think that is definitely possible, but as you mentioned above, most of the code duplication is coming from ThinLTOCodeGenerator. Most of the duplication is to help me maintain is downstream so the patch can be able to constantly rebasing it to master. I want to see how much value does this have to upstream for other legacy C API users before I refactor both new and old API to share more interfaces. I will ping Sony linker team offline to see if they have any comments on this.

The other possibility is to refactor the ThinLTOCodegenator to use ThinBackendProc as a starting point. I haven't done any research to see how easy it is to adapt the C APIs to the new ones but I think that should be doable as well.

include/llvm/LTO/legacy/ThinLTOCodeGenerator.h

 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_LTO_THINLTOCODEGENERATOR_H
 #define LLVM_LTO_THINLTOCODEGENERATOR_H
 
 #include "llvm-c/lto.h"
 #include "llvm/ADT/StringSet.h"
 #include "llvm/ADT/Triple.h"
+#include "llvm/IR/DiagnosticInfo.h"
+#include "llvm/IR/DiagnosticPrinter.h"
 #include "llvm/IR/ModuleSummaryIndex.h"
 #include "llvm/Support/CachePruning.h"
 #include "llvm/Support/CodeGen.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Target/TargetOptions.h"
+#include "llvm/Transforms/IPO/FunctionImport.h"
 
 #include <string>
 
 namespace llvm {
 class StringRef;
 class LLVMContext;
 class TargetMachine;
 
+class ThinLTODiagnosticInfo : public DiagnosticInfo {
+  const Twine &Msg;
+
+public:
+  ThinLTODiagnosticInfo(const Twine &DiagMsg,
+                        DiagnosticSeverity Severity = DS_Error)
+      : DiagnosticInfo(DK_Linker, Severity), Msg(DiagMsg) {}
+  void print(DiagnosticPrinter &DP) const override { DP << Msg; }
+};
+
 /// Wrapper around MemoryBufferRef, owning the identifier
 class ThinLTOBuffer {
   std::string OwnedIdentifier;
   StringRef Buffer;
 
 public:
   ThinLTOBuffer(StringRef Buffer, StringRef Identifier)
       : OwnedIdentifier(Identifier), Buffer(Buffer) {}
 
   MemoryBufferRef getMemBuffer() const {
     return MemoryBufferRef(Buffer,
                            {OwnedIdentifier.c_str(), OwnedIdentifier.size()});
   }
   StringRef getBuffer() const { return Buffer; }
   StringRef getBufferIdentifier() const { return OwnedIdentifier; }
+  void setBufferIdentifier(StringRef Name) { OwnedIdentifier = Name; }
 };
 
 /// Helper to gather options relevant to the target machine creation
 struct TargetMachineBuilder {
   Triple TheTriple;
   std::string MCpu;
   std::string MAttr;
   TargetOptions Options;
   Optional<Reloc::Model> RelocModel;
   CodeGenOpt::Level CGOptLevel = CodeGenOpt::Aggressive;
 
+  // Initialize the TargetMachine builder for a given Triple
+  void init(const Triple &TheTriple);
   std::unique_ptr<TargetMachine> create() const;
 };
 
+/// Manage caching for a single Module.
+class ModuleCacheEntry {
+protected:
+  SmallString<128> EntryPath;
+
+public:
+  // Create a cache entry. This compute a unique hash for the Module considering
+  // the current list of export/import, and offer an interface to query to
+  // access the content in the cache.
+  ModuleCacheEntry(
+      StringRef CachePath, const ModuleSummaryIndex &Index, StringRef ModuleID,
+      const FunctionImporter::ImportMapTy &ImportList,
+      const FunctionImporter::ExportSetTy &ExportList,
+      const std::map<GlobalValue::GUID, GlobalValue::LinkageTypes> &ResolvedODR,
+      const GVSummaryMapTy &DefinedGVSummaries, unsigned OptLevel,
+      bool Freestanding, const TargetMachineBuilder &TMBuilder);
+
+  // Access the path to this entry in the cache.
+  StringRef getEntryPath() { return EntryPath; }
+
+  // Try loading the buffer for this cache entry.
+  ErrorOr<std::unique_ptr<MemoryBuffer>> tryLoadingBuffer();
+
+  // Cache the Produced object file from MemoryBuffer.
+  void write(const MemoryBuffer &OutputBuffer);
+
+  // Cache the Produced object file from file.
+  void write(StringRef FilePath);
+};
+
 /// This class define an interface similar to the LTOCodeGenerator, but adapted
 /// for ThinLTO processing.
 /// The ThinLTOCodeGenerator is not intended to be reuse for multiple
 /// compilation: the model is that the client adds modules to the generator and
 /// ask to perform the ThinLTO optimizations / codegen, and finally destroys the
 /// codegenerator.
 class ThinLTOCodeGenerator {
 public:
+  virtual ~ThinLTOCodeGenerator() = default;
   /// Add given module to the code generator.
   void addModule(StringRef Identifier, StringRef Data);
 
   /**
    * Adds to a list of all global symbols that must exist in the final generated
    * code. If a symbol is not listed there, it will be optimized away if it is
    * inlined into every usage.
    */
public:
 
   /**
    * Process all the modules that were added to the code generator in parallel.
    *
    * Client can access the resulting object files using getProducedBinaries(),
    * unless setGeneratedObjectsDirectory() has been called, in which case
    * results are available through getProducedBinaryFiles().
    */
-  void run();
+  virtual void run();
 
   /**
    * Return the "in memory" binaries produced by the code generator. This is
    * filled after run() unless setGeneratedObjectsDirectory() has been
    * called, in which case results are available through
    * getProducedBinaryFiles().
    */
   std::vector<std::unique_ptr<MemoryBuffer>> &getProducedBinaries() {
public:
    *  - The pruning expiration time indicates to the garbage collector how old
    *    an entry needs to be to be removed.
    *  - Finally, the garbage collector can be instructed to prune the cache till
    *    the occupied space goes below a threshold.
    * @{
    */
 
   struct CachingOptions {
-    std::string Path;                    // Path to the cache, empty to disable.
+    std::string Path; // Path to the cache, empty to disable.
     CachePruningPolicy Policy;
   };
 
   /// Provide a path to a directory where to store the cached files for
   /// incremental build.
   void setCacheDir(std::string Path) { CacheOptions.Path = std::move(Path); }
 
   /// Cache policy: interval (seconds) between two prunes of the cache. Set to a
   /// negative value to disable pruning. A value of 0 will force pruning to
   /// occur.
   void setCachePruningInterval(int Interval) {
-    if(Interval < 0)
+    if (Interval < 0)
       CacheOptions.Policy.Interval.reset();
     else
       CacheOptions.Policy.Interval = std::chrono::seconds(Interval);
   }
 
   /// Cache policy: expiration (in seconds) for an entry.
   /// A value of 0 will be ignored.
   void setCacheEntryExpiration(unsigned Expiration) {
public:
 
   /**
    * Perform post-importing ThinLTO optimizations.
    */
   void optimize(Module &Module);
 
   /**@}*/
 
-private:
+protected:
+  /// Simple helper to save temporary files for debug.
+  void saveTempBitcode(const Module &TheModule, StringRef TempDir,
+                       unsigned count, StringRef Suffix);
+
+  StringMap<MemoryBufferRef>
+  generateModuleMap(const std::vector<ThinLTOBuffer> &Modules);
+
+  std::unique_ptr<Module> loadModuleFromBuffer(const MemoryBufferRef &Buffer,
+                                               LLVMContext &Context, bool Lazy,
+                                               bool IsImporting);
+
+  void crossImportIntoModule(Module &TheModule, const ModuleSummaryIndex &Index,
+                             StringMap<MemoryBufferRef> &ModuleMap,
+                             const FunctionImporter::ImportMapTy &ImportList);
+
+  DenseSet<GlobalValue::GUID>
+  computeGUIDPreservedSymbols(const StringSet<> &PreservedSymbols,
+                              const Triple &TheTriple);
+
+  std::unique_ptr<MemoryBuffer>
+  ProcessThinLTOModule(Module &TheModule, ModuleSummaryIndex &Index,
+                       StringMap<MemoryBufferRef> &ModuleMap, TargetMachine &TM,
+                       const FunctionImporter::ImportMapTy &ImportList,
+                       const FunctionImporter::ExportSetTy &ExportList,
+                       const DenseSet<GlobalValue::GUID> &GUIDPreservedSymbols,
+                       const GVSummaryMapTy &DefinedGlobals,
+                       const ThinLTOCodeGenerator::CachingOptions &CacheOptions,
+                       bool DisableCodeGen, StringRef SaveTempsDir,
+                       bool Freestanding, unsigned OptLevel, unsigned count);
+
+  /// Resolve LinkOnce/Weak symbols. Record resolutions in the \p ResolvedODR map
+  /// for caching, and in the \p Index for application during the ThinLTO
+  /// backends. This is needed for correctness for exported symbols (ensure
+  /// at least one copy kept) and a compile-time optimization (to drop duplicate
+  /// copies when possible).
+  void resolvePrevailingInIndex(
+      ModuleSummaryIndex &Index,
+      StringMap<std::map<GlobalValue::GUID, GlobalValue::LinkageTypes>>
+          &ResolvedODR);
+
+  void internalizeAndPromoteInIndex(
+      const StringMap<FunctionImporter::ExportSetTy> &ExportLists,
+      const DenseSet<GlobalValue::GUID> &GUIDPreservedSymbols,
+      ModuleSummaryIndex &Index);
+
+  void computeDeadSymbolsInIndex(
+      ModuleSummaryIndex &Index,
+      const DenseSet<GlobalValue::GUID> &GUIDPreservedSymbols);
+
+  std::unique_ptr<MemoryBuffer> codegen(Module &TheModule);
+
+  /// Write out the generated object file, either from CacheEntryPath or from
+  /// OutputBuffer, preferring hard-link when possible.
+  /// Returns the path to the generated file in SavedObjectsDirectoryPath.
+  std::string writeGeneratedObject(int count, StringRef CacheEntryPath,
+                                   StringRef SavedObjectsDirectoryPath,
+                                   const MemoryBuffer &OutputBuffer);
+
   /// Helper factory to build a TargetMachine
   TargetMachineBuilder TMBuilder;
 
-  /// Vector holding the in-memory buffer containing the produced binaries, when
-  /// SavedObjectsDirectoryPath isn't set.
+  /// Vector holding the in-memory buffer containing the produced binaries,
+  /// when SavedObjectsDirectoryPath isn't set.
   std::vector<std::unique_ptr<MemoryBuffer>> ProducedBinaries;
 
-  /// Path to generated files in the supplied SavedObjectsDirectoryPath if any.
+  /// Path to generated files in the supplied SavedObjectsDirectoryPath if
+  /// any.
   std::vector<std::string> ProducedBinaryFiles;
 
   /// Vector holding the input buffers containing the bitcode modules to
   /// process.
   std::vector<ThinLTOBuffer> Modules;
 
   /// Set of symbols that need to be preserved outside of the set of bitcode
   /// files.
   StringSet<> PreservedSymbols;
 
   /// Set of symbols that are cross-referenced between bitcode files.
   StringSet<> CrossReferencedSymbols;
 
   /// Control the caching behavior.
   CachingOptions CacheOptions;
 
   /// Path to a directory to save the temporary bitcode files.
   std::string SaveTempsDir;
 
   /// Path to a directory to save the generated object files.
   std::string SavedObjectsDirectoryPath;
 
-  /// Flag to enable/disable CodeGen. When set to true, the process stops after
-  /// optimizations and a bitcode is produced.
+  /// Flag to enable/disable CodeGen. When set to true, the process stops
+  /// after optimizations and a bitcode is produced.
   bool DisableCodeGen = false;
 
-  /// Flag to indicate that only the CodeGen will be performed, no cross-module
-  /// importing or optimization.
+  /// Flag to indicate that only the CodeGen will be performed, no
+  /// cross-module importing or optimization.
   bool CodeGenOnly = false;
 
-  /// Flag to indicate that the optimizer should not assume builtins are present
-  /// on the target.
+  /// Flag to indicate that the optimizer should not assume builtins are
+  /// present on the target.
   bool Freestanding = false;
 
   /// IR Optimization Level [0-3].
   unsigned OptLevel = 3;
 };
-}
+} // namespace llvm
 #endif

include/llvm/LTO/legacy/ThinLTOOutOfProcessCodeGenerator.h

+//===-ThinLTOOutOfProcessCodeGenerator.h - LLVM Link Time Optimizer
+//-------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file declares the ThinLTOOutOfProcessCodeGenerator class, similar to the
+// LTOCodeGenerator but for the ThinLTO scheme. It provides an interface for
+// linker plugin.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LTO_THINLTOOUTOFPROCESSCODEGENERATOR_H
+#define LLVM_LTO_THINLTOOUTOFPROCESSCODEGENERATOR_H
+
+#include "llvm-c/lto.h"
+#include "llvm/ADT/StringSet.h"
+#include "llvm/ADT/Triple.h"
+#include "llvm/IR/ModuleSummaryIndex.h"
+#include "llvm/LTO/legacy/ThinLTOCodeGenerator.h"
+#include "llvm/Support/CachePruning.h"
+#include "llvm/Support/CodeGen.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/Program.h"
+#include "llvm/Support/ThreadPool.h"
+#include "llvm/Target/TargetOptions.h"
+
+#include <condition_variable>
+#include <utility>
+
+#include <string>
+
+namespace llvm {
+class StringRef;
+class LLVMContext;
+class TargetMachine;
+
+/// Wrapper class to construct a clang invocation suitable for thinlto
+struct ClangThinLTOInvocation {
+  using Callback = std::function<void(const ClangThinLTOInvocation &)>;
+  llvm::ThinLTOBuffer InputBuffer;
+  llvm::StringRef InputTmpFileName;
+  llvm::SmallString<PATH_MAX> IndexPath;
+  llvm::SmallString<PATH_MAX> OutputPath;
+  std::vector<StringRef> ImportFileList;
+
+  ClangThinLTOInvocation(llvm::ThinLTOBuffer const &InputModuleBuffer,
+                         llvm::StringRef InputTmpFileName,
+                         llvm::StringRef IndexPath,
+                         std::vector<StringRef> &&ImportFileList);
+};
+
+class ThinLTOCodegenManager {
+public:
+  virtual ~ThinLTOCodegenManager() {}
+
+  void createInvocation(llvm::ThinLTOBuffer const &InputModuleBuffer,
+                        llvm::StringRef InputTmpFileName,
+                        llvm::StringRef IndexPath,
+                        std::vector<llvm::StringRef> ImportFileList,
+                        ClangThinLTOInvocation::Callback CompletionCallback);
+
+  virtual void
+  setTargetMachineBuilder(const llvm::TargetMachineBuilder &Builder) {
+    TMBuilder = Builder;
+  }
+
+  // Pure virtual function. Subclass needs to provide implementation.
+  virtual void materializeModule(llvm::ThinLTOBuffer &Module) = 0;
+  virtual void execute(const ClangThinLTOInvocation &Invocation,
+                       ClangThinLTOInvocation::Callback Callback) = 0;
+
+protected:
+  llvm::TargetMachineBuilder TMBuilder;
+};
+
+class LocalProcessCodegenManager : public ThinLTOCodegenManager {
+public:
+  void materializeModule(llvm::ThinLTOBuffer &Module) override;
+
+  void execute(const ClangThinLTOInvocation &Invocation,
+               ClangThinLTOInvocation::Callback Callback) override;
+
+private:
+  StringRef getExecutablePath();
+  static std::string ExecutablePath;
+};
+
+/// This class exposes the same interface as ThinLTOCodeGenerator, but performs
+/// the code generation steps in a distributed way by delegating to an XPC
+/// service
+class ThinLTOOutOfProcessCodeGenerator : public ThinLTOCodeGenerator {
+public:
+  ThinLTOOutOfProcessCodeGenerator(ThinLTOCodegenManager *Manager) {
+    CodegenManager.reset(Manager);
+  }
+
+  /**
+   * Process all the modules that were added to the code generator in
+   * parallel.
+   *
+   * Client can access the resulting object files using getProducedBinaries(),
+   * unless setGeneratedObjectsDirectory() has been called, in which case
+   * results are available through getProducedBinaryFiles().
+   */
+  void run() override;
+
+private:
+  /// CodegenManager.
+  std::unique_ptr<ThinLTOCodegenManager> CodegenManager;
+
+  /// Thread Lock.
+  std::mutex ThreadLock;
+};
+} // namespace llvm
+#endif

lib/LTO/CMakeLists.txt

 add_llvm_library(LLVMLTO
   Caching.cpp
   LTO.cpp
   LTOBackend.cpp
   LTOModule.cpp
   LTOCodeGenerator.cpp
   UpdateCompilerUsed.cpp
   ThinLTOCodeGenerator.cpp
+  ThinLTOOutOfProcessCodeGenerator.cpp
 
   ADDITIONAL_HEADER_DIRS
   ${LLVM_MAIN_INCLUDE_DIR}/llvm/LTO
 
   DEPENDS
   intrinsics_gen
   llvm_vcsrevision_h
 )

lib/LTO/ThinLTOCodeGenerator.cpp

 
 #define DEBUG_TYPE "thinlto"
 
 namespace llvm {
 // Flags -discard-value-names, defined in LTOCodeGenerator.cpp
 extern cl::opt<bool> LTODiscardValueNames;
 extern cl::opt<std::string> LTORemarksFilename;
 extern cl::opt<bool> LTOPassRemarksWithHotness;
-}
-
-namespace {
 
-static cl::opt<int>
-    ThreadCount("threads", cl::init(llvm::heavyweight_hardware_concurrency()));
-
-// Simple helper to save temporary files for debug.
-static void saveTempBitcode(const Module &TheModule, StringRef TempDir,
-                            unsigned count, StringRef Suffix) {
-  if (TempDir.empty())
-    return;
-  // User asked to save temps, let dump the bitcode file after import.
-  std::string SaveTempPath = (TempDir + llvm::Twine(count) + Suffix).str();
-  std::error_code EC;
-  raw_fd_ostream OS(SaveTempPath, EC, sys::fs::F_None);
-  if (EC)
-    report_fatal_error(Twine("Failed to open ") + SaveTempPath +
-                       " to save optimized bitcode\n");
-  WriteBitcodeToFile(TheModule, OS, /* ShouldPreserveUseListOrder */ true);
-}
+cl::opt<int> ThreadCount("threads",
+                         cl::init(llvm::heavyweight_hardware_concurrency()));
+} // namespace llvm
 
+namespace {
 static const GlobalValueSummary *
 getFirstDefinitionForLinker(const GlobalValueSummaryList &GVSummaryList) {
   // If there is any strong definition anywhere, get it.
   auto StrongDefForLinker = llvm::find_if(
       GVSummaryList, [](const std::unique_ptr<GlobalValueSummary> &Summary) {
         auto Linkage = Summary->linkage();
         return !GlobalValue::isAvailableExternallyLinkage(Linkage) &&
                !GlobalValue::isWeakForLinker(Linkage);
static void computePrevailingCopies(
 
   for (auto &I : Index) {
     if (HasMultipleCopies(I.second.SummaryList))
       PrevailingCopy[I.first] =
           getFirstDefinitionForLinker(I.second.SummaryList);
   }
 }
 
-static StringMap<MemoryBufferRef>
-generateModuleMap(const std::vector<ThinLTOBuffer> &Modules) {
-  StringMap<MemoryBufferRef> ModuleMap;
-  for (auto &ModuleBuffer : Modules) {
-    assert(ModuleMap.find(ModuleBuffer.getBufferIdentifier()) ==
-               ModuleMap.end() &&
-           "Expect unique Buffer Identifier");
-    ModuleMap[ModuleBuffer.getBufferIdentifier()] = ModuleBuffer.getMemBuffer();
-  }
-  return ModuleMap;
-}
-
 static void promoteModule(Module &TheModule, const ModuleSummaryIndex &Index) {
   if (renameModuleForThinLTO(TheModule, Index))
     report_fatal_error("renameModuleForThinLTO failed");
 }
 
-namespace {
-class ThinLTODiagnosticInfo : public DiagnosticInfo {
-  const Twine &Msg;
-public:
-  ThinLTODiagnosticInfo(const Twine &DiagMsg,
-                        DiagnosticSeverity Severity = DS_Error)
-      : DiagnosticInfo(DK_Linker, Severity), Msg(DiagMsg) {}
-  void print(DiagnosticPrinter &DP) const override { DP << Msg; }
-};
-}
-
 /// Verify the module and strip broken debug info.
 static void verifyLoadedModule(Module &TheModule) {
   bool BrokenDebugInfo = false;
   if (verifyModule(TheModule, &dbgs(), &BrokenDebugInfo))
     report_fatal_error("Broken module found, compilation aborted!");
   if (BrokenDebugInfo) {
     TheModule.getContext().diagnose(ThinLTODiagnosticInfo(
         "Invalid debug info found, debug info will be stripped", DS_Warning));
     StripDebugInfo(TheModule);
   }
 }
 
-static std::unique_ptr<Module>
-loadModuleFromBuffer(const MemoryBufferRef &Buffer, LLVMContext &Context,
-                     bool Lazy, bool IsImporting) {
-  SMDiagnostic Err;
-  Expected<std::unique_ptr<Module>> ModuleOrErr =
-      Lazy
-          ? getLazyBitcodeModule(Buffer, Context,
-                                 /* ShouldLazyLoadMetadata */ true, IsImporting)
-          : parseBitcodeFile(Buffer, Context);
-  if (!ModuleOrErr) {
-    handleAllErrors(ModuleOrErr.takeError(), [&](ErrorInfoBase &EIB) {
-      SMDiagnostic Err = SMDiagnostic(Buffer.getBufferIdentifier(),
-                                      SourceMgr::DK_Error, EIB.message());
-      Err.print("ThinLTO", errs());
-    });
-    report_fatal_error("Can't load module, abort.");
-  }
-  if (!Lazy)
-    verifyLoadedModule(*ModuleOrErr.get());
-  return std::move(ModuleOrErr.get());
-}
-
-static void
-crossImportIntoModule(Module &TheModule, const ModuleSummaryIndex &Index,
-                      StringMap<MemoryBufferRef> &ModuleMap,
-                      const FunctionImporter::ImportMapTy &ImportList) {
-  auto Loader = [&](StringRef Identifier) {
-    return loadModuleFromBuffer(ModuleMap[Identifier], TheModule.getContext(),
-                                /*Lazy=*/true, /*IsImporting*/ true);
-  };
-
-  FunctionImporter Importer(Index, Loader);
-  Expected<bool> Result = Importer.importFunctions(TheModule, ImportList);
-  if (!Result) {
-    handleAllErrors(Result.takeError(), [&](ErrorInfoBase &EIB) {
-      SMDiagnostic Err = SMDiagnostic(TheModule.getModuleIdentifier(),
-                                      SourceMgr::DK_Error, EIB.message());
-      Err.print("ThinLTO", errs());
-    });
-    report_fatal_error("importFunctions failed");
-  }
-  // Verify again after cross-importing.
-  verifyLoadedModule(TheModule);
-}
-
 static void optimizeModule(Module &TheModule, TargetMachine &TM,
                            unsigned OptLevel, bool Freestanding) {
   // Populate the PassManager
   PassManagerBuilder PMB;
   PMB.LibraryInfo = new TargetLibraryInfoImpl(TM.getTargetTriple());
   if (Freestanding)
     PMB.LibraryInfo->disableAllFunctions();
   PMB.Inliner = createFunctionInliningPass();
static void optimizeModule(Module &TheModule, TargetMachine &TM,
   PM.add(createTargetTransformInfoWrapperPass(TM.getTargetIRAnalysis()));
 
   // Add optimizations
   PMB.populateThinLTOPassManager(PM);
 
   PM.run(TheModule);
 }
 
-// Convert the PreservedSymbols map from "Name" based to "GUID" based.
-static DenseSet<GlobalValue::GUID>
-computeGUIDPreservedSymbols(const StringSet<> &PreservedSymbols,
-                            const Triple &TheTriple) {
-  DenseSet<GlobalValue::GUID> GUIDPreservedSymbols(PreservedSymbols.size());
-  for (auto &Entry : PreservedSymbols) {
-    StringRef Name = Entry.first();
-    if (TheTriple.isOSBinFormatMachO() && Name.size() > 0 && Name[0] == '_')
-      Name = Name.drop_front();
-    GUIDPreservedSymbols.insert(GlobalValue::getGUID(Name));
-  }
-  return GUIDPreservedSymbols;
-}
-
-std::unique_ptr<MemoryBuffer> codegenModule(Module &TheModule,
+static std::unique_ptr<MemoryBuffer> codegenModule(Module &TheModule,
                                             TargetMachine &TM) {
   SmallVector<char, 128> OutputBuffer;
 
   // CodeGen
   {
     raw_svector_ostream OS(OutputBuffer);
     legacy::PassManager PM;
 
-    // If the bitcode files contain ARC code and were compiled with optimization,
-    // the ObjCARCContractPass must be run, so do it unconditionally here.
+    // If the bitcode files contain ARC code and were compiled with
+    // optimization, the ObjCARCContractPass must be run, so do it
+    // unconditionally here.
     PM.add(createObjCARCContractPass());
 
     // Setup the codegen now.
     if (TM.addPassesToEmitFile(PM, OS, nullptr, TargetMachine::CGFT_ObjectFile,
                                /* DisableVerify */ true))
       report_fatal_error("Failed to setup codegen");
 
     // Run codegen now. resulting binary is in OutputBuffer.
     PM.run(TheModule);
   }
   return make_unique<SmallVectorMemoryBuffer>(std::move(OutputBuffer));
 }
+} // end anonymous namespace
 
-/// Manage caching for a single Module.
-class ModuleCacheEntry {
-  SmallString<128> EntryPath;
-
-public:
-  // Create a cache entry. This compute a unique hash for the Module considering
-  // the current list of export/import, and offer an interface to query to
-  // access the content in the cache.
-  ModuleCacheEntry(
-      StringRef CachePath, const ModuleSummaryIndex &Index, StringRef ModuleID,
-      const FunctionImporter::ImportMapTy &ImportList,
-      const FunctionImporter::ExportSetTy &ExportList,
-      const std::map<GlobalValue::GUID, GlobalValue::LinkageTypes> &ResolvedODR,
-      const GVSummaryMapTy &DefinedGVSummaries, unsigned OptLevel,
-      bool Freestanding, const TargetMachineBuilder &TMBuilder) {
-    if (CachePath.empty())
-      return;
+ModuleCacheEntry::ModuleCacheEntry(
 
-    if (!Index.modulePaths().count(ModuleID))
-      // The module does not have an entry, it can't have a hash at all
-      return;
 
-    if (all_of(Index.getModuleHash(ModuleID),
-               [](uint32_t V) { return V == 0; }))
-      // No hash entry, no caching!
-      return;
+  if (all_of(Index.getModuleHash(ModuleID), [](uint32_t V) { return V == 0; }))
 
-    llvm::lto::Config Conf;
-    Conf.OptLevel = OptLevel;
-    Conf.Options = TMBuilder.Options;
-    Conf.CPU = TMBuilder.MCpu;
-    Conf.MAttrs.push_back(TMBuilder.MAttr);
-    Conf.RelocModel = TMBuilder.RelocModel;
-    Conf.CGOptLevel = TMBuilder.CGOptLevel;
-    Conf.Freestanding = Freestanding;
-    SmallString<40> Key;
-    computeLTOCacheKey(Key, Conf, Index, ModuleID, ImportList, ExportList,
-                       ResolvedODR, DefinedGVSummaries);
 
-    // This choice of file name allows the cache to be pruned (see pruneCache()
-    // in include/llvm/Support/CachePruning.h).
-    sys::path::append(EntryPath, CachePath, "llvmcache-" + Key);
-  }
 
-  // Access the path to this entry in the cache.
-  StringRef getEntryPath() { return EntryPath; }
-
   // Try loading the buffer for this cache entry.
-  ErrorOr<std::unique_ptr<MemoryBuffer>> tryLoadingBuffer() {
-    if (EntryPath.empty())
-      return std::error_code();
-    int FD;
-    SmallString<64> ResultPath;
-    std::error_code EC = sys::fs::openFileForRead(
-        Twine(EntryPath), FD, sys::fs::OF_UpdateAtime, &ResultPath);
-    if (EC)
-      return EC;
-    ErrorOr<std::unique_ptr<MemoryBuffer>> MBOrErr =
-        MemoryBuffer::getOpenFile(FD, EntryPath,
-                                  /*FileSize*/ -1,
-                                  /*RequiresNullTerminator*/ false);
-    close(FD);
-    return MBOrErr;
-  }
+ErrorOr<std::unique_ptr<MemoryBuffer>> ModuleCacheEntry::tryLoadingBuffer() {
 
-  // Cache the Produced object file
-  void write(const MemoryBuffer &OutputBuffer) {
-    if (EntryPath.empty())
-      return;
+void ModuleCacheEntry::write(const MemoryBuffer &OutputBuffer) {
 
-    // Write to a temporary to avoid race condition
-    SmallString<128> TempFilename;
-    SmallString<128> CachePath(EntryPath);
-    int TempFD;
-    llvm::sys::path::remove_filename(CachePath);
-    sys::path::append(TempFilename, CachePath, "Thin-%%%%%%.tmp.o");
-    std::error_code EC =
       sys::fs::createUniqueFile(TempFilename, TempFD, TempFilename);
-    if (EC) {
-      errs() << "Error: " << EC.message() << "\n";
-      report_fatal_error("ThinLTO: Can't get a temporary file");
-    }
-    {
-      raw_fd_ostream OS(TempFD, /* ShouldClose */ true);
-      OS << OutputBuffer.getBuffer();
-    }
-    // Rename temp file to final destination; rename is atomic
-    EC = sys::fs::rename(TempFilename, EntryPath);
-    if (EC)
-      sys::fs::remove(TempFilename);
-  }
+
+void ModuleCacheEntry::write(StringRef FilePath) {
+  if (EntryPath.empty())
+    return;
+
+  // Create a hard link to the cache. It is ok to be failed because of the file
+  // can be generated by another process. Ignore the failure and keep going.
+  if (auto EC = sys::fs::create_hard_link(FilePath, EntryPath))
+    LLVM_DEBUG(
+        dbgs()
+        << "Could not materialize the hard link to create a cache entry for"
+        << FilePath << " with error message " << EC.message() << "\n");
+}
+
+void TargetMachineBuilder::init(const Triple &Triple) {
+  // Set a default CPU for Darwin triples (copied from LTOCodeGenerator).
+  // FIXME this looks pretty terrible...
+  if (MCpu.empty() && Triple.isOSDarwin()) {
+    if (TheTriple.getArch() == llvm::Triple::x86_64)
+      MCpu = "core2";
+    else if (Triple.getArch() == llvm::Triple::x86)
+      MCpu = "yonah";
+    else if (Triple.getArchName() == "arm64e" ||
+             Triple.getArchName() == "arm64_32")
+      MCpu = "vortex";
+    else if (Triple.getArch() == llvm::Triple::aarch64)
+      MCpu = "cyclone";
+  }
+  TheTriple = std::move(Triple);
+}
+
+void ThinLTOCodeGenerator::saveTempBitcode(const Module &TheModule,
+                                           StringRef TempDir, unsigned count,
+                                           StringRef Suffix) {
+  if (TempDir.empty())
+    return;
+  // User asked to save temps, let dump the bitcode file after import.
+  std::string SaveTempPath = (TempDir + llvm::Twine(count) + Suffix).str();
+  std::error_code EC;
+  raw_fd_ostream OS(SaveTempPath, EC, sys::fs::F_None);
+  if (EC)
+    report_fatal_error(Twine("Failed to open ") + SaveTempPath +
+                       " to save optimized bitcode\n");
+  WriteBitcodeToFile(TheModule, OS, /* ShouldPreserveUseListOrder */ true);
+}
+
+StringMap<MemoryBufferRef> ThinLTOCodeGenerator::generateModuleMap(
+    const std::vector<ThinLTOBuffer> &Modules) {
+  StringMap<MemoryBufferRef> ModuleMap;
+  for (auto &ModuleBuffer : Modules) {
+    assert(ModuleMap.find(ModuleBuffer.getBufferIdentifier()) ==
+           ModuleMap.end() &&
+           "Expect unique Buffer Identifier");
+    ModuleMap[ModuleBuffer.getBufferIdentifier()] = ModuleBuffer.getMemBuffer();
+  }
+  return ModuleMap;
+}
+
+std::unique_ptr<Module>
+ThinLTOCodeGenerator::loadModuleFromBuffer(const MemoryBufferRef &Buffer,
+                                           LLVMContext &Context, bool Lazy,
+                                           bool IsImporting) {
+  SMDiagnostic Err;
+  Expected<std::unique_ptr<Module>> ModuleOrErr =
+    Lazy
+    ? getLazyBitcodeModule(Buffer, Context,
+                           /* ShouldLazyLoadMetadata */ true, IsImporting)
+    : parseBitcodeFile(Buffer, Context);
+  if (!ModuleOrErr) {
+    handleAllErrors(ModuleOrErr.takeError(), [&](ErrorInfoBase &EIB) {
+                                               SMDiagnostic Err = SMDiagnostic(Buffer.getBufferIdentifier(),
+                                                                               SourceMgr::DK_Error, EIB.message());
+                                               Err.print("ThinLTO", errs());
+                                             });
+    report_fatal_error("Can't load module, abort.");
+  }
+  if (!Lazy)
+    verifyLoadedModule(*ModuleOrErr.get());
+  return std::move(ModuleOrErr.get());
+}
+
+void ThinLTOCodeGenerator::crossImportIntoModule(
+    Module &TheModule, const ModuleSummaryIndex &Index,
+    StringMap<MemoryBufferRef> &ModuleMap,
+    const FunctionImporter::ImportMapTy &ImportList) {
+  auto Loader = [&](StringRef Identifier) {
+                  return loadModuleFromBuffer(ModuleMap[Identifier], TheModule.getContext(),
+                                              /*Lazy=*/true, /*IsImporting*/ true);
                 };
 
-static std::unique_ptr<MemoryBuffer>
-ProcessThinLTOModule(Module &TheModule, ModuleSummaryIndex &Index,
+  FunctionImporter Importer(Index, Loader);
+  Expected<bool> Result = Importer.importFunctions(TheModule, ImportList);
+  if (!Result) {
+    handleAllErrors(Result.takeError(), [&](ErrorInfoBase &EIB) {
+                                          SMDiagnostic Err = SMDiagnostic(TheModule.getModuleIdentifier(),
+                                                                          SourceMgr::DK_Error, EIB.message());
+                                          Err.print("ThinLTO", errs());
+                                        });
+    report_fatal_error("importFunctions failed");
+  }
+  // Verify again after cross-importing.
+  verifyLoadedModule(TheModule);
+}
+
+// Convert the PreservedSymbols map from "Name" based to "GUID" based.
+DenseSet<GlobalValue::GUID> ThinLTOCodeGenerator::computeGUIDPreservedSymbols(
+    const StringSet<> &PreservedSymbols, const Triple &TheTriple) {
+  DenseSet<GlobalValue::GUID> GUIDPreservedSymbols(PreservedSymbols.size());
+  for (auto &Entry : PreservedSymbols) {
+    StringRef Name = Entry.first();
+    if (TheTriple.isOSBinFormatMachO() && Name.size() > 0 && Name[0] == '_')
+      Name = Name.drop_front();
+    GUIDPreservedSymbols.insert(GlobalValue::getGUID(Name));
+  }
+  return GUIDPreservedSymbols;
+}
+
+std::unique_ptr<MemoryBuffer>
+ThinLTOCodeGenerator::ProcessThinLTOModule(Module &TheModule, ModuleSummaryIndex &Index,
                      StringMap<MemoryBufferRef> &ModuleMap, TargetMachine &TM,
                      const FunctionImporter::ImportMapTy &ImportList,
                      const FunctionImporter::ExportSetTy &ExportList,
                      const DenseSet<GlobalValue::GUID> &GUIDPreservedSymbols,
                      const GVSummaryMapTy &DefinedGlobals,
                      const ThinLTOCodeGenerator::CachingOptions &CacheOptions,
                      bool DisableCodeGen, StringRef SaveTempsDir,
                      bool Freestanding, unsigned OptLevel, unsigned count) {
ThinLTOCodeGenerator::ProcessThinLTOModule(Module &TheModule, ModuleSummaryIndex &Index,
   return codegenModule(TheModule, TM);
 }
 
 /// Resolve prevailing symbols. Record resolutions in the \p ResolvedODR map
 /// for caching, and in the \p Index for application during the ThinLTO
 /// backends. This is needed for correctness for exported symbols (ensure
 /// at least one copy kept) and a compile-time optimization (to drop duplicate
 /// copies when possible).
-static void resolvePrevailingInIndex(
+void ThinLTOCodeGenerator::resolvePrevailingInIndex(
     ModuleSummaryIndex &Index,
     StringMap<std::map<GlobalValue::GUID, GlobalValue::LinkageTypes>>
         &ResolvedODR) {
-
   DenseMap<GlobalValue::GUID, const GlobalValueSummary *> PrevailingCopy;
   computePrevailingCopies(Index, PrevailingCopy);
 
   auto isPrevailing = [&](GlobalValue::GUID GUID, const GlobalValueSummary *S) {
     const auto &Prevailing = PrevailingCopy.find(GUID);
     // Not in map means that there was only one copy, which must be prevailing.
     if (Prevailing == PrevailingCopy.end())
       return true;
     return Prevailing->second == S;
   };
 
   auto recordNewLinkage = [&](StringRef ModuleIdentifier,
                               GlobalValue::GUID GUID,
                               GlobalValue::LinkageTypes NewLinkage) {
     ResolvedODR[ModuleIdentifier][GUID] = NewLinkage;
   };
 
   thinLTOResolvePrevailingInIndex(Index, isPrevailing, recordNewLinkage);
 }
 
-// Initialize the TargetMachine builder for a given Triple
-static void initTMBuilder(TargetMachineBuilder &TMBuilder,
-                          const Triple &TheTriple) {
-  // Set a default CPU for Darwin triples (copied from LTOCodeGenerator).
-  // FIXME this looks pretty terrible...
-  if (TMBuilder.MCpu.empty() && TheTriple.isOSDarwin()) {
-    if (TheTriple.getArch() == llvm::Triple::x86_64)
-      TMBuilder.MCpu = "core2";
-    else if (TheTriple.getArch() == llvm::Triple::x86)
-      TMBuilder.MCpu = "yonah";
-    else if (TheTriple.getArch() == llvm::Triple::aarch64)
-      TMBuilder.MCpu = "cyclone";
-  }
-  TMBuilder.TheTriple = std::move(TheTriple);
-}
-
-} // end anonymous namespace
-
 void ThinLTOCodeGenerator::addModule(StringRef Identifier, StringRef Data) {
   ThinLTOBuffer Buffer(Data, Identifier);
   LLVMContext Context;
   StringRef TripleStr;
   ErrorOr<std::string> TripleOrErr = expectedToErrorOrAndEmitErrors(
       Context, getBitcodeTargetTriple(Buffer.getMemBuffer()));
 
   if (TripleOrErr)
     TripleStr = *TripleOrErr;
 
   Triple TheTriple(TripleStr);
 
   if (Modules.empty())
-    initTMBuilder(TMBuilder, Triple(TheTriple));
+    TMBuilder.init(Triple(TheTriple));
   else if (TMBuilder.TheTriple != TheTriple) {
     if (!TMBuilder.TheTriple.isCompatibleWith(TheTriple))
       report_fatal_error("ThinLTO modules with incompatible triples not "
                          "supported");
-    initTMBuilder(TMBuilder, Triple(TMBuilder.TheTriple.merge(TheTriple)));
+    auto TheMergedTriple = Triple(TMBuilder.TheTriple.merge(TheTriple));
+    TMBuilder.init(TheMergedTriple);
   }
 
   Modules.push_back(Buffer);
 }
 
 void ThinLTOCodeGenerator::preserveSymbol(StringRef Name) {
   PreservedSymbols.insert(Name);
 }
if (Error Err = readModuleSummaryIndex(ModuleBuffer.getMemBuffer(),
           std::move(Err), errs(),
           "error: can't create module summary index for buffer: ");
       return nullptr;
     }
   }
   return CombinedIndex;
 }
 
-static void internalizeAndPromoteInIndex(
+void ThinLTOCodeGenerator::internalizeAndPromoteInIndex(
     const StringMap<FunctionImporter::ExportSetTy> &ExportLists,
     const DenseSet<GlobalValue::GUID> &GUIDPreservedSymbols,
     ModuleSummaryIndex &Index) {
   auto isExported = [&](StringRef ModuleIdentifier, GlobalValue::GUID GUID) {
     const auto &ExportList = ExportLists.find(ModuleIdentifier);
     return (ExportList != ExportLists.end() &&
             ExportList->second.count(GUID)) ||
            GUIDPreservedSymbols.count(GUID);
   };
 
   thinLTOInternalizeAndPromoteInIndex(Index, isExported);
 }
 
-static void computeDeadSymbolsInIndex(
+void ThinLTOCodeGenerator::computeDeadSymbolsInIndex(
     ModuleSummaryIndex &Index,
     const DenseSet<GlobalValue::GUID> &GUIDPreservedSymbols) {
   // We have no symbols resolution available. And can't do any better now in the
   // case where the prevailing symbol is in a native object. It can be refined
   // with linker information in the future.
   auto isPrevailing = [&](GlobalValue::GUID G) {
     return PrevailingType::Unknown;
   };
report_fatal_error(Twine("Failed to open ") + OutputName +
                        " to save imports lists\n");
 }
 
 /**
  * Perform internalization. Index is updated to reflect linkage changes.
  */
 void ThinLTOCodeGenerator::internalize(Module &TheModule,
                                        ModuleSummaryIndex &Index) {
-  initTMBuilder(TMBuilder, Triple(TheModule.getTargetTriple()));
+  TMBuilder.init(Triple(TheModule.getTargetTriple()));
   auto ModuleCount = Index.modulePaths().size();
   auto ModuleIdentifier = TheModule.getModuleIdentifier();
 
   // Convert the preserved symbols set from string to GUID
   auto GUIDPreservedSymbols =
       computeGUIDPreservedSymbols(PreservedSymbols, TMBuilder.TheTriple);
 
   // Collect for each module the list of function it defines (GUID -> Summary).
void ThinLTOCodeGenerator::internalize(Module &TheModule,
   thinLTOInternalizeModule(TheModule,
                            ModuleToDefinedGVSummaries[ModuleIdentifier]);
 }
 
 /**
  * Perform post-importing ThinLTO optimizations.
  */
 void ThinLTOCodeGenerator::optimize(Module &TheModule) {
-  initTMBuilder(TMBuilder, Triple(TheModule.getTargetTriple()));
+  TMBuilder.init(Triple(TheModule.getTargetTriple()));
 
   // Optimize now
   optimizeModule(TheModule, *TMBuilder.create(), OptLevel, Freestanding);
 }
 
-/// Write out the generated object file, either from CacheEntryPath or from
-/// OutputBuffer, preferring hard-link when possible.
-/// Returns the path to the generated file in SavedObjectsDirectoryPath.
-static std::string writeGeneratedObject(int count, StringRef CacheEntryPath,
+/**
+ * Perform ThinLTO CodeGen.
+ */
+std::unique_ptr<MemoryBuffer> ThinLTOCodeGenerator::codegen(Module &TheModule) {
+  TMBuilder.init(Triple(TheModule.getTargetTriple()));
+  return codegenModule(TheModule, *TMBuilder.create());
+}
+
+std::string
+ThinLTOCodeGenerator::writeGeneratedObject(int count, StringRef CacheEntryPath,
                                            StringRef SavedObjectsDirectoryPath,
                                            const MemoryBuffer &OutputBuffer) {
   SmallString<128> OutputPath(SavedObjectsDirectoryPath);
   llvm::sys::path::append(OutputPath, Twine(count) + ".thinlto.o");
   OutputPath.c_str(); // Ensure the string is null terminated.
   if (sys::fs::exists(OutputPath))
     sys::fs::remove(OutputPath);
 
   // We don't return a memory buffer to the linker, just a list of files.
   if (!CacheEntryPath.empty()) {
if (!SaveTempsDir.empty()) {
     std::error_code EC;
     raw_fd_ostream OS(SaveTempPath, EC, sys::fs::F_None);
     if (EC)
       report_fatal_error(Twine("Failed to open ") + SaveTempPath +
                          " to save optimized bitcode\n");
     WriteIndexToFile(*Index, OS);
   }
 
-
   // Prepare the module map.
   auto ModuleMap = generateModuleMap(Modules);
   auto ModuleCount = Modules.size();
 
   // Collect for each module the list of function it defines (GUID -> Summary).
   StringMap<GVSummaryMapTy> ModuleToDefinedGVSummaries(ModuleCount);
   Index->collectDefinedGVSummariesPerModule(ModuleToDefinedGVSummaries);
 
for (auto IndexCount : ModulesOrdering) {
             Context, LTORemarksFilename, LTOPassRemarksWithHotness, count);
         if (!DiagFileOrErr) {
           errs() << "Error: " << toString(DiagFileOrErr.takeError()) << "\n";
           report_fatal_error("ThinLTO: Can't get an output file for the "
                              "remarks");
         }
 
         // Parse module now
-        auto TheModule =
-            loadModuleFromBuffer(ModuleBuffer.getMemBuffer(), Context, false,
+        auto TheModule = loadModuleFromBuffer(ModuleBuffer.getMemBuffer(),
+                                              Context, false,
                                               /*IsImporting*/ false);
 
         // Save temps: original file.
         saveTempBitcode(*TheModule, SaveTempsDir, count, ".0.original.bc");
 
         auto &ImportList = ImportLists[ModuleIdentifier];
         // Run the main process now, and generates a binary
         auto OutputBuffer = ProcessThinLTOModule(
             *TheModule, *Index, ModuleMap, *TMBuilder.create(), ImportList,
             ExportList, GUIDPreservedSymbols,
             ModuleToDefinedGVSummaries[ModuleIdentifier], CacheOptions,
             DisableCodeGen, SaveTempsDir, Freestanding, OptLevel, count);
 
         // Commit to the cache (if enabled)
         CacheEntry.write(*OutputBuffer);
 
         if (SavedObjectsDirectoryPath.empty()) {
           // We need to generated a memory buffer for the linker.
           if (!CacheEntryPath.empty()) {
             // When cache is enabled, reload from the cache if possible.
             // Releasing the buffer from the heap and reloading it from the
             // cache file with mmap helps us to lower memory pressure.
             // The freed memory can be used for the next input file.
-            // The final binary link will read from the VFS cache (hopefully!)
-            // or from disk (if the memory pressure was too high).
+            // The final binary link will read from the VFS cache
+            // (hopefully!) or from disk (if the memory pressure was too
+            // high).
             auto ReloadedBufferOrErr = CacheEntry.tryLoadingBuffer();
             if (auto EC = ReloadedBufferOrErr.getError()) {
-              // On error, keep the preexisting buffer and print a diagnostic.
-              errs() << "error: can't reload cached file '" << CacheEntryPath
-                     << "': " << EC.message() << "\n";
+              // On error, keep the preexisting buffer and print a
+              // diagnostic.
+              errs() << "error: can't reload cached file '"
+                     << CacheEntryPath << "': " << EC.message() << "\n";
             } else {
               OutputBuffer = std::move(*ReloadedBufferOrErr);
             }
           }
           ProducedBinaries[count] = std::move(OutputBuffer);
           return;
         }
-        ProducedBinaryFiles[count] = writeGeneratedObject(
-            count, CacheEntryPath, SavedObjectsDirectoryPath, *OutputBuffer);
+        ProducedBinaryFiles[count] =
+            writeGeneratedObject(count, CacheEntryPath,
+                                 SavedObjectsDirectoryPath, *OutputBuffer);
       }, IndexCount);
     }
   }
 
   pruneCache(CacheOptions.Path, CacheOptions.Policy);
 
   // If statistics were requested, print them out now.
   if (llvm::AreStatisticsEnabled())
     llvm::PrintStatistics();
   reportAndResetTimings();
 }

lib/LTO/ThinLTOOutOfProcessCodeGenerator.cpp

+//===-ThinLTOOutOfProcessCodeGenerator.cpp - LLVM Link Time Optimizer
+//-----------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements the Thin Link Time Optimization library. This library is
+// intended to be used by linker to optimize code at link time.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/LTO/legacy/ThinLTOOutOfProcessCodeGenerator.h"
+
+#include "llvm/ADT/Statistic.h"
+#include "llvm/ADT/StringExtras.h"
+#include "llvm/Analysis/ModuleSummaryAnalysis.h"
+#include "llvm/Analysis/ProfileSummaryInfo.h"
+#include "llvm/Analysis/TargetLibraryInfo.h"
+#include "llvm/Analysis/TargetTransformInfo.h"
+#include "llvm/Bitcode/BitcodeReader.h"
+#include "llvm/Bitcode/BitcodeWriter.h"
+#include "llvm/Bitcode/BitcodeWriterPass.h"
+#include "llvm/Config/llvm-config.h"
+#include "llvm/IR/DebugInfo.h"
+#include "llvm/IR/DiagnosticPrinter.h"
+#include "llvm/IR/LLVMContext.h"
+#include "llvm/IR/LegacyPassManager.h"
+#include "llvm/IR/Mangler.h"
+#include "llvm/IR/PassTimingInfo.h"
+#include "llvm/IR/Verifier.h"
+#include "llvm/IRReader/IRReader.h"
+#include "llvm/LTO/LTO.h"
+#include "llvm/MC/SubtargetFeature.h"
+#include "llvm/Object/IRObjectFile.h"
+#include "llvm/Support/CachePruning.h"
+#include "llvm/Support/Debug.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/Program.h"
+#include "llvm/Support/SHA1.h"
+#include "llvm/Support/Signals.h"
+#include "llvm/Support/SmallVectorMemoryBuffer.h"
+#include "llvm/Support/TargetRegistry.h"
+#include "llvm/Support/ThreadPool.h"
+#include "llvm/Support/Threading.h"
+#include "llvm/Support/ToolOutputFile.h"
+#include "llvm/Support/VCSRevision.h"
+#include "llvm/Target/TargetMachine.h"
+#include "llvm/Transforms/IPO.h"
+#include "llvm/Transforms/IPO/FunctionImport.h"
+#include "llvm/Transforms/IPO/Internalize.h"
+#include "llvm/Transforms/IPO/PassManagerBuilder.h"
+#include "llvm/Transforms/ObjCARC.h"
+#include "llvm/Transforms/Utils/FunctionImportUtils.h"
+
+#include <limits.h>
+#include <numeric>
+
+using namespace llvm;
+
+#define DEBUG_TYPE "thinlto-outofprocess"
+
+namespace llvm {
+// Flags -discard-value-names, defined in LTOCodeGenerator.cpp
+extern cl::opt<bool> LTODiscardValueNames;
+extern cl::opt<std::string> LTORemarksFilename;
+extern cl::opt<bool> LTOPassRemarksWithHotness;
+
+// Defined in ThinLTOCodeGenerator.cpp
+extern cl::opt<int> ThreadCount;
+} // namespace llvm
+
+void ThinLTOCodegenManager::createInvocation(
+    const llvm::ThinLTOBuffer &InputModuleBuffer,
+    llvm::StringRef InputTmpFileName, llvm::StringRef IndexPath,
+    std::vector<llvm::StringRef> ImportFileList,
+    ClangThinLTOInvocation::Callback CompletionCallback) {
+
+  ClangThinLTOInvocation Entry(InputModuleBuffer, InputTmpFileName, IndexPath,
+                               std::move(ImportFileList));
+  this->execute(Entry, CompletionCallback);
+}
+
+void LocalProcessCodegenManager::materializeModule(ThinLTOBuffer &Module) {
+  SmallString<PATH_MAX> Path;
+  auto EC = sys::fs::createTemporaryFile(
+      llvm::sys::path::stem(Module.getBufferIdentifier()), "bc", Path);
+  if (EC)
+    report_fatal_error("Could not create temporary file to materialize module");
+  raw_fd_ostream OS(Path, EC, sys::fs::F_None);
+  if (EC)
+    report_fatal_error(Twine("Failed to open ") + Path +
+                       " to save module bitcode\n");
+  OS << Module.getBuffer();
+  // Update the buffer identifier and pointing it to the temp file.
+  Module.setBufferIdentifier(Path);
+  // Register the temp file to be removed on signal.
+  llvm::sys::RemoveFileOnSignal(Path);
+}
+
+void LocalProcessCodegenManager::execute(
+    const llvm::ClangThinLTOInvocation &Invocation,
+    ClangThinLTOInvocation::Callback Callback) {
+  auto IndexPathArg = llvm::Twine("-fthinlto-index=") + Invocation.IndexPath;
+  llvm::SmallString<PATH_MAX> IndexPathArgData;
+  auto IndexPathArgRef = IndexPathArg.toStringRef(IndexPathArgData);
+
+  auto OptLevelArg = llvm::Twine("-O") + llvm::Twine(TMBuilder.CGOptLevel);
+  llvm::SmallString<8> OptLevelArgData;
+  llvm::StringRef OptLevelArgRef = OptLevelArg.toStringRef(OptLevelArgData);
+
+  llvm::SmallString<32> RelocModelArgString;
+
+  switch (*TMBuilder.RelocModel) {
+  case llvm::Reloc::Static:
+    RelocModelArgString += "static";
+    break;
+  case llvm::Reloc::PIC_:
+    RelocModelArgString += "pic";
+    break;
+  case llvm::Reloc::DynamicNoPIC:
+    RelocModelArgString += "dynamic-no-pic";
+    break;
+  case llvm::Reloc::ROPI:
+    RelocModelArgString += "ropi";
+    break;
+  case llvm::Reloc::RWPI:
+    RelocModelArgString += "rwpi";
+    break;
+  case llvm::Reloc::ROPI_RWPI:
+    RelocModelArgString += "ropi-rwpi";
+    break;
+  }
+
+  std::vector<llvm::StringRef> Arguments = {
+      {getExecutablePath(), OptLevelArgRef, "-x", "ir",
+       Invocation.InputTmpFileName, "-c", "-Xclang", "-mrelocation-model",
+       "-Xclang", RelocModelArgString, IndexPathArgRef, "-o",
+       Invocation.OutputPath}};
+
+  if (llvm::sys::ExecuteAndWait(Arguments[0], Arguments, llvm::None))
+    llvm::report_fatal_error("Clang Invocation failed for Input file: " +
+                             Invocation.InputTmpFileName + "\n");
+  Callback(Invocation);
+}
+
+std::string LocalProcessCodegenManager::ExecutablePath = "";
+
+StringRef LocalProcessCodegenManager::getExecutablePath() {
+  if (ExecutablePath.empty()) {
+    if (llvm::ErrorOr<std::string> P = llvm::sys::findProgramByName("clang")) {
+      ExecutablePath = *P;
+      LLVM_DEBUG(llvm::dbgs() << "Using clang executable: "
+                              << ExecutablePath << "\n");
+    } else
+      llvm::report_fatal_error("Could not find clang executable!");
+  }
+  return ExecutablePath;
+}
+
+ClangThinLTOInvocation::ClangThinLTOInvocation(
+    llvm::ThinLTOBuffer const &InputModuleBuffer,
+    llvm::StringRef InputTmpFileName, llvm::StringRef IndexPath,
+    std::vector<StringRef> &&ImportFileList)
+    : InputBuffer(InputModuleBuffer), InputTmpFileName(InputTmpFileName),
+      IndexPath(IndexPath), ImportFileList(std::move(ImportFileList)) {
+  auto EC = llvm::sys::fs::createTemporaryFile(
+      llvm::sys::path::stem(InputModuleBuffer.getBufferIdentifier()),
+      "thinlto.o", OutputPath);
+  LLVM_DEBUG(llvm::dbgs() << "Just created the following file to store outputs "
+                          << OutputPath << "\n");
+  if (EC)
+    llvm::report_fatal_error("Could not create temporary file to store thinlto "
+                             "codegen output");
+}
+
+// Main entry point for the ThinLTO processing
+void ThinLTOOutOfProcessCodeGenerator::run() {
+  LLVM_DEBUG(
+      dbgs() << "Out of process thinlto optimzation and codegen path!\n");
+
+  CodegenManager->setTargetMachineBuilder(TMBuilder);
+
+  // Prepare the resulting object vector
+  assert(ProducedBinaries.empty() && "The generator should not be reused");
+  if (SavedObjectsDirectoryPath.empty())
+    ProducedBinaries.resize(Modules.size());
+  else {
+    sys::fs::create_directories(SavedObjectsDirectoryPath);
+    bool IsDir;
+    sys::fs::is_directory(SavedObjectsDirectoryPath, IsDir);
+    if (!IsDir)
+      report_fatal_error("Unexistent dir: '" + SavedObjectsDirectoryPath + "'");
+    ProducedBinaryFiles.resize(Modules.size());
+  }
+
+  if (CodeGenOnly) {
+    // Perform only parallel codegen and return.
+    ThreadPool Pool;
+    int count = 0;
+    for (auto &ModuleBuffer : Modules) {
+      Pool.async(
+          [&](int count) {
+            LLVMContext Context;
+            Context.setDiscardValueNames(LTODiscardValueNames);
+
+            // Parse module now
+            auto TheModule = loadModuleFromBuffer(ModuleBuffer.getMemBuffer(),
+                                                  Context, false,
+                                                  /*IsImporting*/ false);
+
+            // CodeGen
+            auto OutputBuffer = codegen(*TheModule);
+            if (SavedObjectsDirectoryPath.empty())
+              ProducedBinaries[count] = std::move(OutputBuffer);
+            else
+              ProducedBinaryFiles[count] = writeGeneratedObject(
+                  count, "", SavedObjectsDirectoryPath, *OutputBuffer);
+          },
+          count++);
+    }
+
+    return;
+  }
+
+  // Materialize the file.
+  for (auto &Module : Modules)
+    CodegenManager->materializeModule(Module);
+
+  // Sequential linking phase
+  auto Index = linkCombinedIndex();
+
+  // Save temps: index.
+  if (!SaveTempsDir.empty()) {
+    auto SaveTempPath = SaveTempsDir + "index.bc";
+    std::error_code EC;
+    raw_fd_ostream OS(SaveTempPath, EC, sys::fs::F_None);
+    if (EC)
+      report_fatal_error(Twine("Failed to open ") + SaveTempPath +
+                         " to save optimized bitcode\n");
+    WriteIndexToFile(*Index, OS);
+  }
+
+  // Prepare the module map.
+  auto ModuleMap = generateModuleMap(Modules);
+  auto ModuleCount = Modules.size();
+
+  // Collect for each module the list of function it defines (GUID ->
+  // Summary).
+  StringMap<GVSummaryMapTy> ModuleToDefinedGVSummaries(ModuleCount);
+  Index->collectDefinedGVSummariesPerModule(ModuleToDefinedGVSummaries);
+
+  // Convert the preserved symbols set from string to GUID, this is needed
+  // for computing the caching hash and the internalization.
+  auto GUIDPreservedSymbols =
+      computeGUIDPreservedSymbols(PreservedSymbols, TMBuilder.TheTriple);
+
+  // Compute "dead" symbols, we don't want to import/export these!
+  computeDeadSymbolsInIndex(*Index, GUIDPreservedSymbols);
+
+  // Collect the import/export lists for all modules from the call-graph in
+  // the combined index.
+  StringMap<FunctionImporter::ImportMapTy> ImportLists(ModuleCount);
+  StringMap<FunctionImporter::ExportSetTy> ExportLists(ModuleCount);
+  ComputeCrossModuleImport(*Index, ModuleToDefinedGVSummaries, ImportLists,
+                           ExportLists);
+
+  // We use a std::map here to be able to have a defined ordering when
+  // producing a hash for the cache entry.
+  // FIXME: we should be able to compute the caching hash for the entry
+  // based on the index, and nuke this map.
+  StringMap<std::map<GlobalValue::GUID, GlobalValue::LinkageTypes>> ResolvedODR;
+
+  // Resolve LinkOnce/Weak symbols, this has to be computed early because it
+  // impacts the caching.
+  resolvePrevailingInIndex(*Index, ResolvedODR);
+
+  // Use global summary-based analysis to identify symbols that can be
+  // internalized (because they aren't exported or preserved as per
+  // callback). Changes are made in the index, consumed in the ThinLTO
+  // backends.
+  internalizeAndPromoteInIndex(ExportLists, GUIDPreservedSymbols, *Index);
+
+  // Make sure that every module has an entry in the ExportLists and
+  // ResolvedODR maps to enable threaded access to these maps below.
+  for (auto &DefinedGVSummaries : ModuleToDefinedGVSummaries) {
+    ExportLists[DefinedGVSummaries.first()];
+    ResolvedODR[DefinedGVSummaries.first()];
+  }
+
+  // Compute the ordering we will process the inputs: the rough heuristic
+  // here is to sort them per size so that the largest module get schedule
+  // as soon as possible. This is purely a compile-time optimization.
+  std::vector<int> ModulesOrdering;
+  ModulesOrdering.resize(Modules.size());
+  std::iota(ModulesOrdering.begin(), ModulesOrdering.end(), 0);
+  llvm::sort(ModulesOrdering.begin(), ModulesOrdering.end(),
+             [&](int LeftIndex, int RightIndex) {
+               auto LSize = Modules[LeftIndex].getBuffer().size();
+               auto RSize = Modules[RightIndex].getBuffer().size();
+               return LSize > RSize;
+             });
+
+  // Parallel optimizer + codegen
+  {
+    ThreadPool Pool(ThreadCount);
+    for (auto IndexCount : ModulesOrdering) {
+      Pool.async(
+          [&](int ModuleIndex) {
+            auto &ModuleBuffer = Modules[ModuleIndex];
+            auto ModuleIdentifier = ModuleBuffer.getBufferIdentifier();
+            auto &ExportList = ExportLists[ModuleIdentifier];
+
+            auto &DefinedFunctions =
+                ModuleToDefinedGVSummaries[ModuleIdentifier];
+
+            // The module may be cached, this helps handling it.
+            ModuleCacheEntry CacheEntry(
+                CacheOptions.Path, *Index, ModuleIdentifier,
+                ImportLists[ModuleIdentifier], ExportList,
+                ResolvedODR[ModuleIdentifier], DefinedFunctions, OptLevel,
+                Freestanding, TMBuilder);
+
+            auto CacheEntryPath = CacheEntry.getEntryPath();
+            {
+              auto ErrOrBuffer = CacheEntry.tryLoadingBuffer();
+              LLVM_DEBUG(dbgs()
+                         << "Cache " << (ErrOrBuffer ? "hit" : "miss") << " '"
+                         << CacheEntryPath << "' for buffer " << ModuleIndex
+                         << " " << ModuleIdentifier << "\n");
+
+              if (ErrOrBuffer) {
+                // Cache Hit!
+                if (SavedObjectsDirectoryPath.empty())
+                  ProducedBinaries[ModuleIndex] = std::move(ErrOrBuffer.get());
+                else
+                  ProducedBinaryFiles[ModuleIndex] = writeGeneratedObject(
+                      ModuleIndex, CacheEntryPath, SavedObjectsDirectoryPath,
+                      *ErrOrBuffer.get());
+                return;
+              }
+            }
+
+            LLVMContext Context;
+            Context.setDiscardValueNames(LTODiscardValueNames);
+            Context.enableDebugTypeODRUniquing();
+            auto DiagFileOrErr = lto::setupOptimizationRemarks(
+                Context, LTORemarksFilename, LTOPassRemarksWithHotness,
+                ModuleIndex);
+            if (!DiagFileOrErr) {
+              errs() << "Error: " << toString(DiagFileOrErr.takeError())
+                     << "\n";
+              report_fatal_error("ThinLTO: Can't get an output file for the "
+                                 "remarks");
+            }
+
+            std::map<std::string, GVSummaryMapTy> ModuleToSummariesForIndex;
+            llvm::gatherImportedSummariesForModule(
+                ModuleIdentifier, ModuleToDefinedGVSummaries,
+                ImportLists[ModuleIdentifier], ModuleToSummariesForIndex);
+            SmallString<PATH_MAX> IndexFilePath;
+            auto EC = sys::fs::createTemporaryFile(llvm::Twine("index-") +
+                                                       llvm::Twine(ModuleIndex),
+                                                   "bc", IndexFilePath);
+            if (EC)
+              report_fatal_error(
+                  "could not create temporary file for thinlto codegen "
+                  "output");
+            {
+              raw_fd_ostream OS(IndexFilePath, EC, sys::fs::F_None);
+              if (EC)
+                report_fatal_error(
+                    "could not open temporary file to write thinlto "
+                    "codegen output");
+              WriteIndexToFile(*Index, OS, &ModuleToSummariesForIndex);
+              // Remove the temporary index file on signal.
+              {
+                std::lock_guard<std::mutex> LockGuard(ThreadLock);
+                llvm::sys::RemoveFileOnSignal(IndexFilePath);
+              }
+            }
+
+            std::vector<StringRef> ImportFileList;
+            for (auto &ILI : ModuleToSummariesForIndex) {
+              // We don't want to include the current module in it's imports
+              // list
+              if (ILI.first != ModuleBuffer.getBufferIdentifier())
+                ImportFileList.emplace_back(ILI.first);
+            }
+
+            CodegenManager->createInvocation(
+                ModuleBuffer, ModuleIdentifier,
+                IndexFilePath, std::move(ImportFileList),
+                [=](const ClangThinLTOInvocation &Invocation) mutable {
+                  CacheEntry.write(Invocation.OutputPath);
+
+                  if (SavedObjectsDirectoryPath.empty()) {
+                    // We need to generate a memory buffer for the linker.
+                    auto ErrorOrOutputBuffer =
+                        MemoryBuffer::getFile(Invocation.OutputPath);
+                    if (auto EC = ErrorOrOutputBuffer.getError())
+                      report_fatal_error(
+                          Twine("Could not load output memory buffer for ") +
+                          Invocation.OutputPath + "': " + EC.message() + "\n");
+                    else
+                      ProducedBinaries[ModuleIndex] =
+                          std::move(*ErrorOrOutputBuffer);
+
+                    sys::fs::remove(Invocation.OutputPath);
+                  } else {
+                    // Create a hard link from the temporary output file to
+                    // the correct location in SavedObjectsDirectoryPath.
+                    SmallString<PATH_MAX> SavedObjectPath(
+                        SavedObjectsDirectoryPath);
+                    sys::path::append(SavedObjectPath,
+                                      llvm::Twine(ModuleIndex) +
+                                          llvm::Twine("thinlto.bc"));
+                    if (auto EC = sys::fs::rename(Invocation.OutputPath,
+                                                  SavedObjectPath))
+                      report_fatal_error(
+                          Twine("Could not write output to output "
+                                "file directory for") +
+                          Invocation.OutputPath + ": " + EC.message() + "\n");
+
+                    ProducedBinaryFiles[ModuleIndex] = SavedObjectPath.str();
+                  }
+                });
+          },
+          IndexCount);
+    }
+  }
+
+  pruneCache(CacheOptions.Path, CacheOptions.Policy);
+
+  // Run interrupt handler to remove all the temp files.
+  llvm::sys::RunInterruptHandlers();
+
+  // If statistics were requested, print them out now.
+  if (llvm::AreStatisticsEnabled())
+    llvm::PrintStatistics();
+  reportAndResetTimings();
+}

tools/lto/lto.cpp

 #include "llvm/Bitcode/BitcodeReader.h"
 #include "llvm/CodeGen/CommandFlags.inc"
 #include "llvm/IR/DiagnosticInfo.h"
 #include "llvm/IR/DiagnosticPrinter.h"
 #include "llvm/IR/LLVMContext.h"
 #include "llvm/LTO/legacy/LTOCodeGenerator.h"
 #include "llvm/LTO/legacy/LTOModule.h"
 #include "llvm/LTO/legacy/ThinLTOCodeGenerator.h"
+#include "llvm/LTO/legacy/ThinLTOOutOfProcessCodeGenerator.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Signals.h"
 #include "llvm/Support/TargetSelect.h"
 #include "llvm/Support/raw_ostream.h"
 
 // extra command-line flags needed for LTOCodeGenerator
 static cl::opt<char>
 OptLevel("O",

 #else
 static bool VerifyByDefault = true;
 #endif
 
 static cl::opt<bool> DisableVerify(
     "disable-llvm-verifier", cl::init(!VerifyByDefault),
     cl::desc("Don't run the LLVM verifier during the optimization pipeline"));
 
+namespace {
+enum ThinLTOMode {
+  Thread,
+  OutOfProcess
+};
+} // end anonymous namespace.
+
+static cl::opt<ThinLTOMode> ThinLTOCGMode(
+    "thinlto-mode", cl::init(Thread),
+    cl::desc(
+        "ThinLTO Code Generation Mode. Options: thread, out-of-process, xpc"),
+    cl::values(clEnumValN(Thread, "thread", "Threaded codegen"),
+               clEnumValN(OutOfProcess, "out-of-process",
+                          "Local out of process using clang")));
+
 // Holds most recent error string.
 // *** Not thread safe ***
 static std::string sLastErrorString;
 
 // Holds the initialization state of the LTO module.
 // *** Not thread safe ***
 static bool initialized = false;
 
struct LibLTOCodeGenerator : LTOCodeGenerator {
   ~LibLTOCodeGenerator() { resetMergedModule(); }
 
   void init() { setDiagnosticHandler(handleLibLTODiagnostic, nullptr); }
 
   std::unique_ptr<MemoryBuffer> NativeObjectFile;
   std::unique_ptr<LLVMContext> OwnedContext;
 };
 
-}
+} // namespace
 
 DEFINE_SIMPLE_CONVERSION_FUNCTIONS(LibLTOCodeGenerator, lto_code_gen_t)
 DEFINE_SIMPLE_CONVERSION_FUNCTIONS(ThinLTOCodeGenerator, thinlto_code_gen_t)
 DEFINE_SIMPLE_CONVERSION_FUNCTIONS(LTOModule, lto_module_t)
 
 // Convert the subtarget features into a string to pass to LTOCodeGenerator.
 static void lto_add_attrs(lto_code_gen_t cg) {
   LTOCodeGenerator *CG = unwrap(cg);
void lto_codegen_set_should_embed_uselists(lto_code_gen_t cg,
                                            lto_bool_t ShouldEmbedUselists) {
   unwrap(cg)->setShouldEmbedUselists(ShouldEmbedUselists);
 }
 
 // ThinLTO API below
 
 thinlto_code_gen_t thinlto_create_codegen(void) {
   lto_initialize();
-  ThinLTOCodeGenerator *CodeGen = new ThinLTOCodeGenerator();
+  ThinLTOCodeGenerator *CodeGen = nullptr;
+  switch(ThinLTOCGMode) {
+  case Thread:
+    CodeGen = new ThinLTOCodeGenerator();
+    break;
+  case OutOfProcess:
+    CodeGen =
+        new ThinLTOOutOfProcessCodeGenerator(new LocalProcessCodegenManager());
+    break;
+  }
+
   CodeGen->setTargetOptions(InitTargetOptionsFromCodeGenFlags());
   CodeGen->setFreestanding(EnableFreestanding);
 
   if (OptLevel.getNumOccurrences()) {
     if (OptLevel < '0' || OptLevel > '3')
       report_fatal_error("Optimization level must be between 0 and 3");
     CodeGen->setOptLevel(OptLevel - '0');
     switch (OptLevel) {