[ThinLTO] Add an API to trigger file-based API for returning objects to the linker

[mehdi_amini]

Summary

The motivation is to support better the -object_path_lto option on
Darwin. The linker needs to write down the generate object files on
disk for later use by lldb or dsymutil (debug info are not present
in the final binary). We're moving this into libLTO so that we can
be smarter when a cache is enabled and hard-link when possible
instead of duplicating the files.

[mehdi_amini]

Fix typo in the C API

[mehdi_amini]

Ping!

[tejohnson]

Looks good generally, but a few questions/comments.

llvm/include/llvm-c/lto.h

651 ↗

(On Diff #80557)

Doc needs update - returns path to ...

658 ↗

(On Diff #80557)

Since version should be 21?

llvm/include/llvm/LTO/legacy/ThinLTOCodeGenerator.h

87 ↗

(On Diff #80557)

Give an error here if the wrong routine called? I.e. if getProducedBinaries called and setGeneratedObjectsDirectory was called. I see that the C api will assert if the wrong interface is called because the index will be out of range, but perhaps a more meaningful error could be given.

97 ↗

(On Diff #80557)

Ditto

294 ↗

(On Diff #80557)

This variable name was confusing to me - can you change to something like SavedObjectsDirectoryPath or something like that (and change corresponding parameter names where this is passed in)?

llvm/lib/LTO/ThinLTOCodeGenerator.cpp

740 ↗

(On Diff #80557)

Document return value

746 ↗

(On Diff #80557)

What is the point of this line?

762 ↗

(On Diff #80557)

I guess this is the case where the cache entry was somehow pruned between the time the OutputBuffer was loaded and when the copy/hardlink was attempted? Can you add a comment to that effect?

953 ↗

(On Diff #80557)

Remind we why we do this instead of just using the existing OutputBuffer?

llvm/test/ThinLTO/X86/save_objects.ll

6 ↗

(On Diff #80557)

Nit: why use ".o" extension for save objects which is a directory?

[mehdi_amini]

Address Teresa's comments

[mehdi_amini]

Thanks for the review! See inline.

llvm/include/llvm/LTO/legacy/ThinLTOCodeGenerator.h

87 ↗

(On Diff #80557)

There shouldn't be an assert in the C API if correctly used:

The client is supposed to iterate over the range given by thinlto_module_get_num_object_files() and call thinlto_module_get_object_file() (similarly thinlto_module_get_object/thinlto_module_get_num_objects).

The way I implemented it in ld64 is basically:

for (int BufId = 0; BufID < thinlto_module_get_num_objects; ++BufID) {
   process(thinlto_module_get_object(BufID));
}
for (int FileID = 0; FileID < thinlto_module_get_num_object_files; ++FileID) {
   process(thinlto_module_get_object_file(FileID));
}

Does it make sense?

llvm/lib/LTO/ThinLTOCodeGenerator.cpp

746 ↗

(On Diff #80557)

This ensure that the SmallString is null terminated before calling sys::fs::exists (added a comment)

953 ↗

(On Diff #80557)

Lower memory consumption: the buffer is on the heap and releasing this memory can be used for the next file. The final binary link will reload from disk (or from the VFS cache if the memory pressure wasn't too high).
(Added comment to this effect)

llvm/test/ThinLTO/X86/save_objects.ll

6 ↗

(On Diff #80557)

This is just what Xcode is doing, because originally it was implemented for Monolithic LTO where there is a single output file ;)
Changed it to %t.thin.out

[tejohnson]

LGTM, see small comments below. Error message I was suggesting is not a blocker, up to you if you agree.

llvm/include/llvm/LTO/legacy/ThinLTOCodeGenerator.h

87 ↗

(On Diff #80557)

Sure, it won't assert if invoked properly, I was suggesting a more meaningful error in case it is not.

llvm/lib/LTO/ThinLTOCodeGenerator.cpp

746 ↗

(On Diff #80557)

Ok, thanks

953 ↗

(On Diff #80557)

I see, because reloading will attempt to mmap the file. Nit: typo in new comment "pressuree"

[mehdi_amini]

llvm/include/llvm/LTO/legacy/ThinLTOCodeGenerator.h

87 ↗

(On Diff #80557)

I just don't understand what error you're suggesting here, can you clarify?

Right now this API is always OK to be called, and both the thinlto_module_get_num_object_files and thinlto_module_get_num_objects are invoking it during the normal use I mentioned above (loop bound check).
That's why the C API asserts only when you access out-of-bound and not here when you access the vector itself (to check for size for instance).

[tejohnson]

llvm/include/llvm/LTO/legacy/ThinLTOCodeGenerator.h

87 ↗

(On Diff #80557)

Oh I see, you need to access the maps to get the size in order to guard/index the calls appropriately. So what I had in mind won't work. I was just trying to see if there was a more meaningful error other than the ones in thinlto_module_get_object_file() and the like than "Index overflow" when the wrong map was accessed. Maybe not worth it...

llvm/trunk/include/llvm-c/lto.h

 
 /**
  * @defgroup LLVMCLTO LTO
  * @ingroup LLVMC
  *
  * @{
  */
 
-#define LTO_API_VERSION 20
+#define LTO_API_VERSION 21
 
 /**
  * \since prior to LTO_API_VERSION=3
  */
 typedef enum {
     LTO_SYMBOL_ALIGNMENT_MASK              = 0x0000001F, /* log2 of alignment */
     LTO_SYMBOL_PERMISSIONS_MASK            = 0x000000E0,
     LTO_SYMBOL_PERMISSIONS_CODE            = 0x000000A0,

  * available objects.
  *
  * \since LTO_API_VERSION=18
  */
 extern LTOObjectBuffer thinlto_module_get_object(thinlto_code_gen_t cg,
                                                  unsigned int index);
 
 /**
+ * Returns the number of object files produced by the ThinLTO CodeGenerator.
+ *
+ * It usually matches the number of input files, but this is not a guarantee of
+ * the API and may change in future implementation, so the client should not
+ * assume it.
+ *
+ * \since LTO_API_VERSION=21
+ */
+unsigned int thinlto_module_get_num_object_files(thinlto_code_gen_t cg);
+
+/**
+ * Returns the path to the ith object file produced by the ThinLTO
+ * CodeGenerator.
+ *
+ * Client should use \p thinlto_module_get_num_object_files() to get the number
+ * of available objects.
+ *
+ * \since LTO_API_VERSION=21
+ */
+const char *thinlto_module_get_object_file(thinlto_code_gen_t cg,
+                                           unsigned int index);
+
+/**
  * Sets which PIC code model to generate.
  * Returns true on error (check lto_get_error_message() for details).
  *
  * \since LTO_API_VERSION=18
  */
 extern lto_bool_t thinlto_codegen_set_pic_model(thinlto_code_gen_t cg,
                                                 lto_codegen_model);
 

  * stage of the pipeline.
  *
  * \since LTO_API_VERSION=18
  */
 extern void thinlto_codegen_set_savetemps_dir(thinlto_code_gen_t cg,
                                               const char *save_temps_dir);
 
 /**
+ * Set the path to a directory where to save generated object files. This
+ * path can be used by a linker to request on-disk files instead of in-memory
+ * buffers. When set, results are available through
+ * thinlto_module_get_object_file() instead of thinlto_module_get_object().
+ *
+ * \since LTO_API_VERSION=21
+ */
+void thinlto_set_generated_objects_dir(thinlto_code_gen_t cg,
+                                       const char *save_temps_dir);
+
+/**
  * Sets the cpu to generate code for.
  *
  * \since LTO_API_VERSION=18
  */
 extern void thinlto_codegen_set_cpu(thinlto_code_gen_t cg, const char *cpu);
 
 /**
  * Disable CodeGen, only run the stages till codegen and stop. The output will

llvm/trunk/include/llvm/LTO/legacy/ThinLTOCodeGenerator.h

public:
    * references from a ThinLTO module to this symbol is optimized away, then
    * the symbol can be discarded.
    */
   void crossReferenceSymbol(StringRef Name);
 
   /**
    * Process all the modules that were added to the code generator in parallel.
    *
-   * Client can access the resulting object files using getProducedBinaries()
+   * Client can access the resulting object files using getProducedBinaries(),
+   * unless setGeneratedObjectsDirectory() has been called, in which case
+   * results are available through getProducedBinaryFiles().
    */
   void run();
 
   /**
-   * Return the "in memory" binaries produced by the code generator.
+   * Return the "in memory" binaries produced by the code generator. This is
+   * filled after run() unless setGeneratedObjectsDirectory() has been
+   * called, in which case results are available through
+   * getProducedBinaryFiles().
    */
   std::vector<std::unique_ptr<MemoryBuffer>> &getProducedBinaries() {
     return ProducedBinaries;
   }
 
   /**
+   * Return the "on-disk" binaries produced by the code generator. This is
+   * filled after run() when setGeneratedObjectsDirectory() has been
+   * called, in which case results are available through getProducedBinaries().
+   */
+  std::vector<std::string> &getProducedBinaryFiles() {
+    return ProducedBinaryFiles;
+  }
+
+  /**
    * \defgroup Options setters
    * @{
    */
 
   /**
    * \defgroup Cache controlling options
    *
    * These entry points control the ThinLTO cache. The cache is intended to
public:
   }
 
   /**@}*/
 
   /// Set the path to a directory where to save temporaries at various stages of
   /// the processing.
   void setSaveTempsDir(std::string Path) { SaveTempsDir = std::move(Path); }
 
+  /// Set the path to a directory where to save generated object files. This
+  /// path can be used by a linker to request on-disk files instead of in-memory
+  /// buffers. When set, results are available through getProducedBinaryFiles()
+  /// instead of getProducedBinaries().
+  void setGeneratedObjectsDirectory(std::string Path) {
+    SavedObjectsDirectoryPath = std::move(Path);
+  }
+
   /// CPU to use to initialize the TargetMachine
   void setCpu(std::string Cpu) { TMBuilder.MCpu = std::move(Cpu); }
 
   /// Subtarget attributes
   void setAttr(std::string MAttr) { TMBuilder.MAttr = std::move(MAttr); }
 
   /// TargetMachine options
   void setTargetOptions(TargetOptions Options) {
public:
   std::unique_ptr<MemoryBuffer> codegen(Module &Module);
 
   /**@}*/
 
 private:
   /// Helper factory to build a TargetMachine
   TargetMachineBuilder TMBuilder;
 
-  /// Vector holding the in-memory buffer containing the produced binaries.
+  /// Vector holding the in-memory buffer containing the produced binaries, when
+  /// SavedObjectsDirectoryPath isn't set.
   std::vector<std::unique_ptr<MemoryBuffer>> ProducedBinaries;
 
+  /// Path to generated files in the supplied SavedObjectsDirectoryPath if any.
+  std::vector<std::string> ProducedBinaryFiles;
+
   /// Vector holding the input buffers containing the bitcode modules to
   /// process.
   std::vector<MemoryBufferRef> Modules;
 
   /// Set of symbols that need to be preserved outside of the set of bitcode
   /// files.
   StringSet<> PreservedSymbols;
 
   /// Set of symbols that are cross-referenced between bitcode files.
   StringSet<> CrossReferencedSymbols;
 
   /// Control the caching behavior.
   CachingOptions CacheOptions;
 
   /// Path to a directory to save the temporary bitcode files.
   std::string SaveTempsDir;
 
+  /// Path to a directory to save the generated object files.
+  std::string SavedObjectsDirectoryPath;
+
   /// Flag to enable/disable CodeGen. When set to true, the process stops after
   /// optimizations and a bitcode is produced.
   bool DisableCodeGen = false;
 
   /// Flag to indicate that only the CodeGen will be performed, no cross-module
   /// importing or optimization.
   bool CodeGenOnly = false;
 };
 }
 #endif

llvm/trunk/include/llvm/Support/FileSystem.h

 /// hard links don't work on SMB file systems.
 ///
 /// @param to The path to hard link to.
 /// @param from The path to hard link from. This is created.
 /// @returns errc::success if the link was created, otherwise a platform
 /// specific error_code.
 std::error_code create_link(const Twine &to, const Twine &from);
 
+/// Create a hard link from \a from to \a to, or return an error.
+///
+/// @param to The path to hard link to.
+/// @param from The path to hard link from. This is created.
+/// @returns errc::success if the link was created, otherwise a platform
+/// specific error_code.
+std::error_code create_hard_link(const Twine &to, const Twine &from);
+
 /// @brief Get the current path.
 ///
 /// @param result Holds the current path on return.
 /// @returns errc::success if the current path has been stored in result,
 ///          otherwise a platform-specific error_code.
 std::error_code current_path(SmallVectorImpl<char> &result);
 
 /// @brief Remove path. Equivalent to POSIX remove().

llvm/trunk/lib/LTO/ThinLTOCodeGenerator.cpp

#endif
   // Try loading the buffer for this cache entry.
   ErrorOr<std::unique_ptr<MemoryBuffer>> tryLoadingBuffer() {
     if (EntryPath.empty())
       return std::error_code();
     return MemoryBuffer::getFile(EntryPath);
   }
 
   // Cache the Produced object file
-  std::unique_ptr<MemoryBuffer>
-  write(std::unique_ptr<MemoryBuffer> OutputBuffer) {
+  void write(const MemoryBuffer &OutputBuffer) {
     if (EntryPath.empty())
-      return OutputBuffer;
+      return;
 
     // Write to a temporary to avoid race condition
     SmallString<128> TempFilename;
     int TempFD;
     std::error_code EC =
         sys::fs::createTemporaryFile("Thin", "tmp.o", TempFD, TempFilename);
     if (EC) {
       errs() << "Error: " << EC.message() << "\n";
       report_fatal_error("ThinLTO: Can't get a temporary file");
     }
     {
       raw_fd_ostream OS(TempFD, /* ShouldClose */ true);
-      OS << OutputBuffer->getBuffer();
+      OS << OutputBuffer.getBuffer();
     }
     // Rename to final destination (hopefully race condition won't matter here)
     EC = sys::fs::rename(TempFilename, EntryPath);
     if (EC) {
       sys::fs::remove(TempFilename);
       raw_fd_ostream OS(EntryPath, EC, sys::fs::F_None);
       if (EC)
         report_fatal_error(Twine("Failed to open ") + EntryPath +
                            " to save cached entry\n");
-      OS << OutputBuffer->getBuffer();
-    }
-    auto ReloadedBufferOrErr = MemoryBuffer::getFile(EntryPath);
-    if (auto EC = ReloadedBufferOrErr.getError()) {
-      // FIXME diagnose
-      errs() << "error: can't reload cached file '" << EntryPath
-             << "': " << EC.message() << "\n";
-      return OutputBuffer;
+      OS << OutputBuffer.getBuffer();
     }
-    return std::move(*ReloadedBufferOrErr);
   }
 };
 
 static std::unique_ptr<MemoryBuffer>
 ProcessThinLTOModule(Module &TheModule, ModuleSummaryIndex &Index,
                      StringMap<MemoryBufferRef> &ModuleMap, TargetMachine &TM,
                      const FunctionImporter::ImportMapTy &ImportList,
                      const FunctionImporter::ExportSetTy &ExportList,

 /**
  * Perform ThinLTO CodeGen.
  */
 std::unique_ptr<MemoryBuffer> ThinLTOCodeGenerator::codegen(Module &TheModule) {
   initTMBuilder(TMBuilder, Triple(TheModule.getTargetTriple()));
   return codegenModule(TheModule, *TMBuilder.create());
 }
 
+/// Write out the generated object file, either from CacheEntryPath or from
+/// OutputBuffer, preferring hard-link when possible.
+/// Returns the path to the generated file in SavedObjectsDirectoryPath.
+static std::string writeGeneratedObject(int count, StringRef CacheEntryPath,
+                                        StringRef SavedObjectsDirectoryPath,
+                                        const MemoryBuffer &OutputBuffer) {
+  SmallString<128> OutputPath(SavedObjectsDirectoryPath);
+  llvm::sys::path::append(OutputPath, Twine(count) + ".thinlto.o");
+  OutputPath.c_str(); // Ensure the string is null terminated.
+  if (sys::fs::exists(OutputPath))
+    sys::fs::remove(OutputPath);
+
+  // We don't return a memory buffer to the linker, just a list of files.
+  if (!CacheEntryPath.empty()) {
+    // Cache is enabled, hard-link the entry (or copy if hard-link fails).
+    auto Err = sys::fs::create_hard_link(CacheEntryPath, OutputPath);
+    if (!Err)
+      return OutputPath.str();
+    // Hard linking failed, try to copy.
+    Err = sys::fs::copy_file(CacheEntryPath, OutputPath);
+    if (!Err)
+      return OutputPath.str();
+    // Copy failed (could be because the CacheEntry was removed from the cache
+    // in the meantime by another process), fall back and try to write down the
+    // buffer to the output.
+    errs() << "error: can't link or copy from cached entry '" << CacheEntryPath
+           << "' to '" << OutputPath << "'\n";
+  }
+  // No cache entry, just write out the buffer.
+  std::error_code Err;
+  raw_fd_ostream OS(OutputPath, Err, sys::fs::F_None);
+  if (Err)
+    report_fatal_error("Can't open output '" + OutputPath + "'\n");
+  OS << OutputBuffer.getBuffer();
+  return OutputPath.str();
+}
+
 // Main entry point for the ThinLTO processing
 void ThinLTOCodeGenerator::run() {
   if (CodeGenOnly) {
     // Perform only parallel codegen and return.
     ThreadPool Pool;
     assert(ProducedBinaries.empty() && "The generator should not be reused");
     ProducedBinaries.resize(Modules.size());
     int count = 0;
if (!SaveTempsDir.empty()) {
     if (EC)
       report_fatal_error(Twine("Failed to open ") + SaveTempPath +
                          " to save optimized bitcode\n");
     WriteIndexToFile(*Index, OS);
   }
 
   // Prepare the resulting object vector
   assert(ProducedBinaries.empty() && "The generator should not be reused");
+  if (SavedObjectsDirectoryPath.empty())
     ProducedBinaries.resize(Modules.size());
+  else {
+    sys::fs::create_directories(SavedObjectsDirectoryPath);
+    bool IsDir;
+    sys::fs::is_directory(SavedObjectsDirectoryPath, IsDir);
+    if (!IsDir)
+      report_fatal_error("Unexistent dir: '" + SavedObjectsDirectoryPath + "'");
+    ProducedBinaryFiles.resize(Modules.size());
+  }
 
   // Prepare the module map.
   auto ModuleMap = generateModuleMap(Modules);
   auto ModuleCount = Modules.size();
 
   // Collect for each module the list of function it defines (GUID -> Summary).
   StringMap<GVSummaryMapTy> ModuleToDefinedGVSummaries(ModuleCount);
   Index->collectDefinedGVSummariesPerModule(ModuleToDefinedGVSummaries);
for (auto IndexCount : ModulesOrdering) {
 
         auto &DefinedFunctions = ModuleToDefinedGVSummaries[ModuleIdentifier];
 
         // The module may be cached, this helps handling it.
         ModuleCacheEntry CacheEntry(CacheOptions.Path, *Index, ModuleIdentifier,
                                     ImportLists[ModuleIdentifier], ExportList,
                                     ResolvedODR[ModuleIdentifier],
                                     DefinedFunctions, GUIDPreservedSymbols);
+        auto CacheEntryPath = CacheEntry.getEntryPath();
 
         {
           auto ErrOrBuffer = CacheEntry.tryLoadingBuffer();
           DEBUG(dbgs() << "Cache " << (ErrOrBuffer ? "hit" : "miss") << " '"
-                       << CacheEntry.getEntryPath() << "' for buffer " << count
-                       << " " << ModuleIdentifier << "\n");
+                       << CacheEntryPath << "' for buffer " << count << " "
+                       << ModuleIdentifier << "\n");
 
           if (ErrOrBuffer) {
             // Cache Hit!
+            if (SavedObjectsDirectoryPath.empty())
               ProducedBinaries[count] = std::move(ErrOrBuffer.get());
+            else
+              ProducedBinaryFiles[count] = writeGeneratedObject(
+                  count, CacheEntryPath, SavedObjectsDirectoryPath,
+                  *ErrOrBuffer.get());
             return;
           }
         }
 
         LLVMContext Context;
         Context.setDiscardValueNames(LTODiscardValueNames);
         Context.enableDebugTypeODRUniquing();
         auto DiagFileOrErr = setupOptimizationRemarks(Context, count);
for (auto IndexCount : ModulesOrdering) {
         auto &ImportList = ImportLists[ModuleIdentifier];
         // Run the main process now, and generates a binary
         auto OutputBuffer = ProcessThinLTOModule(
             *TheModule, *Index, ModuleMap, *TMBuilder.create(), ImportList,
             ExportList, GUIDPreservedSymbols,
             ModuleToDefinedGVSummaries[ModuleIdentifier], CacheOptions,
             DisableCodeGen, SaveTempsDir, count);
 
-        OutputBuffer = CacheEntry.write(std::move(OutputBuffer));
+        // Commit to the cache (if enabled)
+        CacheEntry.write(*OutputBuffer);
+
+        if (SavedObjectsDirectoryPath.empty()) {
+          // We need to generated a memory buffer for the linker.
+          if (!CacheEntryPath.empty()) {
+            // Cache is enabled, reload from the cache
+            // We do this to lower memory pressuree: the buffer is on the heap
+            // and releasing it frees memory that can be used for the next input
+            // file. The final binary link will read from the VFS cache
+            // (hopefully!) or from disk if the memory pressure wasn't too high.
+            auto ReloadedBufferOrErr = CacheEntry.tryLoadingBuffer();
+            if (auto EC = ReloadedBufferOrErr.getError()) {
+              // On error, keeping the preexisting buffer and printing a
+              // diagnostic is more friendly than just crashing.
+              errs() << "error: can't reload cached file '" << CacheEntryPath
+                     << "': " << EC.message() << "\n";
+            } else {
+              OutputBuffer = std::move(*ReloadedBufferOrErr);
+            }
+          }
           ProducedBinaries[count] = std::move(OutputBuffer);
+          return;
+        }
+        ProducedBinaryFiles[count] = writeGeneratedObject(
+            count, CacheEntryPath, SavedObjectsDirectoryPath, *OutputBuffer);
       }, IndexCount);
     }
   }
 
   CachePruning(CacheOptions.Path)
       .setPruningInterval(std::chrono::seconds(CacheOptions.PruningInterval))
       .setEntryExpiration(std::chrono::seconds(CacheOptions.Expiration))
       .setMaxSize(CacheOptions.MaxPercentageOfAvailableSpace)
       .prune();
 
   // If statistics were requested, print them out now.
   if (llvm::AreStatisticsEnabled())
     llvm::PrintStatistics();
 }

llvm/trunk/lib/Support/Unix/Path.inc

std::error_code create_link(const Twine &to, const Twine &from) {
   StringRef t = to.toNullTerminatedStringRef(to_storage);
 
   if (::symlink(t.begin(), f.begin()) == -1)
     return std::error_code(errno, std::generic_category());
 
   return std::error_code();
 }
 
+std::error_code create_hard_link(const Twine &to, const Twine &from) {
+  // Get arguments.
+  SmallString<128> from_storage;
+  SmallString<128> to_storage;
+  StringRef f = from.toNullTerminatedStringRef(from_storage);
+  StringRef t = to.toNullTerminatedStringRef(to_storage);
+
+  if (::link(t.begin(), f.begin()) == -1)
+    return std::error_code(errno, std::generic_category());
+
+  return std::error_code();
+}
+
 std::error_code remove(const Twine &path, bool IgnoreNonExisting) {
   SmallString<128> path_storage;
   StringRef p = path.toNullTerminatedStringRef(path_storage);
 
   struct stat buf;
   if (lstat(p.begin(), &buf) != 0) {
     if (errno != ENOENT || !IgnoreNonExisting)
       return std::error_code(errno, std::generic_category());

llvm/trunk/lib/Support/Windows/Path.inc

if (std::error_code ec = widenPath(to, wide_to))
     return ec;
 
   if (!::CreateHardLinkW(wide_from.begin(), wide_to.begin(), NULL))
     return mapWindowsError(::GetLastError());
 
   return std::error_code();
 }
 
+std::error_code create_hard_link(const Twine &to, const Twine &from) {
+  return create_link(to, from);
+}
+
 std::error_code remove(const Twine &path, bool IgnoreNonExisting) {
   SmallVector<wchar_t, 128> path_utf16;
 
   file_status ST;
   if (std::error_code EC = status(path, ST)) {
     if (EC != errc::no_such_file_or_directory || !IgnoreNonExisting)
       return EC;
     return std::error_code();

llvm/trunk/test/ThinLTO/X86/save_objects.ll

+; RUN: opt -module-hash -module-summary %s -o %t.bc
+; RUN: opt -module-hash -module-summary %p/Inputs/cache.ll -o %t2.bc
+
+; Check that the generating object files is working without cache
+; RUN: rm -Rf %t.thin.out
+; RUN: llvm-lto -thinlto-save-objects=%t.thin.out -thinlto-action=run %t2.bc  %t.bc -exported-symbol=main 
+; RUN: ls %t.thin.out | count 2
+
+; Same with cache
+; RUN: rm -Rf %t.thin.out
+; RUN: rm -Rf %t.cache && mkdir %t.cache
+; RUN: llvm-lto -thinlto-save-objects=%t.thin.out -thinlto-action=run %t2.bc  %t.bc -exported-symbol=main -thinlto-cache-dir %t.cache 
+; RUN: ls %t.thin.out | count 2
+; RUN: ls %t.cache | count 3
+
+; Same with hot cache
+; RUN: rm -Rf %t.thin.out
+; RUN: rm -Rf %t.cache && mkdir %t.cache
+; RUN: llvm-lto -thinlto-save-objects=%t.thin.out -thinlto-action=run %t2.bc  %t.bc -exported-symbol=main -thinlto-cache-dir %t.cache 
+; RUN: ls %t.thin.out | count 2
+; RUN: ls %t.cache | count 3
+
+
+target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-apple-macosx10.11.0"
+
+define void @globalfunc() #0 {
+entry:
+  ret void
+}

llvm/trunk/tools/llvm-lto/llvm-lto.cpp

 static cl::opt<std::string>
     ThinLTOCacheDir("thinlto-cache-dir", cl::desc("Enable ThinLTO caching."));
 
 static cl::opt<std::string> ThinLTOSaveTempsPrefix(
     "thinlto-save-temps",
     cl::desc("Save ThinLTO temp files using filenames created by adding "
              "suffixes to the given file path prefix."));
 
+static cl::opt<std::string> ThinLTOGeneratedObjectsDir(
+    "thinlto-save-objects",
+    cl::desc("Save ThinLTO generated object files using filenames created in "
+             "the given directory."));
+
 static cl::opt<bool>
     SaveModuleFile("save-merged-module", cl::init(false),
                    cl::desc("Write merged LTO module to file before CodeGen"));
 
 static cl::list<std::string> InputFilenames(cl::Positional, cl::OneOrMore,
                                             cl::desc("<input bitcode files>"));
 
 static cl::opt<std::string> OutputFilename("o", cl::init(""),
for (unsigned i = 0; i < InputFilenames.size(); ++i) {
       auto InputOrErr = MemoryBuffer::getFile(Filename);
       error(InputOrErr, "error " + CurrentActivity);
       InputBuffers.push_back(std::move(*InputOrErr));
       ThinGenerator.addModule(Filename, InputBuffers.back()->getBuffer());
     }
 
     if (!ThinLTOSaveTempsPrefix.empty())
       ThinGenerator.setSaveTempsDir(ThinLTOSaveTempsPrefix);
+
+    if (!ThinLTOGeneratedObjectsDir.empty()) {
+      ThinGenerator.setGeneratedObjectsDirectory(ThinLTOGeneratedObjectsDir);
+      ThinGenerator.run();
+      return;
+    }
+
     ThinGenerator.run();
 
     auto &Binaries = ThinGenerator.getProducedBinaries();
     if (Binaries.size() != InputFilenames.size())
       report_fatal_error("Number of output objects does not match the number "
                          "of inputs");
 
     for (unsigned BufID = 0; BufID < Binaries.size(); ++BufID) {

llvm/trunk/tools/lto/lto.cpp

 LTOObjectBuffer thinlto_module_get_object(thinlto_code_gen_t cg,
                                           unsigned int index) {
   assert(index < unwrap(cg)->getProducedBinaries().size() && "Index overflow");
   auto &MemBuffer = unwrap(cg)->getProducedBinaries()[index];
   return LTOObjectBuffer{MemBuffer->getBufferStart(),
                          MemBuffer->getBufferSize()};
 }
 
+unsigned int thinlto_module_get_num_object_files(thinlto_code_gen_t cg) {
+  return unwrap(cg)->getProducedBinaryFiles().size();
+}
+const char *thinlto_module_get_object_file(thinlto_code_gen_t cg,
+                                           unsigned int index) {
+  assert(index < unwrap(cg)->getProducedBinaryFiles().size() &&
+         "Index overflow");
+  return unwrap(cg)->getProducedBinaryFiles()[index].c_str();
+}
+
 void thinlto_codegen_disable_codegen(thinlto_code_gen_t cg,
                                      lto_bool_t disable) {
   unwrap(cg)->disableCodeGen(disable);
 }
 
 void thinlto_codegen_set_codegen_only(thinlto_code_gen_t cg,
                                       lto_bool_t CodeGenOnly) {
   unwrap(cg)->setCodeGenOnly(CodeGenOnly);
void thinlto_codegen_set_final_cache_size_relative_to_available_space(
   return unwrap(cg)->setMaxCacheSizeRelativeToAvailableSpace(Percentage);
 }
 
 void thinlto_codegen_set_savetemps_dir(thinlto_code_gen_t cg,
                                        const char *save_temps_dir) {
   return unwrap(cg)->setSaveTempsDir(save_temps_dir);
 }
 
+void thinlto_set_generated_objects_dir(thinlto_code_gen_t cg,
+                                       const char *save_temps_dir) {
+  unwrap(cg)->setGeneratedObjectsDirectory(save_temps_dir);
+}
+
 lto_bool_t thinlto_codegen_set_pic_model(thinlto_code_gen_t cg,
                                          lto_codegen_model model) {
   switch (model) {
   case LTO_CODEGEN_PIC_MODEL_STATIC:
     unwrap(cg)->setCodePICModel(Reloc::Static);
     return false;
   case LTO_CODEGEN_PIC_MODEL_DYNAMIC:
     unwrap(cg)->setCodePICModel(Reloc::PIC_);

llvm/trunk/tools/lto/lto.exports

 thinlto_codegen_set_savetemps_dir
 thinlto_codegen_set_cpu
 thinlto_debug_options
 lto_module_is_thinlto
 thinlto_codegen_add_must_preserve_symbol
 thinlto_codegen_add_cross_referenced_symbol
 thinlto_codegen_set_final_cache_size_relative_to_available_space
 thinlto_codegen_set_codegen_only
-thinlto_codegen_disable_codegen
+thinlto_codegen_disable_codegen
+thinlto_module_get_num_object_files
+thinlto_module_get_object_file
+thinlto_set_generated_objects_dir
 No newline at end of file