[ThinLTO/gold] Add caching support to gold-plugin

[tejohnson]

Summary

With support now in the new LTO API for caching (r279576), add
optional ThinLTO caching in the gold-plugin.

[mehdi_amini]

LGTM.

(Note: you'll have to prune the cache dir manually)

[mehdi_amini]

llvm/trunk/tools/gold/gold-plugin.cpp

812

I just noticed: this is largely suboptimal, the Buffer will be a mmap'd file in memory, and you're duplicating it here.

[mehdi_amini]

llvm/trunk/tools/gold/gold-plugin.cpp

812

I didn't notice earlier because the libLTO interface returns a pointer to the memory to the linker, so the MemoryBuffer is untouched till the linker needs it (it does not need to be paged in memory, which can contributes to explains why you have much more memory pressure)

[tejohnson]

llvm/trunk/tools/gold/gold-plugin.cpp

812

I just noticed: this is largely suboptimal, the Buffer will be a mmap'd file in memory, and you're duplicating it here.

Dumb question - where is the duplication happening? getBuffer will give back a StringRef to the memory pointed to by the MemoryBuffer. And we then stream it to an output file (which we would have done earlier without caching).

[mehdi_amini]

llvm/trunk/tools/gold/gold-plugin.cpp

812

The StringRef points to the memory mapped from the cache. So this is data on disk that you load in memory while you access it here.
The duplication is that you're creating a new file, while there is already a file on disk, this wouldn't happen in the libLTO implementation.

[tejohnson]

llvm/trunk/tools/gold/gold-plugin.cpp

812

Ah ok, yes the file is duplicated unfortunately (temporarily), gold cleans up the temp object files written here. I didn't noticed a memory increase from this change.

Not sure if there is a way for gold-plugin to pass back the native object in memory, but I don't see a hook for that unfortunately.

[mehdi_amini]

llvm/trunk/tools/gold/gold-plugin.cpp

812

Even if the peak memory does not increase, the memory traffic (and cache trashing) is probably important (let along that these data need to be flushed to disk...)

llvm/trunk/test/tools/gold/X86/Inputs/cache.ll

+target triple = "x86_64-unknown-linux-gnu"
+
+define i32 @main() {
+entry:
+  call void (...) @globalfunc()
+  ret i32 0
+}
+
+declare void @globalfunc(...)

llvm/trunk/test/tools/gold/X86/cache.ll

+; RUN: opt -module-summary %s -o %t.o
+; RUN: opt -module-summary %p/Inputs/cache.ll -o %t2.o
+
+; Verify that enabling caching is working
+; RUN: rm -Rf %t.cache && mkdir %t.cache
+; RUN: %gold -m elf_x86_64 -plugin %llvmshlibdir/LLVMgold.so \
+; RUN:     --plugin-opt=thinlto \
+; RUN:     --plugin-opt=cache-dir=%t.cache \
+; RUN:     -o %t3.o %t2.o %t.o
+
+; RUN: ls %t.cache | count 2
+
+target triple = "x86_64-unknown-linux-gnu"
+
+define void @globalfunc() #0 {
+entry:
+  ret void
+}

llvm/trunk/tools/gold/gold-plugin.cpp

 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/Bitcode/ReaderWriter.h"
 #include "llvm/CodeGen/CommandFlags.h"
 #include "llvm/Config/config.h" // plugin-api.h requires HAVE_STDINT_H
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/DiagnosticPrinter.h"
+#include "llvm/LTO/Caching.h"
 #include "llvm/LTO/LTO.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/ManagedStatic.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/TargetSelect.h"
 #include "llvm/Support/raw_ostream.h"
 #include <list>
#endif
   static bool thinlto_emit_imports_files = false;
   // Option to control where files for a distributed backend (the individual
   // index files and optional imports files) are created.
   // If specified, expects a string of the form "oldprefix:newprefix", and
   // instead of generating these files in the same directory path as the
   // corresponding bitcode file, will use a path formed by replacing the
   // bitcode file's path prefix matching oldprefix with newprefix.
   static std::string thinlto_prefix_replace;
+  // Optional path to a directory for caching ThinLTO objects.
+  static std::string cache_dir;
   // Additional options to pass into the code generator.
   // Note: This array will contain all plugin options which are not claimed
   // as plugin exclusive to pass to the code generator.
   static std::vector<const char *> extra;
 
   static void process_plugin_option(const char *opt_)
   {
     if (opt_ == nullptr)
if (opt.startswith("mcpu=")) {
       thinlto_index_only = true;
       thinlto_linked_objects_file = opt.substr(strlen("thinlto-index-only="));
     } else if (opt == "thinlto-emit-imports-files") {
       thinlto_emit_imports_files = true;
     } else if (opt.startswith("thinlto-prefix-replace=")) {
       thinlto_prefix_replace = opt.substr(strlen("thinlto-prefix-replace="));
       if (thinlto_prefix_replace.find(";") == std::string::npos)
         message(LDPL_FATAL, "thinlto-prefix-replace expects 'old;new' format");
+    } else if (opt.startswith("cache-dir=")) {
+      cache_dir = opt.substr(strlen("cache-dir="));
     } else if (opt.size() == 2 && opt[0] == 'O') {
       if (opt[1] < '0' || opt[1] > '3')
         message(LDPL_FATAL, "Optimization level must be between 0 and 3");
       OptLevel = opt[1] - '0';
     } else if (opt.startswith("jobs=")) {
       if (StringRef(opt_ + 5).getAsInteger(10, Parallelism))
         message(LDPL_FATAL, "Invalid parallelism level: %s", opt_ + 5);
     } else if (opt == "disable-verify") {
static ld_plugin_status allSymbolsReadHook() {
   else if (options::TheOutputType == options::OT_SAVE_TEMPS)
     Filename = output_name + ".o";
   bool SaveTemps = !Filename.empty();
 
   MaxTasks = Lto->getMaxTasks();
   std::vector<uintptr_t> IsTemporary(MaxTasks);
   std::vector<SmallString<128>> Filenames(MaxTasks);
 
-  auto AddOutput = [&](size_t Task) {
+  auto AddOutput =
+      [&](size_t Task) -> std::unique_ptr<lto::NativeObjectOutput> {
     auto &OutputName = Filenames[Task];
     getOutputFileName(Filename, /*TempOutFile=*/!SaveTemps, OutputName,
                       MaxTasks > 1 ? Task : -1);
     IsTemporary[Task] = !SaveTemps;
+    if (options::cache_dir.empty())
       return llvm::make_unique<LTOOutput>(OutputName);
+
+    return llvm::make_unique<CacheObjectOutput>(
+        options::cache_dir, [OutputName](std::unique_ptr<MemoryBuffer> Buffer) {
+          *LTOOutput(OutputName).getStream() << Buffer->getBuffer();
+        });
   };
 
   check(Lto->run(AddOutput));
 
   if (options::TheOutputType == options::OT_DISABLE ||
       options::TheOutputType == options::OT_BC_ONLY)
     return LDPS_OK;