[ThinLTO] Add option to emit imports files for distributed backends

[tejohnson]

Summary

Add a new thinlto-emit-imports-files plugin option to enable emission of
plaintext lists of the imported files for each distributed backend compilation.
Used for distributed build file staging. This option is only valid with
thinlto-index-only (i.e. for distributed builds).

Depends on D19556.

[mehdi_amini]

It'd be nice to have this mirrored in llvm-lto.

Actually I have the impression that the gold-plugin is getting too fat: these functionality should be abstracted in LLVM an not be implemented here.

[tejohnson]

In D19636#415046, @joker.eph wrote:

It'd be nice to have this mirrored in llvm-lto.

I will do that and add a test.

Actually I have the impression that the gold-plugin is getting too fat: these functionality should be abstracted in LLVM an not be implemented here.

I could move this particular code into a helper in FunctionImport.cpp that takes in the ImportsLists and a file name as input, and do the file I/O there.
(Unrelated to this patch, but to the above comment: as noted in my response to D19556 this morning, I will pull the preceding code to compute the ModuleToSummariesForIndexMap into a helper in FunctionImport.cpp.)

[mehdi_amini]

Actually I have the impression that the gold-plugin is getting too fat: these functionality should be abstracted in LLVM an not be implemented here.

I could move this particular code into a helper in FunctionImport.cpp that takes in the ImportsLists and a file name as input, and do the file I/O there.
(Unrelated to this patch, but to the above comment: as noted in my response to D19556 this morning, I will pull the preceding code to compute the ModuleToSummariesForIndexMap into a helper in FunctionImport.cpp.)

Yes this is not about this patch in particular, this is about this plugin file that drifted over time. Its scope should be very narrowed to the interaction with gold IMO. All other logic should be refactored in some LLVM libraries.

[tejohnson]

• Address comments: refactor into a helper and add libLTO and llvm-lto support and tests

[tejohnson]

Ping

[mehdi_amini]

Oh sorry I missed the update to the patch, will look this morning (hopefully)

[tejohnson]

In D19636#421031, @joker.eph wrote:

Oh sorry I missed the update to the patch, will look this morning (hopefully)

Thanks, actually I meant to ping the one this is dependent on first (D19556 for emitting individual backend index files), but they were both updated last week (along with D19644) and are ready for review.

[mehdi_amini]

LGTM, with two remarks.

include/llvm/LTO/ThinLTOCodeGenerator.h

198 ↗

(On Diff #55528)

for *each* module? I'd say for \p ModulePath instead.

201 ↗

(On Diff #55528)

Same remark as the previous path: this does not really use anything specific to the ThinLTOCodeGenerator.

[tejohnson]

thanks

include/llvm/LTO/ThinLTOCodeGenerator.h

198 ↗

(On Diff #55528)

Yes the old comment wasn't really correct, will fix

201 ↗

(On Diff #55528)

Will change this to a static method too.

[vitalybuka]

This patch is the reason for
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fast/builds/12540

[tejohnson]

In D19636#422547, @vitalybuka wrote:

This patch is the reason for
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fast/builds/12540

This patch is not yet committed. However, another patch I committed this morning (D19556) is the cause of this. I'm working on tracking it down (unfortunately, it isn't obvious since we shouldn't even execute the routine in question and don't with a non-msan compiler. Trying to build an msan compiler right now to track down further). Should I revert in the meantime?

[vitalybuka]

My mistake, it is D19556. I searched by sub-string in the title.
It's better to revert if a fix is not obvious.

llvm/trunk/include/llvm/LTO/ThinLTOCodeGenerator.h

public:
   std::unique_ptr<ModuleSummaryIndex> linkCombinedIndex();
 
   /**
    * Perform promotion and renaming of exported internal functions.
    */
   void promote(Module &Module, ModuleSummaryIndex &Index);
 
   /**
+   * Compute and emit the imported files for module at \p ModulePath.
+   */
+  static void emitImports(StringRef ModulePath, StringRef OutputName,
+                          ModuleSummaryIndex &Index);
+
+  /**
    * Perform cross-module importing for the module identified by
    * ModuleIdentifier.
    */
   void crossModuleImport(Module &Module, ModuleSummaryIndex &Index);
 
   /**
    * Compute the list of summaries needed for importing into module.
    */

llvm/trunk/include/llvm/Transforms/IPO/FunctionImport.h

 /// \p ModuleToSummariesForIndex will be populated with the needed summaries
 /// from each required module path. Use a std::map instead of StringMap to get
 /// stable order for bitcode emission.
 void gatherImportedSummariesForModule(
     StringRef ModulePath,
     const StringMap<GVSummaryMapTy> &ModuleToDefinedGVSummaries,
     const StringMap<FunctionImporter::ImportMapTy> &ImportLists,
     std::map<std::string, GVSummaryMapTy> &ModuleToSummariesForIndex);
+
+std::error_code
+EmitImportsFiles(StringRef ModulePath, StringRef OutputFilename,
+                 const StringMap<FunctionImporter::ImportMapTy> &ImportLists);
 }
 
 #endif // LLVM_FUNCTIONIMPORT_H

llvm/trunk/lib/LTO/ThinLTOCodeGenerator.cpp

ComputeCrossModuleImport(Index, ModuleToDefinedGVSummaries, ImportLists,
                            ExportLists);
 
   llvm::gatherImportedSummariesForModule(ModulePath, ModuleToDefinedGVSummaries,
                                          ImportLists,
                                          ModuleToSummariesForIndex);
 }
 
 /**
+ * Emit the list of files needed for importing into module.
+ */
+void ThinLTOCodeGenerator::emitImports(StringRef ModulePath,
+                                       StringRef OutputName,
+                                       ModuleSummaryIndex &Index) {
+  auto ModuleCount = Index.modulePaths().size();
+
+  // Collect for each module the list of function it defines (GUID -> Summary).
+  StringMap<GVSummaryMapTy> ModuleToDefinedGVSummaries(ModuleCount);
+  Index.collectDefinedGVSummariesPerModule(ModuleToDefinedGVSummaries);
+
+  // Generate import/export list
+  StringMap<FunctionImporter::ImportMapTy> ImportLists(ModuleCount);
+  StringMap<FunctionImporter::ExportSetTy> ExportLists(ModuleCount);
+  ComputeCrossModuleImport(Index, ModuleToDefinedGVSummaries, ImportLists,
+                           ExportLists);
+
+  std::error_code EC;
+  if ((EC = EmitImportsFiles(ModulePath, OutputName, ImportLists)))
+    report_fatal_error(Twine("Failed to open ") + OutputName +
+                       " to save imports lists\n");
+}
+
+/**
  * Perform internalization.
  */
 void ThinLTOCodeGenerator::internalize(Module &TheModule,
                                        ModuleSummaryIndex &Index) {
   initTMBuilder(TMBuilder, Triple(TheModule.getTargetTriple()));
   auto ModuleCount = Index.modulePaths().size();
   auto ModuleIdentifier = TheModule.getModuleIdentifier();
 

llvm/trunk/lib/Transforms/IPO/FunctionImport.cpp

for (auto &ILI : ModuleImports->second) {
         assert(DS != DefinedGVSummaries.end() &&
                "Expected a defined summary for imported global value");
         SummariesForIndex[GI.first] = DS->second;
       }
     }
   }
 }
 
+/// Emit the files \p ModulePath will import from into \p OutputFilename.
+std::error_code llvm::EmitImportsFiles(
+    StringRef ModulePath, StringRef OutputFilename,
+    const StringMap<FunctionImporter::ImportMapTy> &ImportLists) {
+  auto ModuleImports = ImportLists.find(ModulePath);
+  std::error_code EC;
+  raw_fd_ostream ImportsOS(OutputFilename, EC, sys::fs::OpenFlags::F_None);
+  if (EC)
+    return EC;
+  if (ModuleImports != ImportLists.end())
+    for (auto &ILI : ModuleImports->second)
+      ImportsOS << ILI.first() << "\n";
+  return std::error_code();
+}
+
 // Automatically import functions in Module \p DestModule based on the summaries
 // index.
 //
 bool FunctionImporter::importFunctions(
     Module &DestModule, const FunctionImporter::ImportMapTy &ImportList,
     bool ForceImportReferencedDiscardableSymbols) {
   DEBUG(dbgs() << "Starting import for Module "
                << DestModule.getModuleIdentifier() << "\n");

llvm/trunk/test/ThinLTO/X86/Inputs/emit_imports.ll

+define void @g() {
+entry:
+  ret void
+}

llvm/trunk/test/ThinLTO/X86/emit_imports.ll

+; RUN: opt -module-summary %s -o %t1.bc
+; RUN: opt -module-summary %p/Inputs/emit_imports.ll -o %t2.bc
+; RUN: llvm-lto -thinlto-action=thinlink -o %t.index.bc %t1.bc %t2.bc
+; RUN: llvm-lto -thinlto-action=emitimports -thinlto-index %t.index.bc %t1.bc %t2.bc
+
+; The imports file for this module contains the bitcode file for
+; Inputs/emit_imports.ll
+; RUN: cat %t1.bc.imports | count 1
+; RUN: cat %t1.bc.imports | FileCheck %s --check-prefix=IMPORTS1
+; IMPORTS1: emit_imports.ll.tmp2.bc
+
+; The imports file for Input/emit_imports.ll is empty as it does not import anything.
+; RUN: cat %t2.bc.imports | count 0
+
+declare void @g(...)
+
+define void @f() {
+entry:
+  call void (...) @g()
+  ret void
+}

llvm/trunk/test/tools/gold/X86/thinlto_emit_imports.ll

+; Generate summary sections and test gold handling.
+; RUN: opt -module-summary %s -o %t.o
+; RUN: opt -module-summary %p/Inputs/thinlto.ll -o %t2.o
+
+; Ensure gold generates imports files if requested for distributed backends.
+; RUN: %gold -plugin %llvmshlibdir/LLVMgold.so \
+; RUN:    --plugin-opt=thinlto \
+; RUN:    --plugin-opt=thinlto-index-only \
+; RUN:    --plugin-opt=thinlto-emit-imports-files \
+; RUN:    -shared %t.o %t2.o -o %t3
+
+; The imports file for this module contains the bitcode file for
+; Inputs/thinlto.ll
+; RUN: cat %t.o.imports | count 1
+; RUN: cat %t.o.imports | FileCheck %s --check-prefix=IMPORTS1
+; IMPORTS1: test/tools/gold/X86/Output/thinlto_emit_imports.ll.tmp2.o
+
+; The imports file for Input/thinlto.ll is empty as it does not import anything.
+; RUN: cat %t2.o.imports | count 0
+
+declare void @g(...)
+
+define void @f() {
+entry:
+  call void (...) @g()
+  ret void
+}

llvm/trunk/tools/gold/gold-plugin.cpp

#endif
   // global index for the ThinLTO backends.
   static bool thinlto = false;
   // If false, all ThinLTO backend compilations through code gen are performed
   // using multiple threads in the gold-plugin, before handing control back to
   // gold. If true, write individual backend index files which reflect
   // the import decisions, and exit afterwards. The assumption is
   // that the build system will launch the backend processes.
   static bool thinlto_index_only = false;
+  // If true, when generating individual index files for distributed backends,
+  // also generate a "${bitcodefile}.imports" file at the same location for each
+  // bitcode file, listing the files it imports from in plain text. This is to
+  // support distributed build file staging.
+  static bool thinlto_emit_imports_files = false;
   // Additional options to pass into the code generator.
   // Note: This array will contain all plugin options which are not claimed
   // as plugin exclusive to pass to the code generator.
   // For example, "generate-api-file" and "as"options are for the plugin
   // use only and will not be passed.
   static std::vector<const char *> extra;
 
   static void process_plugin_option(const char *opt_)
static void process_plugin_option(const char *opt_)
     } else if (opt == "save-temps") {
       TheOutputType = OT_SAVE_TEMPS;
     } else if (opt == "disable-output") {
       TheOutputType = OT_DISABLE;
     } else if (opt == "thinlto") {
       thinlto = true;
     } else if (opt == "thinlto-index-only") {
       thinlto_index_only = true;
+    } else if (opt == "thinlto-emit-imports-files") {
+      thinlto_emit_imports_files = true;
     } else if (opt.size() == 2 && opt[0] == 'O') {
       if (opt[1] < '0' || opt[1] > '3')
         message(LDPL_FATAL, "Optimization level must be between 0 and 3");
       OptLevel = opt[1] - '0';
     } else if (opt.startswith("jobs=")) {
       if (StringRef(opt_ + 5).getAsInteger(10, Parallelism))
         message(LDPL_FATAL, "Invalid parallelism level: %s", opt_ + 5);
     } else if (opt == "disable-verify") {
for (claimed_file &F : Modules) {
     std::unique_ptr<ModuleSummaryIndex> Index =
         getModuleSummaryIndexForFile(F, InputFile.file());
 
     // Skip files without a module summary.
     if (Index)
       CombinedIndex.mergeFrom(std::move(Index), ++NextModuleId);
   }
 
+  if (options::thinlto_emit_imports_files && !options::thinlto_index_only)
+    message(LDPL_WARNING,
+            "thinlto-emit-imports-files ignored unless thinlto-index-only");
+
   if (options::thinlto_index_only) {
     // Collect for each module the list of function it defines (GUID ->
     // Summary).
     StringMap<std::map<GlobalValue::GUID, GlobalValueSummary *>>
         ModuleToDefinedGVSummaries(NextModuleId);
     CombinedIndex.collectDefinedGVSummariesPerModule(
         ModuleToDefinedGVSummaries);
 
for (claimed_file &F : Modules) {
                 InputFile.file().name, EC.message().c_str());
       // Build a map of module to the GUIDs and summary objects that should
       // be written to its index.
       std::map<std::string, GVSummaryMapTy> ModuleToSummariesForIndex;
       gatherImportedSummariesForModule(InputFile.file().name,
                                        ModuleToDefinedGVSummaries, ImportLists,
                                        ModuleToSummariesForIndex);
       WriteIndexToFile(CombinedIndex, OS, &ModuleToSummariesForIndex);
+
+      if (options::thinlto_emit_imports_files) {
+        if ((EC = EmitImportsFiles(
+                 InputFile.file().name,
+                 (Twine(InputFile.file().name) + ".imports").str(),
+                 ImportLists)))
+          message(LDPL_FATAL, "Unable to open %s.imports",
+                  InputFile.file().name, EC.message().c_str());
+      }
     }
 
     cleanup_hook();
     exit(0);
   }
 
   // Create OS in nested scope so that it will be closed on destruction.
   {

llvm/trunk/tools/llvm-lto/llvm-lto.cpp

 
 static cl::opt<bool>
     ThinLTO("thinlto", cl::init(false),
             cl::desc("Only write combined global index for ThinLTO backends"));
 
 enum ThinLTOModes {
   THINLINK,
   THINDISTRIBUTE,
+  THINEMITIMPORTS,
   THINPROMOTE,
   THINIMPORT,
   THININTERNALIZE,
   THINOPT,
   THINCODEGEN,
   THINALL
 };
 
 cl::opt<ThinLTOModes> ThinLTOMode(
     "thinlto-action", cl::desc("Perform a single ThinLTO stage:"),
     cl::values(
         clEnumValN(
             THINLINK, "thinlink",
             "ThinLink: produces the index by linking only the summaries."),
         clEnumValN(THINDISTRIBUTE, "distributedindexes",
                    "Produces individual indexes for distributed backends."),
+        clEnumValN(THINEMITIMPORTS, "emitimports",
+                   "Emit imports files for distributed backends."),
         clEnumValN(THINPROMOTE, "promote",
                    "Perform pre-import promotion (requires -thinlto-index)."),
         clEnumValN(THINIMPORT, "import", "Perform both promotion and "
                                          "cross-module importing (requires "
                                          "-thinlto-index)."),
         clEnumValN(THININTERNALIZE, "internalize",
                    "Perform internalization driven by -exported-symbol "
                    "(requires -thinlto-index)."),
public:
   }
 
   void run() {
     switch (ThinLTOMode) {
     case THINLINK:
       return thinLink();
     case THINDISTRIBUTE:
       return distributedIndexes();
+    case THINEMITIMPORTS:
+      return emitImports();
     case THINPROMOTE:
       return promote();
     case THINIMPORT:
       return import();
     case THININTERNALIZE:
       return internalize();
     case THINOPT:
       return optimize();
for (auto &Filename : InputFilenames) {
       }
       std::error_code EC;
       raw_fd_ostream OS(OutputName, EC, sys::fs::OpenFlags::F_None);
       error(EC, "error opening the file '" + OutputName + "'");
       WriteIndexToFile(*Index, OS, &ModuleToSummariesForIndex);
     }
   }
 
+  /// Load the combined index from disk, compute the imports, and emit
+  /// the import file lists for each module to disk.
+  void emitImports() {
+    if (InputFilenames.size() != 1 && !OutputFilename.empty())
+      report_fatal_error("Can't handle a single output filename and multiple "
+                         "input files, do not provide an output filename and "
+                         "the output files will be suffixed from the input "
+                         "ones.");
+
+    auto Index = loadCombinedIndex();
+    for (auto &Filename : InputFilenames) {
+      std::string OutputName = OutputFilename;
+      if (OutputName.empty()) {
+        OutputName = Filename + ".imports";
+      }
+      ThinLTOCodeGenerator::emitImports(Filename, OutputName, *Index);
+    }
+  }
+
   /// Load the combined index from disk, then load every file referenced by
   /// the index and add them to the generator, finally perform the promotion
   /// on the files mentioned on the command line (these must match the index
   /// content).
   void promote() {
     if (InputFilenames.size() != 1 && !OutputFilename.empty())
       report_fatal_error("Can't handle a single output filename and multiple "
                          "input files, do not provide an output filename and "