[lld-link] implement -thinlto-index-only

[inglorion]

Summary

This implements -thinlto-index-only, -thinlto-index-only:,
and -thinlto-emit-imports-files options in lld-link. They are
analogous to their counterparts in ld.lld: -thinlto-index-only
causes us to perform ThinLTO's thin link and write index files,
but not perform code generation. -thinlto-index-only: does the
same, but also writes a text file listing the native object
files expected to be generated. -thinlto-emit-imports-files
creates a text file next to each index file, listing the files
to import from.

[rnk]

lld/COFF/LTO.cpp

57

I guess this is mainly a future extension point for prefix rewriting, so we can have different .thinlto.bc index files for different executable targets.

[rnk]

Hit send without saying things...

This basically looks good to me, but I figured you should wait to get feedback from other reviewers.

[inglorion]

lld/COFF/LTO.cpp

57

Exactly. The ELF linker allows prefix and suffix replace here, and we'll want that for the COFF linker, too. Patch is coming.

[inglorion]

lld/COFF/LTO.cpp

57

Originally I had the prefix and suffix replace code in this patch, but I broke it up into smaller pieces.

[ruiu]

LGTM

lld/COFF/LTO.cpp

57

Please move this https://reviews.llvm.org/D64542 so that this identity function is not included in the first patch.

[inglorion]

removed getThinLTOOutputFile identity function

[inglorion]

new variable naming style

[smeenai]

Distributed ThinLTO doesn't support archives: http://lists.llvm.org/pipermail/llvm-dev/2019-June/133145.html. @tejohnson suggested -start-lib -end-lib as a workaround, but that's not a thing for COFF. Do you have plans for archive support, or is that not something you need to deal with for your use cases?

[rnk]

In D64461#1586526, @smeenai wrote:

Distributed ThinLTO doesn't support archives: http://lists.llvm.org/pipermail/llvm-dev/2019-June/133145.html. @tejohnson suggested -start-lib -end-lib as a workaround, but that's not a thing for COFF. Do you have plans for archive support, or is that not something you need to deal with for your use cases?

Personally I believe we should prioritize fixing that. It's trivial to fix for thin archives, since those are practically equivalent to the command line object groups.

[tejohnson]

In D64461#1586582, @rnk wrote:

In D64461#1586526, @smeenai wrote:

Distributed ThinLTO doesn't support archives: http://lists.llvm.org/pipermail/llvm-dev/2019-June/133145.html. @tejohnson suggested -start-lib -end-lib as a workaround, but that's not a thing for COFF. Do you have plans for archive support, or is that not something you need to deal with for your use cases?

Personally I believe we should prioritize fixing that. It's trivial to fix for thin archives, since those are practically equivalent to the command line object groups.

Someone asked about this on the llvm-dev mailing list recently, pasting my response below:

Somehow, the module identifier for each constituent object would need to be both unique (we currently generate this from the name of the archive plus the offset in the archive plus the name of the source file IIRC), but also correctly identify the extracted bitcode object used in the post-thinlink backend invocation. This is so it can write out the distributed index file with a filename that gets consumed by the associated backend invocation (passed to -fthinlto-index=), and so that the module paths emitted in those index files correctly identify where we can import functions from. Since the bitcode objects need to be extracted for the corresponding backend clang invocations, a couple possibilities come to mind:

1. Do it outside the compiler/linker: wrap the whole thing in a script that does the extraction, invokes the link with extracted constituents surrounded by --start-lib/-end-lib pairs, invokes each backend through some parallel or distributed mechanism, and then invokes the final link; or
2. Add support to pass some kind of mapping file into LTO that maps from each archive constituent to the extracted filename including path that the corresponding ThinLTO backend clang invocation will use, and have LTO set the module identifiers accordingly so that everything "just works" (in theory). We already support some munging of these names (see the thinlto_object_suffix_replace plugin option in either gold-plugin.cpp or in lld), but what you need here is a bit more complicated than a simple suffix change. But since there is already support for adjusting the name, it might not be too bad to add this support.

[rnk]

In D64461#1587532, @tejohnson wrote:

In D64461#1586582, @rnk wrote:

Personally I believe we should prioritize fixing that. It's trivial to fix for thin archives, since those are practically equivalent to the command line object groups.

Someone asked about this on the llvm-dev mailing list recently, pasting my response below:

Somehow, the module identifier for each constituent object would need to be both unique (we currently generate this from the name of the archive plus the offset in the archive plus the name of the source file IIRC), but also correctly identify the extracted bitcode object used in the post-thinlink backend invocation. This is so it can write out the distributed index file with a filename that gets consumed by the associated backend invocation (passed to -fthinlto-index=), and so that the module paths emitted in those index files correctly identify where we can import functions from. Since the bitcode objects need to be extracted for the corresponding backend clang invocations, a couple possibilities come to mind:

1. Do it outside the compiler/linker: wrap the whole thing in a script that does the extraction, invokes the link with extracted constituents surrounded by --start-lib/-end-lib pairs, invokes each backend through some parallel or distributed mechanism, and then invokes the final link; or
2. Add support to pass some kind of mapping file into LTO that maps from each archive constituent to the extracted filename including path that the corresponding ThinLTO backend clang invocation will use, and have LTO set the module identifiers accordingly so that everything "just works" (in theory). We already support some munging of these names (see the thinlto_object_suffix_replace plugin option in either gold-plugin.cpp or in lld), but what you need here is a bit more complicated than a simple suffix change. But since there is already support for adjusting the name, it might not be too bad to add this support.

For thin archives, the bitcode files should already be extracted, it's just a matter of adjusting the path to the thin archive input. I think what you described for true static archives is quite a bit of work, and I don't think it's worth doing. But, making thin archives work should be feasible, and it moving a build system from fat to thin archives should just be a matter of passing a flag to the archiver. I think that extra ease of use is worth a lot.

To @ruiu's point on the llvm-commits thread, I'm also in favor of adding --start-lib --end-lib to LLD COFF. There's no reason we can't do both.

lld/COFF/Config.h

struct Configuration {
   std::map<StringRef, StringRef> AlternateNames;
 
   // Used for /order.
   llvm::StringMap<int> Order;
 
   // Used for /lldmap.
   std::string MapFile;
 
+  // Used for /thinlto-index-only:
+  llvm::StringRef ThinLTOIndexOnlyArg;
+
   uint64_t ImageBase = -1;
   uint64_t FileAlign = 512;
   uint64_t StackReserve = 1024 * 1024;
   uint64_t StackCommit = 4096;
   uint64_t HeapReserve = 1024 * 1024;
   uint64_t HeapCommit = 4096;
   uint32_t MajorImageVersion = 0;
   uint32_t MinorImageVersion = 0;
struct Configuration {
   bool WarnLocallyDefinedImported = true;
   bool WarnDebugInfoUnusable = true;
   bool Incremental = true;
   bool IntegrityCheck = false;
   bool KillAt = false;
   bool Repro = false;
   bool SwaprunCD = false;
   bool SwaprunNet = false;
+  bool ThinLTOEmitImportsFiles;
+  bool ThinLTOIndexOnly;
 };
 
 extern Configuration *Config;
 
 } // namespace coff
 } // namespace lld
 
 #endif

lld/COFF/Driver.cpp

void LinkerDriver::link(ArrayRef<const char *> ArgsArr) {
   for (auto *Arg : Args.filtered(OPT_manifestinput))
     Config->ManifestInput.push_back(Arg->getValue());
 
   if (!Config->ManifestInput.empty() &&
       Config->Manifest != Configuration::Embed) {
     fatal("/manifestinput: requires /manifest:embed");
   }
 
+  Config->ThinLTOEmitImportsFiles = Args.hasArg(OPT_thinlto_emit_imports_files);
+  Config->ThinLTOIndexOnly = Args.hasArg(OPT_thinlto_index_only) ||
+                             Args.hasArg(OPT_thinlto_index_only_arg);
+  Config->ThinLTOIndexOnlyArg =
+      Args.getLastArgValue(OPT_thinlto_index_only_arg);
   // Handle miscellaneous boolean flags.
   Config->AllowBind = Args.hasFlag(OPT_allowbind, OPT_allowbind_no, true);
   Config->AllowIsolation =
       Args.hasFlag(OPT_allowisolation, OPT_allowisolation_no, true);
   Config->Incremental =
       Args.hasFlag(OPT_incremental, OPT_incremental_no,
                    !Config->DoGC && !Config->DoICF && !Args.hasArg(OPT_order) &&
                        !Args.hasArg(OPT_profile));
do {
     if (Symtab->findUnderscore("_load_config_used"))
       addUndefined(mangle("_load_config_used"));
   } while (run());
 
   if (errorCount())
     return;
 
   // Do LTO by compiling bitcode input files to a set of native COFF files then
-  // link those files.
+  // link those files (unless -thinlto-index-only was given, in which case we
+  // resolve symbols and write indices, but don't generate native code or link).
   Symtab->addCombinedLTOObjects();
+
+  // If -thinlto-index-only is given, we should create only "index
+  // files" and not object files. Index file creation is already done
+  // in addCombinedLTOObject, so we are done if that's the case.
+  if (Config->ThinLTOIndexOnly)
+    return;
+
+  // If we generated native object files from bitcode files, this resolves
+  // references to the symbols we use from them.
   run();
 
   if (Args.hasArg(OPT_include_optional)) {
     // Handle /includeoptional
     for (auto *Arg : Args.filtered(OPT_include_optional))
       if (dyn_cast_or_null<Lazy>(Symtab->find(Arg->getValue())))
         addUndefined(Arg->getValue());
     while (run());

lld/COFF/LTO.h

 // functionality.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLD_COFF_LTO_H
 #define LLD_COFF_LTO_H
 
 #include "lld/Common/LLVM.h"
+#include "llvm/ADT/DenseSet.h"
 #include "llvm/ADT/SmallString.h"
+#include "llvm/Support/raw_ostream.h"
 #include <memory>
 #include <vector>
 
 namespace llvm {
 namespace lto {
 class LTO;
 }
 }
public:
 
   void add(BitcodeFile &F);
   std::vector<StringRef> compile();
 
 private:
   std::unique_ptr<llvm::lto::LTO> LTOObj;
   std::vector<SmallString<0>> Buf;
   std::vector<std::unique_ptr<MemoryBuffer>> Files;
+  std::unique_ptr<llvm::raw_fd_ostream> IndexFile;
+  llvm::DenseSet<StringRef> ThinIndices;
 };
 }
 }
 
 #endif

lld/COFF/LTO.cpp

 #include "lld/Common/Args.h"
 #include "lld/Common/ErrorHandler.h"
 #include "lld/Common/Strings.h"
 #include "lld/Common/TargetOptionsCommandFlags.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/Twine.h"
+#include "llvm/Bitcode/BitcodeWriter.h"
 #include "llvm/IR/DiagnosticPrinter.h"
 #include "llvm/LTO/Caching.h"
 #include "llvm/LTO/Config.h"
 #include "llvm/LTO/LTO.h"
 #include "llvm/Object/SymbolicFile.h"
 #include "llvm/Support/CodeGen.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/raw_ostream.h"
 #include <algorithm>
 #include <cstddef>
 #include <memory>
 #include <string>
 #include <system_error>
 #include <vector>
 
 using namespace llvm;
 using namespace llvm::object;
 
 using namespace lld;
 using namespace lld::coff;
 
-static std::unique_ptr<lto::LTO> createLTO() {
+// Creates an empty file to and returns a raw_fd_ostream to write to it.
+static std::unique_ptr<raw_fd_ostream> openFile(StringRef File) {
+  std::error_code EC;
+  auto Ret =
+      llvm::make_unique<raw_fd_ostream>(File, EC, sys::fs::OpenFlags::F_None);
+  if (EC) {
+    error("cannot open " + File + ": " + EC.message());
+    return nullptr;
+  }
+  return Ret;
+}
+
+static std::string getThinLTOOutputFile(StringRef ModulePath) {
+  return ModulePath;
+}
+
+static lto::Config createConfig() {
   lto::Config C;
   C.Options = initTargetOptionsFromCodeGenFlags();
 
   // Always emit a section per function/datum with LTO. LLVM LTO should get most
   // of the benefit of linker GC, but there are still opportunities for ICF.
   C.Options.FunctionSections = true;
   C.Options.DataSections = true;
 
static lto::Config createConfig() {
   C.OptLevel = Config->LTOO;
   C.CPU = getCPUStr();
   C.MAttrs = getMAttrs();
   C.CGOptLevel = args::getCGOptLevel(Config->LTOO);
 
   if (Config->SaveTemps)
     checkError(C.addSaveTemps(std::string(Config->OutputFile) + ".",
                               /*UseInputModulePath*/ true));
+  return C;
+}
+
+BitcodeCompiler::BitcodeCompiler() {
+  // Initialize IndexFile.
+  if (!Config->ThinLTOIndexOnlyArg.empty())
+    IndexFile = openFile(Config->ThinLTOIndexOnlyArg);
+
+  // Initialize LTOObj.
   lto::ThinBackend Backend;
-  if (Config->ThinLTOJobs != 0)
+  if (Config->ThinLTOIndexOnly) {
+    auto OnIndexWrite = [&](StringRef S) { ThinIndices.erase(S); };
+    Backend = lto::createWriteIndexesThinBackend(
+        "", "", Config->ThinLTOEmitImportsFiles, IndexFile.get(), OnIndexWrite);
+  } else if (Config->ThinLTOJobs != 0) {
     Backend = lto::createInProcessThinBackend(Config->ThinLTOJobs);
-  return llvm::make_unique<lto::LTO>(std::move(C), Backend,
-                                     Config->LTOPartitions);
   }
 
-BitcodeCompiler::BitcodeCompiler() : LTOObj(createLTO()) {}
+  LTOObj = llvm::make_unique<lto::LTO>(createConfig(), Backend,
+                                       Config->LTOPartitions);
+}
 
 BitcodeCompiler::~BitcodeCompiler() = default;
 
 static void undefine(Symbol *S) { replaceSymbol<Undefined>(S, S->getName()); }
 
 void BitcodeCompiler::add(BitcodeFile &F) {
   lto::InputFile &Obj = *F.Obj;
   unsigned SymNum = 0;
   std::vector<Symbol *> SymBodies = F.getSymbols();
   std::vector<lto::SymbolResolution> Resols(SymBodies.size());
 
+  if (Config->ThinLTOIndexOnly)
+    ThinIndices.insert(Obj.getName());
+
   // Provide a resolution to the LTO API for each symbol.
   for (const lto::InputFile::Symbol &ObjSym : Obj.symbols()) {
     Symbol *Sym = SymBodies[SymNum];
     lto::SymbolResolution &R = Resols[SymNum];
     ++SymNum;
 
     // Ideally we shouldn't check for SF_Undefined but currently IRObjectFile
     // reports two symbols for module ASM defined. Without this check, lld
std::vector<StringRef> BitcodeCompiler::compile() {
 
   checkError(LTOObj->run(
       [&](size_t Task) {
         return llvm::make_unique<lto::NativeObjectStream>(
             llvm::make_unique<raw_svector_ostream>(Buf[Task]));
       },
       Cache));
 
+  // Emit empty index files for non-indexed files
+  for (StringRef S : ThinIndices) {
+    std::string Path = getThinLTOOutputFile(S);
+    openFile(Path + ".thinlto.bc");
+    if (Config->ThinLTOEmitImportsFiles)
+      openFile(Path + ".imports");
+  }
+
+  // ThinLTO with index only option is required to generate only the index
+  // files. After that, we exit from linker and ThinLTO backend runs in a
+  // distributed environment.
+  if (Config->ThinLTOIndexOnly) {
+    if (IndexFile)
+      IndexFile->close();
+    return {};
+  }
+
   if (!Config->LTOCache.empty())
     pruneCache(Config->LTOCache, Config->LTOCachePolicy);
 
   std::vector<StringRef> Ret;
   for (unsigned I = 0; I != MaxTasks; ++I) {
     if (Buf[I].empty())
       continue;
     if (Config->SaveTemps) {

lld/COFF/Options.td

def include_optional : Joined<["/", "-", "/?", "-?"], "includeoptional:">,
     HelpText<"Add symbol as undefined, but allow it to remain undefined">;
 def kill_at : F<"kill-at">;
 def lldmingw : F<"lldmingw">;
 def output_def : Joined<["/", "-", "/?", "-?"], "output-def:">;
 def pdb_source_path : P<"pdbsourcepath",
                         "Base path used to make relative source file path absolute in PDB">;
 def rsp_quoting : Joined<["--"], "rsp-quoting=">,
   HelpText<"Quoting style for response files, 'windows' (default) or 'posix'">;
+def thinlto_emit_imports_files :
+    F<"thinlto-emit-imports-files">,
+    HelpText<"Emit .imports files with -thinlto-index-only">;
+def thinlto_index_only :
+    F<"thinlto-index-only">,
+    HelpText<"Instead of linking, emit ThinLTO index files">;
+def thinlto_index_only_arg : P<
+    "thinlto-index-only",
+    "-thinlto-index-only and also write native module names to file">;
 def dash_dash_version : Flag<["--"], "version">,
   HelpText<"Print version information">;
 defm threads: B<"threads",
     "Run the linker multi-threaded (default)",
     "Do not run the linker multi-threaded">;
 
 // Flags for debugging
 def lldmap : F<"lldmap">;

lld/test/COFF/Inputs/thinlto-empty.ll

+target datalayout = "e-m:w-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-pc-windows-msvc19.0.24215"

lld/test/COFF/Inputs/thinlto.ll

+target datalayout = "e-m:w-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-pc-windows-msvc19.0.24215"
+
+define void @g() {
+  ret void
+}

lld/test/COFF/thinlto-emit-imports.ll

+; REQUIRES: x86
+
+; Generate summary sections and test lld handling.
+; RUN: opt -module-summary %s -o %t1.obj
+; RUN: opt -module-summary %p/Inputs/thinlto.ll -o %t2.obj
+
+; Include a file with an empty module summary index, to ensure that the expected
+; output files are created regardless, for a distributed build system.
+; RUN: opt -module-summary %p/Inputs/thinlto-empty.ll -o %t3.obj
+
+; Ensure lld generates imports files if requested for distributed backends.
+; RUN: rm -f %t3.obj.imports %t3.obj.thinlto.bc
+; RUN: lld-link -entry:main -thinlto-index-only \
+; RUN:     -thinlto-emit-imports-files %t1.obj %t2.obj %t3.obj -out:%t4.exe
+
+; The imports file for this module contains the bitcode file for
+; Inputs/thinlto.ll
+; RUN: cat %t1.obj.imports | count 1
+; RUN: cat %t1.obj.imports | FileCheck %s --check-prefix=IMPORTS1
+; IMPORTS1: thinlto-emit-imports.ll.tmp2.obj
+
+; The imports file for Input/thinlto.ll is empty as it does not import anything.
+; RUN: cat %t2.obj.imports | count 0
+
+; The imports file for Input/thinlto_empty.ll is empty but should exist.
+; RUN: cat %t3.obj.imports | count 0
+
+; The index file should be created even for the input with an empty summary.
+; RUN: ls %t3.obj.thinlto.bc
+
+; Ensure lld generates error if unable to write to imports file.
+; RUN: rm -f %t3.obj.imports
+; RUN: touch %t3.obj.imports
+; RUN: chmod 400 %t3.obj.imports
+; RUN: not lld-link -entry:main -thinlto-index-only \
+; RUN:     -thinlto-emit-imports-files %t1.obj %t2.obj %t3.obj \
+; RUN:     -out:%t4.exe 2>&1 | FileCheck %s --check-prefix=ERR
+; ERR: cannot open {{.*}}3.obj.imports: {{P|p}}ermission denied
+
+; Ensure lld doesn't generate import files when thinlto-index-only is not enabled
+; RUN: rm -f %t1.obj.imports
+; RUN: rm -f %t2.obj.imports
+; RUN: rm -f %t3.obj.imports
+; RUN: lld-link -entry:main -thinlto-emit-imports-files \
+; RUN:     %t1.obj %t2.obj %t3.obj -out:%t4.exe
+; RUN: not ls %t1.obj.imports
+; RUN: not ls %t2.obj.imports
+; RUN: not ls %t3.obj.imports
+
+target datalayout = "e-m:w-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-pc-windows-msvc19.0.24215"
+
+declare void @g(...)
+
+define void @main() {
+entry:
+  call void (...) @g()
+  ret void
+}

lld/test/COFF/thinlto-index-only.ll

+; REQUIRES: x86
+
+; Basic ThinLTO tests.
+; RUN: opt -thinlto-bc %s -o %t1.obj
+; RUN: opt -thinlto-bc %p/Inputs/thinlto.ll -o %t2.obj
+; RUN: opt -thinlto-bc %p/Inputs/thinlto-empty.ll -o %t3.obj
+
+; Ensure lld generates an index and not a binary if requested.
+; RUN: rm -f %t4.exe
+; RUN: lld-link -thinlto-index-only -entry:main %t1.obj %t2.obj -out:%t4.exe
+; RUN: llvm-bcanalyzer -dump %t1.obj.thinlto.bc | FileCheck %s --check-prefix=BACKEND1
+; RUN: llvm-bcanalyzer -dump %t2.obj.thinlto.bc | FileCheck %s --check-prefix=BACKEND2
+; RUN: not test -e %t4.exe
+
+; The backend index for this module contains summaries from itself and
+; Inputs/thinlto.ll, as it imports from the latter.
+; BACKEND1: <MODULE_STRTAB_BLOCK
+; BACKEND1-NEXT: <ENTRY {{.*}} record string = '{{.*}}thinlto-index-only.ll.tmp{{.*}}.obj'
+; BACKEND1: <ENTRY {{.*}} record string = '{{.*}}thinlto-index-only.ll.tmp{{.*}}.obj'
+; BACKEND1-NOT: <ENTRY
+; BACKEND1: </MODULE_STRTAB_BLOCK
+; BACKEND1: <GLOBALVAL_SUMMARY_BLOCK
+; BACKEND1: <VERSION
+; BACKEND1: <FLAGS
+; BACKEND1: <VALUE_GUID op0={{1|2}} op1={{-5300342847281564238|-2624081020897602054}}
+; BACKEND1: <VALUE_GUID op0={{1|2}} op1={{-5300342847281564238|-2624081020897602054}}
+; BACKEND1: <COMBINED
+; BACKEND1: <COMBINED
+; BACKEND1: </GLOBALVAL_SUMMARY_BLOCK
+
+; The backend index for Input/thinlto.ll contains summaries from itself only,
+; as it does not import anything.
+; BACKEND2: <MODULE_STRTAB_BLOCK
+; BACKEND2-NEXT: <ENTRY {{.*}} record string = '{{.*}}thinlto-index-only.ll.tmp2.obj'
+; BACKEND2-NOT: <ENTRY
+; BACKEND2: </MODULE_STRTAB_BLOCK
+; BACKEND2-NEXT: <GLOBALVAL_SUMMARY_BLOCK
+; BACKEND2-NEXT: <VERSION
+; BACKEND2-NEXT: <FLAGS
+; BACKEND2-NEXT: <VALUE_GUID op0=1 op1=-5300342847281564238
+; BACKEND2-NEXT: <COMBINED
+; BACKEND2-NEXT: </GLOBALVAL_SUMMARY_BLOCK
+
+target datalayout = "e-m:w-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-pc-windows-msvc19.0.24215"
+
+declare void @g(...)
+
+define void @main() {
+  call void (...) @g()
+  ret void
+}