Support for distributed ThinLTO options

[rdhindsa]

Summary

Added support for options thinlto-index-only and thinlto-prefix-replace.

When thinlto-index-only option is specified, the thin LTO backend is created to generate the index files.
thinlto-prefix-replace controls where the files of distributed backend are created.

[ruiu]

There are a few things to note:

1. When you upload a patch, please include all context. Are you using git? If so, use git diff -u 10000 to include virtually the entire file in your diff.
2. Please read the LLVM coding style and stick with it. The easiest way of doing it is to use clang-format-diff to automatically format your patch according to the LLVM coding style.

lld/ELF/Driver.cpp

798

Please append {} to all these if ~ else if ~ else.

802

Basically all members in Config class are named after their corresponding command line options. This is a special case, but I'm reluctant to give a totally different name than the corresponding option. How about ThinLTOIndexOnlyFile?

803

I'd just write 19 instead of strlen(...).

806

Ditto

807

Use StringRef::contains.

808

fatal() is not for reporting a wrong usage. Use error() instead.

1015

I do understand that you need this or otherwise ThinLTO doesn't work. But this comment doesn't explain why we really need this. Could you expand the comment so that why ThinLTO needs to see all bitcode files even within --start-lib/end-lib?

1016

Flip the condition and break early.

if (Config->ThinLTOIndexOnly)
  break;

lld/ELF/LTO.cpp

72

Please stick with the LLVM/lld function naming scheme.

73

Please use clang-format.

74

I don't think you need this assertion.

88–89

Ditto

90

Why do you need a temporary variable?

101

Format

142

You can replace this with

std::string Old;
std::string New;
std::tie(Old, New) = Config->ThinLTOPrefixReplace.split(';');

156

Please follow the LLVM coding style.

158

Please fix style.

270

Well, you need to use clang-format.

lld/ELF/LTO.h

56

What is this for?

[rdhindsa]

lld/ELF/Driver.cpp

802

I have updated name to ThinLTOIndexOnlyObjectsFile. This file contains the list of object files. I think keeping objects in the name makes it easy to understand.

lld/ELF/LTO.cpp

74

Removed it.

90

Removed the getThinLTOOldAndNewPrefix function, since the old and new prefix can just be retreived by std::tie(OldPrefix, NewPrefix) = Config->ThinLTOPrefixReplace.split(';');

lld/ELF/LTO.h

56

I have removed ObjectToIndexFileState for now, I will be adding it in subsequent patch. It is supposed to track all the object files that are indexed.

[pcc]

lld/ELF/Driver.cpp

1015

The problem as I understand it is not related to files with no symbols but rather that if an object file between --start-lib and --end-lib is not selected using the usual rules for adding archive members to the link, the distributed build system will still want to see a .thinlto.bc file for it, even if it is unused.

1018

This part seems to be equivalent to writeEmptyDistributedBuildOutputs in the gold plugin. It's unfortunate that we need this in order to work around a deficiency in the distributed build system, but if we are going to do this, can we instead do something equivalent to that in order to avoid linking more than we need?

I think we should be able to do this by keeping track of which LazyObjFiles have been added to the link, and emitting a .thinlto.bc file for every file that has not been added to the link.

[rdhindsa]

lld/ELF/Driver.cpp

1018

I intended to add similar functionality as to writeEmptyDistributedBuildOutputs in the LTO.cpp while trying to add files, so that we emit .thinlto.bc for every file, instead of doing more linking. Will that be okay?

[tejohnson]

lld/ELF/Driver.cpp

1015

There are two cases where we may need to write out an empty file:

1. object between --start-lib and --end-lib is not selected by the link
2. The case handled by https://reviews.llvm.org/D42514, when the object is skipped by ThinLTO because it doesn't have a summary, which I think is due to side effects of module splitting for types?

It's also not so much that we need to generate object files, this step is not generating object files in a distributed build, but rather that we are expected to generate the necessary index files. It's this indexing step (the thin link) of the distributed build system that wants to validate that all the expected output files have been created.

Perhaps the comment can then be changed to something along the lines of "With the ThinLTOIndexOnly option, only the thin link is performed, and will generate index files for the ThinLTO backends in a distributed build system. The distributed build system may expect that index files are created for all input bitcode objects provided to the linker for the thin link. However, index files will not normally be created for input bitcode objects that either aren't selected by the linker (i.e. in a static library and not needed), or because they don't have a summary. Therefore we need to create empty dummy index file outputs in those cases".

[pcc]

lld/ELF/Driver.cpp

1018

It depends what you mean by "every file". We can't just do it for files added using BitcodeCompiler::add because that wouldn't include non-selected files and then you're back to the original problem. And we can't do it for everything in LinkerDriver::Files because that would include regular object files like crtbegin.o. I think that we can do it by filtering LinkerDriver::Files along the lines that I mentioned.

[rdhindsa]

For the LazyObjFiles which are not added for link, this change ensures that empty index files are generated.

In the next patch, I will update it to include imports file as well, along with the EmitImports option.

[pcc]

lld/ELF/InputFiles.h

262

Instead of adding this function, can you give Seen a clearer name and make it public?

263

I don't think you need this function. You should be able to use isBitcode to test whether MB->File is bitcode directly in the caller.

lld/ELF/LTO.cpp

85

This function is always called with SkipModule = true. There's also a case in the gold plugin where SkipModule is false (which just creates an empty file); that's the case where the file was added to the link but we end up not emitting an index for it. There's no easy way to detect this case using the current API, so probably the easiest thing to do would be to create the empty files in BitcodeCompiler::add and let them be overwritten later.

175

I don't think you need the if.

195

This part doesn't seem right. We don't want to skip input files without symbols because they could contain section contributions that could be significant.

lld/ELF/SymbolTable.cpp

132

Consider moving this part to BitcodeCompiler::compile().

135

Remove unnecessary parens.

[grimar]

lld/ELF/Driver.cpp

806

Please provide the testcase for this error.

lld/ELF/LTO.cpp

87

You do not need llvm:: prefix I think.

93

Error messages start from the lower case usually.
Also, the test would be nice to have if possible (i think you could create the file and so this code should error out when try to overwrite it, but I am not sure).

99

The same.

[grimar]

lld/ELF/LTO.cpp

99

I mean you can remove llvm:: I think.

[rdhindsa]

lld/ELF/InputFiles.h

263

isBitcode is a static function in InputFiles.cpp. In order to be able to use that function in LTO.cpp :

1. Either I move the function to InputFiles.h, in which case it will get imported to all other files including InputFiles.h ,which will increase code size.

In my opinion, we can simply call it for LazyObjFile. Please let me know if you still have concerns.

[pcc]

lld/ELF/InputFiles.h

263

You can just declare it as inline bool isBitcode(...) { ... }. That way it should be fine to put it in a header file.

[rdhindsa]

lld/ELF/Driver.cpp

806

Added the testcase to thinlto_prefix_replace.ll

lld/ELF/LTO.cpp

93

Added the test case to thinlto.ll
By changing the permissions of the file, it is not available to write and test produces error.

[pcc]

lld/ELF/InputFiles.h

47

Remove this line and qualify with llvm:: below.

262

Rename to AddedToLink to match local style.

355

Do you need this line? Both identify_magic and file_magic are in the llvm namespace.

lld/ELF/LTO.cpp

259–260

Remove braces

262

Remove this->

lld/test/ELF/lto/thinlto_prefix_replace.ll

10

I would also test that the index file is still created if the file is wrapped in --start-lib..--end-lib, or if -module-summary is not passed to opt.

[pcc]

LGTM

lld/ELF/LTO.cpp

259

Remove this one as well.

lld/ELF/Options.td

428–431

These lines aren't needed, we only support the new flags as plugin-opts.

[sbc100]

I think this broke the BUILD_SHARED_LIB build: https://reviews.llvm.org/D46381

[tejohnson]

Couple of post-commit suggestions for improvements below.

lld/ELF/LTO.cpp

199

Sorry for not noticing this earlier - I was looking at the code again in the context of D46400. This seems inefficient - we have large links of tens of thousands of files, and writing empty index (and with D46400, empty imports) for each will slow down the thin link. In gold-plugin, only the necessary empty files are written. The createLTO function takes an optional callback function, which can be used to track which files get index files written for them. Take a look at how gold-plugin uses this to optionally call it's version of writeEmptyDistributedBuildOutputs (specifically, see how ObjectToIndexFileState is used). Can you do a follow-up change to implement the same support here so it is only called when needed?

302

Can this be moved up above the cache pruning and the iteration over all the MaxTasks to create output files and save temps? Those should never be needed in the index-only case, and it would be more efficient to skip the MaxTasks Buff[I].empty() checks.

[rdhindsa]

I will send out a patch to address both concerns. Thank you for reviewing.

lld/ELF/Config.h

struct Configuration {
   llvm::StringRef LTOSampleProfile;
   llvm::StringRef MapFile;
   llvm::StringRef OutputFile;
   llvm::StringRef OptRemarksFilename;
   llvm::StringRef ProgName;
   llvm::StringRef SoName;
   llvm::StringRef Sysroot;
   llvm::StringRef ThinLTOCacheDir;
+  llvm::StringRef ThinLTOIndexOnlyObjectsFile;
+  llvm::StringRef ThinLTOPrefixReplace;
   std::string Rpath;
   std::vector<VersionDefinition> VersionDefinitions;
   std::vector<llvm::StringRef> AuxiliaryList;
   std::vector<llvm::StringRef> FilterList;
   std::vector<llvm::StringRef> SearchPaths;
   std::vector<llvm::StringRef> SymbolOrderingFile;
   std::vector<llvm::StringRef> Undefined;
   std::vector<SymbolVersion> DynamicList;
struct Configuration {
   bool Relocatable;
   bool SaveTemps;
   bool SingleRoRx;
   bool Shared;
   bool Static = false;
   bool SysvHash = false;
   bool Target1Rel;
   bool Trace;
+  bool ThinLTOIndexOnly;
   bool UndefinedVersion;
   bool WarnBackrefs;
   bool WarnCommon;
   bool WarnMissingEntry;
   bool WarnSymbolOrdering;
   bool WriteAddends;
   bool ZCombreloc;
   bool ZCopyreloc;

lld/ELF/Driver.cpp

void LinkerDriver::readConfigs(opt::InputArgList &Args) {
   Config->ZRodynamic = hasZOption(Args, "rodynamic");
   Config->ZStackSize = args::getZOptionValue(Args, OPT_z, "stack-size", 0);
   Config->ZText = getZFlag(Args, "text", "notext", true);
   Config->ZWxneeded = hasZOption(Args, "wxneeded");
 
   // Parse LTO plugin-related options for compatibility with gold.
   for (auto *Arg : Args.filtered(OPT_plugin_opt)) {
     StringRef S = Arg->getValue();
-    if (S == "disable-verify")
+    if (S == "disable-verify") {
       Config->DisableVerify = true;
-    else if (S == "save-temps")
+    } else if (S == "save-temps") {
       Config->SaveTemps = true;
-    else if (S.startswith("O"))
+    } else if (S.startswith("O")) {
       Config->LTOO = parseInt(S.substr(1), Arg);
-    else if (S.startswith("lto-partitions="))
+    } else if (S.startswith("lto-partitions=")) {
       Config->LTOPartitions = parseInt(S.substr(15), Arg);
-    else if (S.startswith("jobs="))
+    } else if (S.startswith("jobs=")) {
       Config->ThinLTOJobs = parseInt(S.substr(5), Arg);
-    else if (S.startswith("mcpu="))
+    } else if (S.startswith("mcpu=")) {
       parseClangOption(Saver.save("-" + S), Arg->getSpelling());
-    else if (S == "new-pass-manager")
+    } else if (S == "new-pass-manager") {
       Config->LTONewPassManager = true;
-    else if (S == "debug-pass-manager")
+    } else if (S == "debug-pass-manager") {
       Config->LTODebugPassManager = true;
-    else if (S.startswith("sample-profile="))
-      Config->LTOSampleProfile = S.substr(strlen("sample-profile="));
-    else if (!S.startswith("/") && !S.startswith("-fresolution=") &&
-             !S.startswith("-pass-through=") && !S.startswith("thinlto"))
+    } else if (S.startswith("sample-profile=")) {
+      Config->LTOSampleProfile = S.substr(15);
+    } else if (S == "thinlto-index-only") {
+      Config->ThinLTOIndexOnly = true;
+    } else if (S.startswith("thinlto-index-only=")) {
+      Config->ThinLTOIndexOnly = true;
+      Config->ThinLTOIndexOnlyObjectsFile = S.substr(19);
+    } else if (S.startswith("thinlto-prefix-replace=")) {
+      Config->ThinLTOPrefixReplace = S.substr(23);
+      if (!Config->ThinLTOPrefixReplace.contains(';'))
+        error("thinlto-prefix-replace expects 'old;new' format");
+    } else if (!S.startswith("/") && !S.startswith("-fresolution=") &&
+               !S.startswith("-pass-through=") && !S.startswith("thinlto")) {
       parseClangOption(S, Arg->getSpelling());
     }
+  }
 
   // Parse -mllvm options.
   for (auto *Arg : Args.filtered(OPT_mllvm))
     parseClangOption(Arg->getValue(), Arg->getSpelling());
 
   if (Config->LTOO > 3)
     error("invalid optimization level for LTO: " + Twine(Config->LTOO));
   if (Config->LTOPartitions == 0)
for (auto *Arg : Args) {
     case OPT_end_group:
       if (!InputFile::IsInGroup)
         error("stray --end-group");
       InputFile::IsInGroup = false;
       ++InputFile::NextGroupId;
       break;
     case OPT_start_lib:
       if (InLib)
         error("nested --start-lib");
       if (InputFile::IsInGroup)
         error("may not nest --start-lib in --start-group");
       InLib = true;
       InputFile::IsInGroup = true;
       break;
     case OPT_end_lib:
       if (!InLib)
         error("stray --end-lib");
       InLib = false;
       InputFile::IsInGroup = false;
       ++InputFile::NextGroupId;

lld/ELF/InputFiles.h

 class InputSectionBase;
 }
 
 // Returns "<internal>", "foo.a(bar.o)" or "baz.o".
 std::string toString(const elf::InputFile *F);
 
 namespace elf {
 
 using llvm::object::Archive;
 
 class Symbol;
 
 // If -reproduce option is given, all input files are written
 // to this tar archive.
 extern llvm::TarWriter *Tar;
 
 // Opens a given file.
LazyObjFile(MemoryBufferRef M, StringRef ArchiveName,
     this->ArchiveName = ArchiveName;
   }
 
   static bool classof(const InputFile *F) { return F->kind() == LazyObjKind; }
 
   template <class ELFT> void parse();
   MemoryBufferRef getBuffer();
   InputFile *fetch();
+  bool AddedToLink = false;
 
 private:
   template <class ELFT> void addElfSymbols();
 
-  bool Seen = false;
   uint64_t OffsetInArchive;
 };
 
 // An ArchiveFile object represents a .a file.
 class ArchiveFile : public InputFile {
 public:
   explicit ArchiveFile(std::unique_ptr<Archive> &&File);
   static bool classof(const InputFile *F) { return F->kind() == ArchiveKind; }
public:
   static bool classof(const InputFile *F) { return F->kind() == BinaryKind; }
   void parse();
 };
 
 InputFile *createObjectFile(MemoryBufferRef MB, StringRef ArchiveName = "",
                             uint64_t OffsetInArchive = 0);
 InputFile *createSharedFile(MemoryBufferRef MB, StringRef DefaultSoName);
 
+inline bool isBitcode(MemoryBufferRef MB) {
+  return identify_magic(MB.getBuffer()) == llvm::file_magic::bitcode;
+}
+
 extern std::vector<BinaryFile *> BinaryFiles;
 extern std::vector<BitcodeFile *> BitcodeFiles;
+extern std::vector<LazyObjFile *> LazyObjFiles;
 extern std::vector<InputFile *> ObjectFiles;
 extern std::vector<InputFile *> SharedFiles;
 
 } // namespace elf
 } // namespace lld
 
 #endif

lld/ELF/InputFiles.cpp

 
 using namespace lld;
 using namespace lld::elf;
 
 bool InputFile::IsInGroup;
 uint32_t InputFile::NextGroupId;
 std::vector<BinaryFile *> elf::BinaryFiles;
 std::vector<BitcodeFile *> elf::BitcodeFiles;
+std::vector<LazyObjFile *> elf::LazyObjFiles;
 std::vector<InputFile *> elf::ObjectFiles;
 std::vector<InputFile *> elf::SharedFiles;
 
 TarWriter *elf::Tar;
 
 InputFile::InputFile(Kind K, MemoryBufferRef M)
     : MB(M), GroupId(NextGroupId), FileKind(K) {
   // All files within the same --{start,end}-group get the same group ID.
BitcodeFile::BitcodeFile(MemoryBufferRef MB, StringRef ArchiveName,
   // Here we pass a new MemoryBufferRef which is identified by ArchiveName
   // (the fully resolved path of the archive) + member name + offset of the
   // member in the archive.
   // ThinLTO uses the MemoryBufferRef identifier to access its internal
   // data structures and if two archives define two members with the same name,
   // this causes a collision which result in only one of the objects being
   // taken into consideration at LTO time (which very likely causes undefined
   // symbols later in the link stage).
-  MemoryBufferRef MBRef(MB.getBuffer(),
-                        Saver.save(ArchiveName + MB.getBufferIdentifier() +
-                                   utostr(OffsetInArchive)));
+  MemoryBufferRef MBRef(
+      MB.getBuffer(),
+                 (ArchiveName.empty() ? "" : utostr(OffsetInArchive))));
   Obj = CHECK(lto::InputFile::create(MBRef), this);
 
   Triple T(Obj->getTargetTriple());
   EKind = getBitcodeELFKind(T);
   EMachine = getBitcodeMachineKind(MB.getBufferIdentifier(), T);
 }
 
 static uint8_t mapVisibility(GlobalValue::VisibilityTypes GvVisibility) {
void BinaryFile::parse() {
   Symtab->addRegular(Saver.save(S + "_start"), STV_DEFAULT, STT_OBJECT, 0, 0,
                      STB_GLOBAL, Section, nullptr);
   Symtab->addRegular(Saver.save(S + "_end"), STV_DEFAULT, STT_OBJECT,
                      Data.size(), 0, STB_GLOBAL, Section, nullptr);
   Symtab->addRegular(Saver.save(S + "_size"), STV_DEFAULT, STT_OBJECT,
                      Data.size(), 0, STB_GLOBAL, nullptr, nullptr);
 }
 
-static bool isBitcode(MemoryBufferRef MB) {
-  using namespace sys::fs;
-  return identify_magic(MB.getBuffer()) == file_magic::bitcode;
-}
-
 InputFile *elf::createObjectFile(MemoryBufferRef MB, StringRef ArchiveName,
                                  uint64_t OffsetInArchive) {
   if (isBitcode(MB))
     return make<BitcodeFile>(MB, ArchiveName, OffsetInArchive);
 
   switch (getELFKind(MB)) {
   case ELF32LEKind:
     return make<ObjFile<ELF32LE>>(MB, ArchiveName);
InputFile *elf::createSharedFile(MemoryBufferRef MB, StringRef DefaultSoName) {
   case ELF64BEKind:
     return make<SharedFile<ELF64BE>>(MB, DefaultSoName);
   default:
     llvm_unreachable("getELFKind");
   }
 }
 
 MemoryBufferRef LazyObjFile::getBuffer() {
-  if (Seen)
+  if (AddedToLink)
     return MemoryBufferRef();
-  Seen = true;
+  AddedToLink = true;
   return MB;
 }
 
 InputFile *LazyObjFile::fetch() {
   MemoryBufferRef MBRef = getBuffer();
   if (MBRef.getBuffer().empty())
     return nullptr;
 

lld/ELF/LTO.h

 //===----------------------------------------------------------------------===//
 
 #ifndef LLD_ELF_LTO_H
 #define LLD_ELF_LTO_H
 
 #include "lld/Common/LLVM.h"
 #include "llvm/ADT/DenseSet.h"
 #include "llvm/ADT/SmallString.h"
+#include "llvm/Support/raw_ostream.h"
 #include <memory>
 #include <vector>
 
 namespace llvm {
 namespace lto {
 class LTO;
 }
 } // namespace llvm
 
 namespace lld {
 namespace elf {
 
 class BitcodeFile;
 class InputFile;
+class LazyObjFile;
 
 class BitcodeCompiler {
 public:
   BitcodeCompiler();
   ~BitcodeCompiler();
 
   void add(BitcodeFile &F);
   std::vector<InputFile *> compile();
+  void addLazyObjFile(LazyObjFile *File);
 
 private:
   std::unique_ptr<llvm::lto::LTO> LTOObj;
   std::vector<SmallString<0>> Buff;
   std::vector<std::unique_ptr<MemoryBuffer>> Files;
   llvm::DenseSet<StringRef> UsedStartStop;
+  std::unique_ptr<llvm::raw_fd_ostream> LinkedObjects;
 };
 } // namespace elf
 } // namespace lld
 
 #endif

lld/ELF/LTO.cpp

 #include "Symbols.h"
 #include "lld/Common/ErrorHandler.h"
 #include "lld/Common/TargetOptionsCommandFlags.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/Twine.h"
 #include "llvm/BinaryFormat/ELF.h"
+#include "llvm/Bitcode/BitcodeReader.h"
+#include "llvm/Bitcode/BitcodeWriter.h"
 #include "llvm/IR/DiagnosticPrinter.h"
 #include "llvm/LTO/Caching.h"
 #include "llvm/LTO/Config.h"
 #include "llvm/LTO/LTO.h"
 #include "llvm/Object/SymbolicFile.h"
 #include "llvm/Support/CodeGen.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/MemoryBuffer.h"
-#include "llvm/Support/raw_ostream.h"
 #include <algorithm>
 #include <cstddef>
 #include <memory>
 #include <string>
 #include <system_error>
 #include <vector>
 
 using namespace llvm;
static void diagnosticHandler(const DiagnosticInfo &DI) {
   warn(ErrStorage);
 }
 
 static void checkError(Error E) {
   handleAllErrors(std::move(E),
                   [&](ErrorInfoBase &EIB) { error(EIB.message()); });
 }
 
-static std::unique_ptr<lto::LTO> createLTO() {
+// With the ThinLTOIndexOnly option, only the thin link is performed, and will
+// generate index files for the ThinLTO backends in a distributed build system.
+// The distributed build system may expect that index files are created for all
+// input bitcode objects provided to the linker for the thin link. However,
+// index files will not normally be created for input bitcode objects that
+// either aren't selected by the linker (i.e. in a static library and not
+// needed), or because they don't have a summary. Therefore we need to create
+// empty dummy index file outputs in those cases.
+// If SkipModule is true then .thinlto.bc should contain just
+// SkipModuleByDistributedBackend flag which requests distributed backend
+// to skip the compilation of the corresponding module and produce an empty
+// object file.
+static void writeEmptyDistributedBuildOutputs(const std::string &ModulePath,
+                                              const std::string &OldPrefix,
+                                              const std::string &NewPrefix,
+                                              bool SkipModule) {
+  std::string NewModulePath =
+      lto::getThinLTOOutputFile(ModulePath, OldPrefix, NewPrefix);
+  std::error_code EC;
+
+  raw_fd_ostream OS(NewModulePath + ".thinlto.bc", EC,
+                    sys::fs::OpenFlags::F_None);
+  if (EC)
+    error("failed to write " + NewModulePath + ".thinlto.bc" + ": " +
+          EC.message());
+
+  if (SkipModule) {
+    ModuleSummaryIndex Index(false);
+    Index.setSkipModuleByDistributedBackend();
+    WriteIndexToFile(Index, OS);
+  }
+}
+
+// Creates and returns output stream with a list of object files for final
+// linking of distributed ThinLTO.
+static std::unique_ptr<raw_fd_ostream> createLinkedObjectsFile() {
+  if (Config->ThinLTOIndexOnlyObjectsFile.empty())
+    return nullptr;
+  std::error_code EC;
+  auto LinkedObjectsFile = llvm::make_unique<raw_fd_ostream>(
+      Config->ThinLTOIndexOnlyObjectsFile, EC, sys::fs::OpenFlags::F_None);
+  if (EC)
+    error("cannot create " + Config->ThinLTOIndexOnlyObjectsFile + ": " +
+          EC.message());
+  return LinkedObjectsFile;
+}
+
+// Creates instance of LTO.
+// LinkedObjectsFile is an output stream to write the list of object files for
+// the final ThinLTO linking. Can be nullptr.
+static std::unique_ptr<lto::LTO> createLTO(raw_fd_ostream *LinkedObjectsFile) {
   lto::Config Conf;
 
   // LLD supports the new relocations.
   Conf.Options = InitTargetOptionsFromCodeGenFlags();
   Conf.Options.RelaxELFRelocations = true;
 
   // Always emit a section per function/datum with LTO.
   Conf.Options.FunctionSections = true;
   Conf.Options.DataSections = true;
 
   if (Config->Relocatable)
     Conf.RelocModel = None;
   else if (Config->Pic)
     Conf.RelocModel = Reloc::PIC_;
   else
     Conf.RelocModel = Reloc::Static;
   Conf.CodeModel = GetCodeModelFromCMModel();
   Conf.DisableVerify = Config->DisableVerify;
   Conf.DiagHandler = diagnosticHandler;
   Conf.OptLevel = Config->LTOO;
   Conf.CPU = GetCPUStr();
 
   // Set up a custom pipeline if we've been asked to.
   Conf.OptPipeline = Config->LTONewPmPasses;
   Conf.AAPipeline = Config->LTOAAPipeline;
 
   // Set up optimization remarks if we've been asked to.
   Conf.RemarksFilename = Config->OptRemarksFilename;
   Conf.RemarksWithHotness = Config->OptRemarksWithHotness;
 
   if (Config->SaveTemps)
     checkError(Conf.addSaveTemps(std::string(Config->OutputFile) + ".",
                                  /*UseInputModulePath*/ true));
 
   lto::ThinBackend Backend;
   if (Config->ThinLTOJobs != -1u)
     Backend = lto::createInProcessThinBackend(Config->ThinLTOJobs);
 
+  if (Config->ThinLTOIndexOnly) {
+    std::string OldPrefix, NewPrefix;
+    std::tie(OldPrefix, NewPrefix) = Config->ThinLTOPrefixReplace.split(';');
+    Backend = lto::createWriteIndexesThinBackend(OldPrefix, NewPrefix, true,
+                                                 LinkedObjectsFile, nullptr);
+  }
+
   Conf.SampleProfile = Config->LTOSampleProfile;
   Conf.UseNewPM = Config->LTONewPassManager;
   Conf.DebugPassManager = Config->LTODebugPassManager;
 
   return llvm::make_unique<lto::LTO>(std::move(Conf), Backend,
                                      Config->LTOPartitions);
 }
 
-BitcodeCompiler::BitcodeCompiler() : LTOObj(createLTO()) {
+BitcodeCompiler::BitcodeCompiler() {
+  LinkedObjects = createLinkedObjectsFile();
+  LTOObj = createLTO(LinkedObjects.get());
   for (Symbol *Sym : Symtab->getSymbols()) {
     StringRef Name = Sym->getName();
     for (StringRef Prefix : {"__start_", "__stop_"})
       if (Name.startswith(Prefix))
         UsedStartStop.insert(Name.substr(Prefix.size()));
   }
 }
 
 BitcodeCompiler::~BitcodeCompiler() = default;
 
 static void undefine(Symbol *S) {
   replaceSymbol<Undefined>(S, nullptr, S->getName(), STB_GLOBAL, STV_DEFAULT,
                            S->Type);
 }
 
 void BitcodeCompiler::add(BitcodeFile &F) {
   lto::InputFile &Obj = *F.Obj;
+
+  std::string OldPrefix, NewPrefix;
+  std::tie(OldPrefix, NewPrefix) = Config->ThinLTOPrefixReplace.split(';');
+
+  // Create the empty files which, if indexed, will be overwritten later.
+  writeEmptyDistributedBuildOutputs(Obj.getName(), OldPrefix, NewPrefix, false);
+
   unsigned SymNum = 0;
   std::vector<Symbol *> Syms = F.getSymbols();
   std::vector<lto::SymbolResolution> Resols(Syms.size());
 
   bool IsExecutable = !Config->Shared && !Config->Relocatable;
 
   // Provide a resolution to the LTO API for each symbol.
   for (const lto::InputFile::Symbol &ObjSym : Obj.symbols()) {

 // Merge all the bitcode files we have seen, codegen the result
 // and return the resulting ObjectFile(s).
 std::vector<InputFile *> BitcodeCompiler::compile() {
   std::vector<InputFile *> Ret;
   unsigned MaxTasks = LTOObj->getMaxTasks();
   Buff.resize(MaxTasks);
   Files.resize(MaxTasks);
 
+  // If LazyObjFile has not been added to link, emit empty index files
+  if (Config->ThinLTOIndexOnly) {
+    for (LazyObjFile *F : LazyObjFiles)
+      if (!F->AddedToLink && isBitcode(F->MB))
+        addLazyObjFile(F);
+  }
+
   // The --thinlto-cache-dir option specifies the path to a directory in which
   // to cache native object files for ThinLTO incremental builds. If a path was
   // specified, configure LTO to use it as the cache directory.
   lto::NativeObjectCache Cache;
   if (!Config->ThinLTOCacheDir.empty())
     Cache = check(
         lto::localCache(Config->ThinLTOCacheDir,
                         [&](size_t Task, std::unique_ptr<MemoryBuffer> MB) {
                           Files[Task] = std::move(MB);
                         }));
 
   checkError(LTOObj->run(
       [&](size_t Task) {
         return llvm::make_unique<lto::NativeObjectStream>(
if (Config->SaveTemps) {
         saveBuffer(Buff[I], Config->OutputFile + ".lto.o");
       else
         saveBuffer(Buff[I], Config->OutputFile + Twine(I) + ".lto.o");
     }
     InputFile *Obj = createObjectFile(MemoryBufferRef(Buff[I], "lto.tmp"));
     Ret.push_back(Obj);
   }
 
+  // ThinLTO with index only option is required to generate only the index
+  // files. After that, we exit from linker and ThinLTO backend runs in a
+  // distributed environment.
+  if (Config->ThinLTOIndexOnly)
+    exit(0);
+
   for (std::unique_ptr<MemoryBuffer> &File : Files)
     if (File)
       Ret.push_back(createObjectFile(*File));
 
   return Ret;
 }
+
+// For lazy object files not added to link, adds empty index files
+void BitcodeCompiler::addLazyObjFile(LazyObjFile *File) {
+  StringRef Identifier = File->getBuffer().getBufferIdentifier();
+  std::string OldPrefix, NewPrefix;
+  std::tie(OldPrefix, NewPrefix) = Config->ThinLTOPrefixReplace.split(';');
+  writeEmptyDistributedBuildOutputs(Identifier, OldPrefix, NewPrefix,
+                                    /* SkipModule */ true);
+}

lld/ELF/Options.td

 defm plugin_opt: Eq<"plugin-opt">,
   HelpText<"specifies LTO options for compatibility with GNU linkers">;
 def save_temps: F<"save-temps">;
 def thinlto_cache_dir: J<"thinlto-cache-dir=">,
   HelpText<"Path to ThinLTO cached object file directory">;
 defm thinlto_cache_policy: Eq<"thinlto-cache-policy">,
   HelpText<"Pruning policy for the ThinLTO cache">;
 def thinlto_jobs: J<"thinlto-jobs=">, HelpText<"Number of ThinLTO jobs">;
+def thinlto_index_only: J<"thinlto-index-only=">,
+  HelpText<"Write individual backend index files for LTO ">;
+def thinlto_prefix_replace : F<"thinlto-prefix-replace">,
+  HelpText<"Controls location of distributed backend files">;
 
 // Ignore LTO plugin-related options.
 // clang -flto passes -plugin and -plugin-opt to the linker. This is required
 // for ld.gold and ld.bfd to get LTO working. But it's not for lld which doesn't
 // rely on a plugin. Instead of detecting which linker is used on clang side we
 // just ignore the option on lld side as it's easier. In fact, the linker could
 // be called 'ld' and understanding which linker is used would require parsing of
 // --version output.

lld/ELF/SymbolTable.cpp

template <class ELFT> void SymbolTable::addFile(InputFile *File) {
   // .a file
   if (auto *F = dyn_cast<ArchiveFile>(File)) {
     F->parse<ELFT>();
     return;
   }
 
   // Lazy object file
   if (auto *F = dyn_cast<LazyObjFile>(File)) {
+    LazyObjFiles.push_back(F);
     F->parse<ELFT>();
     return;
   }
 
   if (Config->Trace)
     message(toString(File));
 
   // .so file
template <class ELFT> void SymbolTable::addCombinedLTOObject() {
   if (BitcodeFiles.empty())
     return;
 
   // Compile bitcode files and replace bitcode symbols.
   LTO.reset(new BitcodeCompiler);
   for (BitcodeFile *F : BitcodeFiles)
     LTO->add(*F);
 
   for (InputFile *File : LTO->compile()) {
     DenseSet<CachedHashStringRef> DummyGroups;
     auto *Obj = cast<ObjFile<ELFT>>(File);
     Obj->parse(DummyGroups);
     for (Symbol *Sym : Obj->getGlobalSymbols())
       Sym->parseSymbolVersion();
     ObjectFiles.push_back(File);
   }
 }
 
 Defined *SymbolTable::addAbsolute(StringRef Name, uint8_t Visibility,
                                   uint8_t Binding) {

lld/test/ELF/lto/Inputs/thinlto_empty.ll

+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"

lld/test/ELF/lto/thinlto.ll

 ; REQUIRES: x86
+
+; First ensure that the ThinLTO handling in lld handles
+; bitcode without summary sections gracefully and generates index file.
+; RUN: llvm-as %s -o %t.o
+; RUN: llvm-as %p/Inputs/thinlto.ll -o %t2.o
+; RUN: ld.lld -m elf_x86_64 --plugin-opt=thinlto-index-only -shared %t.o %t2.o -o %t3
+; RUN: ls %t2.o.thinlto.bc
+; RUN: not test -e %t3
+; RUN: ld.lld -m elf_x86_64 -shared %t.o %t2.o -o %t4
+; RUN: llvm-nm %t4 | FileCheck %s --check-prefix=NM
+
 ; Basic ThinLTO tests.
 ; RUN: opt -module-summary %s -o %t.o
 ; RUN: opt -module-summary %p/Inputs/thinlto.ll -o %t2.o
+; RUN: opt -module-summary %p/Inputs/thinlto_empty.ll -o %t4.o
+
+; Ensure lld generates an index and not a binary if requested.
+; RUN: ld.lld -m elf_x86_64 --plugin-opt=thinlto-index-only -shared %t.o %t2.o -o %t3
+; RUN: llvm-bcanalyzer -dump %t.o.thinlto.bc | FileCheck %s --check-prefix=BACKEND1
+; RUN: llvm-bcanalyzer -dump %t2.o.thinlto.bc | FileCheck %s --check-prefix=BACKEND2
+; RUN: not test -e %t3
+
+; Ensure lld generates an index even if the file is wrapped in --start-lib/--end-lib
+; RUN: rm -f %t2.o.thinlto.bc
+; RUN: ld.lld -m elf_x86_64 --plugin-opt=thinlto-index-only -shared %t.o %t4.o --start-lib %t2.o --end-lib -o %t5
+; RUN: ls %t2.o.thinlto.bc
+
+; Ensure lld generates error if unable to write to index file
+; RUN: rm -f %t4.o.thinlto.bc
+; RUN: touch %t4.o.thinlto.bc
+; RUN: chmod 400 %t4.o.thinlto.bc
+; RUN: ld.lld -m elf_x86_64 --plugin-opt=thinlto-index-only -shared %t.o %t4.o -o %t5 2>&1 | FileCheck %s --check-prefix=ERR
+; ERR: failed to write {{.*}}4.o.thinlto.bc: Permission denied
 
 ; First force single-threaded mode
 ; RUN: rm -f %t.lto.o %t1.lto.o
 ; RUN: ld.lld -save-temps --thinlto-jobs=1 -shared %t.o %t2.o -o %t
 ; RUN: llvm-nm %t1.lto.o | FileCheck %s --check-prefix=NM1
 ; RUN: llvm-nm %t2.lto.o | FileCheck %s --check-prefix=NM2
 
 ; Next force multi-threaded mode
 ; RUN: rm -f %t2.lto.o %t21.lto.o
 ; RUN: ld.lld -save-temps --thinlto-jobs=2 -shared %t.o %t2.o -o %t2
 ; RUN: llvm-nm %t21.lto.o | FileCheck %s --check-prefix=NM1
 ; RUN: llvm-nm %t22.lto.o | FileCheck %s --check-prefix=NM2
 
-; NM1: T f
-; NM1-NOT: U g
-
-; NM2: T g
-
 ; Then check without --thinlto-jobs (which currently default to hardware_concurrency)
 ; We just check that we don't crash or fail (as it's not sure which tests are
 ; stable on the final output file itself.
 ; RUN: ld.lld -shared %t.o %t2.o -o %t2
 
+; NM: T f
+; NM1: T f
+; NM1-NOT: U g
+; NM2: T g
+
+; The backend index for this module contains summaries from itself and
+; Inputs/thinlto.ll, as it imports from the latter.
+; BACKEND1: <MODULE_STRTAB_BLOCK
+; BACKEND1-NEXT: <ENTRY {{.*}} record string = '{{.*}}/thinlto.ll.tmp{{.*}}.o'
+; BACKEND1-NEXT: <ENTRY {{.*}} record string = '{{.*}}/thinlto.ll.tmp{{.*}}.o'
+; BACKEND1-NEXT: </MODULE_STRTAB_BLOCK
+; BACKEND1: <GLOBALVAL_SUMMARY_BLOCK
+; BACKEND1: <VERSION
+; BACKEND1: <FLAGS
+; BACKEND1: <VALUE_GUID op0={{1|2}} op1={{-3706093650706652785|-5300342847281564238}}
+; BACKEND1: <VALUE_GUID op0={{1|2}} op1={{-3706093650706652785|-5300342847281564238}}
+; BACKEND1: <COMBINED
+; BACKEND1: <COMBINED
+; BACKEND1: </GLOBALVAL_SUMMARY_BLOCK
+
+; The backend index for Input/thinlto.ll contains summaries from itself only,
+; as it does not import anything.
+; BACKEND2: <MODULE_STRTAB_BLOCK
+; BACKEND2-NEXT: <ENTRY {{.*}} record string = '{{.*}}/thinlto.ll.tmp2.o'
+; BACKEND2-NEXT: </MODULE_STRTAB_BLOCK
+; BACKEND2-NEXT: <GLOBALVAL_SUMMARY_BLOCK
+; BACKEND2-NEXT: <VERSION
+; BACKEND2-NEXT: <FLAGS
+; BACKEND2-NEXT: <VALUE_GUID op0=1 op1=-5300342847281564238
+; BACKEND2-NEXT: <COMBINED
+; BACKEND2-NEXT: </GLOBALVAL_SUMMARY_BLOCK
+
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 declare void @g(...)
 
 define void @f() {
 entry:
   call void (...) @g()
   ret void
 }

lld/test/ELF/lto/thinlto_prefix_replace.ll

+; REQUIRES: x86
+; Check that changing the output path via thinlto-prefix-replace works
+; RUN: mkdir -p %t/oldpath
+; RUN: opt -module-summary %s -o %t/oldpath/thinlto_prefix_replace.o
+
+; Ensure that there is no existing file at the new path, so we properly
+; test the creation of the new file there.
+; RUN: rm -f %t/newpath/thinlto_prefix_replace.o.thinlto.bc
+; RUN: ld.lld --plugin-opt=thinlto-index-only --plugin-opt=thinlto-prefix-replace="%t/oldpath/;%t/newpath/" -shared %t/oldpath/thinlto_prefix_replace.o -o %t/thinlto_prefix_replace
+; RUN: ls %t/newpath/thinlto_prefix_replace.o.thinlto.bc
+
+; Ensure that lld generates error if prefix replace option does not have 'old;new' format
+; RUN: rm -f %t/newpath/thinlto_prefix_replace.o.thinlto.bc
+; RUN: not ld.lld --plugin-opt=thinlto-index-only --plugin-opt=thinlto-prefix-replace="%t/oldpath/:%t/newpath/" -shared %t/oldpath/thinlto_prefix_replace.o -o %t/thinlto_prefix_replace 2>&1 | FileCheck %s --check-prefix=ERR
+; ERR: thinlto-prefix-replace expects 'old;new' format
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+define void @f() {
+entry:
+  ret void
+}