[LTO] Switch to the new resolution-based API.

[davide]

Summary

This should be all the logic. Before reviewing the style/code appeareance, I'd like to ensure that this is the right direction, thanks.

Left to be done:

• There are two tests failing, the one which include inline asm (currently investigating).
• The old LTO code wasn't removed yet. I found that it makes the patch less readable, and that can be removed separately.

[ruiu]

I want to see the old code being removed in this patch to see deltas.

[davide]

In D24492#540718, @ruiu wrote:

I want to see the old code being removed in this patch to see deltas.

Fair enough, I'll do that tomorrow.

[davide]

Back to more interesting business, the two asm inline tests that are failing are also failing with the gold plugin, so that might inidcate a bug in lib/LTO. Here's a synthetic testcase.

$ /usr/bin/ld.gold -shared -m elf_x86_64 -o /home/davide/work/llvm/build/test/tools/gold/X86/Output/module_asm.ll.tmp2 -plugin /home/davide/work/llvm/build/./lib/LLVMgold.so /home/davide/work/llvm/build/test/tools/gold/X86/Output/module_asm.ll.tmp.o
ld.gold: ../lib/Linker/IRMover.cpp:1240: llvm::Error (anonymous namespace)::IRLinker::run(): Assertion `!GV->isDeclaration()' failed.

; RUN: llvm-as %s -o %t.o
; RUN: %gold -shared -m elf_x86_64 -o %t2 -plugin %llvmshlibdir/LLVMgold.so %t.o

target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-unknown-linux-gnu"

module asm ".weak patatino"
module asm ".equ patatino, foo"

declare void @patatino()

define void @foo() {
  ret void
}

define void @_start() {
  call void @patatino()
  ret void
}

[davide]

I opened a report for the gold-plugin bug (https://llvm.org/bugs/show_bug.cgi?id=30363). I'll take a closer look tomorrow.

[ruiu]

Davide,

Sure. Go ahead with your patch first.

[davide]

Removed dead code as per Rui's request.
Also, for whoever cares, here's the output of diffstat:

33 files changed, 209 insertions(+), 432 deletions(-)

[ruiu]

This patch contains complex code with almost no comment. Please explain in the code about what you are doing.

ELF/SymbolTable.cpp

116–120

Why don't you keep the old interface and replace only the content of LTO.cpp? I think that was a good separation.

[pcc]

ELF/SymbolTable.cpp

213

You should replace all uses of IRObjectFile elsewhere with lto::InputFile. Then you won't need to create a new InputFile here, you can just reuse the one in the BitcodeFile.

225

You shouldn't need to do this, see r281437.

234

This isn't right, this should test whether the SymbolBody is a DefinedBitcode and is defined by this BitcodeFile.

252

There's no need to save objects to files, you can just use raw_string_ostream and operate on the object file directly in memory.

[davide]

Addressed comments. I'm working to remove the call to getComdatSymbolTable(). Other than that it should be ready for another look.

[• rafael]

lld/ELF/InputFiles.cpp

629 ↗

(On Diff #72753)

You can use GlobalValue::VisibilityTypes to make the argument type clear.

642 ↗

(On Diff #72753)

static

644 ↗

(On Diff #72753)

You don't need the llvm:: prefixes.
Do you really want a StringSaver by value?

655 ↗

(On Diff #72753)

Simplify to just

uint8_t Type = ObjSym.isTLS() ? STT_TLS : STT_NOTYPE;

656 ↗

(On Diff #72753)

Simplify to

uint8_t Visibility = mapVisibility(ObjSym.getVisibility());

661 ↗

(On Diff #72753)

You have lost HasUnnamedAddr. Where do you compensate for that?

[davide]

Rafael's comments.

lld/ELF/InputFiles.cpp

661 ↗

(On Diff #72753)

I think it's not needed anymore becuase lib/LTO will compute it based on GlobalResolution.
I'm garbage collecting it for now.

[ruiu]

lld/ELF/Error.h

36 ↗

(On Diff #72851)

You are not using Msg.

lld/ELF/LTO.cpp

150 ↗

(On Diff #72851)

Filenames is not a large vector, so please use std::string for simplicity.

155–156 ↗

(On Diff #72851)

I'd make getOutputFilename a pure function that returns std::string so that we can do

Filenames[Task] = getOutputFilename(Filename, MaxTasks > 1 ? Task : -1);

[davide]

Rui's comments.

[pcc]

lld/ELF/LTO.cpp

150 ↗

(On Diff #72851)

Once you make the change below, Filenames will be unused and you can remove it.

166 ↗

(On Diff #72851)

This could be if (!Buff[I].empty()), then you can make this a for-range loop.

[pcc]

lld/ELF/LTO.cpp

151 ↗

(On Diff #72860)

Rename to AddStream.

[• rafael]

lld/ELF/InputFiles.cpp

672 ↗

(On Diff #72851)

You don't need the llvm prefix.

lld/ELF/LTO.cpp

67 ↗

(On Diff #72851)

Just inline the variable into the single use.

88 ↗

(On Diff #72851)

You don't need the ()

122 ↗

(On Diff #72851)

Add a comment on why you need the undefined check.

[ruiu]

lld/ELF/Error.h

38 ↗

(On Diff #72860)

Do you need c_str()?

[pcc]

lld/ELF/LTO.cpp

79 ↗

(On Diff #72860)

You could just assign without the if statement here and below.

[davide]

lld/ELF/LTO.cpp

154 ↗

(On Diff #72867)

hmm, I still need I to be passed to saveBuffer, no? Do you want me to use a separate variable incremented at each iteration?

[pcc]

lld/ELF/LTO.cpp

154 ↗

(On Diff #72867)

No, sorry, what you have is fine.

155 ↗

(On Diff #72867)

I would remove this comment as it doesn't help readers understand the code.

[davide]

Rebased on top of

commit de07c6a10ee2cbf5877e74fdbbe1d8d3f1d343a4
Author: Etienne Bergeron <[e-mail]>
Date:   Wed Sep 28 18:04:07 2016 +0000

+ removed comment as suggested by Peter.

[ruiu]

lld/ELF/InputFiles.cpp

653 ↗

(On Diff #72870)

Remove ().

lld/ELF/LTO.cpp

85–88 ↗

(On Diff #72870)

Remove llvm::.

I don't think your code is prepared to see LtoObj being null. Maybe you want to return a new instance regardless of HasError value?

101–102 ↗

(On Diff #72870)

Why do you want to return here immediately if there was an error before this function?

[davide]

lld/ELF/LTO.cpp

101–102 ↗

(On Diff #72870)

Ooops, fixed, thanks.

[ruiu]

lld/ELF/LTO.cpp

107 ↗

(On Diff #72883)

auto -> BasicSymbolRef &

126 ↗

(On Diff #72883)

Stray }?

148 ↗

(On Diff #72883)

if (Buff[I].empty())
  continue;

[davide]

lld/ELF/LTO.cpp

131–132 ↗

(On Diff #72886)

hmm, no, just clang-formatted incorrectly. Weird as I have a pre-hook for that that didn't kick in.

[ruiu]

lld/ELF/LTO.cpp

131–132 ↗

(On Diff #72886)

It seems that it is still formatted weirdly.

[davide]

run clang-format manually and it seems fine on my side.

[pcc]

LGTM with nit. Thanks for working on this!

lld/ELF/InputFiles.cpp

646 ↗

(On Diff #72890)

Unused argument

[ruiu]

LGTM

ELF/CMakeLists.txt

   ICF.cpp
   InputFiles.cpp
   InputSection.cpp
-  LTO.cpp
   LinkerScript.cpp
   MarkLive.cpp
   Mips.cpp

ELF/Driver.h

 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/StringSet.h"
+#include "llvm/IR/LLVMContext.h"
 #include "llvm/Option/ArgList.h"
 #include "llvm/Support/raw_ostream.h"
 

ELF/Error.h

 
 #include "lld/Core/LLVM.h"
 
+#include "llvm/Support/Error.h"
+
 namespace lld {
 namespace elf {
 

 LLVM_ATTRIBUTE_NORETURN void fatal(const Twine &Msg);
 LLVM_ATTRIBUTE_NORETURN void fatal(const Twine &Msg, const Twine &Prefix);
 
+inline void check(Error E, std::string Msg = "") {
+  handleAllErrors(std::move(E), [&](llvm::ErrorInfoBase &EIB) {
+    error(EIB.message().c_str());
+    return Error::success();
+  });
+}
+
 template <class T> T check(ErrorOr<T> E) {
   if (auto EC = E.getError())
     fatal(EC.message());

ELF/InputFiles.h

   ArrayRef<Symbol *> getSymbols() { return Symbols; }
   static bool shouldSkip(uint32_t Flags);
   std::unique_ptr<llvm::object::IRObjectFile> Obj;
+  MemoryBufferRef getBuffer() { return MB; }
 
 private:
   std::vector<Symbol *> Symbols;

ELF/LTO.h

-//===- LTO.h ----------------------------------------------------*- C++ -*-===//
-//
-//                             The LLVM Linker
-//
-// This file is distributed under the University of Illinois Open Source
-// License. See LICENSE.TXT for details.
-//
-//===----------------------------------------------------------------------===//
-//
-// This file provides a way to combine bitcode files into one ELF
-// file by compiling them using LLVM.
-//
-// If LTO is in use, your input files are not in regular ELF files
-// but instead LLVM bitcode files. In that case, the linker has to
-// convert bitcode files into the native format so that we can create
-// an ELF file that contains native code. This file provides that
-// functionality.
-//
-//===----------------------------------------------------------------------===//
-
-#ifndef LLD_ELF_LTO_H
-#define LLD_ELF_LTO_H
-
-#include "lld/Core/LLVM.h"
-#include "llvm/ADT/SmallString.h"
-#include "llvm/ADT/StringSet.h"
-#include "llvm/IR/Module.h"
-#include "llvm/Linker/IRMover.h"
-
-namespace lld {
-namespace elf {
-
-class BitcodeFile;
-class InputFile;
-
-class BitcodeCompiler {
-public:
-  BitcodeCompiler();
-  void add(BitcodeFile &F);
-  std::vector<std::unique_ptr<InputFile>> compile();
-
-private:
-  std::vector<std::unique_ptr<InputFile>> runSplitCodegen(
-      const std::function<std::unique_ptr<llvm::TargetMachine>()> &TMFactory);
-
-  std::unique_ptr<llvm::Module> Combined;
-  std::vector<SmallString<0>> OwningData;
-  llvm::StringSet<> InternalizedSyms;
-  llvm::StringSet<> AsmUndefinedRefs;
-};
-}
-}
-
-#endif

ELF/LTO.cpp

-//===- LTO.cpp ------------------------------------------------------------===//
-//
-//                             The LLVM Linker
-//
-// This file is distributed under the University of Illinois Open Source
-// License. See LICENSE.TXT for details.
-//
-//===----------------------------------------------------------------------===//
-
-#include "LTO.h"
-#include "Config.h"
-#include "Driver.h"
-#include "Error.h"
-#include "InputFiles.h"
-#include "Symbols.h"
-#include "llvm/Analysis/AliasAnalysis.h"
-#include "llvm/Analysis/CGSCCPassManager.h"
-#include "llvm/Analysis/LoopPassManager.h"
-#include "llvm/Analysis/TargetLibraryInfo.h"
-#include "llvm/Analysis/TargetTransformInfo.h"
-#include "llvm/Bitcode/ReaderWriter.h"
-#include "llvm/CodeGen/CommandFlags.h"
-#include "llvm/CodeGen/ParallelCG.h"
-#include "llvm/IR/AutoUpgrade.h"
-#include "llvm/IR/LegacyPassManager.h"
-#include "llvm/IR/PassManager.h"
-#include "llvm/IR/Verifier.h"
-#include "llvm/LTO/legacy/UpdateCompilerUsed.h"
-#include "llvm/Linker/IRMover.h"
-#include "llvm/Passes/PassBuilder.h"
-#include "llvm/Support/StringSaver.h"
-#include "llvm/Support/TargetRegistry.h"
-#include "llvm/Target/TargetMachine.h"
-#include "llvm/Transforms/IPO.h"
-#include "llvm/Transforms/IPO/PassManagerBuilder.h"
-#include "llvm/Transforms/Utils/ModuleUtils.h"
-
-using namespace llvm;
-using namespace llvm::object;
-using namespace llvm::ELF;
-
-using namespace lld;
-using namespace lld::elf;
-
-// This is for use when debugging LTO.
-static void saveBuffer(StringRef Buffer, const Twine &Path) {
-  std::error_code EC;
-  raw_fd_ostream OS(Path.str(), EC, sys::fs::OpenFlags::F_None);
-  if (EC)
-    error(EC, "cannot create " + Path);
-  OS << Buffer;
-}
-
-// This is for use when debugging LTO.
-static void saveBCFile(Module &M, const Twine &Path) {
-  std::error_code EC;
-  raw_fd_ostream OS(Path.str(), EC, sys::fs::OpenFlags::F_None);
-  if (EC)
-    error(EC, "cannot create " + Path);
-  WriteBitcodeToFile(&M, OS, /* ShouldPreserveUseListOrder */ true);
-}
-
-static void runNewCustomLtoPasses(Module &M, TargetMachine &TM) {
-  PassBuilder PB(&TM);
-
-  AAManager AA;
-
-  // Parse a custom AA pipeline if asked to.
-  if (!PB.parseAAPipeline(AA, Config->LtoAAPipeline)) {
-    error("Unable to parse AA pipeline description: " + Config->LtoAAPipeline);
-    return;
-  }
-
-  LoopAnalysisManager LAM;
-  FunctionAnalysisManager FAM;
-  CGSCCAnalysisManager CGAM;
-  ModuleAnalysisManager MAM;
-
-  // Register the AA manager first so that our version is the one used.
-  FAM.registerPass([&] { return std::move(AA); });
-
-  // Register all the basic analyses with the managers.
-  PB.registerModuleAnalyses(MAM);
-  PB.registerCGSCCAnalyses(CGAM);
-  PB.registerFunctionAnalyses(FAM);
-  PB.registerLoopAnalyses(LAM);
-  PB.crossRegisterProxies(LAM, FAM, CGAM, MAM);
-
-  ModulePassManager MPM;
-  if (!Config->DisableVerify)
-    MPM.addPass(VerifierPass());
-
-  // Now, add all the passes we've been requested to.
-  if (!PB.parsePassPipeline(MPM, Config->LtoNewPmPasses)) {
-    error("unable to parse pass pipeline description: " +
-          Config->LtoNewPmPasses);
-    return;
-  }
-
-  if (!Config->DisableVerify)
-    MPM.addPass(VerifierPass());
-  MPM.run(M, MAM);
-}
-
-static void runOldLtoPasses(Module &M, TargetMachine &TM) {
-  // Note that the gold plugin has a similar piece of code, so
-  // it is probably better to move this code to a common place.
-  legacy::PassManager LtoPasses;
-  LtoPasses.add(createTargetTransformInfoWrapperPass(TM.getTargetIRAnalysis()));
-  PassManagerBuilder PMB;
-  PMB.LibraryInfo = new TargetLibraryInfoImpl(Triple(TM.getTargetTriple()));
-  PMB.Inliner = createFunctionInliningPass();
-  PMB.VerifyInput = PMB.VerifyOutput = !Config->DisableVerify;
-  PMB.LoopVectorize = true;
-  PMB.SLPVectorize = true;
-  PMB.OptLevel = Config->LtoO;
-  PMB.populateLTOPassManager(LtoPasses);
-  LtoPasses.run(M);
-}
-
-static void runLTOPasses(Module &M, TargetMachine &TM) {
-  if (!Config->LtoNewPmPasses.empty()) {
-    // The user explicitly asked for a set of passes to be run.
-    // This needs the new PM to work as there's no clean way to
-    // pass a set of passes to run in the legacy PM.
-    runNewCustomLtoPasses(M, TM);
-    if (HasError)
-      return;
-  } else {
-    // Run the 'default' set of LTO passes. This code still uses
-    // the legacy PM as the new one is not the default.
-    runOldLtoPasses(M, TM);
-  }
-
-  if (Config->SaveTemps)
-    saveBCFile(M, Config->OutputFile + ".lto.opt.bc");
-}
-
-static bool shouldInternalize(const SmallPtrSet<GlobalValue *, 8> &Used,
-                              Symbol *S, GlobalValue *GV) {
-  if (S->IsUsedInRegularObj || Used.count(GV))
-    return false;
-  return !S->includeInDynsym();
-}
-
-BitcodeCompiler::BitcodeCompiler()
-    : Combined(new Module("ld-temp.o", Driver->Context)) {}
-
-static void undefine(Symbol *S) {
-  replaceBody<Undefined>(S, S->body()->getName(), STV_DEFAULT, S->body()->Type,
-                         nullptr);
-}
-
-static void handleUndefinedAsmRefs(const BasicSymbolRef &Sym, GlobalValue *GV,
-                                   StringSet<> &AsmUndefinedRefs) {
-  // GV associated => not an assembly symbol, bail out.
-  if (GV)
-    return;
-
-  // This is an undefined reference to a symbol in asm. We put that in
-  // compiler.used, so that we can preserve it from being dropped from
-  // the output, without necessarily preventing its internalization.
-  SmallString<64> Name;
-  raw_svector_ostream OS(Name);
-  Sym.printName(OS);
-  AsmUndefinedRefs.insert(Name.str());
-}
-
-void BitcodeCompiler::add(BitcodeFile &F) {
-  std::unique_ptr<IRObjectFile> Obj = std::move(F.Obj);
-  std::vector<GlobalValue *> Keep;
-  unsigned BodyIndex = 0;
-  ArrayRef<Symbol *> Syms = F.getSymbols();
-
-  Module &M = Obj->getModule();
-  if (M.getDataLayoutStr().empty())
-    fatal("invalid bitcode file: " + F.getName() + " has no datalayout");
-
-  // Discard non-compatible debug infos if necessary.
-  M.materializeMetadata();
-  UpgradeDebugInfo(M);
-
-  // If a symbol appears in @llvm.used, the linker is required
-  // to treat the symbol as there is a reference to the symbol
-  // that it cannot see. Therefore, we can't internalize.
-  SmallPtrSet<GlobalValue *, 8> Used;
-  collectUsedGlobalVariables(M, Used, /* CompilerUsed */ false);
-
-  for (const BasicSymbolRef &Sym : Obj->symbols()) {
-    uint32_t Flags = Sym.getFlags();
-    GlobalValue *GV = Obj->getSymbolGV(Sym.getRawDataRefImpl());
-    if (GV && GV->hasAppendingLinkage())
-      Keep.push_back(GV);
-    if (BitcodeFile::shouldSkip(Flags))
-      continue;
-    Symbol *S = Syms[BodyIndex++];
-    if (GV)
-      GV->setUnnamedAddr(S->HasUnnamedAddr ? GlobalValue::UnnamedAddr::Global
-                                           : GlobalValue::UnnamedAddr::None);
-    if (Flags & BasicSymbolRef::SF_Undefined) {
-      handleUndefinedAsmRefs(Sym, GV, AsmUndefinedRefs);
-      continue;
-    }
-    SymbolBody *B = S->body();
-    if (B->File != &F)
-      continue;
-
-    // We collect the set of symbols we want to internalize here
-    // and change the linkage after the IRMover executed, i.e. after
-    // we imported the symbols and satisfied undefined references
-    // to it. We can't just change linkage here because otherwise
-    // the IRMover will just rename the symbol.
-    if (GV && shouldInternalize(Used, S, GV))
-      InternalizedSyms.insert(GV->getName());
-
-    // At this point we know that either the combined LTO object will provide a
-    // definition of a symbol, or we will internalize it. In either case, we
-    // need to undefine the symbol. In the former case, the real definition
-    // needs to be able to replace the original definition without conflicting.
-    // In the latter case, we need to allow the combined LTO object to provide a
-    // definition with the same name, for example when doing parallel codegen.
-    if (auto *C = dyn_cast<DefinedCommon>(B)) {
-      if (auto *GO = dyn_cast<GlobalObject>(GV))
-        GO->setAlignment(C->Alignment);
-    } else {
-      undefine(S);
-    }
-
-    if (!GV)
-      // Module asm symbol.
-      continue;
-
-    switch (GV->getLinkage()) {
-    default:
-      break;
-    case GlobalValue::LinkOnceAnyLinkage:
-      GV->setLinkage(GlobalValue::WeakAnyLinkage);
-      break;
-    case GlobalValue::LinkOnceODRLinkage:
-      GV->setLinkage(GlobalValue::WeakODRLinkage);
-      break;
-    }
-
-    Keep.push_back(GV);
-  }
-
-  IRMover Mover(*Combined);
-  if (Error E = Mover.move(Obj->takeModule(), Keep,
-                           [](GlobalValue &, IRMover::ValueAdder) {})) {
-    handleAllErrors(std::move(E), [&](const ErrorInfoBase &EIB) {
-      fatal("failed to link module " + F.getName() + ": " + EIB.message());
-    });
-  }
-}
-
-static void internalize(GlobalValue &GV) {
-  assert(!GV.hasLocalLinkage() &&
-         "Trying to internalize a symbol with local linkage!");
-  GV.setLinkage(GlobalValue::InternalLinkage);
-}
-
-std::vector<std::unique_ptr<InputFile>> BitcodeCompiler::runSplitCodegen(
-    const std::function<std::unique_ptr<TargetMachine>()> &TMFactory) {
-  unsigned NumThreads = Config->LtoJobs;
-  OwningData.resize(NumThreads);
-
-  std::list<raw_svector_ostream> OSs;
-  std::vector<raw_pwrite_stream *> OSPtrs;
-  for (SmallString<0> &Obj : OwningData) {
-    OSs.emplace_back(Obj);
-    OSPtrs.push_back(&OSs.back());
-  }
-
-  splitCodeGen(std::move(Combined), OSPtrs, {}, TMFactory);
-
-  std::vector<std::unique_ptr<InputFile>> ObjFiles;
-  for (SmallString<0> &Obj : OwningData)
-    ObjFiles.push_back(createObjectFile(
-        MemoryBufferRef(Obj, "LLD-INTERNAL-combined-lto-object")));
-
-  // If -save-temps is given, we need to save temporary objects to files.
-  // This is for debugging.
-  if (Config->SaveTemps) {
-    if (NumThreads == 1) {
-      saveBuffer(OwningData[0], Config->OutputFile + ".lto.o");
-    } else {
-      for (unsigned I = 0; I < NumThreads; ++I)
-        saveBuffer(OwningData[I], Config->OutputFile + Twine(I) + ".lto.o");
-    }
-  }
-
-  return ObjFiles;
-}
-
-// Merge all the bitcode files we have seen, codegen the result
-// and return the resulting ObjectFile.
-std::vector<std::unique_ptr<InputFile>> BitcodeCompiler::compile() {
-  for (const auto &Name : InternalizedSyms) {
-    GlobalValue *GV = Combined->getNamedValue(Name.first());
-    assert(GV);
-    internalize(*GV);
-  }
-
-  std::string TheTriple = Combined->getTargetTriple();
-  std::string Msg;
-  const Target *T = TargetRegistry::lookupTarget(TheTriple, Msg);
-  if (!T)
-    fatal("target not found: " + Msg);
-
-  // LLD supports the new relocations.
-  TargetOptions Options = InitTargetOptionsFromCodeGenFlags();
-  Options.RelaxELFRelocations = true;
-
-  auto CreateTargetMachine = [&]() {
-    return std::unique_ptr<TargetMachine>(T->createTargetMachine(
-        TheTriple, "", "", Options, Config->Pic ? Reloc::PIC_ : Reloc::Static));
-  };
-
-  std::unique_ptr<TargetMachine> TM = CreateTargetMachine();
-
-  // Update llvm.compiler.used so that optimizations won't strip
-  // off AsmUndefinedReferences.
-  updateCompilerUsed(*Combined, *TM, AsmUndefinedRefs);
-
-  if (Config->SaveTemps)
-    saveBCFile(*Combined, Config->OutputFile + ".lto.bc");
-
-  runLTOPasses(*Combined, *TM);
-  if (HasError)
-    return {};
-
-  return runSplitCodegen(CreateTargetMachine);
-}

ELF/SymbolTable.h

 #define LLD_ELF_SYMBOL_TABLE_H
 
 #include "InputFiles.h"
-#include "LTO.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/Support/Regex.h"
 

   // Set of .so files to not link the same shared object file more than once.
   llvm::DenseSet<StringRef> SoNames;
 
-  std::unique_ptr<BitcodeCompiler> Lto;
+  // Set of compiler LTO bitcode files.
+  std::vector<std::unique_ptr<llvm::MemoryBuffer>> CompiledBitcodeFiles;
 };
 
 template <class ELFT> struct Symtab { static SymbolTable<ELFT> *X; };

ELF/SymbolTable.cpp

 
 #include "SymbolTable.h"
 #include "Config.h"
+#include "Driver.h"
 #include "Error.h"
 #include "LinkerScript.h"
 #include "SymbolListFile.h"
 #include "Symbols.h"
 #include "llvm/Bitcode/ReaderWriter.h"
+#include "llvm/CodeGen/CommandFlags.h"
+#include "llvm/IR/DiagnosticPrinter.h"
+#include "llvm/LTO/LTO.h"
+#include "llvm/LTO/LTOBackend.h"
 #include "llvm/Support/StringSaver.h"
 
 using namespace llvm;

   F->parse(ComdatGroups);
 }
 
+static void diagnosticHandler(const DiagnosticInfo &DI) {
+  if (const auto *BDI = dyn_cast<BitcodeDiagnosticInfo>(&DI)) {
+    std::error_code EC = BDI->getError();
+    if (EC == BitcodeError::InvalidBitcodeSignature)
+      return;
+  }
+
+  std::string ErrStorage;
+  {
+    raw_string_ostream OS(ErrStorage);
+    DiagnosticPrinterRawOStream DP(OS);
+    DI.print(DP);
+  }
+  warning(ErrStorage.c_str());
+}
+
+static std::unique_ptr<lto::LTO> createLTO() {
+  lto::Config Conf;
+  lto::ThinBackend Backend;
+  unsigned ParallelCodeGenParallelismLevel = Config->LtoJobs;
+
+  // LLD supports the new relocations.
+  Conf.Options = InitTargetOptionsFromCodeGenFlags();
+  Conf.Options.RelaxELFRelocations = true;
+
+  Conf.RelocModel = Config->Pic ? Reloc::PIC_ : Reloc::Static;
+  Conf.DisableVerify = Config->DisableVerify;
+  Conf.DiagHandler = diagnosticHandler;
+  Conf.OptLevel = Config->LtoO;
+
+  // Set up a custom pipeline if we've been asked to.
+  if (!Config->LtoNewPmPasses.empty())
+    Conf.OptPipeline = Config->LtoNewPmPasses;
+
+  if (Config->SaveTemps)
+    check(Conf.addSaveTemps(std::string(Config->OutputFile) + ".",
+                            /*UseInputModulePath*/ true));
+
+  return (HasError)
+             ? nullptr
+             : llvm::make_unique<lto::LTO>(std::move(Conf), Backend,
+                                           ParallelCodeGenParallelismLevel);
+}
+
+namespace {
+// Define the LTOOutput handling
+class LTOOutput : public lto::NativeObjectOutput {
+  StringRef Path;
+
+public:
+  LTOOutput(StringRef Path) : Path(Path) {}
+  // Open the filename \p Path and allocate a stream.
+  std::unique_ptr<raw_pwrite_stream> getStream() override {
+    int FD;
+    std::error_code EC = sys::fs::openFileForWrite(Path, FD, sys::fs::F_None);
+    if (EC)
+      llvm_unreachable("can't open the file");
+    return llvm::make_unique<llvm::raw_fd_ostream>(FD, true);
+  }
+};
+}
+
+/// Return the desired output filename given a base input name, a flag
+/// indicating whether a temp file should be generated, and an optional task id.
+/// The new filename generated is returned in \p NewFilename.
+static void getOutputFileName(SmallString<128> InFilename,
+                              SmallString<128> &NewFilename, int TaskID = -1) {
+  NewFilename = InFilename;
+  if (TaskID >= 0)
+    NewFilename += utostr(TaskID);
+}
+
+static void undefine(Symbol *S) {
+  replaceBody<Undefined>(S, S->body()->getName(), STV_DEFAULT, S->body()->Type,
+                         nullptr);
+}
+
+// This is for use when debugging LTO.
+static void saveBuffer(StringRef Buffer, const Twine &Path) {
+  std::error_code EC;
+  raw_fd_ostream OS(Path.str(), EC, sys::fs::OpenFlags::F_None);
+  if (EC)
+    error(EC, "cannot create " + Path);
+  OS << Buffer;
+}
+
 // This function is where all the optimizations of link-time
 // optimization happens. When LTO is in use, some input files are
 // not in native object file format but in the LLVM bitcode format.

   if (BitcodeFiles.empty())
     return;
 
-  // Compile bitcode files.
-  Lto.reset(new BitcodeCompiler);
-  for (const std::unique_ptr<BitcodeFile> &F : BitcodeFiles)
-    Lto->add(*F);
-  std::vector<std::unique_ptr<InputFile>> IFs = Lto->compile();
+  std::unique_ptr<lto::LTO> LtoObj = createLTO();
+  if (HasError)
+    return;
+
+  for (const std::unique_ptr<BitcodeFile> &F : BitcodeFiles) {
+    Expected<std::unique_ptr<lto::InputFile>> ObjOrErr =
+        lto::InputFile::create(F->getBuffer());
+    if (!ObjOrErr)
+      error(toString(ObjOrErr.takeError()).c_str());
+    if (HasError)
+      return;
+    lto::InputFile &Obj = **ObjOrErr;
+    if (Obj.getDataLayoutStr().empty())
+      fatal("invalid bitcode file: " + F->getName() + " has no datalayout");
+
+    unsigned SymNum = 0;
+    std::vector<Symbol *> Syms = F->getSymbols();
+    std::vector<lto::SymbolResolution> Resols(Syms.size());
+    for (LLVM_ATTRIBUTE_UNUSED auto &ObjSym : Obj.symbols()) {
+      Symbol *Sym = Syms[SymNum];
+      lto::SymbolResolution &R = Resols[SymNum];
+      ++SymNum;
+      SymbolBody *B = Sym->body();
+      if (B->File != F.get())
+        continue;
+      if (Sym->body()->isUndefined())
+        continue;
+      R.Prevailing = Sym->body()->isDefined();
+      R.VisibleToRegularObj = Sym->IsUsedInRegularObj || Sym->includeInDynsym();
+      if (Sym->body()->isDefined())
+        undefine(Sym);
+    }
+    check(LtoObj->add(std::move(*ObjOrErr), Resols));
+    if (HasError)
+      return;
+  }
+
+  SmallString<128> Filename;
+  Filename = Config->OutputFile;
+  unsigned MaxTasks = LtoObj->getMaxTasks();
+  std::vector<SmallString<128>> Filenames(MaxTasks);
+
+  auto AddOutput =
+      [&](size_t Task) -> std::unique_ptr<lto::NativeObjectOutput> {
+    auto &OutputName = Filenames[Task];
+    getOutputFileName(Filename, OutputName, MaxTasks > 1 ? Task : -1);
+    return llvm::make_unique<LTOOutput>(OutputName);
+  };
 
-  // Replace bitcode symbols.
-  for (auto &IF : IFs) {
-    ObjectFile<ELFT> *Obj = cast<ObjectFile<ELFT>>(IF.release());
+  check(LtoObj->run(AddOutput));
+  if (HasError)
+    return;
 
-    DenseSet<StringRef> DummyGroups;
-    Obj->parse(DummyGroups);
-    ObjectFiles.emplace_back(Obj);
+  for (unsigned I = 0; I != MaxTasks; ++I) {
+    if (!Filenames[I].empty()) {
+      auto MBOrErr = MemoryBuffer::getFile(Filenames[I]);
+      if (auto EC = MBOrErr.getError()) {
+        error(EC, "cannot open " + Filenames[I]);
+        return;
+      }
+      MemoryBufferRef MB = (*MBOrErr)->getMemBufferRef();
+
+      // Do we want to move this to the LTO API?
+      if (Config->SaveTemps) {
+        if (MaxTasks == 1)
+          saveBuffer(MB.getBuffer(), Config->OutputFile + ".lto.o");
+        else
+          saveBuffer(MB.getBuffer(), Config->OutputFile + Twine(I) + ".lto.o");
+      }
+
+      CompiledBitcodeFiles.push_back(std::move(*MBOrErr));
+      ObjectFile<ELFT> *Obj =
+          cast<ObjectFile<ELFT>>(createObjectFile(MB).release());
+      DenseSet<StringRef> DummyGroups;
+      Obj->parse(DummyGroups);
+      ObjectFiles.emplace_back(Obj);
+    }
   }
 }
 

test/ELF/lto/archive-3.ll

 ; RUN: llvm-as %s -o %t2.o
 
 ; RUN: ld.lld -m elf_x86_64 %t1.o %t2.o  -o %t3 -save-temps
-; RUN: llvm-dis %t3.lto.bc -o - | FileCheck %s
+; RUN: llvm-dis %t3.0.2.internalize.bc -o - | FileCheck %s
 
 ; RUN: rm -f %t.a
 ; RUN: llvm-ar rcs %t.a %t1.o
 ; RUN: ld.lld -m elf_x86_64 %t.a %t1.o %t2.o  -o %t3 -save-temps
-; RUN: llvm-dis %t3.lto.bc -o - | FileCheck %s
+; RUN: llvm-dis %t3.0.2.internalize.bc -o - | FileCheck %s
 
 ; CHECK: define internal void @foo() {
 

test/ELF/lto/available-externally.ll

 ; RUN: llvm-as %s -o %t1.o
 ; RUN: llvm-as %p/Inputs/available-externally.ll -o %t2.o
 ; RUN: ld.lld %t1.o %t2.o -m elf_x86_64 -o %t.so -shared -save-temps
-; RUN: llvm-dis < %t.so.lto.bc | FileCheck %s
+; RUN: llvm-dis < %t.so.0.2.internalize.bc | FileCheck %s
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"

test/ELF/lto/common2.ll

 ; RUN: llvm-as %s -o %t1.o
 ; RUN: ld.lld -m elf_x86_64 %t1.o -o %t -shared -save-temps
-; RUN: llvm-dis < %t.lto.bc | FileCheck %s
+; RUN: llvm-dis < %t.0.2.internalize.bc | FileCheck %s
 ; RUN: llvm-readobj -t %t | FileCheck %s --check-prefix=SHARED
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"

test/ELF/lto/common3.ll

 ; RUN: llvm-as %s -o %t1.o
 ; RUN: llvm-as %S/Inputs/common3.ll -o %t2.o
 ; RUN: ld.lld -m elf_x86_64 %t1.o %t2.o -o %t -shared -save-temps
-; RUN: llvm-dis < %t.lto.bc | FileCheck %s
+; RUN: llvm-dis < %t.0.2.internalize.bc | FileCheck %s
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"

test/ELF/lto/discard-value-names.ll

 ; RUN: llvm-as %s -o %t.o
 
 ; RUN: ld.lld -m elf_x86_64 -shared -save-temps %t.o -o %t2.o
-; RUN: llvm-dis < %t2.o.lto.bc | FileCheck %s
+; RUN: llvm-dis < %t2.o.0.0.preopt.bc | FileCheck %s
 
 ; CHECK: @GlobalValueName
 ; CHECK: @foo(i32 %in)

test/ELF/lto/drop-debug-info.ll

 ;
 ; RUN: ld.lld -m elf_x86_64 -shared %p/Inputs/drop-debug-info.bc \
 ; RUN: -disable-verify 2>&1 | FileCheck %s
-; CHECK: warning: ignoring debug info with an invalid version (1) in {{.*}}drop-debug-info.bc
+; CHECK: ignoring debug info with an invalid version (1) in {{.*}}drop-debug-info.bc
 

test/ELF/lto/drop-linkage.ll

 ; RUN: llc %s -o %t.o -filetype=obj
 ; RUN: llvm-as %p/Inputs/drop-linkage.ll -o %t2.o
 ; RUN: ld.lld %t.o %t2.o -o %t.so -save-temps -shared
-; RUN: llvm-dis %t.so.lto.opt.bc -o - | FileCheck %s
+; RUN: llvm-dis %t.so.0.4.opt.bc -o - | FileCheck %s
 
 define void @foo() {
   ret void

test/ELF/lto/internalize-basic.ll

 ; REQUIRES: x86
 ; RUN: llvm-as %s -o %t.o
 ; RUN: ld.lld -m elf_x86_64 %t.o -o %t2 -save-temps
-; RUN: llvm-dis < %t2.lto.bc | FileCheck %s
+; RUN: llvm-dis < %t2.0.2.internalize.bc | FileCheck %s
 
 target triple = "x86_64-unknown-linux-gnu"
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"

test/ELF/lto/internalize-exportdyn.ll

 ; RUN: llvm-as %s -o %t.o
 ; RUN: llvm-as %p/Inputs/internalize-exportdyn.ll -o %t2.o
 ; RUN: ld.lld -m elf_x86_64 %t.o %t2.o -o %t2 --export-dynamic -save-temps
-; RUN: llvm-dis < %t2.lto.bc | FileCheck %s
+; RUN: llvm-dis < %t2.0.2.internalize.bc | FileCheck %s
 
 target triple = "x86_64-unknown-linux-gnu"
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"

test/ELF/lto/internalize-llvmused.ll

 ; REQUIRES: x86
 ; RUN: llvm-as %s -o %t.o
 ; RUN: ld.lld -m elf_x86_64 %t.o -o %t2 -save-temps
-; RUN: llvm-dis < %t2.lto.bc | FileCheck %s
+; RUN: llvm-dis < %t2.0.2.internalize.bc | FileCheck %s
 
 target triple = "x86_64-unknown-linux-gnu"
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"

test/ELF/lto/internalize-undef.ll

 ; RUN: llvm-as %s -o %t.o
 ; RUN: llvm-as %p/Inputs/internalize-undef.ll -o %t2.o
 ; RUN: ld.lld -m elf_x86_64 %t.o %t2.o -o %t -save-temps
-; RUN: llvm-dis < %t.lto.bc | FileCheck %s
+; RUN: llvm-dis < %t.0.2.internalize.bc | FileCheck %s
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"

test/ELF/lto/internalize-version-script.ll

 ; RUN: llvm-as %s -o %t.o
 ; RUN: echo "{ global: foo; local: *; };" > %t.script
 ; RUN: ld.lld -m elf_x86_64 %t.o -o %t2 -shared --version-script %t.script -save-temps
-; RUN: llvm-dis < %t2.lto.bc | FileCheck %s
+; RUN: llvm-dis < %t2.0.2.internalize.bc | FileCheck %s
 
 target triple = "x86_64-unknown-linux-gnu"
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"

test/ELF/lto/irmover-error.ll

 ; RUN: llvm-as -o %t2.bc %S/Inputs/irmover-error.ll
 ; RUN: not ld.lld -m elf_x86_64 %t1.bc %t2.bc -o %t 2>&1 | FileCheck %s
 
-; CHECK: failed to link module {{.*}}2.bc: linking module flags 'foo': IDs have conflicting values
+; CHECK: linking module flags 'foo': IDs have conflicting values
 
 target triple = "x86_64-unknown-linux-gnu"
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"

test/ELF/lto/linkonce-odr.ll

 ; RUN: llvm-as %p/Inputs/linkonce-odr.ll -o %t1.o
 ; RUN: llc -relocation-model=pic %s -o %t2.o -filetype=obj
 ; RUN: ld.lld %t1.o %t2.o -o %t.so -shared -save-temps
-; RUN: llvm-dis %t.so.lto.opt.bc -o - | FileCheck %s
+; RUN: llvm-dis %t.so.0.4.opt.bc -o - | FileCheck %s
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"

test/ELF/lto/linkonce.ll

 ; RUN: llvm-as %p/Inputs/linkonce.ll -o %t1.o
 ; RUN: llc -relocation-model=pic %s -o %t2.o -filetype=obj
 ; RUN: ld.lld %t1.o %t2.o -o %t.so -shared -save-temps
-; RUN: llvm-dis %t.so.lto.opt.bc -o - | FileCheck %s
+; RUN: llvm-dis %t.so.0.4.opt.bc -o - | FileCheck %s
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"

test/ELF/lto/ltopasses-basic.ll

 ; REQUIRES: x86
-; RUN: rm -f %t.so.lto.bc %t.so.lto.opt.bc %t.so.lto.o
 ; RUN: llvm-as %s -o %t.o
 ; RUN: ld.lld -m elf_x86_64 %t.o -o %t.so -save-temps -mllvm -debug-pass=Arguments -shared 2>&1 | FileCheck %s --check-prefix=MLLVM
-; RUN: llvm-dis %t.so.lto.opt.bc -o - | FileCheck %s
+; RUN: llvm-dis %t.so.0.4.opt.bc -o - | FileCheck %s
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"

test/ELF/lto/ltopasses-custom.ll

 ; RUN: ld.lld -m elf_x86_64 %t.o -o %t.so -save-temps --lto-aa-pipeline=basic-aa \
 ; RUN: --lto-newpm-passes=ipsccp -shared
 ; RUN: ld.lld -m elf_x86_64 %t.o -o %t2.so -save-temps --lto-newpm-passes=loweratomic -shared
-; RUN: llvm-dis %t.so.lto.opt.bc -o - | FileCheck %s
-; RUN: llvm-dis %t2.so.lto.opt.bc -o - | FileCheck %s --check-prefix=ATOMIC
+; RUN: llvm-dis %t.so.0.4.opt.bc -o - | FileCheck %s
+; RUN: llvm-dis %t2.so.0.4.opt.bc -o - | FileCheck %s --check-prefix=ATOMIC
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"

test/ELF/lto/save-temps.ll

 ; RUN: llvm-as %p/Inputs/save-temps.ll -o %t2.o
 ; RUN: ld.lld -shared -m elf_x86_64 %t.o %t2.o -save-temps
 ; RUN: llvm-nm a.out | FileCheck %s
-; RUN: llvm-nm a.out.lto.bc | FileCheck %s
+; RUN: llvm-nm a.out.0.0.preopt.bc | FileCheck %s
 ; RUN: llvm-nm a.out.lto.o | FileCheck %s
-; RUN: llvm-dis a.out.lto.bc
+; RUN: llvm-dis a.out.0.0.preopt.bc
 
 target triple = "x86_64-unknown-linux-gnu"
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"

test/ELF/lto/type-merge.ll

 ; RUN: llvm-as %s -o %t.o
 ; RUN: llvm-as %p/Inputs/type-merge.ll -o %t2.o
 ; RUN: ld.lld -m elf_x86_64 %t.o %t2.o -o %t -shared -save-temps
-; RUN: llvm-dis < %t.lto.bc | FileCheck %s
+; RUN: llvm-dis < %t.0.0.preopt.bc | FileCheck %s
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"

test/ELF/lto/type-merge2.ll

 ; RUN: llvm-as %s -o %t.o
 ; RUN: llvm-as %p/Inputs/type-merge2.ll -o %t2.o
 ; RUN: ld.lld -m elf_x86_64 %t.o %t2.o -o %t.so -shared -save-temps
-; RUN: llvm-dis %t.so.lto.bc -o - | FileCheck %s
+; RUN: llvm-dis %t.so.0.0.preopt.bc -o - | FileCheck %s
 
 target triple = "x86_64-unknown-linux-gnu"
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"

test/ELF/lto/unnamed-addr-comdat.ll

 ; RUN: llvm-as %s -o %t.o
 ; RUN: ld.lld -m elf_x86_64 %t.o %t.o -o %t.so -save-temps -shared
-; RUN: llvm-dis %t.so.lto.bc -o - | FileCheck %s
+; RUN: llvm-dis %t.so.0.2.internalize.bc -o - | FileCheck %s
 
 target triple = "x86_64-unknown-linux-gnu"
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"

test/ELF/lto/unnamed-addr-drop.ll

 ; RUN: llvm-as %s -o %t1.o
 ; RUN: llvm-as %S/Inputs/unnamed-addr-drop.ll -o %t2.o
 ; RUN: ld.lld -m elf_x86_64 %t1.o %t2.o -o %t.so -save-temps -shared
-; RUN: llvm-dis %t.so.lto.bc -o - | FileCheck %s
+; RUN: llvm-dis %t.so.0.2.internalize.bc -o - | FileCheck %s
 
 target triple = "x86_64-unknown-linux-gnu"
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"

test/ELF/lto/unnamed-addr-lib.ll

 ; RUN: llvm-mc %p/Inputs/unnamed-addr-lib.s -o %t2.o -filetype=obj -triple=x86_64-pc-linux
 ; RUN: ld.lld %t2.o -shared -o %t2.so
 ; RUN: ld.lld -m elf_x86_64 %t.o %t2.so -o %t.so -save-temps -shared
-; RUN: llvm-dis %t.so.lto.bc -o - | FileCheck %s
+; RUN: llvm-dis %t.so.0.2.internalize.bc -o - | FileCheck %s
 
 ; This documents a small limitation of lld's internalization logic. We decide
 ; that bar should be in the symbol table because if it is it will preempt the

 ; We could add one extra bit for ODR so that we know that preemption is not
 ; necessary, but that is probably not worth it.
 
-; CHECK: @foo = internal constant i8 42
-; CHECK: @bar = weak_odr constant i8 42
+; CHECK: @foo = internal unnamed_addr constant i8 42
+; CHECK: @bar = weak_odr unnamed_addr constant i8 42
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"

test/ELF/lto/unnamed-addr.ll

 ; RUN: llvm-as %s -o %t.o
 ; RUN: ld.lld -m elf_x86_64 %t.o -o %t.so -save-temps -shared
-; RUN: llvm-dis %t.so.lto.opt.bc -o - | FileCheck %s
+; RUN: llvm-dis %t.so.0.4.opt.bc -o - | FileCheck %s
 
 target triple = "x86_64-unknown-linux-gnu"
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"

test/ELF/lto/version-script.ll

 ; RUN: llvm-as %s -o %t.o
 ; RUN: echo "VERSION_1.0{ global: foo; local: *; }; VERSION_2.0{ global: bar; local: *; };" > %t.script
 ; RUN: ld.lld -m elf_x86_64 %t.o -o %t2 -shared --version-script %t.script -save-temps
-; RUN: llvm-dis < %t2.lto.bc | FileCheck %s
+; RUN: llvm-dis < %t2.0.0.preopt.bc | FileCheck %s
 ; RUN: llvm-readobj -V -dyn-symbols %t2 | FileCheck --check-prefix=DSO %s
 
 target triple = "x86_64-unknown-linux-gnu"