[ThinLTO] Fix ThinLTOCodegenerator to export llvm.used symbols

[steven_wu]

Summary

Reapply r357931 with fixes to ThinLTO testcases and llvm-lto tool.

ThinLTOCodeGenerator currently does not preserve llvm.used symbols and
it can internalize them. In order to pass the necessary information to the
legacy ThinLTOCodeGenerator, the input to the code generator is
rewritten to be based on lto::InputFile.

Now ThinLTO using the legacy LTO API will requires data layout in
Module.

"internalize" thinlto action in llvm-lto is updated to run both
"promote" and "internalize" with the same configuration as
ThinLTOCodeGenerator. The old "promote" + "internalize" option does not
produce the same output as ThinLTOCodeGenerator.

This fixes: PR41236
rdar://problem/49293439

[tejohnson]

What part of the patch caused the need to change the internalize action to do promote+internalize in one go?

[steven_wu]

In D60421#1458858, @tejohnson wrote:

What part of the patch caused the need to change the internalize action to do promote+internalize in one go?

It is to test the fix in the patch (llvm.used) which doesn't reproduce with llvm-lto without change. In the previous patch, I updated promote directly to behave the same as ThinLTOCodegenerator and I thought no tests is broken. It turns out my local configuration is wrong and some tests are labelled unsupported.
It turns out there are lots of weak resolution tests rely on the current configuration of promote function, which is testing the output from a pipeline doesn't match ThinLTOCodeGenerator. Merging promote + internalize and matching that with ThinLTOCodeGenerator is the best solution I have right now.

[steven_wu]

Add one file that was missing from previous patch

[bd1976llvm]

Hi Steven,

I missed that this work was being done - sorry.

Currently the legacy api does not require the IRSymtab to be written to the bitcode files so there is potentially a performance degradation due to this approach and the generated bitcode files will be larger, did you measure that? There is also a hidden performance cost that if the bitcode needs upgrading then the IRSymtab must be regenerated which can be expensive.

An alternative approach is for the legacy api to get the list of used symbols from the bitcode and expose them via the lto_symbol_attributes; the system linker can then add "used" symbols to the preserve list directly. This seems more in-keeping with the original design IMO.

[steven_wu]

In D60421#1461407, @bd1976llvm wrote:

Hi Steven,

I missed that this work was being done - sorry.

Currently the legacy api does not require the IRSymtab to be written to the bitcode files so there is potentially a performance degradation due to this approach and the generated bitcode files will be larger, did you measure that?

The legacy API is not writing out IRSymtab but build that in memory during code generation. The only requirement change is that now data layout is required for thin LTO .

There is also a hidden performance cost that if the bitcode needs upgrading then the IRSymtab must be regenerated which can be expensive.

I did a simple performance test in https://reviews.llvm.org/D60226. Yes, the cost of hitting the upgrade path is quite significant (50% thin link time increase) but not measurable if no upgrade is hit. I don’t have big concerns over this. Let me know if you think it is harmful.

An alternative approach is for the legacy api to get the list of used symbols from the bitcode and expose them via the lto_symbol_attributes; the system linker can then add "used" symbols to the preserve list directly. This seems more in-keeping with the original design IMO.

That was one of the proposals I had but some change needs to made to encode and convey this info. See previous review for more info.

[bd1976llvm]

Sorry for the delay in replying. I was trying to understand if the upgrade path was important. Comments inline:

In D60421#1461442, @steven_wu wrote:

In D60421#1461407, @bd1976llvm wrote:

Hi Steven,

I missed that this work was being done - sorry.

Currently the legacy api does not require the IRSymtab to be written to the bitcode files so there is potentially a performance degradation due to this approach and the generated bitcode files will be larger, did you measure that?

The legacy API is not writing out IRSymtab but build that in memory during code generation. The only requirement change is that now data layout is required for thin LTO .

My understanding is that the on disk bitcode contains a binary representation of the IRSymtab and this is read into memory when loading the file (which is why the upgrade path is more expensive than the no-upgrade path). I am saying that currently the legacy api does not require the IRSymtab to be written to disk. Writing it may cost more time and make the input files bigger.

There is also a hidden performance cost that if the bitcode needs upgrading then the IRSymtab must be regenerated which can be expensive.

I did a simple performance test in https://reviews.llvm.org/D60226. Yes, the cost of hitting the upgrade path is quite significant (50% thin link time increase) but not measurable if no upgrade is hit. I don’t have big concerns over this. Let me know if you think it is harmful.

My concern is with patch releases. These might hit the upgrade path for thirdparty bitcode libraries. This might be inconvenient for developers to work around. I will try to get back to you on Monday after I have gathered more information on this.

An alternative approach is for the legacy api to get the list of used symbols from the bitcode and expose them via the lto_symbol_attributes; the system linker can then add "used" symbols to the preserve list directly. This seems more in-keeping with the original design IMO.

That was one of the proposals I had but some change needs to made to encode and convey this info. See previous review for more info.

I can have a go at implementing this to see what would be involved, if that would help?

[steven_wu]

In D60421#1464989, @bd1976llvm wrote:

Sorry for the delay in replying. I was trying to understand if the upgrade path was important. Comments inline:

In D60421#1461442, @steven_wu wrote:

In D60421#1461407, @bd1976llvm wrote:

Hi Steven,

I missed that this work was being done - sorry.

Currently the legacy api does not require the IRSymtab to be written to the bitcode files so there is potentially a performance degradation due to this approach and the generated bitcode files will be larger, did you measure that?

The legacy API is not writing out IRSymtab but build that in memory during code generation. The only requirement change is that now data layout is required for thin LTO .

My understanding is that the on disk bitcode contains a binary representation of the IRSymtab and this is read into memory when loading the file (which is why the upgrade path is more expensive than the no-upgrade path). I am saying that currently the legacy api does not require the IRSymtab to be written to disk. Writing it may cost more time and make the input files bigger.

My patch didn't change that. IRSymtabs are built during thin-link time, hold in memory and it is not written out the disk. Any additional IO during thinLTO can potentially be a huge overhead so we are definitely introducing that. Am I misunderstanding your question?

There is also a hidden performance cost that if the bitcode needs upgrading then the IRSymtab must be regenerated which can be expensive.

I did a simple performance test in https://reviews.llvm.org/D60226. Yes, the cost of hitting the upgrade path is quite significant (50% thin link time increase) but not measurable if no upgrade is hit. I don’t have big concerns over this. Let me know if you think it is harmful.

My concern is with patch releases. These might hit the upgrade path for thirdparty bitcode libraries. This might be inconvenient for developers to work around. I will try to get back to you on Monday after I have gathered more information on this.

Sure. @dexonsmith, do you have any concerns on this?

An alternative approach is for the legacy api to get the list of used symbols from the bitcode and expose them via the lto_symbol_attributes; the system linker can then add "used" symbols to the preserve list directly. This seems more in-keeping with the original design IMO.

That was one of the proposals I had but some change needs to made to encode and convey this info. See previous review for more info.

I can have a go at implementing this to see what would be involved, if that would help?

Sure. I think my initial proposal was in the bug report: https://bugs.llvm.org/show_bug.cgi?id=41236
You can take a look and see if you agree with our discussion.

[bd1976llvm]

LGTM! Thanks for waiting for me to confirm. Comments inline:

In D60421#1465029, @steven_wu wrote:

In D60421#1464989, @bd1976llvm wrote:

Sorry for the delay in replying. I was trying to understand if the upgrade path was important. Comments inline:

In D60421#1461442, @steven_wu wrote:

In D60421#1461407, @bd1976llvm wrote:

Hi Steven,

I missed that this work was being done - sorry.

Currently the legacy api does not require the IRSymtab to be written to the bitcode files so there is potentially a performance degradation due to this approach and the generated bitcode files will be larger, did you measure that?

The legacy API is not writing out IRSymtab but build that in memory during code generation. The only requirement change is that now data layout is required for thin LTO .

My understanding is that the on disk bitcode contains a binary representation of the IRSymtab and this is read into memory when loading the file (which is why the upgrade path is more expensive than the no-upgrade path). I am saying that currently the legacy api does not require the IRSymtab to be written to disk. Writing it may cost more time and make the input files bigger.

My patch didn't change that. IRSymtabs are built during thin-link time, hold in memory and it is not written out the disk. Any additional IO during thinLTO can potentially be a huge overhead so we are definitely introducing that. Am I misunderstanding your question?

Sorry for the confusion. I see that this patch hasn't changed the behavior. This is because the IRSymtab is always written to the bitcode files. I imagined that, downstream, if you are shipping a toolchain that only uses the legacy api you would have disabled the generation of the IRSymtab, as it is not required. That's why I thought that there would be an impact from requiring the IRSymtab.

There is also a hidden performance cost that if the bitcode needs upgrading then the IRSymtab must be regenerated which can be expensive.

I did a simple performance test in https://reviews.llvm.org/D60226. Yes, the cost of hitting the upgrade path is quite significant (50% thin link time increase) but not measurable if no upgrade is hit. I don’t have big concerns over this. Let me know if you think it is harmful.

My concern is with patch releases. These might hit the upgrade path for thirdparty bitcode libraries. This might be inconvenient for developers to work around. I will try to get back to you on Monday after I have gathered more information on this.

Sure. @dexonsmith, do you have any concerns on this?

As things are now, this patch would mean that customers who receive patch releases would hit the upgrade path. However, we think that by the time we ship a toolchain containing this patch, we should either be able to change our release processes so that this is not an issue (by guaranteeing that patch releases will not need to upgrade the IRSymtab), or we will commit to improving the performance of the upgrade path. Therefore, this is no longer a concern for me.

An alternative approach is for the legacy api to get the list of used symbols from the bitcode and expose them via the lto_symbol_attributes; the system linker can then add "used" symbols to the preserve list directly. This seems more in-keeping with the original design IMO.

That was one of the proposals I had but some change needs to made to encode and convey this info. See previous review for more info.

I can have a go at implementing this to see what would be involved, if that would help?

Sure. I think my initial proposal was in the bug report: https://bugs.llvm.org/show_bug.cgi?id=41236
You can take a look and see if you agree with our discussion.

Thanks. Actually, after looking at this more, I want to accept this patch rather than any of the options that you list on the bug. Using the IRSymtab is a departure from the original design of the legacy api... but I now believe that this is a "good thing", as it means that the legacy and the resolution api's will use the same mechanism. I would like the legacy api and the resolution api to share as much code as possible, as this will improve the situation with maintaining and extending LTO. I am also interested in speeding up access to the "llvm.linker.options" metadata from the legacy api. In Sony we use llvm.linker.options for "dependent libraries" - #pragma comment(lib, ...). Currently, accessing this metadata is slow; but, if we use the IRSymtab access the dependent libraries, then we will be able to increase performance without having to make any changes to the bitcode loader.

[tejohnson]

LGTM
Sorry for the delay, just catching up after a vacation.

[steven_wu]

Thanks Teresa and Ben for the feedback! I am landing this change.

llvm/trunk/include/llvm/LTO/LTO.h

public:
   public:
     Symbol(const irsymtab::Symbol &S) : irsymtab::Symbol(S) {}
 
     using irsymtab::Symbol::isUndefined;
     using irsymtab::Symbol::isCommon;
     using irsymtab::Symbol::isWeak;
     using irsymtab::Symbol::isIndirect;
     using irsymtab::Symbol::getName;
+    using irsymtab::Symbol::getIRName;
     using irsymtab::Symbol::getVisibility;
     using irsymtab::Symbol::canBeOmittedFromSymbolTable;
     using irsymtab::Symbol::isTLS;
     using irsymtab::Symbol::getComdatIndex;
     using irsymtab::Symbol::getCommonSize;
     using irsymtab::Symbol::getCommonAlignment;
     using irsymtab::Symbol::getCOFFWeakExternalFallback;
     using irsymtab::Symbol::getSectionName;
public:
   StringRef getTargetTriple() const { return TargetTriple; }
 
   /// Returns the source file path specified at compile time.
   StringRef getSourceFileName() const { return SourceFileName; }
 
   // Returns a table with all the comdats used by this file.
   ArrayRef<StringRef> getComdatTable() const { return ComdatTable; }
 
+  // Returns the only BitcodeModule from InputFile.
+  BitcodeModule &getSingleBitcodeModule();
+
 private:
   ArrayRef<Symbol> module_symbols(unsigned I) const {
     const auto &Indices = ModuleSymIndices[I];
     return {Symbols.data() + Indices.first, Symbols.data() + Indices.second};
   }
 };
 
 /// This class wraps an output stream for a native object. Most clients should

llvm/trunk/include/llvm/LTO/legacy/ThinLTOCodeGenerator.h

 
 #ifndef LLVM_LTO_THINLTOCODEGENERATOR_H
 #define LLVM_LTO_THINLTOCODEGENERATOR_H
 
 #include "llvm-c/lto.h"
 #include "llvm/ADT/StringSet.h"
 #include "llvm/ADT/Triple.h"
 #include "llvm/IR/ModuleSummaryIndex.h"
+#include "llvm/LTO/LTO.h"
 #include "llvm/Support/CachePruning.h"
 #include "llvm/Support/CodeGen.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Target/TargetOptions.h"
 
 #include <string>
 
 namespace llvm {
 class StringRef;
 class LLVMContext;
 class TargetMachine;
 
-/// Wrapper around MemoryBufferRef, owning the identifier
-class ThinLTOBuffer {
-  std::string OwnedIdentifier;
-  StringRef Buffer;
-
-public:
-  ThinLTOBuffer(StringRef Buffer, StringRef Identifier)
-      : OwnedIdentifier(Identifier), Buffer(Buffer) {}
-
-  MemoryBufferRef getMemBuffer() const {
-    return MemoryBufferRef(Buffer,
-                           {OwnedIdentifier.c_str(), OwnedIdentifier.size()});
-  }
-  StringRef getBuffer() const { return Buffer; }
-  StringRef getBufferIdentifier() const { return OwnedIdentifier; }
-};
-
 /// Helper to gather options relevant to the target machine creation
 struct TargetMachineBuilder {
   Triple TheTriple;
   std::string MCpu;
   std::string MAttr;
   TargetOptions Options;
   Optional<Reloc::Model> RelocModel;
   CodeGenOpt::Level CGOptLevel = CodeGenOpt::Aggressive;
public:
    */
   std::unique_ptr<ModuleSummaryIndex> linkCombinedIndex();
 
   /**
    * Perform promotion and renaming of exported internal functions,
    * and additionally resolve weak and linkonce symbols.
    * Index is updated to reflect linkage changes from weak resolution.
    */
-  void promote(Module &Module, ModuleSummaryIndex &Index);
+  void promote(Module &Module, ModuleSummaryIndex &Index,
+               const lto::InputFile &File);
 
   /**
    * Compute and emit the imported files for module at \p ModulePath.
    */
   void emitImports(Module &Module, StringRef OutputName,
-                   ModuleSummaryIndex &Index);
+                   ModuleSummaryIndex &Index,
+                   const lto::InputFile &File);
 
   /**
    * Perform cross-module importing for the module identified by
    * ModuleIdentifier.
    */
-  void crossModuleImport(Module &Module, ModuleSummaryIndex &Index);
+  void crossModuleImport(Module &Module, ModuleSummaryIndex &Index,
+                         const lto::InputFile &File);
 
   /**
    * Compute the list of summaries needed for importing into module.
    */
   void gatherImportedSummariesForModule(
       Module &Module, ModuleSummaryIndex &Index,
-      std::map<std::string, GVSummaryMapTy> &ModuleToSummariesForIndex);
+      std::map<std::string, GVSummaryMapTy> &ModuleToSummariesForIndex,
+      const lto::InputFile &File);
 
   /**
    * Perform internalization. Index is updated to reflect linkage changes.
    */
-  void internalize(Module &Module, ModuleSummaryIndex &Index);
+  void internalize(Module &Module, ModuleSummaryIndex &Index,
+                   const lto::InputFile &File);
 
   /**
    * Perform post-importing ThinLTO optimizations.
    */
   void optimize(Module &Module);
 
   /**@}*/
 
 private:
   /// Helper factory to build a TargetMachine
   TargetMachineBuilder TMBuilder;
 
   /// Vector holding the in-memory buffer containing the produced binaries, when
   /// SavedObjectsDirectoryPath isn't set.
   std::vector<std::unique_ptr<MemoryBuffer>> ProducedBinaries;
 
   /// Path to generated files in the supplied SavedObjectsDirectoryPath if any.
   std::vector<std::string> ProducedBinaryFiles;
 
   /// Vector holding the input buffers containing the bitcode modules to
   /// process.
-  std::vector<ThinLTOBuffer> Modules;
+  std::vector<std::unique_ptr<lto::InputFile>> Modules;
 
   /// Set of symbols that need to be preserved outside of the set of bitcode
   /// files.
   StringSet<> PreservedSymbols;
 
   /// Set of symbols that are cross-referenced between bitcode files.
   StringSet<> CrossReferencedSymbols;
 

llvm/trunk/lib/LTO/LTO.cpp

Expected<std::unique_ptr<InputFile>> InputFile::create(MemoryBufferRef Object) {
   File->Strtab = std::move(FOrErr->Strtab);
   return std::move(File);
 }
 
 StringRef InputFile::getName() const {
   return Mods[0].getModuleIdentifier();
 }
 
+BitcodeModule &InputFile::getSingleBitcodeModule() {
+  assert(Mods.size() == 1 && "Expect only one bitcode module");
+  return Mods[0];
+}
+
 LTO::RegularLTOState::RegularLTOState(unsigned ParallelCodeGenParallelismLevel,
                                       Config &Conf)
     : ParallelCodeGenParallelismLevel(ParallelCodeGenParallelismLevel),
       Ctx(Conf), CombinedModule(llvm::make_unique<Module>("ld-temp.o", Ctx)),
       Mover(llvm::make_unique<IRMover>(*CombinedModule)) {}
 
 LTO::ThinLTOState::ThinLTOState(ThinBackend Backend)
     : Backend(Backend), CombinedIndex(/*HaveGVs*/ false) {

llvm/trunk/lib/LTO/ThinLTOCodeGenerator.cpp

static void computePrevailingCopies(
 
   for (auto &I : Index) {
     if (HasMultipleCopies(I.second.SummaryList))
       PrevailingCopy[I.first] =
           getFirstDefinitionForLinker(I.second.SummaryList);
   }
 }
 
-static StringMap<MemoryBufferRef>
-generateModuleMap(const std::vector<ThinLTOBuffer> &Modules) {
-  StringMap<MemoryBufferRef> ModuleMap;
-  for (auto &ModuleBuffer : Modules) {
-    assert(ModuleMap.find(ModuleBuffer.getBufferIdentifier()) ==
-               ModuleMap.end() &&
+static StringMap<lto::InputFile *>
+generateModuleMap(std::vector<std::unique_ptr<lto::InputFile>> &Modules) {
+  StringMap<lto::InputFile *> ModuleMap;
+  for (auto &M : Modules) {
+    assert(ModuleMap.find(M->getName()) == ModuleMap.end() &&
            "Expect unique Buffer Identifier");
-    ModuleMap[ModuleBuffer.getBufferIdentifier()] = ModuleBuffer.getMemBuffer();
+    ModuleMap[M->getName()] = M.get();
   }
   return ModuleMap;
 }
 
 static void promoteModule(Module &TheModule, const ModuleSummaryIndex &Index) {
   if (renameModuleForThinLTO(TheModule, Index))
     report_fatal_error("renameModuleForThinLTO failed");
 }
if (verifyModule(TheModule, &dbgs(), &BrokenDebugInfo))
     report_fatal_error("Broken module found, compilation aborted!");
   if (BrokenDebugInfo) {
     TheModule.getContext().diagnose(ThinLTODiagnosticInfo(
         "Invalid debug info found, debug info will be stripped", DS_Warning));
     StripDebugInfo(TheModule);
   }
 }
 
-static std::unique_ptr<Module>
-loadModuleFromBuffer(const MemoryBufferRef &Buffer, LLVMContext &Context,
-                     bool Lazy, bool IsImporting) {
+static std::unique_ptr<Module> loadModuleFromInput(lto::InputFile *Input,
+                                                   LLVMContext &Context,
+                                                   bool Lazy,
+                                                   bool IsImporting) {
+  auto &Mod = Input->getSingleBitcodeModule();
   SMDiagnostic Err;
   Expected<std::unique_ptr<Module>> ModuleOrErr =
-      Lazy
-          ? getLazyBitcodeModule(Buffer, Context,
-                                 /* ShouldLazyLoadMetadata */ true, IsImporting)
-          : parseBitcodeFile(Buffer, Context);
+      Lazy ? Mod.getLazyModule(Context,
+           : Mod.parseModule(Context);
   if (!ModuleOrErr) {
     handleAllErrors(ModuleOrErr.takeError(), [&](ErrorInfoBase &EIB) {
-      SMDiagnostic Err = SMDiagnostic(Buffer.getBufferIdentifier(),
+      SMDiagnostic Err = SMDiagnostic(Mod.getModuleIdentifier(),
                                       SourceMgr::DK_Error, EIB.message());
       Err.print("ThinLTO", errs());
     });
     report_fatal_error("Can't load module, abort.");
   }
   if (!Lazy)
     verifyLoadedModule(*ModuleOrErr.get());
-  return std::move(ModuleOrErr.get());
+  return std::move(*ModuleOrErr);
 }
 
 static void
 crossImportIntoModule(Module &TheModule, const ModuleSummaryIndex &Index,
-                      StringMap<MemoryBufferRef> &ModuleMap,
+                      StringMap<lto::InputFile*> &ModuleMap,
                       const FunctionImporter::ImportMapTy &ImportList) {
   auto Loader = [&](StringRef Identifier) {
-    return loadModuleFromBuffer(ModuleMap[Identifier], TheModule.getContext(),
-                                /*Lazy=*/true, /*IsImporting*/ true);
+    auto &Input = ModuleMap[Identifier];
+    return loadModuleFromInput(Input, TheModule.getContext(),
   };
 
   FunctionImporter Importer(Index, Loader);
   Expected<bool> Result = Importer.importFunctions(TheModule, ImportList);
   if (!Result) {
     handleAllErrors(Result.takeError(), [&](ErrorInfoBase &EIB) {
       SMDiagnostic Err = SMDiagnostic(TheModule.getModuleIdentifier(),
                                       SourceMgr::DK_Error, EIB.message());
static void optimizeModule(Module &TheModule, TargetMachine &TM,
   PM.add(createTargetTransformInfoWrapperPass(TM.getTargetIRAnalysis()));
 
   // Add optimizations
   PMB.populateThinLTOPassManager(PM);
 
   PM.run(TheModule);
 }
 
+static void
+addUsedSymbolToPreservedGUID(const lto::InputFile &File,
+                             DenseSet<GlobalValue::GUID> &PreservedGUID) {
+  for (const auto &Sym : File.symbols()) {
+    if (Sym.isUsed())
+      PreservedGUID.insert(GlobalValue::getGUID(Sym.getIRName()));
+  }
+}
+
 // Convert the PreservedSymbols map from "Name" based to "GUID" based.
 static DenseSet<GlobalValue::GUID>
 computeGUIDPreservedSymbols(const StringSet<> &PreservedSymbols,
                             const Triple &TheTriple) {
   DenseSet<GlobalValue::GUID> GUIDPreservedSymbols(PreservedSymbols.size());
   for (auto &Entry : PreservedSymbols) {
     StringRef Name = Entry.first();
     if (TheTriple.isOSBinFormatMachO() && Name.size() > 0 && Name[0] == '_')
void write(const MemoryBuffer &OutputBuffer) {
     EC = sys::fs::rename(TempFilename, EntryPath);
     if (EC)
       sys::fs::remove(TempFilename);
   }
 };
 
 static std::unique_ptr<MemoryBuffer>
 ProcessThinLTOModule(Module &TheModule, ModuleSummaryIndex &Index,
-                     StringMap<MemoryBufferRef> &ModuleMap, TargetMachine &TM,
+                     StringMap<lto::InputFile *> &ModuleMap, TargetMachine &TM,
                      const FunctionImporter::ImportMapTy &ImportList,
                      const FunctionImporter::ExportSetTy &ExportList,
                      const DenseSet<GlobalValue::GUID> &GUIDPreservedSymbols,
                      const GVSummaryMapTy &DefinedGlobals,
                      const ThinLTOCodeGenerator::CachingOptions &CacheOptions,
                      bool DisableCodeGen, StringRef SaveTempsDir,
                      bool Freestanding, unsigned OptLevel, unsigned count) {
 
else if (TheTriple.getArch() == llvm::Triple::aarch64)
       TMBuilder.MCpu = "cyclone";
   }
   TMBuilder.TheTriple = std::move(TheTriple);
 }
 
 } // end anonymous namespace
 
 void ThinLTOCodeGenerator::addModule(StringRef Identifier, StringRef Data) {
-  ThinLTOBuffer Buffer(Data, Identifier);
-  LLVMContext Context;
-  StringRef TripleStr;
-  ErrorOr<std::string> TripleOrErr = expectedToErrorOrAndEmitErrors(
-      Context, getBitcodeTargetTriple(Buffer.getMemBuffer()));
+  MemoryBufferRef Buffer(Data, Identifier);
 
-  if (TripleOrErr)
-    TripleStr = *TripleOrErr;
+  auto InputOrError = lto::InputFile::create(Buffer);
+  if (!InputOrError)
+    report_fatal_error("ThinLTO cannot create input file: " +
+                       toString(InputOrError.takeError()));
 
+  auto TripleStr = (*InputOrError)->getTargetTriple();
   Triple TheTriple(TripleStr);
 
   if (Modules.empty())
     initTMBuilder(TMBuilder, Triple(TheTriple));
   else if (TMBuilder.TheTriple != TheTriple) {
     if (!TMBuilder.TheTriple.isCompatibleWith(TheTriple))
       report_fatal_error("ThinLTO modules with incompatible triples not "
                          "supported");
     initTMBuilder(TMBuilder, Triple(TMBuilder.TheTriple.merge(TheTriple)));
   }
 
-  Modules.push_back(Buffer);
+  Modules.emplace_back(std::move(*InputOrError));
 }
 
 void ThinLTOCodeGenerator::preserveSymbol(StringRef Name) {
   PreservedSymbols.insert(Name);
 }
 
 void ThinLTOCodeGenerator::crossReferenceSymbol(StringRef Name) {
   // FIXME: At the moment, we don't take advantage of this extra information,

 /**
  * Produce the combined summary index from all the bitcode files:
  * "thin-link".
  */
 std::unique_ptr<ModuleSummaryIndex> ThinLTOCodeGenerator::linkCombinedIndex() {
   std::unique_ptr<ModuleSummaryIndex> CombinedIndex =
       llvm::make_unique<ModuleSummaryIndex>(/*HaveGVs=*/false);
   uint64_t NextModuleId = 0;
-  for (auto &ModuleBuffer : Modules) {
-    if (Error Err = readModuleSummaryIndex(ModuleBuffer.getMemBuffer(),
-                                           *CombinedIndex, NextModuleId++)) {
+  for (auto &Mod : Modules) {
+    auto &M = Mod->getSingleBitcodeModule();
+    if (Error Err =
+            M.readSummary(*CombinedIndex, Mod->getName(), NextModuleId++)) {
       // FIXME diagnose
       logAllUnhandledErrors(
           std::move(Err), errs(),
           "error: can't create module summary index for buffer: ");
       return nullptr;
     }
   }
   return CombinedIndex;
static void computeDeadSymbolsInIndex(
   computeDeadSymbolsWithConstProp(Index, GUIDPreservedSymbols, isPrevailing,
                                   /* ImportEnabled = */ true);
 }
 
 /**
  * Perform promotion and renaming of exported internal functions.
  * Index is updated to reflect linkage changes from weak resolution.
  */
-void ThinLTOCodeGenerator::promote(Module &TheModule,
-                                   ModuleSummaryIndex &Index) {
+void ThinLTOCodeGenerator::promote(Module &TheModule, ModuleSummaryIndex &Index,
+                                   const lto::InputFile &File) {
   auto ModuleCount = Index.modulePaths().size();
   auto ModuleIdentifier = TheModule.getModuleIdentifier();
 
   // Collect for each module the list of function it defines (GUID -> Summary).
   StringMap<GVSummaryMapTy> ModuleToDefinedGVSummaries;
   Index.collectDefinedGVSummariesPerModule(ModuleToDefinedGVSummaries);
 
   // Convert the preserved symbols set from string to GUID
   auto GUIDPreservedSymbols = computeGUIDPreservedSymbols(
       PreservedSymbols, Triple(TheModule.getTargetTriple()));
 
+  // Add used symbol to the preserved symbols.
+  addUsedSymbolToPreservedGUID(File, GUIDPreservedSymbols);
+
   // Compute "dead" symbols, we don't want to import/export these!
   computeDeadSymbolsInIndex(Index, GUIDPreservedSymbols);
 
   // Generate import/export list
   StringMap<FunctionImporter::ImportMapTy> ImportLists(ModuleCount);
   StringMap<FunctionImporter::ExportSetTy> ExportLists(ModuleCount);
   ComputeCrossModuleImport(Index, ModuleToDefinedGVSummaries, ImportLists,
                            ExportLists);
void ThinLTOCodeGenerator::promote(Module &TheModule, ModuleSummaryIndex &Index,
 
   promoteModule(TheModule, Index);
 }
 
 /**
  * Perform cross-module importing for the module identified by ModuleIdentifier.
  */
 void ThinLTOCodeGenerator::crossModuleImport(Module &TheModule,
-                                             ModuleSummaryIndex &Index) {
+                                             ModuleSummaryIndex &Index,
+                                             const lto::InputFile &File) {
   auto ModuleMap = generateModuleMap(Modules);
   auto ModuleCount = Index.modulePaths().size();
 
   // Collect for each module the list of function it defines (GUID -> Summary).
   StringMap<GVSummaryMapTy> ModuleToDefinedGVSummaries(ModuleCount);
   Index.collectDefinedGVSummariesPerModule(ModuleToDefinedGVSummaries);
 
   // Convert the preserved symbols set from string to GUID
   auto GUIDPreservedSymbols = computeGUIDPreservedSymbols(
       PreservedSymbols, Triple(TheModule.getTargetTriple()));
 
+  addUsedSymbolToPreservedGUID(File, GUIDPreservedSymbols);
+
   // Compute "dead" symbols, we don't want to import/export these!
   computeDeadSymbolsInIndex(Index, GUIDPreservedSymbols);
 
   // Generate import/export list
   StringMap<FunctionImporter::ImportMapTy> ImportLists(ModuleCount);
   StringMap<FunctionImporter::ExportSetTy> ExportLists(ModuleCount);
   ComputeCrossModuleImport(Index, ModuleToDefinedGVSummaries, ImportLists,
                            ExportLists);
   auto &ImportList = ImportLists[TheModule.getModuleIdentifier()];
 
   crossImportIntoModule(TheModule, Index, ModuleMap, ImportList);
 }
 
 /**
  * Compute the list of summaries needed for importing into module.
  */
 void ThinLTOCodeGenerator::gatherImportedSummariesForModule(
     Module &TheModule, ModuleSummaryIndex &Index,
-    std::map<std::string, GVSummaryMapTy> &ModuleToSummariesForIndex) {
+    std::map<std::string, GVSummaryMapTy> &ModuleToSummariesForIndex,
+    const lto::InputFile &File) {
   auto ModuleCount = Index.modulePaths().size();
   auto ModuleIdentifier = TheModule.getModuleIdentifier();
 
   // Collect for each module the list of function it defines (GUID -> Summary).
   StringMap<GVSummaryMapTy> ModuleToDefinedGVSummaries(ModuleCount);
   Index.collectDefinedGVSummariesPerModule(ModuleToDefinedGVSummaries);
 
   // Convert the preserved symbols set from string to GUID
   auto GUIDPreservedSymbols = computeGUIDPreservedSymbols(
       PreservedSymbols, Triple(TheModule.getTargetTriple()));
 
+  addUsedSymbolToPreservedGUID(File, GUIDPreservedSymbols);
+
   // Compute "dead" symbols, we don't want to import/export these!
   computeDeadSymbolsInIndex(Index, GUIDPreservedSymbols);
 
   // Generate import/export list
   StringMap<FunctionImporter::ImportMapTy> ImportLists(ModuleCount);
   StringMap<FunctionImporter::ExportSetTy> ExportLists(ModuleCount);
   ComputeCrossModuleImport(Index, ModuleToDefinedGVSummaries, ImportLists,
                            ExportLists);
 
   llvm::gatherImportedSummariesForModule(
       ModuleIdentifier, ModuleToDefinedGVSummaries,
       ImportLists[ModuleIdentifier], ModuleToSummariesForIndex);
 }
 
 /**
  * Emit the list of files needed for importing into module.
  */
 void ThinLTOCodeGenerator::emitImports(Module &TheModule, StringRef OutputName,
-                                       ModuleSummaryIndex &Index) {
+                                       ModuleSummaryIndex &Index,
+                                       const lto::InputFile &File) {
   auto ModuleCount = Index.modulePaths().size();
   auto ModuleIdentifier = TheModule.getModuleIdentifier();
 
   // Collect for each module the list of function it defines (GUID -> Summary).
   StringMap<GVSummaryMapTy> ModuleToDefinedGVSummaries(ModuleCount);
   Index.collectDefinedGVSummariesPerModule(ModuleToDefinedGVSummaries);
 
   // Convert the preserved symbols set from string to GUID
   auto GUIDPreservedSymbols = computeGUIDPreservedSymbols(
       PreservedSymbols, Triple(TheModule.getTargetTriple()));
 
+  addUsedSymbolToPreservedGUID(File, GUIDPreservedSymbols);
+
   // Compute "dead" symbols, we don't want to import/export these!
   computeDeadSymbolsInIndex(Index, GUIDPreservedSymbols);
 
   // Generate import/export list
   StringMap<FunctionImporter::ImportMapTy> ImportLists(ModuleCount);
   StringMap<FunctionImporter::ExportSetTy> ExportLists(ModuleCount);
   ComputeCrossModuleImport(Index, ModuleToDefinedGVSummaries, ImportLists,
                            ExportLists);
 
   std::map<std::string, GVSummaryMapTy> ModuleToSummariesForIndex;
   llvm::gatherImportedSummariesForModule(
       ModuleIdentifier, ModuleToDefinedGVSummaries,
       ImportLists[ModuleIdentifier], ModuleToSummariesForIndex);
 
   std::error_code EC;
   if ((EC = EmitImportsFiles(ModuleIdentifier, OutputName,
                              ModuleToSummariesForIndex)))
     report_fatal_error(Twine("Failed to open ") + OutputName +
                        " to save imports lists\n");
 }
 
 /**
- * Perform internalization. Index is updated to reflect linkage changes.
+ * Perform internalization. Runs promote and internalization together.
+ * Index is updated to reflect linkage changes.
  */
 void ThinLTOCodeGenerator::internalize(Module &TheModule,
-                                       ModuleSummaryIndex &Index) {
+                                       ModuleSummaryIndex &Index,
+                                       const lto::InputFile &File) {
   initTMBuilder(TMBuilder, Triple(TheModule.getTargetTriple()));
   auto ModuleCount = Index.modulePaths().size();
   auto ModuleIdentifier = TheModule.getModuleIdentifier();
 
   // Convert the preserved symbols set from string to GUID
   auto GUIDPreservedSymbols =
       computeGUIDPreservedSymbols(PreservedSymbols, TMBuilder.TheTriple);
 
+  addUsedSymbolToPreservedGUID(File, GUIDPreservedSymbols);
+
   // Collect for each module the list of function it defines (GUID -> Summary).
   StringMap<GVSummaryMapTy> ModuleToDefinedGVSummaries(ModuleCount);
   Index.collectDefinedGVSummariesPerModule(ModuleToDefinedGVSummaries);
 
   // Compute "dead" symbols, we don't want to import/export these!
   computeDeadSymbolsInIndex(Index, GUIDPreservedSymbols);
 
   // Generate import/export list
   StringMap<FunctionImporter::ImportMapTy> ImportLists(ModuleCount);
   StringMap<FunctionImporter::ExportSetTy> ExportLists(ModuleCount);
   ComputeCrossModuleImport(Index, ModuleToDefinedGVSummaries, ImportLists,
                            ExportLists);
   auto &ExportList = ExportLists[ModuleIdentifier];
 
   // Be friendly and don't nuke totally the module when the client didn't
   // supply anything to preserve.
   if (ExportList.empty() && GUIDPreservedSymbols.empty())
     return;
 
-  // Internalization
+  // Resolve prevailing symbols
+  StringMap<std::map<GlobalValue::GUID, GlobalValue::LinkageTypes>> ResolvedODR;
+  resolvePrevailingInIndex(Index, ResolvedODR);
+
+  // Promote the exported values in the index, so that they are promoted
+  // in the module.
   internalizeAndPromoteInIndex(ExportLists, GUIDPreservedSymbols, Index);
+
+  promoteModule(TheModule, Index);
+
+  // Internalization
+  thinLTOResolvePrevailingInModule(
+      TheModule, ModuleToDefinedGVSummaries[ModuleIdentifier]);
+
   thinLTOInternalizeModule(TheModule,
                            ModuleToDefinedGVSummaries[ModuleIdentifier]);
 }
 
 /**
  * Perform post-importing ThinLTO optimizations.
  */
 void ThinLTOCodeGenerator::optimize(Module &TheModule) {
if (!IsDir)
       report_fatal_error("Unexistent dir: '" + SavedObjectsDirectoryPath + "'");
     ProducedBinaryFiles.resize(Modules.size());
   }
 
   if (CodeGenOnly) {
     // Perform only parallel codegen and return.
     ThreadPool Pool;
     int count = 0;
-    for (auto &ModuleBuffer : Modules) {
+    for (auto &Mod : Modules) {
       Pool.async([&](int count) {
         LLVMContext Context;
         Context.setDiscardValueNames(LTODiscardValueNames);
 
         // Parse module now
-        auto TheModule =
-            loadModuleFromBuffer(ModuleBuffer.getMemBuffer(), Context, false,
+        auto TheModule = loadModuleFromInput(Mod.get(), Context, false,
                                              /*IsImporting*/ false);
 
         // CodeGen
         auto OutputBuffer = codegenModule(*TheModule, *TMBuilder.create());
         if (SavedObjectsDirectoryPath.empty())
           ProducedBinaries[count] = std::move(OutputBuffer);
         else
           ProducedBinaryFiles[count] = writeGeneratedObject(
               count, "", SavedObjectsDirectoryPath, *OutputBuffer);
void ThinLTOCodeGenerator::run() {
   StringMap<GVSummaryMapTy> ModuleToDefinedGVSummaries(ModuleCount);
   Index->collectDefinedGVSummariesPerModule(ModuleToDefinedGVSummaries);
 
   // Convert the preserved symbols set from string to GUID, this is needed for
   // computing the caching hash and the internalization.
   auto GUIDPreservedSymbols =
       computeGUIDPreservedSymbols(PreservedSymbols, TMBuilder.TheTriple);
 
+  // Add used symbol from inputs to the preserved symbols.
+  for (const auto &M : Modules)
+    addUsedSymbolToPreservedGUID(*M, GUIDPreservedSymbols);
+
   // Compute "dead" symbols, we don't want to import/export these!
   computeDeadSymbolsInIndex(*Index, GUIDPreservedSymbols);
 
   // Synthesize entry counts for functions in the combined index.
   computeSyntheticCounts(*Index);
 
   // Collect the import/export lists for all modules from the call-graph in the
   // combined index.
void ThinLTOCodeGenerator::run() {
   // internalized (because they aren't exported or preserved as per callback).
   // Changes are made in the index, consumed in the ThinLTO backends.
   internalizeAndPromoteInIndex(ExportLists, GUIDPreservedSymbols, *Index);
 
   // Make sure that every module has an entry in the ExportLists, ImportList,
   // GVSummary and ResolvedODR maps to enable threaded access to these maps
   // below.
   for (auto &Module : Modules) {
-    auto ModuleIdentifier = Module.getBufferIdentifier();
+    auto ModuleIdentifier = Module->getName();
     ExportLists[ModuleIdentifier];
     ImportLists[ModuleIdentifier];
     ResolvedODR[ModuleIdentifier];
     ModuleToDefinedGVSummaries[ModuleIdentifier];
   }
 
   // Compute the ordering we will process the inputs: the rough heuristic here
   // is to sort them per size so that the largest module get schedule as soon as
   // possible. This is purely a compile-time optimization.
   std::vector<int> ModulesOrdering;
   ModulesOrdering.resize(Modules.size());
   std::iota(ModulesOrdering.begin(), ModulesOrdering.end(), 0);
   llvm::sort(ModulesOrdering, [&](int LeftIndex, int RightIndex) {
-    auto LSize = Modules[LeftIndex].getBuffer().size();
-    auto RSize = Modules[RightIndex].getBuffer().size();
+    auto LSize =
+        Modules[LeftIndex]->getSingleBitcodeModule().getBuffer().size();
+    auto RSize =
+        Modules[RightIndex]->getSingleBitcodeModule().getBuffer().size();
     return LSize > RSize;
   });
 
   // Parallel optimizer + codegen
   {
     ThreadPool Pool(ThreadCount);
     for (auto IndexCount : ModulesOrdering) {
-      auto &ModuleBuffer = Modules[IndexCount];
+      auto &Mod = Modules[IndexCount];
       Pool.async([&](int count) {
-        auto ModuleIdentifier = ModuleBuffer.getBufferIdentifier();
+        auto ModuleIdentifier = Mod->getName();
         auto &ExportList = ExportLists[ModuleIdentifier];
 
         auto &DefinedGVSummaries = ModuleToDefinedGVSummaries[ModuleIdentifier];
 
         // The module may be cached, this helps handling it.
         ModuleCacheEntry CacheEntry(CacheOptions.Path, *Index, ModuleIdentifier,
                                     ImportLists[ModuleIdentifier], ExportList,
                                     ResolvedODR[ModuleIdentifier],
for (auto IndexCount : ModulesOrdering) {
             LTOPassRemarksWithHotness, count);
         if (!DiagFileOrErr) {
           errs() << "Error: " << toString(DiagFileOrErr.takeError()) << "\n";
           report_fatal_error("ThinLTO: Can't get an output file for the "
                              "remarks");
         }
 
         // Parse module now
-        auto TheModule =
-            loadModuleFromBuffer(ModuleBuffer.getMemBuffer(), Context, false,
+        auto TheModule = loadModuleFromInput(Mod.get(), Context, false,
                                              /*IsImporting*/ false);
 
         // Save temps: original file.
         saveTempBitcode(*TheModule, SaveTempsDir, count, ".0.original.bc");
 
         auto &ImportList = ImportLists[ModuleIdentifier];
         // Run the main process now, and generates a binary
         auto OutputBuffer = ProcessThinLTOModule(
             *TheModule, *Index, ModuleMap, *TMBuilder.create(), ImportList,

llvm/trunk/test/Bitcode/Inputs/module_hash.ll

 ; Needs a function for the combined index to be populated
+target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
+
 define void @bar() {
     ret void
 }

llvm/trunk/test/Bitcode/module_hash.ll

 ; First capture the value of the hash for the two modules.
 ; COMBINED: <HASH op0=[[HASH1_1:[0-9]*]] op1=[[HASH1_2:[0-9]*]] op2=[[HASH1_3:[0-9]*]] op3=[[HASH1_4:[0-9]*]] op4=[[HASH1_5:[0-9]*]]/>
 ; COMBINED: <HASH op0=[[HASH2_1:[0-9]*]] op1=[[HASH2_2:[0-9]*]] op2=[[HASH2_3:[0-9]*]] op3=[[HASH2_4:[0-9]*]] op4=[[HASH2_5:[0-9]*]]/>
 
 ; Validate against the value extracted from the combined index
 ; COMBINED-DAG: <HASH abbrevid={{[0-9]*}} op0=[[HASH1_1]] op1=[[HASH1_2]] op2=[[HASH1_3]] op3=[[HASH1_4]] op4=[[HASH1_5]]/>
 ; COMBINED-DAG: <HASH abbrevid={{[0-9]*}} op0=[[HASH2_1]] op1=[[HASH2_2]] op2=[[HASH2_3]] op3=[[HASH2_4]] op4=[[HASH2_5]]/>
 
+target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
 
 ; Need a function for the combined index to be populated.
 define void @foo() {
     ret void
 }

llvm/trunk/test/ThinLTO/X86/Inputs/alias_import.ll

-
-
-
+target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
 
 @globalfuncAlias = alias void (...), bitcast (void ()* @globalfunc to void (...)*)
 @globalfuncWeakAlias = weak alias void (...), bitcast (void ()* @globalfunc to void (...)*)
 @globalfuncLinkonceAlias = linkonce alias void (...), bitcast (void ()* @globalfunc to void (...)*)
 @globalfuncWeakODRAlias = weak_odr alias void (...), bitcast (void ()* @globalfunc to void (...)*)
 @globalfuncLinkonceODRAlias = linkonce_odr alias void (...), bitcast (void ()* @globalfunc to void (...)*)
 define void @globalfunc() {
 entry:

llvm/trunk/test/ThinLTO/X86/Inputs/alias_resolution.ll

-
-
-
+target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
 
 @globalfuncAlias = alias void (...), bitcast (void ()* @globalfunc to void (...)*)
 @globalfuncWeakAlias = weak alias void (...), bitcast (void ()* @globalfunc to void (...)*)
 @globalfuncLinkonceAlias = linkonce alias void (...), bitcast (void ()* @globalfunc to void (...)*)
 @globalfuncWeakODRAlias = weak_odr alias void (...), bitcast (void ()* @globalfunc to void (...)*)
 @globalfuncLinkonceODRAlias = linkonce_odr alias void (...), bitcast (void ()* @globalfunc to void (...)*)
 define void @globalfunc() {
 entry:

llvm/trunk/test/ThinLTO/X86/Inputs/distributed_indexes.ll

+target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
+
 define void @g() {
 entry:
   ret void
 }
 
 @analias = alias void (...), bitcast (void ()* @aliasee to void (...)*)
 define void @aliasee() {
 entry:
   ret void
 }

llvm/trunk/test/ThinLTO/X86/Inputs/merge-triple.ll

 target triple = "x86_64-apple-macosx10.11.0"
+target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"

llvm/trunk/test/ThinLTO/X86/Inputs/section.ll

 ; An internal global variable that can't be renamed because it has a section
+target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
 @var_with_section = internal global i32 0, section "some_section"
 
 ; @reference_gv_with_section() can't be imported
 define i32 @reference_gv_with_section() {
     %res = load i32, i32* @var_with_section
     ret i32 %res
 }
 
 ; canary
 define void @foo() {
     ret void
 }

llvm/trunk/test/ThinLTO/X86/Inputs/thinlto-internalize-used2.ll

+target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-apple-macosx10.15.0"
+
+define i32 @main() {
+entry:
+  %call = call i32 @bar()
+  ret i32 0
+}
+
+declare i32 @bar()

llvm/trunk/test/ThinLTO/X86/alias_import.ll

 ; IMPORT-DAG: define available_externally void @weakfuncWeakODRAlias()
 ; IMPORT-DAG: define available_externally void @weakfuncLinkonceODRAlias()
 ; IMPORT-DAG: define available_externally void @linkonceODRfuncAlias()
 ; IMPORT-DAG: declare void @linkonceODRfuncWeakAlias()
 ; IMPORT-DAG: define available_externally void @linkonceODRfuncWeakODRAlias()
 ; IMPORT-DAG: declare void @linkonceODRfuncLinkonceAlias()
 ; IMPORT-DAG: define available_externally void @linkonceODRfuncLinkonceODRAlias()
 
+target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
+
 define i32 @main() #0 {
 entry:
   call void @globalfuncAlias()
   call void @globalfuncWeakAlias()
   call void @globalfuncLinkonceAlias()
   call void @globalfuncWeakODRAlias()
   call void @globalfuncLinkonceODRAlias()
 

llvm/trunk/test/ThinLTO/X86/alias_resolution.ll

 ; PROMOTE_MOD1: @weakfuncLinkonceAlias = weak alias void (...), bitcast (void ()* @weakfunc{{.*}} to void (...)*)
 ; PROMOTE_MOD2: @weakfuncLinkonceAlias = linkonce alias void (...), bitcast (void ()* @weakfunc{{.*}} to void (...)*)
 ; FIXME: The "resolution" should turn one of these to linkonce_odr
 ; PROMOTE_MOD1: @weakfuncWeakODRAlias = weak_odr alias void (...), bitcast (void ()* @weakfunc.mod1 to void (...)*)
 ; PROMOTE_MOD2: @weakfuncWeakODRAlias = weak_odr alias void (...), bitcast (void ()* @weakfunc to void (...)*)
 ; PROMOTE_MOD1: @weakfuncLinkonceODRAlias = weak_odr alias void (...), bitcast (void ()* @weakfunc.mod1 to void (...)*)
 ; PROMOTE_MOD2: @weakfuncLinkonceODRAlias = linkonce_odr alias void (...), bitcast (void ()* @weakfunc to void (...)*)
 
+target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
 
 @linkonceODRfuncAlias = alias void (...), bitcast (void ()* @linkonceODRfunc.mod1 to void (...)*)
 @linkonceODRfuncWeakAlias = weak alias void (...), bitcast (void ()* @linkonceODRfunc.mod1 to void (...)*)
 @linkonceODRfuncLinkonceAlias = linkonce alias void (...), bitcast (void ()* @linkonceODRfunc.mod1 to void (...)*)
 @linkonceODRfuncWeakODRAlias = weak_odr alias void (...), bitcast (void ()* @linkonceODRfunc.mod1 to void (...)*)
 @linkonceODRfuncLinkonceODRAlias = linkonce_odr alias void (...), bitcast (void ()* @linkonceODRfunc.mod1 to void (...)*)
 define linkonce_odr void @linkonceODRfunc.mod1() {
 entry:

llvm/trunk/test/ThinLTO/X86/deadstrip.ll

 ; Require asserts for -debug-only
 ; REQUIRES: asserts
 
 ; RUN: opt -module-summary %s -o %t1.bc
 ; RUN: opt -module-summary %p/Inputs/deadstrip.ll -o %t2.bc
 ; RUN: llvm-lto -thinlto-action=thinlink -o %t.index.bc %t1.bc %t2.bc
 
-; RUN: llvm-lto -exported-symbol=_main -thinlto-action=promote %t1.bc -thinlto-index=%t.index.bc -o - | llvm-lto -exported-symbol=_main -thinlto-action=internalize -thinlto-index %t.index.bc -thinlto-module-id=%t1.bc - -o - | llvm-dis -o - | FileCheck %s
-; RUN: llvm-lto -exported-symbol=_main -thinlto-action=promote %t2.bc -thinlto-index=%t.index.bc -o - | llvm-lto -exported-symbol=_main -thinlto-action=internalize -thinlto-index %t.index.bc -thinlto-module-id=%t2.bc - -o - | llvm-dis -o - | FileCheck %s --check-prefix=CHECK2
+; RUN: llvm-lto -exported-symbol=_main -thinlto-action=internalize %t1.bc -thinlto-index=%t.index.bc -o - | llvm-dis -o - | FileCheck %s
+; RUN: llvm-lto -exported-symbol=_main -thinlto-action=internalize %t2.bc -thinlto-index=%t.index.bc -o - | llvm-dis -o - | FileCheck %s --check-prefix=CHECK2
 
 ; RUN: llvm-lto -exported-symbol=_main -thinlto-action=run -stats %t1.bc %t2.bc 2>&1 | FileCheck %s --check-prefix=STATS
 ; RUN: llvm-nm %t1.bc.thinlto.o | FileCheck %s --check-prefix=CHECK-NM
 
 ; RUN: llvm-lto2 run %t1.bc %t2.bc -o %t.out -save-temps -stats \
 ; RUN:   -r %t1.bc,_main,plx \
 ; RUN:   -r %t1.bc,_bar,pl \
 ; RUN:   -r %t1.bc,_dead_func,pl \

llvm/trunk/test/ThinLTO/X86/distributed_indexes.ll

 ; BACKEND2-NEXT: </GLOBALVAL_SUMMARY_BLOCK
 
 ; Make sure that when the alias is imported as a copy of the aliasee, but the
 ; aliasee is not being imported by itself, that we can still print the summary.
 ; The aliasee should be "null".
 ; RUN: llvm-dis %t1.bc.thinlto.bc -o - | FileCheck %s --check-prefix=DIS
 ; DIS: aliasee: null
 
+target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
+
 declare void @g(...)
 declare void @analias(...)
 
 define void @f() {
 entry:
   call void (...) @g()
   call void (...) @analias()
   ret void
 }

llvm/trunk/test/ThinLTO/X86/internalize.ll

 ; REGULAR: define void @foo
 ; REGULAR: define void @bar
 ; REGULAR: define linkonce void @linkonce_func()
 ; INTERNALIZE: define void @foo
 ; INTERNALIZE: define internal void @bar
 ; INTERNALIZE: define internal void @linkonce_func()
 ; INTERNALIZE-OPTION-DISABLE: define void @foo
 ; INTERNALIZE-OPTION-DISABLE: define void @bar
-; INTERNALIZE-OPTION-DISABLE: define linkonce void @linkonce_func()
+; INTERNALIZE-OPTION-DISABLE: define weak void @linkonce_func()
 ; INTERNALIZE2: define dso_local void @foo
 ; INTERNALIZE2: define internal void @bar
 ; INTERNALIZE2: define internal void @linkonce_func()
 ; INTERNALIZE2-OPTION-DISABLE: define dso_local void @foo
 ; INTERNALIZE2-OPTION-DISABLE: define dso_local void @bar
 ; INTERNALIZE2-OPTION-DISABLE: define weak dso_local void @linkonce_func()
 
 target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"

llvm/trunk/test/ThinLTO/X86/linkonce_odr_unnamed_addr.ll

 ; This test ensures that when linkonce_odr + unnamed_addr symbols promoted to
 ; weak symbols, it preserves the auto hide property.
 
 ; RUN: opt -module-summary %s -o %t.bc
 ; RUN: opt -module-summary %s -o %t2.bc
 ; RUN: llvm-lto -thinlto-action=thinlink -o %t3.bc %t.bc %t2.bc
 ; RUN: llvm-lto -thinlto-action=promote %t.bc -thinlto-index=%t3.bc -o - | llvm-dis -o - | FileCheck %s
 
+target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
+
 ; CHECK: @linkonceodrunnamed = weak_odr hidden unnamed_addr constant i32 0
 @linkonceodrunnamed = linkonce_odr unnamed_addr constant i32 0

llvm/trunk/test/ThinLTO/X86/merge-triple.ll

 ; RUN: opt -module-summary %s -o %t1.bc
 ; RUN: opt -module-summary %p/Inputs/merge-triple.ll -o %t2.bc
 ; RUN: llvm-lto -thinlto-action=optimize %t1.bc %t2.bc
 ; RUN: llvm-dis < %t1.bc.thinlto.imported.bc | FileCheck %s --check-prefix=CHECK1
 ; RUN: llvm-dis < %t2.bc.thinlto.imported.bc | FileCheck %s --check-prefix=CHECK2
 
+target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-apple-macosx10.12.0"
 
 ; CHECK1: target triple = "x86_64-apple-macosx10.12.0"
 ; CHECK2: target triple = "x86_64-apple-macosx10.11.0"

llvm/trunk/test/ThinLTO/X86/prefix_replace.ll

 ; Check that changing the output path via prefix-replace works
 ; Use of '/' in paths created here make this unsuitable for Windows.
 ; RUN: mkdir -p %t/oldpath
 ; RUN: opt -module-summary %s -o %t/oldpath/prefix_replace.o
 ; Ensure that there is no existing file at the new path, so we properly
 ; test the creation of the new file there.
 ; RUN: rm -f %t/newpath/prefix_replace.o.thinlto.bc
 
 ; RUN: llvm-lto -thinlto-action=thinlink -o %t.index.bc %t/oldpath/prefix_replace.o
 ; RUN: llvm-lto -thinlto-action=distributedindexes -thinlto-prefix-replace="%t/oldpath/;%t/newpath/" -thinlto-index %t.index.bc %t/oldpath/prefix_replace.o
 
 ; RUN: ls %t/newpath/prefix_replace.o.thinlto.bc
+target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
 
 define void @f() {
 entry:
   ret void
 }

llvm/trunk/test/ThinLTO/X86/section.ll

 ; Do setup work for all below tests: generate bitcode and combined index
 ; RUN: opt -module-summary %s -o %t.bc
 ; RUN: opt -module-summary %p/Inputs/section.ll -o %t2.bc
 ; RUN: llvm-lto -thinlto-action=thinlink -o %t3.bc %t.bc %t2.bc
 
 ; Check that we don't promote 'var_with_section'
 ; RUN: llvm-lto -thinlto-action=promote %t2.bc -thinlto-index=%t3.bc -o - | llvm-dis -o - | FileCheck %s --check-prefix=PROMOTE
 ; PROMOTE: @var_with_section = internal global i32 0, section "some_section"
 
 ; RUN: llvm-lto -thinlto-action=import %t.bc -thinlto-index=%t3.bc -o - | llvm-dis -o - | FileCheck %s --check-prefix=IMPORT
 ; Check that section prevent import of @reference_gv_with_section.
 ; IMPORT: declare void @reference_gv_with_section()
 ; Canary to check that importing is correctly set up.
 ; IMPORT: define available_externally void @foo()
+target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
 
 
 define i32 @main() {
     call void @reference_gv_with_section()
     call void @foo()
     ret i32 42
 }
 
 
 declare void @reference_gv_with_section()
 declare void @foo()

llvm/trunk/test/ThinLTO/X86/thinlto-internalize-used.ll

+; RUN: opt -module-summary -o %t.bc %s
+; RUN: opt -module-summary -o %t-main.bc %S/Inputs/thinlto-internalize-used2.ll
+; RUN: llvm-lto -thinlto-action=thinlink %t.bc %t-main.bc -o %t-index.bc
+; RUN: llvm-lto -thinlto-action=internalize -thinlto-index %t-index.bc %t.bc -o %t.promote.bc
+; RUN: llvm-dis %t.promote.bc -o - | FileCheck %s
+
+target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-apple-macosx10.15.0"
+
[e-mail] = appending global [1 x i8*] [i8* bitcast (i32 ()* @foo to i8*)], section "llvm.metadata"
+
+; Make sure foo is not internalized.
+; CHECK: define i32 @foo()
+define i32 @foo() {
+  ret i32 0
+}
+
+define hidden i32 @bar() {
+  ret i32 0
+}
+

llvm/trunk/test/ThinLTO/X86/weak_resolution.ll

 ; Do setup work for all below tests: generate bitcode and combined index
 ; RUN: opt -module-summary %s -o %t.bc
 ; RUN: opt -module-summary %p/Inputs/weak_resolution.ll -o %t2.bc
 ; RUN: llvm-lto -thinlto-action=thinlink -o %t3.bc %t.bc %t2.bc
 
 ; Verify that prevailing weak for linker symbol is selected across modules,
 ; non-prevailing ODR are not kept when possible, but non-ODR non-prevailing
 ; are not affected.
 ; RUN: llvm-lto -thinlto-action=promote %t.bc -thinlto-index=%t3.bc -o - | llvm-dis -o - | FileCheck %s --check-prefix=MOD1
-; RUN: llvm-lto -thinlto-action=promote %t.bc -thinlto-index=%t3.bc -exported-symbol=linkoncefunc -o - | llvm-lto -thinlto-action=internalize -thinlto-module-id=%t.bc - -thinlto-index=%t3.bc -exported-symbol=linkoncefunc -o - | llvm-dis -o - | FileCheck %s --check-prefix=MOD1-INT
+; RUN: llvm-lto -thinlto-action=internalize %t.bc -thinlto-index=%t3.bc -exported-symbol=linkoncefunc -o - | llvm-dis -o - | FileCheck %s --check-prefix=MOD1-INT
 ; RUN: llvm-lto -thinlto-action=promote %t2.bc -thinlto-index=%t3.bc -o - | llvm-dis -o - | FileCheck %s --check-prefix=MOD2
 ; When exported, we always preserve a linkonce
 ; RUN: llvm-lto -thinlto-action=promote %t.bc -thinlto-index=%t3.bc -o - --exported-symbol=linkonceodrfuncInSingleModule | llvm-dis -o - | FileCheck %s --check-prefix=EXPORTED
 
 target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-apple-macosx10.11.0"
 
 ; Alias are resolved, but can't be turned into "available_externally"

llvm/trunk/test/ThinLTO/X86/weak_resolution_single.ll

 ; RUN: opt -module-summary %s -o %t.bc
 ; RUN: llvm-lto -thinlto-action=thinlink -o %t2.bc %t.bc
 
-; RUN: llvm-lto -thinlto-action=promote %t.bc -thinlto-index=%t2.bc -exported-symbol=foo -o - | llvm-lto -thinlto-action=internalize -thinlto-module-id=%t.bc - -thinlto-index=%t2.bc -exported-symbol=foo -o - | llvm-dis -o - | FileCheck %s
+; RUN: llvm-lto -thinlto-action=internalize %t.bc -thinlto-index=%t2.bc -exported-symbol=foo -o - | llvm-dis -o - | FileCheck %s
 
 ; CHECK: define weak_odr void @foo()
+target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
 define linkonce_odr void @foo() {
   ret void
 }

llvm/trunk/test/Transforms/ThinLTOBitcodeWriter/no-type-md.ll

 ; Copy the minimized bitcode to the regular bitcode path so the module
 ; paths in the index are the same.
 ; RUN: cp %t.thinlink.bc %t.bc
 ; RUN: llvm-lto -thinlto-action=thinlink -o %t4.thinlto.bc %t.bc
 ; RUN: diff %t3.thinlto.bc %t4.thinlto.bc
 
 ; BCA: <GLOBALVAL_SUMMARY_BLOCK
 
+target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
+
 ; CHECK: @g = global i8 42
 @g = global i8 42
 
 ; CHECK: define void @f()
 define void @f() {
   ret void
 }

llvm/trunk/tools/llvm-lto/llvm-lto.cpp

 std::unique_ptr<ModuleSummaryIndex> loadCombinedIndex() {
   if (ThinLTOIndex.empty())
     report_fatal_error("Missing -thinlto-index for ThinLTO promotion stage");
   ExitOnError ExitOnErr("llvm-lto: error loading file '" + ThinLTOIndex +
                         "': ");
   return ExitOnErr(getModuleSummaryIndexForFile(ThinLTOIndex));
 }
 
-static std::unique_ptr<Module> loadModule(StringRef Filename,
-                                          LLVMContext &Ctx) {
-  SMDiagnostic Err;
-  std::unique_ptr<Module> M(parseIRFile(Filename, Err, Ctx));
-  if (!M) {
-    Err.print("llvm-lto", errs());
-    report_fatal_error("Can't load module for file " + Filename);
-  }
-  maybeVerifyModule(*M);
+static std::unique_ptr<MemoryBuffer> loadFile(StringRef Filename) {
+  ExitOnError ExitOnErr("llvm-lto: error loading file '" + Filename.str() +
+                        "': ");
+  return ExitOnErr(errorOrToExpected(MemoryBuffer::getFileOrSTDIN(Filename)));
 
+static std::unique_ptr<lto::InputFile> loadInputFile(MemoryBufferRef Buffer) {
+  ExitOnError ExitOnErr("llvm-lto: error loading input '" +
+                        Buffer.getBufferIdentifier().str() + "': ");
+  return ExitOnErr(lto::InputFile::create(Buffer));
+}
+
+static std::unique_ptr<Module> loadModuleFromInput(lto::InputFile &File,
+                                                   LLVMContext &CTX) {
+  auto &Mod = File.getSingleBitcodeModule();
+  auto ModuleOrErr = Mod.parseModule(CTX);
+  if (!ModuleOrErr) {
+    handleAllErrors(ModuleOrErr.takeError(), [&](ErrorInfoBase &EIB) {
+      SMDiagnostic Err = SMDiagnostic(Mod.getModuleIdentifier(),
+                                      SourceMgr::DK_Error, EIB.message());
+      Err.print("llvm-lto", errs());
+    });
+    report_fatal_error("Can't load module, abort.");
+  }
+  maybeVerifyModule(**ModuleOrErr);
   if (ThinLTOModuleId.getNumOccurrences()) {
     if (InputFilenames.size() != 1)
       report_fatal_error("Can't override the module id for multiple files");
-    M->setModuleIdentifier(ThinLTOModuleId);
+    (*ModuleOrErr)->setModuleIdentifier(ThinLTOModuleId);
   }
-  return M;
+  return std::move(*ModuleOrErr);
 }
 
 static void writeModuleToFile(Module &TheModule, StringRef Filename) {
   std::error_code EC;
   raw_fd_ostream OS(Filename, EC, sys::fs::OpenFlags::F_None);
   error(EC, "error opening the file '" + Filename + "'");
   maybeVerifyModule(TheModule);
   WriteBitcodeToFile(TheModule, OS, /* ShouldPreserveUseListOrder */ true);
if (InputFilenames.size() != 1 && !OutputFilename.empty())
                          "ones.");
 
     std::string OldPrefix, NewPrefix;
     getThinLTOOldAndNewPrefix(OldPrefix, NewPrefix);
 
     auto Index = loadCombinedIndex();
     for (auto &Filename : InputFilenames) {
       LLVMContext Ctx;
-      auto TheModule = loadModule(Filename, Ctx);
+      auto Buffer = loadFile(Filename);
+      auto Input = loadInputFile(Buffer->getMemBufferRef());
+      auto TheModule = loadModuleFromInput(*Input, Ctx);
 
       // Build a map of module to the GUIDs and summary objects that should
       // be written to its index.
       std::map<std::string, GVSummaryMapTy> ModuleToSummariesForIndex;
-      ThinGenerator.gatherImportedSummariesForModule(*TheModule, *Index,
-                                                     ModuleToSummariesForIndex);
+      ThinGenerator.gatherImportedSummariesForModule(
+          *TheModule, *Index, ModuleToSummariesForIndex, *Input);
 
       std::string OutputName = OutputFilename;
       if (OutputName.empty()) {
         OutputName = Filename + ".thinlto.bc";
       }
       OutputName = getThinLTOOutputFile(OutputName, OldPrefix, NewPrefix);
       std::error_code EC;
       raw_fd_ostream OS(OutputName, EC, sys::fs::OpenFlags::F_None);
if (InputFilenames.size() != 1 && !OutputFilename.empty())
                          "ones.");
 
     std::string OldPrefix, NewPrefix;
     getThinLTOOldAndNewPrefix(OldPrefix, NewPrefix);
 
     auto Index = loadCombinedIndex();
     for (auto &Filename : InputFilenames) {
       LLVMContext Ctx;
-      auto TheModule = loadModule(Filename, Ctx);
+      auto Buffer = loadFile(Filename);
+      auto Input = loadInputFile(Buffer->getMemBufferRef());
+      auto TheModule = loadModuleFromInput(*Input, Ctx);
       std::string OutputName = OutputFilename;
       if (OutputName.empty()) {
         OutputName = Filename + ".imports";
       }
-      OutputName = getThinLTOOutputFile(OutputName, OldPrefix, NewPrefix);
-      ThinGenerator.emitImports(*TheModule, OutputName, *Index);
+      OutputName =
+          getThinLTOOutputFile(OutputName, OldPrefix, NewPrefix);
+      ThinGenerator.emitImports(*TheModule, OutputName, *Index, *Input);
     }
   }
 
   /// Load the combined index from disk, then load every file referenced by
   /// the index and add them to the generator, finally perform the promotion
   /// on the files mentioned on the command line (these must match the index
   /// content).
   void promote() {
     if (InputFilenames.size() != 1 && !OutputFilename.empty())
       report_fatal_error("Can't handle a single output filename and multiple "
                          "input files, do not provide an output filename and "
                          "the output files will be suffixed from the input "
                          "ones.");
 
     auto Index = loadCombinedIndex();
     for (auto &Filename : InputFilenames) {
       LLVMContext Ctx;
-      auto TheModule = loadModule(Filename, Ctx);
+      auto Buffer = loadFile(Filename);
+      auto Input = loadInputFile(Buffer->getMemBufferRef());
+      auto TheModule = loadModuleFromInput(*Input, Ctx);
 
-      ThinGenerator.promote(*TheModule, *Index);
+      ThinGenerator.promote(*TheModule, *Index, *Input);
 
       std::string OutputName = OutputFilename;
       if (OutputName.empty()) {
         OutputName = Filename + ".thinlto.promoted.bc";
       }
       writeModuleToFile(*TheModule, OutputName);
     }
   }
void import() {
     auto Index = loadCombinedIndex();
     auto InputBuffers = loadAllFilesForIndex(*Index);
     for (auto &MemBuffer : InputBuffers)
       ThinGenerator.addModule(MemBuffer->getBufferIdentifier(),
                               MemBuffer->getBuffer());
 
     for (auto &Filename : InputFilenames) {
       LLVMContext Ctx;
-      auto TheModule = loadModule(Filename, Ctx);
+      auto Buffer = loadFile(Filename);
+      auto Input = loadInputFile(Buffer->getMemBufferRef());
+      auto TheModule = loadModuleFromInput(*Input, Ctx);
 
-      ThinGenerator.crossModuleImport(*TheModule, *Index);
+      ThinGenerator.crossModuleImport(*TheModule, *Index, *Input);
 
       std::string OutputName = OutputFilename;
       if (OutputName.empty()) {
         OutputName = Filename + ".thinlto.imported.bc";
       }
       writeModuleToFile(*TheModule, OutputName);
     }
   }
void internalize() {
     auto Index = loadCombinedIndex();
     auto InputBuffers = loadAllFilesForIndex(*Index);
     for (auto &MemBuffer : InputBuffers)
       ThinGenerator.addModule(MemBuffer->getBufferIdentifier(),
                               MemBuffer->getBuffer());
 
     for (auto &Filename : InputFilenames) {
       LLVMContext Ctx;
-      auto TheModule = loadModule(Filename, Ctx);
+      auto Buffer = loadFile(Filename);
+      auto Input = loadInputFile(Buffer->getMemBufferRef());
+      auto TheModule = loadModuleFromInput(*Input, Ctx);
 
-      ThinGenerator.internalize(*TheModule, *Index);
+      ThinGenerator.internalize(*TheModule, *Index, *Input);
 
       std::string OutputName = OutputFilename;
       if (OutputName.empty()) {
         OutputName = Filename + ".thinlto.internalized.bc";
       }
       writeModuleToFile(*TheModule, OutputName);
     }
   }
 
   void optimize() {
     if (InputFilenames.size() != 1 && !OutputFilename.empty())
       report_fatal_error("Can't handle a single output filename and multiple "
                          "input files, do not provide an output filename and "
                          "the output files will be suffixed from the input "
                          "ones.");
     if (!ThinLTOIndex.empty())
       errs() << "Warning: -thinlto-index ignored for optimize stage";
 
     for (auto &Filename : InputFilenames) {
       LLVMContext Ctx;
-      auto TheModule = loadModule(Filename, Ctx);
+      auto Buffer = loadFile(Filename);
+      auto Input = loadInputFile(Buffer->getMemBufferRef());
+      auto TheModule = loadModuleFromInput(*Input, Ctx);
 
       ThinGenerator.optimize(*TheModule);
 
       std::string OutputName = OutputFilename;
       if (OutputName.empty()) {
         OutputName = Filename + ".thinlto.imported.bc";
       }
       writeModuleToFile(*TheModule, OutputName);