[ThinLTO] Import composite types as declarations

[tejohnson]

Summary

When reading the metadata bitcode, create a type declaration when
possible for composite types when we are importing. Doing this in
the bitcode reader saves memory. Also it works naturally in the case
when the type ODR map contains a definition for the same composite type
because it was used in the importing module (buildODRType will
automatically use the existing definition and not create a type
declaration).

For Chromium built with -g2, this reduces the aggregate size of the
generated native object files by 66% (from 31G to 10G). It reduced
the time through the ThinLTO link and backend phases by about 20% on
my machine.

[aprantl]

This sounds like it could break debugging with LLDB. If I understand the description correctly, this is the equivalent of doing -fno-standalone-debug?

[mehdi_amini]

The patch does not seem to apply cleanly?

[tejohnson]

Rebase

[tejohnson]

In D27775#623051, @mehdi_amini wrote:

The patch does not seem to apply cleanly?

Should be fixed by my rebase - there was a merge conflict in LTOBackend.cpp when I tried updating initially.

[aprantl]

Mehdi and I just ran a couple of experiments. LLDB can find forward-declared composite types that are defined in another .o file just fine; the only caveat is that LLDB has to potentially load the accelerator table of each .o file from disk in order to find the .o file with the definition in it. Judging from this, I think that this approach is fine.

[tejohnson]

In D27775#623768, @aprantl wrote:

Mehdi and I just ran a couple of experiments. LLDB can find forward-declared composite types that are defined in another .o file just fine; the only caveat is that LLDB has to potentially load the accelerator table of each .o file from disk in order to find the .o file with the definition in it. Judging from this, I think that this approach is fine.

Does the patch look ok to commit?

[mehdi_amini]

lib/Bitcode/Reader/MetadataLoader.cpp

733 ↗

(On Diff #81513)

Can you avoid calling all the getMD and so above in the case you're nullifying them?
As I'm working on lazy-loading, this would still trigger the load from bitcode and create the MD into memory even if you don't attach them in the MD graph.

[mehdi_amini]

Otherwise, LGTM.

[aprantl]

Yes this is fine.

[tejohnson]

lib/Bitcode/Reader/MetadataLoader.cpp

733 ↗

(On Diff #81513)

Good idea, done.

[tejohnson]

• Avoid getMD calls if we aren't going to use the created metadata
• Add llvm-lto2 based test
• Rebase

[aprantl]

llvm/trunk/lib/Bitcode/Reader/MetadataLoader.cpp

750

Sorry for bringing this up so late, but I just realized that this is only legal if the Identifier field is nonempty. The ODR (or an equivalent rule in the underlying source language) only applies to types with an identifier field. In C, for example, it is legal for multiple translation units to contain definitions of types with the same name, but a different layout. Using just the name it is impossible for the debugger to find the correct definition of a forward-declared type in the other translation units.

[tejohnson]

llvm/trunk/lib/Bitcode/Reader/MetadataLoader.cpp

750

Ok, I can make the decl optimization conditional on the Identifier being non-null. I assume that is not the common case anyway - in which case that change shouldn't hurt the overall impact much?

[tejohnson]

llvm/trunk/lib/Bitcode/Reader/MetadataLoader.cpp

750

r290915

On Chromium, the fix increases the aggregate -g2 object file sizes with ThinLTO by almost 4%, but it just means a 65% reduction over the object files before this patch, instead of the original 66% reduction.

llvm/trunk/include/llvm/Bitcode/BitcodeReader.h

BitcodeModule(ArrayRef<uint8_t> Buffer, StringRef ModuleIdentifier,
                   uint64_t IdentificationBit, uint64_t ModuleBit)
         : Buffer(Buffer), ModuleIdentifier(ModuleIdentifier),
           IdentificationBit(IdentificationBit), ModuleBit(ModuleBit) {}
 
     // Calls the ctor.
     friend Expected<std::vector<BitcodeModule>>
     getBitcodeModuleList(MemoryBufferRef Buffer);
 
-    Expected<std::unique_ptr<Module>>
-    getModuleImpl(LLVMContext &Context, bool MaterializeAll,
-                  bool ShouldLazyLoadMetadata);
+    Expected<std::unique_ptr<Module>> getModuleImpl(LLVMContext &Context,
+                                                    bool MaterializeAll,
+                                                    bool ShouldLazyLoadMetadata,
+                                                    bool IsImporting);
 
   public:
     StringRef getBuffer() const {
       return StringRef((const char *)Buffer.begin(), Buffer.size());
     }
 
     StringRef getModuleIdentifier() const { return ModuleIdentifier; }
 
     /// Read the bitcode module and prepare for lazy deserialization of function
     /// bodies. If ShouldLazyLoadMetadata is true, lazily load metadata as well.
-    Expected<std::unique_ptr<Module>>
-    getLazyModule(LLVMContext &Context, bool ShouldLazyLoadMetadata);
+    /// If IsImporting is true, this module is being parsed for ThinLTO
+    /// importing into another module.
+    Expected<std::unique_ptr<Module>> getLazyModule(LLVMContext &Context,
+                                                    bool ShouldLazyLoadMetadata,
+                                                    bool IsImporting);
 
     /// Read the entire bitcode module and return it.
     Expected<std::unique_ptr<Module>> parseModule(LLVMContext &Context);
 
     /// Check if the given bitcode buffer contains a summary block.
     Expected<bool> hasSummary();
 
     /// Parse the specified bitcode buffer, returning the module summary index.
     Expected<std::unique_ptr<ModuleSummaryIndex>> getSummary();
   };
 
   /// Returns a list of modules in the specified bitcode buffer.
   Expected<std::vector<BitcodeModule>>
   getBitcodeModuleList(MemoryBufferRef Buffer);
 
   /// Read the header of the specified bitcode buffer and prepare for lazy
   /// deserialization of function bodies. If ShouldLazyLoadMetadata is true,
-  /// lazily load metadata as well.
+  /// lazily load metadata as well. If IsImporting is true, this module is
+  /// being parsed for ThinLTO importing into another module.
   Expected<std::unique_ptr<Module>>
   getLazyBitcodeModule(MemoryBufferRef Buffer, LLVMContext &Context,
-                       bool ShouldLazyLoadMetadata = false);
+                       bool ShouldLazyLoadMetadata = false,
+                       bool IsImporting = false);
 
   /// Like getLazyBitcodeModule, except that the module takes ownership of
   /// the memory buffer if successful. If successful, this moves Buffer. On
-  /// error, this *does not* move Buffer.
-  Expected<std::unique_ptr<Module>>
-  getOwningLazyBitcodeModule(std::unique_ptr<MemoryBuffer> &&Buffer,
-                             LLVMContext &Context,
-                             bool ShouldLazyLoadMetadata = false);
+  /// error, this *does not* move Buffer. If IsImporting is true, this module is
+  /// being parsed for ThinLTO importing into another module.
+  Expected<std::unique_ptr<Module>> getOwningLazyBitcodeModule(
+      std::unique_ptr<MemoryBuffer> &&Buffer, LLVMContext &Context,
+      bool ShouldLazyLoadMetadata = false, bool IsImporting = false);
 
   /// Read the header of the specified bitcode buffer and extract just the
   /// triple information. If successful, this returns a string. On error, this
   /// returns "".
   Expected<std::string> getBitcodeTargetTriple(MemoryBufferRef Buffer);
 
   /// Return true if \p Buffer contains a bitcode file with ObjC code (category
   /// or class) in it.

llvm/trunk/lib/Bitcode/Reader/BitcodeReader.cpp

public:
   Error materializeForwardReferencedFunctions();
 
   Error materialize(GlobalValue *GV) override;
   Error materializeModule() override;
   std::vector<StructType *> getIdentifiedStructTypes() const override;
 
   /// \brief Main interface to parsing a bitcode buffer.
   /// \returns true if an error occurred.
-  Error parseBitcodeInto(Module *M, bool ShouldLazyLoadMetadata = false);
+  Error parseBitcodeInto(Module *M, bool ShouldLazyLoadMetadata = false,
+                         bool IsImporting = false);
 
   static uint64_t decodeSignRotatedValue(uint64_t V);
 
   /// Materialize any deferred Metadata block.
   Error materializeMetadata() override;
 
   void setStripDebugInfo() override;
 
case bitc::MODULE_CODE_SOURCE_FILENAME:
         return error("Invalid record");
       TheModule->setSourceFileName(ValueName);
       break;
     }
     Record.clear();
   }
 }
 
-Error BitcodeReader::parseBitcodeInto(Module *M, bool ShouldLazyLoadMetadata) {
+Error BitcodeReader::parseBitcodeInto(Module *M, bool ShouldLazyLoadMetadata,
+                                      bool IsImporting) {
   TheModule = M;
-  MDLoader = MetadataLoader(Stream, *M, ValueList,
+  MDLoader = MetadataLoader(Stream, *M, ValueList, IsImporting,
                             [&](unsigned ID) { return getTypeByID(ID); });
   return parseModule(0, ShouldLazyLoadMetadata);
 }
 
 
 Error BitcodeReader::typeCheckLoadStoreInst(Type *ValType, Type *PtrType) {
   if (!isa<PointerType>(PtrType))
     return error("Load/Store operand is not a pointer type");

 /// This isn't always used in a lazy context.  In particular, it's also used by
 /// \a parseModule().  If this is truly lazy, then we need to eagerly pull
 /// in forward-referenced functions from block address references.
 ///
 /// \param[in] MaterializeAll Set to \c true if we should materialize
 /// everything.
 Expected<std::unique_ptr<Module>>
 BitcodeModule::getModuleImpl(LLVMContext &Context, bool MaterializeAll,
-                             bool ShouldLazyLoadMetadata) {
+                             bool ShouldLazyLoadMetadata, bool IsImporting) {
   BitstreamCursor Stream(Buffer);
 
   std::string ProducerIdentification;
   if (IdentificationBit != -1ull) {
     Stream.JumpToBit(IdentificationBit);
     Expected<std::string> ProducerIdentificationOrErr =
         readIdentificationBlock(Stream);
     if (!ProducerIdentificationOrErr)
       return ProducerIdentificationOrErr.takeError();
 
     ProducerIdentification = *ProducerIdentificationOrErr;
   }
 
   Stream.JumpToBit(ModuleBit);
   auto *R =
       new BitcodeReader(std::move(Stream), ProducerIdentification, Context);
 
   std::unique_ptr<Module> M =
       llvm::make_unique<Module>(ModuleIdentifier, Context);
   M->setMaterializer(R);
 
   // Delay parsing Metadata if ShouldLazyLoadMetadata is true.
-  if (Error Err = R->parseBitcodeInto(M.get(), ShouldLazyLoadMetadata))
+  if (Error Err =
+          R->parseBitcodeInto(M.get(), ShouldLazyLoadMetadata, IsImporting))
     return std::move(Err);
 
   if (MaterializeAll) {
     // Read in the entire module, and destroy the BitcodeReader.
     if (Error Err = M->materializeAll())
       return std::move(Err);
   } else {
     // Resolve forward references from blockaddresses.
     if (Error Err = R->materializeForwardReferencedFunctions())
       return std::move(Err);
   }
   return std::move(M);
 }
 
 Expected<std::unique_ptr<Module>>
-BitcodeModule::getLazyModule(LLVMContext &Context,
-                             bool ShouldLazyLoadMetadata) {
-  return getModuleImpl(Context, false, ShouldLazyLoadMetadata);
+BitcodeModule::getLazyModule(LLVMContext &Context, bool ShouldLazyLoadMetadata,
+                             bool IsImporting) {
+  return getModuleImpl(Context, false, ShouldLazyLoadMetadata, IsImporting);
 }
 
 // Parse the specified bitcode buffer, returning the function info index.
 Expected<std::unique_ptr<ModuleSummaryIndex>> BitcodeModule::getSummary() {
   BitstreamCursor Stream(Buffer);
   Stream.JumpToBit(ModuleBit);
 
   auto Index = llvm::make_unique<ModuleSummaryIndex>();
static Expected<BitcodeModule> getSingleModule(MemoryBufferRef Buffer) {
 
   if (MsOrErr->size() != 1)
     return error("Expected a single module");
 
   return (*MsOrErr)[0];
 }
 
 Expected<std::unique_ptr<Module>>
-llvm::getLazyBitcodeModule(MemoryBufferRef Buffer,
-                           LLVMContext &Context, bool ShouldLazyLoadMetadata) {
+llvm::getLazyBitcodeModule(MemoryBufferRef Buffer, LLVMContext &Context,
+                           bool ShouldLazyLoadMetadata, bool IsImporting) {
   Expected<BitcodeModule> BM = getSingleModule(Buffer);
   if (!BM)
     return BM.takeError();
 
-  return BM->getLazyModule(Context, ShouldLazyLoadMetadata);
+  return BM->getLazyModule(Context, ShouldLazyLoadMetadata, IsImporting);
 }
 
-Expected<std::unique_ptr<Module>>
-llvm::getOwningLazyBitcodeModule(std::unique_ptr<MemoryBuffer> &&Buffer,
-                                 LLVMContext &Context,
-                                 bool ShouldLazyLoadMetadata) {
-  auto MOrErr = getLazyBitcodeModule(*Buffer, Context, ShouldLazyLoadMetadata);
+Expected<std::unique_ptr<Module>> llvm::getOwningLazyBitcodeModule(
+    std::unique_ptr<MemoryBuffer> &&Buffer, LLVMContext &Context,
+    bool ShouldLazyLoadMetadata, bool IsImporting) {
+  auto MOrErr = getLazyBitcodeModule(*Buffer, Context, ShouldLazyLoadMetadata,
+                                     IsImporting);
   if (MOrErr)
     (*MOrErr)->setOwnedMemoryBuffer(std::move(Buffer));
   return MOrErr;
 }
 
 Expected<std::unique_ptr<Module>>
 BitcodeModule::parseModule(LLVMContext &Context) {
-  return getModuleImpl(Context, true, false);
+  return getModuleImpl(Context, true, false, false);
   // TODO: Restore the use-lists to the in-memory state when the bitcode was
   // written.  We must defer until the Module has been fully materialized.
 }
 
 Expected<std::unique_ptr<Module>> llvm::parseBitcodeFile(MemoryBufferRef Buffer,
                                                          LLVMContext &Context) {
   Expected<BitcodeModule> BM = getSingleModule(Buffer);
   if (!BM)

llvm/trunk/lib/Bitcode/Reader/MetadataLoader.h

 class MDNode;
 class Module;
 class Type;
 
 /// Helper class that handles loading Metadatas and keeping them available.
 class MetadataLoader {
   class MetadataLoaderImpl;
   std::unique_ptr<MetadataLoaderImpl> Pimpl;
+  /// True if metadata is being parsed for a module being ThinLTO imported.
+  bool IsImporting = false;
   Error parseMetadata(bool ModuleLevel);
 
 public:
   ~MetadataLoader();
   MetadataLoader(BitstreamCursor &Stream, Module &TheModule,
-                 BitcodeReaderValueList &ValueList,
+                 BitcodeReaderValueList &ValueList, bool IsImporting,
                  std::function<Type *(unsigned)> getTypeByID);
   MetadataLoader &operator=(MetadataLoader &&);
   MetadataLoader(MetadataLoader &&);
 
   // Parse a module metadata block
   Error parseModuleMetadata() { return parseMetadata(true); }
 
   // Parse a function metadata block

llvm/trunk/lib/Bitcode/Reader/MetadataLoader.cpp

 #include <string>
 #include <system_error>
 #include <tuple>
 #include <utility>
 #include <vector>
 
 using namespace llvm;
 
+/// Flag whether we need to import full type definitions for ThinLTO.
+/// Currently needed for Darwin and LLDB.
+static cl::opt<bool> ImportFullTypeDefinitions(
+    "import-full-type-definitions", cl::init(false), cl::Hidden,
+    cl::desc("Import full type definitions for ThinLTO."));
+
 namespace {
 
 static int64_t unrotateSign(uint64_t U) { return U & 1 ? ~(U >> 1) : U >> 1; }
 
 class BitcodeReaderMetadataList {
   unsigned NumFwdRefs;
   bool AnyFwdRefs;
   unsigned MinFwdRef;

 public:
   MetadataLoaderImpl(BitstreamCursor &Stream, Module &TheModule,
                      BitcodeReaderValueList &ValueList,
                      std::function<Type *(unsigned)> getTypeByID)
       : MetadataList(TheModule.getContext()), ValueList(ValueList),
         Stream(Stream), Context(TheModule.getContext()), TheModule(TheModule),
         getTypeByID(getTypeByID) {}
 
-  Error parseMetadata(bool ModuleLevel);
+  Error parseMetadata(bool ModuleLevel, bool IsImporting);
 
   bool hasFwdRefs() const { return MetadataList.hasFwdRefs(); }
   Metadata *getMetadataFwdRef(unsigned Idx) {
     return MetadataList.getMetadataFwdRef(Idx);
   }
 
   MDNode *getMDNodeFwdRefOrNull(unsigned Idx) {
     return MetadataList.getMDNodeFwdRefOrNull(Idx);

 
 Error error(const Twine &Message) {
   return make_error<StringError>(
       Message, make_error_code(BitcodeError::CorruptedBitcode));
 }
 
 /// Parse a METADATA_BLOCK. If ModuleLevel is true then we are parsing
 /// module level metadata.
-Error MetadataLoader::MetadataLoaderImpl::parseMetadata(bool ModuleLevel) {
+Error MetadataLoader::MetadataLoaderImpl::parseMetadata(bool ModuleLevel,
+                                                        bool IsImporting) {
   if (!ModuleLevel && MetadataList.hasFwdRefs())
     return error("Invalid metadata: fwd refs into function blocks");
 
   if (Stream.EnterSubBlock(bitc::METADATA_BLOCK_ID))
     return error("Invalid record");
 
   unsigned NextMetadataNo = MetadataList.size();
   std::vector<std::pair<DICompileUnit *, Metadata *>> CUSubprograms;
case bitc::METADATA_COMPOSITE_TYPE: {
       // mapping.
       IsDistinct = Record[0] & 0x1;
       bool IsNotUsedInTypeRef = Record[0] >= 2;
       unsigned Tag = Record[1];
       MDString *Name = getMDString(Record[2]);
       Metadata *File = getMDOrNull(Record[3]);
       unsigned Line = Record[4];
       Metadata *Scope = getDITypeRefOrNull(Record[5]);
-      Metadata *BaseType = getDITypeRefOrNull(Record[6]);
+      Metadata *BaseType = nullptr;
       uint64_t SizeInBits = Record[7];
       if (Record[8] > (uint64_t)std::numeric_limits<uint32_t>::max())
         return error("Alignment value is too large");
       uint32_t AlignInBits = Record[8];
-      uint64_t OffsetInBits = Record[9];
+      uint64_t OffsetInBits = 0;
       DINode::DIFlags Flags = static_cast<DINode::DIFlags>(Record[10]);
-      Metadata *Elements = getMDOrNull(Record[11]);
+      Metadata *Elements = nullptr;
       unsigned RuntimeLang = Record[12];
-      Metadata *VTableHolder = getDITypeRefOrNull(Record[13]);
-      Metadata *TemplateParams = getMDOrNull(Record[14]);
+      Metadata *VTableHolder = nullptr;
+      Metadata *TemplateParams = nullptr;
       auto *Identifier = getMDString(Record[15]);
+      // If this module is being parsed so that it can be ThinLTO imported
+      // into another module, composite types only need to be imported
+      // as type declarations (unless full type definitions requested).
+      // Create type declarations up front to save memory. Also, buildODRType
+      // handles the case where this is type ODRed with a definition needed
+      // by the importing module, in which case the existing definition is
+      // used.
+      if (IsImporting && !ImportFullTypeDefinitions &&
+          (Tag == dwarf::DW_TAG_enumeration_type ||
+           Tag == dwarf::DW_TAG_class_type ||
+           Tag == dwarf::DW_TAG_structure_type ||
+           Tag == dwarf::DW_TAG_union_type)) {
+        Flags = Flags | DINode::FlagFwdDecl;
+      } else {
+        BaseType = getDITypeRefOrNull(Record[6]);
+        OffsetInBits = Record[9];
+        Elements = getMDOrNull(Record[11]);
+        VTableHolder = getDITypeRefOrNull(Record[13]);
+        TemplateParams = getMDOrNull(Record[14]);
+      }
       DICompositeType *CT = nullptr;
       if (Identifier)
         CT = DICompositeType::buildODRType(
             Context, *Identifier, Tag, Name, File, Line, Scope, BaseType,
             SizeInBits, AlignInBits, OffsetInBits, Flags, Elements, RuntimeLang,
             VTableHolder, TemplateParams);
 
       // Create a node if we didn't get a lazy ODR type.
Error MetadataLoader::MetadataLoaderImpl::parseMetadataKinds() {
   }
 }
 
 MetadataLoader &MetadataLoader::operator=(MetadataLoader &&RHS) {
   Pimpl = std::move(RHS.Pimpl);
   return *this;
 }
 MetadataLoader::MetadataLoader(MetadataLoader &&RHS)
-    : Pimpl(std::move(RHS.Pimpl)) {}
+    : Pimpl(std::move(RHS.Pimpl)), IsImporting(RHS.IsImporting) {}
 
 MetadataLoader::~MetadataLoader() = default;
 MetadataLoader::MetadataLoader(BitstreamCursor &Stream, Module &TheModule,
                                BitcodeReaderValueList &ValueList,
+                               bool IsImporting,
                                std::function<Type *(unsigned)> getTypeByID)
     : Pimpl(llvm::make_unique<MetadataLoaderImpl>(Stream, TheModule, ValueList,
-                                                  getTypeByID)) {}
+                                                  getTypeByID)),
+      IsImporting(IsImporting) {}
 
 Error MetadataLoader::parseMetadata(bool ModuleLevel) {
-  return Pimpl->parseMetadata(ModuleLevel);
+  return Pimpl->parseMetadata(ModuleLevel, IsImporting);
 }
 
 bool MetadataLoader::hasFwdRefs() const { return Pimpl->hasFwdRefs(); }
 
 /// Return the given metadata, creating a replaceable forward reference if
 /// necessary.
 Metadata *MetadataLoader::getMetadataFwdRef(unsigned Idx) {
   return Pimpl->getMetadataFwdRef(Idx);

llvm/trunk/lib/LTO/LTO.cpp

Expected<std::unique_ptr<InputFile>> InputFile::create(MemoryBufferRef Object) {
   if (BMsOrErr->empty())
     return make_error<StringError>("Bitcode file does not contain any modules",
                                    inconvertibleErrorCode());
 
   // Create an InputModule for each module in the InputFile, and add it to the
   // ModuleSymbolTable.
   for (auto BM : *BMsOrErr) {
     Expected<std::unique_ptr<Module>> MOrErr =
-        BM.getLazyModule(File->Ctx, /*ShouldLazyLoadMetadata*/ true);
+        BM.getLazyModule(File->Ctx, /*ShouldLazyLoadMetadata*/ true,
+                         /*IsImporting*/ false);
     if (!MOrErr)
       return MOrErr.takeError();
 
     size_t SymBegin = File->SymTab.symbols().size();
     File->SymTab.addModule(MOrErr->get());
     size_t SymEnd = File->SymTab.symbols().size();
 
     for (const auto &C : (*MOrErr)->getComdatSymbolTable()) {

 Error LTO::addRegularLTO(BitcodeModule BM, const SymbolResolution *&ResI,
                          const SymbolResolution *ResE) {
   if (!RegularLTO.CombinedModule) {
     RegularLTO.CombinedModule =
         llvm::make_unique<Module>("ld-temp.o", RegularLTO.Ctx);
     RegularLTO.Mover = llvm::make_unique<IRMover>(*RegularLTO.CombinedModule);
   }
   Expected<std::unique_ptr<Module>> MOrErr =
-      BM.getLazyModule(RegularLTO.Ctx, /*ShouldLazyLoadMetadata*/ true);
+      BM.getLazyModule(RegularLTO.Ctx, /*ShouldLazyLoadMetadata*/ true,
+                       /*IsImporting*/ false);
   if (!MOrErr)
     return MOrErr.takeError();
 
   Module &M = **MOrErr;
   if (Error Err = M.materializeMetadata())
     return Err;
   UpgradeDebugInfo(M);
 

llvm/trunk/lib/LTO/LTOBackend.cpp

if (Conf.PostInternalizeModuleHook &&
     return Error::success();
 
   auto ModuleLoader = [&](StringRef Identifier) {
     assert(Mod.getContext().isODRUniquingDebugTypes() &&
            "ODR Type uniquing should be enabled on the context");
     auto I = ModuleMap.find(Identifier);
     assert(I != ModuleMap.end());
     return I->second.getLazyModule(Mod.getContext(),
-                                   /*ShouldLazyLoadMetadata=*/true);
+                                   /*ShouldLazyLoadMetadata=*/true,
+                                   /*IsImporting*/ true);
   };
 
   FunctionImporter Importer(CombinedIndex, ModuleLoader);
   if (Error Err = Importer.importFunctions(Mod, ImportList).takeError())
     return Err;
 
   if (Conf.PostImportModuleHook && !Conf.PostImportModuleHook(Task, Mod))
     return Error::success();
 
   if (!opt(Conf, TM.get(), Task, Mod, /*IsThinLTO=*/true))
     return Error::success();
 
   codegen(Conf, TM.get(), AddStream, Task, Mod);
   return Error::success();
 }

llvm/trunk/lib/LTO/ThinLTOCodeGenerator.cpp

 
 static void promoteModule(Module &TheModule, const ModuleSummaryIndex &Index) {
   if (renameModuleForThinLTO(TheModule, Index))
     report_fatal_error("renameModuleForThinLTO failed");
 }
 
 static std::unique_ptr<Module>
 loadModuleFromBuffer(const MemoryBufferRef &Buffer, LLVMContext &Context,
-                     bool Lazy) {
+                     bool Lazy, bool IsImporting) {
   SMDiagnostic Err;
   Expected<std::unique_ptr<Module>> ModuleOrErr =
-      Lazy ? getLazyBitcodeModule(Buffer, Context,
-                                  /* ShouldLazyLoadMetadata */ true)
-           : parseBitcodeFile(Buffer, Context);
+      Lazy
+          ? getLazyBitcodeModule(Buffer, Context,
+                                 /* ShouldLazyLoadMetadata */ true, IsImporting)
   if (!ModuleOrErr) {
     handleAllErrors(ModuleOrErr.takeError(), [&](ErrorInfoBase &EIB) {
       SMDiagnostic Err = SMDiagnostic(Buffer.getBufferIdentifier(),
                                       SourceMgr::DK_Error, EIB.message());
       Err.print("ThinLTO", errs());
     });
     report_fatal_error("Can't load module, abort.");
   }
   return std::move(ModuleOrErr.get());
 }
 
 static void
 crossImportIntoModule(Module &TheModule, const ModuleSummaryIndex &Index,
                       StringMap<MemoryBufferRef> &ModuleMap,
                       const FunctionImporter::ImportMapTy &ImportList) {
   auto Loader = [&](StringRef Identifier) {
     return loadModuleFromBuffer(ModuleMap[Identifier], TheModule.getContext(),
-                                /*Lazy=*/true);
+                                /*Lazy=*/true, /*IsImporting*/ true);
   };
 
   FunctionImporter Importer(Index, Loader);
   if (!Importer.importFunctions(TheModule, ImportList))
     report_fatal_error("importFunctions failed");
 }
 
 static void optimizeModule(Module &TheModule, TargetMachine &TM) {
if (CodeGenOnly) {
     ProducedBinaries.resize(Modules.size());
     int count = 0;
     for (auto &ModuleBuffer : Modules) {
       Pool.async([&](int count) {
         LLVMContext Context;
         Context.setDiscardValueNames(LTODiscardValueNames);
 
         // Parse module now
-        auto TheModule = loadModuleFromBuffer(ModuleBuffer, Context, false);
+        auto TheModule = loadModuleFromBuffer(ModuleBuffer, Context, false,
+                                              /*IsImporting*/ false);
 
         // CodeGen
         ProducedBinaries[count] = codegen(*TheModule);
       }, count++);
     }
 
     return;
   }
for (auto IndexCount : ModulesOrdering) {
         auto DiagFileOrErr = setupOptimizationRemarks(Context, count);
         if (!DiagFileOrErr) {
           errs() << "Error: " << toString(DiagFileOrErr.takeError()) << "\n";
           report_fatal_error("ThinLTO: Can't get an output file for the "
                              "remarks");
         }
 
         // Parse module now
-        auto TheModule = loadModuleFromBuffer(ModuleBuffer, Context, false);
+        auto TheModule = loadModuleFromBuffer(ModuleBuffer, Context, false,
+                                              /*IsImporting*/ false);
 
         // Save temps: original file.
         saveTempBitcode(*TheModule, SaveTempsDir, count, ".0.original.bc");
 
         auto &ImportList = ImportLists[ModuleIdentifier];
         // Run the main process now, and generates a binary
         auto OutputBuffer = ProcessThinLTOModule(
             *TheModule, *Index, ModuleMap, *TMBuilder.create(), ImportList,

llvm/trunk/lib/Object/IRObjectFile.cpp

IRObjectFile::create(MemoryBufferRef Object, LLVMContext &Context) {
   Expected<std::vector<BitcodeModule>> BMsOrErr =
       getBitcodeModuleList(*BCOrErr);
   if (!BMsOrErr)
     return BMsOrErr.takeError();
 
   std::vector<std::unique_ptr<Module>> Mods;
   for (auto BM : *BMsOrErr) {
     Expected<std::unique_ptr<Module>> MOrErr =
-        BM.getLazyModule(Context, /*ShouldLazyLoadMetadata*/ true);
+        BM.getLazyModule(Context, /*ShouldLazyLoadMetadata*/ true,
+                         /*IsImporting*/ false);
     if (!MOrErr)
       return MOrErr.takeError();
 
     Mods.push_back(std::move(*MOrErr));
   }
 
   return std::unique_ptr<IRObjectFile>(
       new IRObjectFile(*BCOrErr, std::move(Mods)));
 }

llvm/trunk/test/ThinLTO/X86/Inputs/debuginfo-compositetype-import.ll

+; ModuleID = 'debuginfo-compositetype-import2.c'
+source_filename = "debuginfo-compositetype-import2.c"
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+; Function Attrs: nounwind uwtable
+define i32 @main() {
+entry:
+  call void (...) @foo()
+  ret i32 0
+}
+
+declare void @foo(...) #1

llvm/trunk/test/ThinLTO/X86/debuginfo-compositetype-import.ll

+; Test to ensure DICompositeType are imported as type declarations
+; for ThinLTO
+
+; RUN: opt -module-summary %s -o %t1.bc
+; RUN: opt -module-summary %p/Inputs/debuginfo-compositetype-import.ll -o %t2.bc
+; RUN: llvm-lto -thinlto-action=thinlink -o %t.index.bc %t1.bc %t2.bc
+
+; By default, composite types are imported as type declarations
+; RUN: llvm-lto -thinlto-action=import %t2.bc -thinlto-index=%t.index.bc -o - | llvm-dis -o - | FileCheck %s
+; RUN: llvm-lto2 %t1.bc %t2.bc -o %t.out -save-temps \
+; RUN:   -r %t2.bc,main,plx \
+; RUN:   -r %t2.bc,foo,l \
+; RUN:   -r %t1.bc,foo,pl
+; RUN: llvm-dis < %t.out.1.3.import.bc | FileCheck %s
+
+; CHECK: distinct !DICompositeType(tag: DW_TAG_enumeration_type, name: "enum", scope: !{{[0-9]+}}, file: !{{[0-9]+}}, line: 50, size: 32, flags: DIFlagFwdDecl, identifier: "enum")
+; CHECK: distinct !DICompositeType(tag: DW_TAG_class_type, name: "class", scope: !{{[0-9]+}}, file: !{{[0-9]+}}, line: 728, size: 448, flags: DIFlagFwdDecl, identifier: "class")
+; CHECK: distinct !DICompositeType(tag: DW_TAG_structure_type, name: "struct", scope: !{{[0-9]+}}, file: !{{[0-9]+}}, line: 309, size: 128, flags: DIFlagFwdDecl, identifier: "list")
+; CHECK: distinct !DICompositeType(tag: DW_TAG_union_type, file: !{{[0-9]+}}, line: 115, size: 384, flags: DIFlagFwdDecl, identifier: "union")
+
+; Ensure that full type definitions of composite types are imported if requested
+; RUN: llvm-lto -import-full-type-definitions -thinlto-action=import %t2.bc -thinlto-index=%t.index.bc -o - | llvm-dis -o - | FileCheck %s --check-prefix=FULL
+; RUN: llvm-lto2 %t1.bc %t2.bc -o %t.out -save-temps \
+; RUN:   -import-full-type-definitions \
+; RUN:   -r %t2.bc,main,plx \
+; RUN:   -r %t2.bc,foo,l \
+; RUN:   -r %t1.bc,foo,pl
+; RUN: llvm-dis < %t.out.1.3.import.bc | FileCheck %s --check-prefix=FULL
+
+; FULL: distinct !DICompositeType(tag: DW_TAG_enumeration_type, name: "enum", scope: !{{[0-9]+}}, file: !{{[0-9]+}}, line: 50, size: 32, elements: !{{[0-9]+}}, identifier: "enum")
+; FULL: distinct !DICompositeType(tag: DW_TAG_class_type, name: "class", scope: !{{[0-9]+}}, file: !{{[0-9]+}}, line: 728, size: 448, elements: !{{[0-9]+}}, identifier: "class")
+; FULL: distinct !DICompositeType(tag: DW_TAG_structure_type, name: "struct", scope: !{{[0-9]+}}, file: !{{[0-9]+}}, line: 309, baseType: !{{[0-9]+}}, size: 128, offset: 64, elements: !{{[0-9]+}}, vtableHolder: !{{[0-9]+}}, templateParams: !{{[0-9]+}}, identifier: "list")
+; FULL: distinct !DICompositeType(tag: DW_TAG_union_type, file: !{{[0-9]+}}, line: 115, size: 384, elements: !{{[0-9]+}}, identifier: "union")
+
+; ModuleID = 'debuginfo-compositetype-import.c'
+source_filename = "debuginfo-compositetype-import.c"
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+; Function Attrs: nounwind uwtable
+define void @foo() #0 !dbg !6 {
+entry:
+    ret void
+}
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!2, !3}
+!llvm.ident = !{!4}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !1, producer: "clang version 4.0.0 (trunk 286863) (llvm/trunk 286875)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug)
+!1 = !DIFile(filename: "debuginfo-compositetype-import.c", directory: "")
+!2 = !{i32 2, !"Dwarf Version", i32 4}
+!3 = !{i32 2, !"Debug Info Version", i32 3}
+!4 = !{!"clang version 4.0.0 (trunk 286863) (llvm/trunk 286875)"}
+!5 = !{}
+!6 = distinct !DISubprogram(name: "foo", scope: !1, file: !1, line: 1, type: !7, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: false, unit: !0, variables: !5)
+!7 = !DISubroutineType(types: !8)
+!8 = !{!9, !10, !11, !12}
+!9 = !DICompositeType(tag: DW_TAG_enumeration_type, name: "enum", scope: !1, file: !1, line: 50, size: 32, elements: !5, identifier: "enum")
+!10 = !DICompositeType(tag: DW_TAG_class_type, name: "class", scope: !1, file: !1, line: 728, size: 448, elements: !5, identifier: "class")
+!11 = distinct !DICompositeType(tag: DW_TAG_structure_type, name: "struct", scope: !1, file: !1, line: 309, baseType: !10, size: 128, offset: 64, elements: !5, vtableHolder: !10, templateParams: !5, identifier: "list")
+!12 = distinct !DICompositeType(tag: DW_TAG_union_type, file: !1, line: 115, size: 384, elements: !5, identifier: "union")