Prototype of modules codegen

[dblaikie]

Summary

First pass at generating weak definitions of inline functions from module files
(& currently skipping definitions of those functions in uses)

I've done some manual testing (haven't delved into the preferred testing
strategy for modules). Seems to work for simplest cases, including transitive
modules.

Also doesn't have an actual flag plumbed through, yet. I can submit that ahead
of this in a separate patch (open to ideas, etc). Is there a goal for how
explicit modules should be used via the driver (rather than cc1)? Otherwise
this would be a cc1 option to match.

One quandry: What to do with non-inline definitions in headers? Currently this
prototype moves their emission (while keeping their linkage) to the module
object. This would remove ODR violations and actually make it 'correct' to
define a non-inline function in a header, which may or may not be desirable. It
could be kept in all the users as it is today.

I'm sure I don't know nearly enough about linkage and about how linkage is
handled in Clang & LLVM. So I'm certainly open to ideas about how this
implementation isn't ideal or correct - or the myriad of interesting linkage
cases I should test/consider.

Perhaps we could chat about this in person/over the shoulder if a faster
iteration would be worthwhile.

[hfinkel]

Can you provide a high-level description of what you're trying to accomplish and the usage model?

[dblaikie]

Add bit to the Module record for when modular codegen decls are included in the MODULAR_CODEGEN_DECLS bitcode record

[dblaikie]

• Move linkage handling into GetGVALinkageForFunction

[dblaikie]

• Add cc1 flag for modular codegen, -fmodule-codegen

[dblaikie]

• Add test coverage

[dblaikie]

• More test coverage (for local static variables)

[rsmith]

include/clang/AST/ASTContext.h

2490

I think the name of this flag might be out of sync with its meaning; it looks like it means "must this declaration be emitted when performing modular codegen?"

include/clang/Basic/Module.h

208

Does this need to be two bits wide? Seems to only store true/false.

include/clang/Driver/CC1Options.td

435–438 ↗

(On Diff #85686)

Maybe module -> modules, to match the other -fmodules-* flags controlling various options.

lib/AST/ASTContext.cpp

9010–9013

I think this return true is unreachable and can be deleted: if we get here with Linkage == GVA_DiscardableODR, then the call to hasExternalDefinitions in GetGVALinkageForFunction must have returned EK_ReplyHazy. (In the case this is checking for, GetGVALinkageForFunction would return GVA_StrongODR, so the check below will return true anyway.)

9019

Pass Linkage in here instead of recomputing it

lib/AST/ExternalASTSource.cpp

33

You should add support for this function to clang::MultiplexExternalSemaSource too.

lib/Serialization/ASTWriterDecl.cpp

2215–2220

I suspect we'll want to seriously prune back the contents of EagerlyDeserializedDecls for the modular codegen case at some point, but we don't need to do that in this change.

[dblaikie]

• Address code review feedback
• Formatting

[dblaikie]

Addressed CR feedback

lib/AST/ExternalASTSource.cpp

33

Done - though is there any test coverage I should add? Not sure exactly when/where/how this is used.

Also only made a rough guess at what the right behavior is here. It could be a bit more obvious if I made "hasExternalDefinitions" return Optional<ExtKind> - then when we find an ExternalASTSource that owns the ID (are the IDs unique across the MultiplexExternalSemaSource? I assume they have to be, but thought they'd only be valid within a single Module... guess I'm confused in a few ways - certainly haven't thought about it in great detail) we'd know to stop asking other sources. Probably not worth it unless there's a lot of sources?

lib/Serialization/ASTWriterDecl.cpp

2215–2220

Right - I was wondering if we'd end up with anything in EagerlyDeserializedDecls when modular codegen is fully implemented. (if that's the case - we could have only one list, and a 'bit' to say whether it's modular codegen decls or EagerlyDeserializedDecls, if these records/blobs/whatnot are expensive to have two of rather than one - but I don't think that's the case so probably more readable/self-documenting to use two lists as we are)

[rsmith]

lib/AST/ExternalASTSource.cpp

33

You could test this with -chain-include. I don't think there's any other in-tree way to get multiple external sources.

include/clang/AST/ASTContext.h

public:
   GVALinkage GetGVALinkageForVariable(const VarDecl *VD);
 
   /// \brief Determines if the decl can be CodeGen'ed or deserialized from PCH
   /// lazily, only when used; this is only relevant for function or file scoped
   /// var definitions.
   ///
   /// \returns true if the function/var must be CodeGen'ed/deserialized even if
   /// it is not used.
-  bool DeclMustBeEmitted(const Decl *D);
+  bool DeclMustBeEmitted(const Decl *D, bool WritingModule = false);
 
   const CXXConstructorDecl *
   getCopyConstructorForExceptionObject(CXXRecordDecl *RD);
 
   void addCopyConstructorForExceptionObject(CXXRecordDecl *RD,
                                             CXXConstructorDecl *CD);
 
   void addTypedefNameForUnnamedTagDecl(TagDecl *TD, TypedefNameDecl *TND);

include/clang/AST/ExternalASTSource.h

 //  construction of AST nodes from some external source.
 //
 //===----------------------------------------------------------------------===//
 #ifndef LLVM_CLANG_AST_EXTERNALASTSOURCE_H
 #define LLVM_CLANG_AST_EXTERNALASTSOURCE_H
 
 #include "clang/AST/CharUnits.h"
 #include "clang/AST/DeclBase.h"
+#include "clang/Basic/Module.h"
 #include "llvm/ADT/DenseMap.h"
 
 namespace clang {
 
 class ASTConsumer;
 class CXXBaseSpecifier;
 class CXXCtorInitializer;
 class DeclarationName;
public:
     StringRef getASTFile() const { return ASTFile; }
     uint64_t getSignature() const { return Signature; }
     const Module *getModuleOrNull() const { return ClangModule; }
   };
 
   /// Return a descriptor for the corresponding module, if one exists.
   virtual llvm::Optional<ASTSourceDescriptor> getSourceDescriptor(unsigned ID);
 
+  virtual Module::ExtKind hasExternalDefinitions(unsigned ID);
+
   /// \brief Finds all declarations lexically contained within the given
   /// DeclContext, after applying an optional filter predicate.
   ///
   /// \param IsKindWeWant a predicate function that returns true if the passed
   /// declaration kind is one we are looking for.
   ///
   /// The default implementation of this method is a no-op.
   virtual void

include/clang/Basic/LangOptions.def

 LANGOPT(CUDAHostDeviceConstexpr, 1, 1, "treating unattributed constexpr functions as __host__ __device__")
 LANGOPT(CUDADeviceFlushDenormalsToZero, 1, 0, "flushing denormals to zero")
 LANGOPT(CUDADeviceApproxTranscendentals, 1, 0, "using approximate transcendental functions")
 
 LANGOPT(SizedDeallocation , 1, 0, "sized deallocation")
 LANGOPT(AlignedAllocation , 1, 0, "aligned allocation")
 LANGOPT(NewAlignOverride  , 32, 0, "maximum alignment guaranteed by '::operator new(size_t)'")
 LANGOPT(ConceptsTS , 1, 0, "enable C++ Extensions for Concepts")
+BENIGN_LANGOPT(ModularCodegen , 1, 1, "Modular codegen")
 BENIGN_LANGOPT(ElideConstructors , 1, 1, "C++ copy constructor elision")
 BENIGN_LANGOPT(DumpRecordLayouts , 1, 0, "dumping the layout of IRgen'd records")
 BENIGN_LANGOPT(DumpRecordLayoutsSimple , 1, 0, "dumping the layout of IRgen'd records in a simple form")
 BENIGN_LANGOPT(DumpVTableLayouts , 1, 0, "dumping the layouts of emitted vtables")
 LANGOPT(NoConstantCFStrings , 1, 0, "no constant CoreFoundation strings")
 BENIGN_LANGOPT(InlineVisibilityHidden , 1, 0, "hidden default visibility for inline C++ methods")
 BENIGN_LANGOPT(ParseUnknownAnytype, 1, 0, "__unknown_anytype")
 BENIGN_LANGOPT(DebuggerSupport , 1, 0, "debugger support")

include/clang/Basic/Module.h

public:
   /// that no identifier not in this list should affect how the module is
   /// built.
   unsigned ConfigMacrosExhaustive : 1;
 
   /// \brief Whether files in this module can only include non-modular headers
   /// and headers from used modules.
   unsigned NoUndeclaredIncludes : 1;
 
+  unsigned ExtDef : 2;
+
   /// \brief Describes the visibility of the various names within a
   /// particular module.
   enum NameVisibilityKind {
     /// \brief All of the names in this module are hidden.
     Hidden,
     /// \brief All of the names in this module are visible.
     AllVisible
   };
struct Conflict {
 
     /// \brief The message provided to the user when there is a conflict.
     std::string Message;
   };
 
   /// \brief The list of conflicts.
   std::vector<Conflict> Conflicts;
 
+  enum ExtKind { EK_Always, EK_Never, EK_ReplyHazy };
+
   /// \brief Construct a new module or submodule.
   Module(StringRef Name, SourceLocation DefinitionLoc, Module *Parent,
          bool IsFramework, bool IsExplicit, unsigned VisibilityID);
-  
   ~Module();
-  
+  void setExternalDefinitions(ExtKind EK) {
+    assert(!Parent);
+    ExtDef = EK;
+  }
+
+  ExtKind hasExternalDefinitions() const {
+    auto *P = this;
+    while (auto *K = P->Parent)
+      P = K;
+    return static_cast<ExtKind>(P->ExtDef);
+  }
+
   /// \brief Determine whether this module is available for use within the
   /// current translation unit.
   bool isAvailable() const { return IsAvailable; }
 
   /// \brief Determine whether this module is available for use within the
   /// current translation unit.
   ///
   /// \param LangOpts The language options used for the current

include/clang/Serialization/ASTBitCodes.h

enum ASTRecordTypes {
       /// directives we've seen.
       CUDA_PRAGMA_FORCE_HOST_DEVICE_DEPTH = 57,
 
       /// \brief Record code for types associated with OpenCL extensions.
       OPENCL_EXTENSION_TYPES = 58,
 
       /// \brief Record code for declarations associated with OpenCL extensions.
       OPENCL_EXTENSION_DECLS = 59,
+
+      MODULAR_CODEGEN_DECLS = 60,
     };
 
     /// \brief Record types used within a source manager block.
     enum SourceManagerRecordTypes {
       /// \brief Describes a source location entry (SLocEntry) for a
       /// file.
       SM_SLOC_FILE_ENTRY = 1,
       /// \brief Describes a source location entry (SLocEntry) for a

include/clang/Serialization/ASTReader.h

private:
   /// \brief The IDs of all declarations that fulfill the criteria of
   /// "interesting" decls.
   ///
   /// This contains the data loaded from all EAGERLY_DESERIALIZED_DECLS blocks
   /// in the chain. The referenced declarations are deserialized and passed to
   /// the consumer eagerly.
   SmallVector<uint64_t, 16> EagerlyDeserializedDecls;
 
+  SmallVector<uint64_t, 16> ModularCodegenDecls;
+
   /// \brief The IDs of all tentative definitions stored in the chain.
   ///
   /// Sema keeps track of all tentative definitions in a TU because it has to
   /// complete them and pass them on to CodeGen. Thus, tentative definitions in
   /// the PCH chain must be eagerly deserialized.
   SmallVector<uint64_t, 16> TentativeDefinitions;
 
   /// \brief The IDs of all CXXRecordDecls stored in the chain whose VTables are
private:
   bool ParseLineTable(ModuleFile &F, const RecordData &Record);
   bool ReadSourceManagerBlock(ModuleFile &F);
   llvm::BitstreamCursor &SLocCursorForID(int ID);
   SourceLocation getImportLocation(ModuleFile *F);
   ASTReadResult ReadModuleMapFileBlock(RecordData &Record, ModuleFile &F,
                                        const ModuleFile *ImportedBy,
                                        unsigned ClientLoadCapabilities);
   ASTReadResult ReadSubmoduleBlock(ModuleFile &F,
-                                   unsigned ClientLoadCapabilities);
+                                   unsigned ClientLoadCapabilities,
+                                   Module *&Root);
   static bool ParseLanguageOptions(const RecordData &Record, bool Complain,
                                    ASTReaderListener &Listener,
                                    bool AllowCompatibleDifferences);
   static bool ParseTargetOptions(const RecordData &Record, bool Complain,
                                  ASTReaderListener &Listener,
                                  bool AllowCompatibleDifferences);
   static bool ParseDiagnosticOptions(const RecordData &Record, bool Complain,
                                      ASTReaderListener &Listener);
public:
   ModuleFile *getLocalModuleFile(ModuleFile &M, unsigned ID);
 
   /// \brief Get an ID for the given module file.
   unsigned getModuleFileID(ModuleFile *M);
 
   /// \brief Return a descriptor for the corresponding module.
   llvm::Optional<ASTSourceDescriptor> getSourceDescriptor(unsigned ID) override;
 
+  Module::ExtKind hasExternalDefinitions(unsigned ID) override;
+
   /// \brief Retrieve a selector from the given module with its local ID
   /// number.
   Selector getLocalSelector(ModuleFile &M, unsigned LocalID);
 
   Selector DecodeSelector(serialization::SelectorID Idx);
 
   Selector GetExternalSelector(serialization::SelectorID ID) override;
   uint32_t GetNumExternalSelectors() override;

include/clang/Serialization/ASTWriter.h

private:
   /// as we are emitting declarations to the AST file. The AST file contains a
   /// separate record for these declarations, which are provided to the AST
   /// consumer by the AST reader. This is behavior is required to properly cope with,
   /// e.g., tentative variable definitions that occur within
   /// headers. The declarations themselves are stored as declaration
   /// IDs, since they will be written out to an EAGERLY_DESERIALIZED_DECLS
   /// record.
   SmallVector<uint64_t, 16> EagerlyDeserializedDecls;
+  SmallVector<uint64_t, 16> ModularCodegenDecls;
 
   /// \brief DeclContexts that have received extensions since their serialized
   /// form.
   ///
   /// For namespaces, when we're chaining and encountering a namespace, we check
   /// if its primary namespace comes from the chain. If it does, we add the
   /// primary to this set, so that we can write out lexical content updates for
   /// it.

lib/AST/ASTContext.cpp

static GVALinkage basicGVALinkageForVariable(const ASTContext &Context,
   llvm_unreachable("Invalid Linkage!");
 }
 
 GVALinkage ASTContext::GetGVALinkageForVariable(const VarDecl *VD) {
   return adjustGVALinkageForAttributes(
       *this, basicGVALinkageForVariable(*this, VD), VD);
 }
 
-bool ASTContext::DeclMustBeEmitted(const Decl *D) {
+bool ASTContext::DeclMustBeEmitted(const Decl *D, bool WritingModule) {
   if (const VarDecl *VD = dyn_cast<VarDecl>(D)) {
     if (!VD->isFileVarDecl())
       return false;
     // Global named register variables (GNU extension) are never emitted.
     if (VD->getStorageClass() == SC_Register)
       return false;
     if (VD->getDescribedVarTemplate() ||
         isa<VarTemplatePartialSpecializationDecl>(VD))
if (getTargetInfo().getCXXABI().canKeyFunctionBeInline()) {
         if (MD->isOutOfLine() && RD->isDynamicClass()) {
           const CXXMethodDecl *KeyFunc = getCurrentKeyFunction(RD);
           if (KeyFunc && KeyFunc->getCanonicalDecl() == MD->getCanonicalDecl())
             return true;
         }
       }
     }
 
+    GVALinkage Linkage = GetGVALinkageForFunction(FD);
+
+    if (Linkage == GVA_DiscardableODR) {
+      if (WritingModule)
+        return true;
+      if (auto *Ext = getExternalSource())
+        if (Ext->hasExternalDefinitions(FD->getOwningModuleID()) ==
+            Module::EK_Never)
+          return true;
+    }
+
     // static, static inline, always_inline, and extern inline functions can
     // always be deferred.  Normal inline functions can be deferred in C99/C++.
     // Implicit template instantiations can also be deferred in C++.
     return !isDiscardableGVALinkage(GetGVALinkageForFunction(FD));
   }
   
   const VarDecl *VD = cast<VarDecl>(D);
   assert(VD->isFileVarDecl() && "Expected file scoped var");
 
   if (VD->isThisDeclarationADefinition() == VarDecl::DeclarationOnly &&
       !isMSStaticDataMemberInlineDefinition(VD))
     return false;

lib/AST/ExternalASTSource.cpp

 
 ExternalASTSource::~ExternalASTSource() { }
 
 llvm::Optional<ExternalASTSource::ASTSourceDescriptor>
 ExternalASTSource::getSourceDescriptor(unsigned ID) {
   return None;
 }
 
+Module::ExtKind ExternalASTSource::hasExternalDefinitions(unsigned ID) {
+  return Module::EK_ReplyHazy;
+}
+
 ExternalASTSource::ASTSourceDescriptor::ASTSourceDescriptor(const Module &M)
   : Signature(M.Signature), ClangModule(&M) {
   if (M.Directory)
     Path = M.Directory->getName();
   if (auto *File = M.getASTFile())
     ASTFile = File->getName();
 }
 

lib/Basic/Module.cpp

Module::Module(StringRef Name, SourceLocation DefinitionLoc, Module *Parent,
                bool IsFramework, bool IsExplicit, unsigned VisibilityID)
     : Name(Name), DefinitionLoc(DefinitionLoc), Parent(Parent), Directory(),
       Umbrella(), Signature(0), ASTFile(nullptr), VisibilityID(VisibilityID),
       IsMissingRequirement(false), HasIncompatibleModuleFile(false),
       IsAvailable(true), IsFromModuleFile(false), IsFramework(IsFramework),
       IsExplicit(IsExplicit), IsSystem(false), IsExternC(false),
       IsInferred(false), InferSubmodules(false), InferExplicitSubmodules(false),
       InferExportWildcard(false), ConfigMacrosExhaustive(false),
-      NoUndeclaredIncludes(false), NameVisibility(Hidden) {
+      NoUndeclaredIncludes(false), ExtDef(EK_ReplyHazy),
+      NameVisibility(Hidden) {
   if (Parent) {
     if (!Parent->isAvailable())
       IsAvailable = false;
     if (Parent->IsSystem)
       IsSystem = true;
     if (Parent->IsExternC)
       IsExternC = true;
     if (Parent->NoUndeclaredIncludes)

lib/CodeGen/CodeGenModule.cpp

llvm::GlobalValue::LinkageTypes CodeGenModule::getLLVMLinkageForDeclarator(
 
   if (D->hasAttr<WeakAttr>()) {
     if (IsConstantVariable)
       return llvm::GlobalVariable::WeakODRLinkage;
     else
       return llvm::GlobalVariable::WeakAnyLinkage;
   }
 
+  if (auto *Ext = getContext().getExternalSource())
+    switch (Ext->hasExternalDefinitions(D->getOwningModuleID())) {
+    case Module::EK_Never:
+      if (Linkage == GVA_DiscardableODR)
+        return llvm::GlobalValue::WeakODRLinkage;
+      break;
+    case Module::EK_Always:
+      return llvm::GlobalValue::AvailableExternallyLinkage;
+    case Module::EK_ReplyHazy:
+      break;
+    }
+
   // We are guaranteed to have a strong definition somewhere else,
   // so we can use available_externally linkage.
   if (Linkage == GVA_AvailableExternally)
-    return llvm::Function::AvailableExternallyLinkage;
+    return llvm::GlobalValue::AvailableExternallyLinkage;
 
   // Note that Apple's kernel linker doesn't support symbol
   // coalescing, so we need to avoid linkonce and weak linkages there.
   // Normally, this means we just map to internal, but for explicit
   // instantiations we'll map to external.
 
   // In C++, the compiler has to emit a definition in every translation unit
   // that references the function.  We should use linkonce_odr because

lib/Lex/ModuleMap.cpp

 
 Module *ModuleMap::lookupModuleQualified(StringRef Name, Module *Context) const{
   if (!Context)
     return findModule(Name);
   
   return Context->findSubmodule(Name);
 }
 
-std::pair<Module *, bool> 
-ModuleMap::findOrCreateModule(StringRef Name, Module *Parent, bool IsFramework,
+std::pair<Module *, bool> ModuleMap::findOrCreateModule(StringRef Name,
+                                                        Module *Parent,
+                                                        bool IsFramework,
                                                         bool IsExplicit) {
   // Try to find an existing module with this name.
   if (Module *Sub = lookupModuleQualified(Name, Parent))
     return std::make_pair(Sub, false);
   
   // Create a new module with this name.
-  Module *Result = new Module(Name, SourceLocation(), Parent,
-                              IsFramework, IsExplicit, NumCreatedModules++);
+  Module *Result = new Module(Name, SourceLocation(), Parent, IsFramework,
+                              IsExplicit, NumCreatedModules++);
   if (!Parent) {
     if (LangOpts.CurrentModule == Name)
       SourceModule = Result;
     Modules[Name] = Result;
   }
   return std::make_pair(Result, true);
 }
 

lib/Serialization/ASTReader.cpp

 ASTReader::ReadASTBlock(ModuleFile &F, unsigned ClientLoadCapabilities) {
   BitstreamCursor &Stream = F.Stream;
 
   if (Stream.EnterSubBlock(AST_BLOCK_ID)) {
     Error("malformed block record in AST file");
     return Failure;
   }
 
+  Module *Root = nullptr;
+  Module::ExtKind EK = Module::EK_ReplyHazy;
+
   // Read all of the records and blocks for the AST file.
   RecordData Record;
   while (true) {
     llvm::BitstreamEntry Entry = Stream.advance();
 
     switch (Entry.Kind) {
     case llvm::BitstreamEntry::Error:
       Error("error at end of module block in AST file");
       return Failure;
     case llvm::BitstreamEntry::EndBlock: {
       // Outside of C++, we do not store a lookup map for the translation unit.
       // Instead, mark it as needing a lookup map to be built if this module
       // contains any declarations lexically within it (which it always does!).
       // This usually has no cost, since we very rarely need the lookup map for
       // the translation unit outside C++.
       DeclContext *DC = Context.getTranslationUnitDecl();
       if (DC->hasExternalLexicalStorage() &&
           !getContext().getLangOpts().CPlusPlus)
         DC->setMustBuildLookupTable();
 
+      if (Root)
+        Root->setExternalDefinitions(EK);
       return Success;
     }
     case llvm::BitstreamEntry::SubBlock:
       switch (Entry.ID) {
       case DECLTYPES_BLOCK_ID:
         // We lazily load the decls block, but we want to set up the
         // DeclsCursor cursor to point into it.  Clone our current bitcode
         // cursor to it, enter the block and read the abbrevs in that block.
case llvm::BitstreamEntry::SubBlock:
         break;
 
       case SOURCE_MANAGER_BLOCK_ID:
         if (ReadSourceManagerBlock(F))
           return Failure;
         break;
 
       case SUBMODULE_BLOCK_ID:
-        if (ASTReadResult Result = ReadSubmoduleBlock(F, ClientLoadCapabilities))
+        if (ASTReadResult Result =
+                ReadSubmoduleBlock(F, ClientLoadCapabilities, Root))
           return Result;
         break;
 
       case COMMENTS_BLOCK_ID: {
         BitstreamCursor C = Stream;
         if (Stream.SkipBlock() ||
             ReadBlockAbbrevs(C, COMMENTS_BLOCK_ID)) {
           Error("malformed comments block in AST file");
while (true) {
 
     case EAGERLY_DESERIALIZED_DECLS:
       // FIXME: Skip reading this record if our ASTConsumer doesn't care
       // about "interesting" decls (for instance, if we're building a module).
       for (unsigned I = 0, N = Record.size(); I != N; ++I)
         EagerlyDeserializedDecls.push_back(getGlobalDeclID(F, Record[I]));
       break;
 
+    case MODULAR_CODEGEN_DECLS:
+      EK = F.Kind == MK_MainFile ? Module::EK_Never : Module::EK_Always;
+      // FIXME: Skip reading this record if our ASTConsumer doesn't care about
+      // them (ie: if we're not codegenerating this module).
+      if (F.Kind == MK_MainFile)
+        for (unsigned I = 0, N = Record.size(); I != N; ++I)
+          EagerlyDeserializedDecls.push_back(getGlobalDeclID(F, Record[I]));
+      break;
+
     case SPECIAL_TYPES:
       if (SpecialTypes.empty()) {
         for (unsigned I = 0, N = Record.size(); I != N; ++I)
           SpecialTypes.push_back(getGlobalTypeID(F, Record[I]));
         break;
       }
 
       if (SpecialTypes.size() != Record.size()) {
SimplePCHValidator validator(LangOpts, TargetOpts, PPOpts,
                                ExistingModuleCachePath, FileMgr);
   return !readASTFileControlBlock(Filename, FileMgr, PCHContainerRdr,
                                   /*FindModuleFileExtensions=*/false,
                                   validator,
                                   /*ValidateDiagnosticOptions=*/true);
 }
 
 ASTReader::ASTReadResult
-ASTReader::ReadSubmoduleBlock(ModuleFile &F, unsigned ClientLoadCapabilities) {
+ASTReader::ReadSubmoduleBlock(ModuleFile &F, unsigned ClientLoadCapabilities,
+                              Module *&Root) {
   // Enter the submodule block.
   if (F.Stream.EnterSubBlock(SUBMODULE_BLOCK_ID)) {
     Error("malformed submodule block record in AST file");
     return Failure;
   }
 
   ModuleMap &ModMap = PP.getHeaderSearchInfo().getModuleMap();
   bool First = true;
case SUBMODULE_DEFINITION: {
       bool ConfigMacrosExhaustive = Record[Idx++];
 
       Module *ParentModule = nullptr;
       if (Parent)
         ParentModule = getSubmodule(Parent);
 
       // Retrieve this (sub)module from the module map, creating it if
       // necessary.
-      CurrentModule = ModMap.findOrCreateModule(Name, ParentModule, IsFramework,
-                                                IsExplicit).first;
+      CurrentModule =
+          ModMap.findOrCreateModule(Name, ParentModule, IsFramework, IsExplicit)
+              .first;
+
+      if (!Parent)
+        Root = CurrentModule;
 
       // FIXME: set the definition loc for CurrentModule, or call
       // ModMap.setInferredModuleAllowedBy()
 
       SubmoduleID GlobalIndex = GlobalID - NUM_PREDEF_SUBMODULE_IDS;
       if (GlobalIndex >= SubmodulesLoaded.size() ||
           SubmodulesLoaded[GlobalIndex]) {
         Error("too many submodules");
if (ModuleMgr.size() == 1) {
     StringRef ModuleName = llvm::sys::path::filename(MF.OriginalSourceFileName);
     StringRef FileName = llvm::sys::path::filename(MF.FileName);
     return ASTReader::ASTSourceDescriptor(ModuleName, MF.OriginalDir, FileName,
                                           MF.Signature);
   }
   return None;
 }
 
+Module::ExtKind ASTReader::hasExternalDefinitions(unsigned ID) {
+  if (const Module *M = getSubmodule(ID))
+    return M->hasExternalDefinitions();
+
+  return Module::EK_ReplyHazy;
+}
+
 Selector ASTReader::getLocalSelector(ModuleFile &M, unsigned LocalID) {
   return DecodeSelector(getGlobalSelectorID(M, LocalID));
 }
 
 Selector ASTReader::DecodeSelector(serialization::SelectorID ID) {
   if (ID == 0)
     return Selector();
 

lib/Serialization/ASTWriter.cpp

#define RECORD(X) EmitRecordID(X, #X, Stream, Record)
 
   // AST Top-Level Block.
   BLOCK(AST_BLOCK);
   RECORD(TYPE_OFFSET);
   RECORD(DECL_OFFSET);
   RECORD(IDENTIFIER_OFFSET);
   RECORD(IDENTIFIER_TABLE);
   RECORD(EAGERLY_DESERIALIZED_DECLS);
+  RECORD(MODULAR_CODEGEN_DECLS);
   RECORD(SPECIAL_TYPES);
   RECORD(STATISTICS);
   RECORD(TENTATIVE_DEFINITIONS);
   RECORD(SELECTOR_OFFSETS);
   RECORD(METHOD_POOL);
   RECORD(PP_COUNTER_VALUE);
   RECORD(SOURCE_LOCATION_OFFSETS);
   RECORD(SOURCE_LOCATION_PRELOADS);
if (WritingModule)
     WriteSubmodules(WritingModule);
 
   Stream.EmitRecord(SPECIAL_TYPES, SpecialTypes);
 
   // Write the record containing external, unnamed definitions.
   if (!EagerlyDeserializedDecls.empty())
     Stream.EmitRecord(EAGERLY_DESERIALIZED_DECLS, EagerlyDeserializedDecls);
 
+  if (Context.getLangOpts().ModularCodegen)
+    Stream.EmitRecord(MODULAR_CODEGEN_DECLS, ModularCodegenDecls);
+
   // Write the record containing tentative definitions.
   if (!TentativeDefinitions.empty())
     Stream.EmitRecord(TENTATIVE_DEFINITIONS, TentativeDefinitions);
 
   // Write the record containing unused file scoped decls.
   if (!UnusedFileScopedDecls.empty())
     Stream.EmitRecord(UNUSED_FILESCOPED_DECLS, UnusedFileScopedDecls);
 

lib/Serialization/ASTWriterDecl.cpp

 /// code generation requirements, if other clients have a different notion of
 /// what is "required" then we may have to consider an alternate scheme where
 /// clients can iterate over the top-level decls and get information on them,
 /// without necessary deserializing them. We could explicitly require such
 /// clients to use a separate API call to "realize" the decl. This should be
 /// relatively painless since they would presumably only do it for top-level
 /// decls.
 static bool isRequiredDecl(const Decl *D, ASTContext &Context,
-                           bool WritingModule) {
+                           bool WritingModule, bool ModularCode) {
   // An ObjCMethodDecl is never considered as "required" because its
   // implementation container always is.
 
   // File scoped assembly or obj-c or OMP declare target implementation must be
   // seen.
   if (isa<FileScopeAsmDecl>(D) || isa<ObjCImplDecl>(D) ||
       D->hasAttr<OMPDeclareTargetDeclAttr>())
     return true;
 
   if (WritingModule && (isa<VarDecl>(D) || isa<ImportDecl>(D))) {
     // These declarations are part of the module initializer, and are emitted
     // if and when the module is imported, rather than being emitted eagerly.
     return false;
   }
 
-  return Context.DeclMustBeEmitted(D);
+  return Context.DeclMustBeEmitted(D, ModularCode);
 }
 
 void ASTWriter::WriteDecl(ASTContext &Context, Decl *D) {
   // Determine the ID for this declaration.
   serialization::DeclID ID;
   assert(!D->isFromASTFile() && "should not be emitting imported decl");
   serialization::DeclID &IDR = DeclIDs[D];
   if (IDR == 0)
void ASTWriter::WriteDecl(ASTContext &Context, Decl *D) {
   }
 
   SourceManager &SM = Context.getSourceManager();
   if (Loc.isValid() && SM.isLocalSourceLocation(Loc))
     associateDeclWithFile(D, ID);
 
   // Note declarations that should be deserialized eagerly so that we can add
   // them to a record in the AST file later.
-  if (isRequiredDecl(D, Context, WritingModule))
+  if (isRequiredDecl(D, Context, WritingModule, false))
     EagerlyDeserializedDecls.push_back(ID);
+  else if (Context.getLangOpts().ModularCodegen && WritingModule &&
+           isRequiredDecl(D, Context, true, true))
+    ModularCodegenDecls.push_back(ID);
 }
 
 void ASTRecordWriter::AddFunctionDefinition(const FunctionDecl *FD) {
   // Switch case IDs are per function body.
   Writer->ClearSwitchCaseIDs();
 
   assert(FD->doesThisDeclarationHaveABody());
   if (auto *CD = dyn_cast<CXXConstructorDecl>(FD)) {
     Record->push_back(CD->getNumCtorInitializers());
     if (CD->getNumCtorInitializers())
       AddCXXCtorInitializers(
           llvm::makeArrayRef(CD->init_begin(), CD->init_end()));
   }
   AddStmt(FD->getBody());
 }