Prototype of modules codegen

[dblaikie]

Summary

First pass at generating weak definitions of inline functions from module files
(& currently skipping definitions of those functions in uses)

I've done some manual testing (haven't delved into the preferred testing
strategy for modules). Seems to work for simplest cases, including transitive
modules.

Also doesn't have an actual flag plumbed through, yet. I can submit that ahead
of this in a separate patch (open to ideas, etc). Is there a goal for how
explicit modules should be used via the driver (rather than cc1)? Otherwise
this would be a cc1 option to match.

One quandry: What to do with non-inline definitions in headers? Currently this
prototype moves their emission (while keeping their linkage) to the module
object. This would remove ODR violations and actually make it 'correct' to
define a non-inline function in a header, which may or may not be desirable. It
could be kept in all the users as it is today.

I'm sure I don't know nearly enough about linkage and about how linkage is
handled in Clang & LLVM. So I'm certainly open to ideas about how this
implementation isn't ideal or correct - or the myriad of interesting linkage
cases I should test/consider.

Perhaps we could chat about this in person/over the shoulder if a faster
iteration would be worthwhile.

[hfinkel]

Can you provide a high-level description of what you're trying to accomplish and the usage model?

[dblaikie]

Add bit to the Module record for when modular codegen decls are included in the MODULAR_CODEGEN_DECLS bitcode record

[dblaikie]

• Move linkage handling into GetGVALinkageForFunction

[dblaikie]

• Add cc1 flag for modular codegen, -fmodule-codegen

[dblaikie]

• Add test coverage

[dblaikie]

• More test coverage (for local static variables)

[rsmith]

include/clang/AST/ASTContext.h

2497

I think the name of this flag might be out of sync with its meaning; it looks like it means "must this declaration be emitted when performing modular codegen?"

include/clang/Basic/Module.h

208

Does this need to be two bits wide? Seems to only store true/false.

include/clang/Driver/CC1Options.td

435–438

Maybe module -> modules, to match the other -fmodules-* flags controlling various options.

lib/AST/ASTContext.cpp

9054–9057

I think this return true is unreachable and can be deleted: if we get here with Linkage == GVA_DiscardableODR, then the call to hasExternalDefinitions in GetGVALinkageForFunction must have returned EK_ReplyHazy. (In the case this is checking for, GetGVALinkageForFunction would return GVA_StrongODR, so the check below will return true anyway.)

9063

Pass Linkage in here instead of recomputing it

lib/AST/ExternalASTSource.cpp

33

You should add support for this function to clang::MultiplexExternalSemaSource too.

lib/Serialization/ASTWriterDecl.cpp

2216–2221

I suspect we'll want to seriously prune back the contents of EagerlyDeserializedDecls for the modular codegen case at some point, but we don't need to do that in this change.

[dblaikie]

• Address code review feedback
• Formatting

[dblaikie]

Addressed CR feedback

lib/AST/ExternalASTSource.cpp

33

Done - though is there any test coverage I should add? Not sure exactly when/where/how this is used.

Also only made a rough guess at what the right behavior is here. It could be a bit more obvious if I made "hasExternalDefinitions" return Optional<ExtKind> - then when we find an ExternalASTSource that owns the ID (are the IDs unique across the MultiplexExternalSemaSource? I assume they have to be, but thought they'd only be valid within a single Module... guess I'm confused in a few ways - certainly haven't thought about it in great detail) we'd know to stop asking other sources. Probably not worth it unless there's a lot of sources?

lib/Serialization/ASTWriterDecl.cpp

2216–2221

Right - I was wondering if we'd end up with anything in EagerlyDeserializedDecls when modular codegen is fully implemented. (if that's the case - we could have only one list, and a 'bit' to say whether it's modular codegen decls or EagerlyDeserializedDecls, if these records/blobs/whatnot are expensive to have two of rather than one - but I don't think that's the case so probably more readable/self-documenting to use two lists as we are)

[rsmith]

lib/AST/ExternalASTSource.cpp

33

You could test this with -chain-include. I don't think there's any other in-tree way to get multiple external sources.

include/clang/AST/ASTContext.h

public:
   GVALinkage GetGVALinkageForVariable(const VarDecl *VD);
 
   /// \brief Determines if the decl can be CodeGen'ed or deserialized from PCH
   /// lazily, only when used; this is only relevant for function or file scoped
   /// var definitions.
   ///
   /// \returns true if the function/var must be CodeGen'ed/deserialized even if
   /// it is not used.
-  bool DeclMustBeEmitted(const Decl *D);
+  bool DeclMustBeEmitted(const Decl *D, bool WritingModule = false);
 
   const CXXConstructorDecl *
   getCopyConstructorForExceptionObject(CXXRecordDecl *RD);
 
   void addCopyConstructorForExceptionObject(CXXRecordDecl *RD,
                                             CXXConstructorDecl *CD);
 
   void addTypedefNameForUnnamedTagDecl(TagDecl *TD, TypedefNameDecl *TND);

include/clang/AST/ExternalASTSource.h

 //  construction of AST nodes from some external source.
 //
 //===----------------------------------------------------------------------===//
 #ifndef LLVM_CLANG_AST_EXTERNALASTSOURCE_H
 #define LLVM_CLANG_AST_EXTERNALASTSOURCE_H
 
 #include "clang/AST/CharUnits.h"
 #include "clang/AST/DeclBase.h"
+#include "clang/Basic/Module.h"
 #include "llvm/ADT/DenseMap.h"
 
 namespace clang {
 
 class ASTConsumer;
 class CXXBaseSpecifier;
 class CXXCtorInitializer;
 class DeclarationName;
public:
     StringRef getASTFile() const { return ASTFile; }
     uint64_t getSignature() const { return Signature; }
     const Module *getModuleOrNull() const { return ClangModule; }
   };
 
   /// Return a descriptor for the corresponding module, if one exists.
   virtual llvm::Optional<ASTSourceDescriptor> getSourceDescriptor(unsigned ID);
 
+  enum ExtKind { EK_Always, EK_Never, EK_ReplyHazy };
+
+  virtual ExtKind hasExternalDefinitions(unsigned ID);
+
   /// \brief Finds all declarations lexically contained within the given
   /// DeclContext, after applying an optional filter predicate.
   ///
   /// \param IsKindWeWant a predicate function that returns true if the passed
   /// declaration kind is one we are looking for.
   ///
   /// The default implementation of this method is a no-op.
   virtual void

include/clang/Basic/LangOptions.def

 LANGOPT(CUDAHostDeviceConstexpr, 1, 1, "treating unattributed constexpr functions as __host__ __device__")
 LANGOPT(CUDADeviceFlushDenormalsToZero, 1, 0, "flushing denormals to zero")
 LANGOPT(CUDADeviceApproxTranscendentals, 1, 0, "using approximate transcendental functions")
 
 LANGOPT(SizedDeallocation , 1, 0, "sized deallocation")
 LANGOPT(AlignedAllocation , 1, 0, "aligned allocation")
 LANGOPT(NewAlignOverride  , 32, 0, "maximum alignment guaranteed by '::operator new(size_t)'")
 LANGOPT(ConceptsTS , 1, 0, "enable C++ Extensions for Concepts")
+BENIGN_LANGOPT(ModularCodegen , 1, 0, "Modular codegen")
 BENIGN_LANGOPT(ElideConstructors , 1, 1, "C++ copy constructor elision")
 BENIGN_LANGOPT(DumpRecordLayouts , 1, 0, "dumping the layout of IRgen'd records")
 BENIGN_LANGOPT(DumpRecordLayoutsSimple , 1, 0, "dumping the layout of IRgen'd records in a simple form")
 BENIGN_LANGOPT(DumpVTableLayouts , 1, 0, "dumping the layouts of emitted vtables")
 LANGOPT(NoConstantCFStrings , 1, 0, "no constant CoreFoundation strings")
 BENIGN_LANGOPT(InlineVisibilityHidden , 1, 0, "hidden default visibility for inline C++ methods")
 BENIGN_LANGOPT(ParseUnknownAnytype, 1, 0, "__unknown_anytype")
 BENIGN_LANGOPT(DebuggerSupport , 1, 0, "debugger support")

include/clang/Basic/Module.h

public:
   /// that no identifier not in this list should affect how the module is
   /// built.
   unsigned ConfigMacrosExhaustive : 1;
 
   /// \brief Whether files in this module can only include non-modular headers
   /// and headers from used modules.
   unsigned NoUndeclaredIncludes : 1;
 
+  unsigned WithCodegen : 2;
+
   /// \brief Describes the visibility of the various names within a
   /// particular module.
   enum NameVisibilityKind {
     /// \brief All of the names in this module are hidden.
     Hidden,
     /// \brief All of the names in this module are visible.
     AllVisible
   };

include/clang/Driver/CC1Options.td

HelpText<"Embed the contents of the specified file into the module file "
            "being compiled.">;
 def fmodules_embed_all_files : Joined<["-"], "fmodules-embed-all-files">,
   HelpText<"Embed the contents of all files read by this compilation into "
            "the produced module file.">;
 def fmodules_local_submodule_visibility :
   Flag<["-"], "fmodules-local-submodule-visibility">,
   HelpText<"Enforce name visibility rules across submodules of the same "
            "top-level module.">;
+def fmodule_codegen :
+  Flag<["-"], "fmodule-codegen">,
+  HelpText<"Generate code for uses of this module that assumes an explicit "
+           "object file will be built for the module">;
 def fmodule_format_EQ : Joined<["-"], "fmodule-format=">,
   HelpText<"Select the container format for clang modules and PCH. "
            "Supported options are 'raw' and 'obj'.">;
 def ftest_module_file_extension_EQ :
   Joined<["-"], "ftest-module-file-extension=">,
   HelpText<"introduce a module file extension for testing purposes. "
            "The argument is parsed as blockname:major:minor:hashed:user info">;
 def fconcepts_ts : Flag<["-"], "fconcepts-ts">,

include/clang/Serialization/ASTBitCodes.h

enum ASTRecordTypes {
       /// directives we've seen.
       CUDA_PRAGMA_FORCE_HOST_DEVICE_DEPTH = 57,
 
       /// \brief Record code for types associated with OpenCL extensions.
       OPENCL_EXTENSION_TYPES = 58,
 
       /// \brief Record code for declarations associated with OpenCL extensions.
       OPENCL_EXTENSION_DECLS = 59,
+
+      MODULAR_CODEGEN_DECLS = 60,
     };
 
     /// \brief Record types used within a source manager block.
     enum SourceManagerRecordTypes {
       /// \brief Describes a source location entry (SLocEntry) for a
       /// file.
       SM_SLOC_FILE_ENTRY = 1,
       /// \brief Describes a source location entry (SLocEntry) for a

include/clang/Serialization/ASTReader.h

private:
   /// \brief The IDs of all declarations that fulfill the criteria of
   /// "interesting" decls.
   ///
   /// This contains the data loaded from all EAGERLY_DESERIALIZED_DECLS blocks
   /// in the chain. The referenced declarations are deserialized and passed to
   /// the consumer eagerly.
   SmallVector<uint64_t, 16> EagerlyDeserializedDecls;
 
+  SmallVector<uint64_t, 16> ModularCodegenDecls;
+
   /// \brief The IDs of all tentative definitions stored in the chain.
   ///
   /// Sema keeps track of all tentative definitions in a TU because it has to
   /// complete them and pass them on to CodeGen. Thus, tentative definitions in
   /// the PCH chain must be eagerly deserialized.
   SmallVector<uint64_t, 16> TentativeDefinitions;
 
   /// \brief The IDs of all CXXRecordDecls stored in the chain whose VTables are
public:
   ModuleFile *getLocalModuleFile(ModuleFile &M, unsigned ID);
 
   /// \brief Get an ID for the given module file.
   unsigned getModuleFileID(ModuleFile *M);
 
   /// \brief Return a descriptor for the corresponding module.
   llvm::Optional<ASTSourceDescriptor> getSourceDescriptor(unsigned ID) override;
 
+  ExtKind hasExternalDefinitions(unsigned ID) override;
+
   /// \brief Retrieve a selector from the given module with its local ID
   /// number.
   Selector getLocalSelector(ModuleFile &M, unsigned LocalID);
 
   Selector DecodeSelector(serialization::SelectorID Idx);
 
   Selector GetExternalSelector(serialization::SelectorID ID) override;
   uint32_t GetNumExternalSelectors() override;

include/clang/Serialization/ASTWriter.h

private:
   /// as we are emitting declarations to the AST file. The AST file contains a
   /// separate record for these declarations, which are provided to the AST
   /// consumer by the AST reader. This is behavior is required to properly cope with,
   /// e.g., tentative variable definitions that occur within
   /// headers. The declarations themselves are stored as declaration
   /// IDs, since they will be written out to an EAGERLY_DESERIALIZED_DECLS
   /// record.
   SmallVector<uint64_t, 16> EagerlyDeserializedDecls;
+  SmallVector<uint64_t, 16> ModularCodegenDecls;
 
   /// \brief DeclContexts that have received extensions since their serialized
   /// form.
   ///
   /// For namespaces, when we're chaining and encountering a namespace, we check
   /// if its primary namespace comes from the chain. If it does, we add the
   /// primary to this set, so that we can write out lexical content updates for
   /// it.

lib/AST/ASTContext.cpp

if (D->hasAttr<DLLImportAttr>()) {
     // visible externally so they can be launched from host.
     if (L == GVA_DiscardableODR || L == GVA_Internal)
       return GVA_StrongODR;
   }
   return L;
 }
 
 GVALinkage ASTContext::GetGVALinkageForFunction(const FunctionDecl *FD) const {
-  return adjustGVALinkageForAttributes(
+  auto L = adjustGVALinkageForAttributes(
        *this, basicGVALinkageForFunction(*this, FD), FD);
+  auto EK = ExternalASTSource::EK_ReplyHazy;
+  if (auto *Ext = getExternalSource())
+    EK = Ext->hasExternalDefinitions(FD->getOwningModuleID());
+  switch (EK) {
+  case ExternalASTSource::EK_Never:
+    if (L == GVA_DiscardableODR)
+      return GVA_StrongODR;
+    break;
+  case ExternalASTSource::EK_Always:
+    return GVA_AvailableExternally;
+  case ExternalASTSource::EK_ReplyHazy:
+    break;
+  }
+  return L;
 }
 
 static GVALinkage basicGVALinkageForVariable(const ASTContext &Context,
                                              const VarDecl *VD) {
   if (!VD->isExternallyVisible())
     return GVA_Internal;
 
   if (VD->isStaticLocal()) {
static GVALinkage basicGVALinkageForVariable(const ASTContext &Context,
   llvm_unreachable("Invalid Linkage!");
 }
 
 GVALinkage ASTContext::GetGVALinkageForVariable(const VarDecl *VD) {
   return adjustGVALinkageForAttributes(
       *this, basicGVALinkageForVariable(*this, VD), VD);
 }
 
-bool ASTContext::DeclMustBeEmitted(const Decl *D) {
+bool ASTContext::DeclMustBeEmitted(const Decl *D, bool WritingModule) {
   if (const VarDecl *VD = dyn_cast<VarDecl>(D)) {
     if (!VD->isFileVarDecl())
       return false;
     // Global named register variables (GNU extension) are never emitted.
     if (VD->getStorageClass() == SC_Register)
       return false;
     if (VD->getDescribedVarTemplate() ||
         isa<VarTemplatePartialSpecializationDecl>(VD))
if (getTargetInfo().getCXXABI().canKeyFunctionBeInline()) {
         if (MD->isOutOfLine() && RD->isDynamicClass()) {
           const CXXMethodDecl *KeyFunc = getCurrentKeyFunction(RD);
           if (KeyFunc && KeyFunc->getCanonicalDecl() == MD->getCanonicalDecl())
             return true;
         }
       }
     }
 
+    GVALinkage Linkage = GetGVALinkageForFunction(FD);
+
+    if (Linkage == GVA_DiscardableODR) {
+      if (WritingModule)
+        return true;
+      if (auto *Ext = getExternalSource())
+        if (Ext->hasExternalDefinitions(FD->getOwningModuleID()) ==
+            ExternalASTSource::EK_Never)
+          return true;
+    }
+
     // static, static inline, always_inline, and extern inline functions can
     // always be deferred.  Normal inline functions can be deferred in C99/C++.
     // Implicit template instantiations can also be deferred in C++.
     return !isDiscardableGVALinkage(GetGVALinkageForFunction(FD));
   }
   
   const VarDecl *VD = cast<VarDecl>(D);
   assert(VD->isFileVarDecl() && "Expected file scoped var");
 
   if (VD->isThisDeclarationADefinition() == VarDecl::DeclarationOnly &&
       !isMSStaticDataMemberInlineDefinition(VD))
     return false;

lib/AST/ExternalASTSource.cpp

 
 ExternalASTSource::~ExternalASTSource() { }
 
 llvm::Optional<ExternalASTSource::ASTSourceDescriptor>
 ExternalASTSource::getSourceDescriptor(unsigned ID) {
   return None;
 }
 
+ExternalASTSource::ExtKind
+ExternalASTSource::hasExternalDefinitions(unsigned ID) {
+  return EK_ReplyHazy;
+}
+
 ExternalASTSource::ASTSourceDescriptor::ASTSourceDescriptor(const Module &M)
   : Signature(M.Signature), ClangModule(&M) {
   if (M.Directory)
     Path = M.Directory->getName();
   if (auto *File = M.getASTFile())
     ASTFile = File->getName();
 }
 

lib/Basic/Module.cpp

Module::Module(StringRef Name, SourceLocation DefinitionLoc, Module *Parent,
                bool IsFramework, bool IsExplicit, unsigned VisibilityID)
     : Name(Name), DefinitionLoc(DefinitionLoc), Parent(Parent), Directory(),
       Umbrella(), Signature(0), ASTFile(nullptr), VisibilityID(VisibilityID),
       IsMissingRequirement(false), HasIncompatibleModuleFile(false),
       IsAvailable(true), IsFromModuleFile(false), IsFramework(IsFramework),
       IsExplicit(IsExplicit), IsSystem(false), IsExternC(false),
       IsInferred(false), InferSubmodules(false), InferExplicitSubmodules(false),
       InferExportWildcard(false), ConfigMacrosExhaustive(false),
-      NoUndeclaredIncludes(false), NameVisibility(Hidden) {
+      NoUndeclaredIncludes(false), WithCodegen(false),
+      NameVisibility(Hidden) {
   if (Parent) {
     if (!Parent->isAvailable())
       IsAvailable = false;
     if (Parent->IsSystem)
       IsSystem = true;
     if (Parent->IsExternC)
       IsExternC = true;
     if (Parent->NoUndeclaredIncludes)

lib/CodeGen/CodeGenModule.cpp

if (IsConstantVariable)
       return llvm::GlobalVariable::WeakODRLinkage;
     else
       return llvm::GlobalVariable::WeakAnyLinkage;
   }
 
   // We are guaranteed to have a strong definition somewhere else,
   // so we can use available_externally linkage.
   if (Linkage == GVA_AvailableExternally)
-    return llvm::Function::AvailableExternallyLinkage;
+    return llvm::GlobalValue::AvailableExternallyLinkage;
 
   // Note that Apple's kernel linker doesn't support symbol
   // coalescing, so we need to avoid linkonce and weak linkages there.
   // Normally, this means we just map to internal, but for explicit
   // instantiations we'll map to external.
 
   // In C++, the compiler has to emit a definition in every translation unit
   // that references the function.  We should use linkonce_odr because

lib/Frontend/CompilerInvocation.cpp

#include "clang/Frontend/LangStandards.def"
   Opts.CoroutinesTS = Args.hasArg(OPT_fcoroutines_ts);
   Opts.ModulesTS = Args.hasArg(OPT_fmodules_ts);
   Opts.Modules = Args.hasArg(OPT_fmodules) || Opts.ModulesTS;
   Opts.ModulesStrictDeclUse = Args.hasArg(OPT_fmodules_strict_decluse);
   Opts.ModulesDeclUse =
       Args.hasArg(OPT_fmodules_decluse) || Opts.ModulesStrictDeclUse;
   Opts.ModulesLocalVisibility =
       Args.hasArg(OPT_fmodules_local_submodule_visibility) || Opts.ModulesTS;
+  Opts.ModularCodegen = Args.hasArg(OPT_fmodule_codegen);
   Opts.ModulesSearchAll = Opts.Modules &&
     !Args.hasArg(OPT_fno_modules_search_all) &&
     Args.hasArg(OPT_fmodules_search_all);
   Opts.ModulesErrorRecovery = !Args.hasArg(OPT_fno_modules_error_recovery);
   Opts.ImplicitModules = !Args.hasArg(OPT_fno_implicit_modules);
   Opts.CharIsSigned = Opts.OpenCL || !Args.hasArg(OPT_fno_signed_char);
   Opts.WChar = Opts.CPlusPlus && !Args.hasArg(OPT_fno_wchar);
   Opts.ShortWChar = Args.hasFlag(OPT_fshort_wchar, OPT_fno_short_wchar, false);

lib/Lex/ModuleMap.cpp

 
 ModuleMap::ModuleMap(SourceManager &SourceMgr, DiagnosticsEngine &Diags,
                      const LangOptions &LangOpts, const TargetInfo *Target,
                      HeaderSearch &HeaderInfo)
     : SourceMgr(SourceMgr), Diags(Diags), LangOpts(LangOpts), Target(Target),
       HeaderInfo(HeaderInfo), BuiltinIncludeDir(nullptr),
       SourceModule(nullptr), NumCreatedModules(0) {
   MMapLangOpts.LineComment = true;
+  MMapLangOpts.ModularCodegen = LangOpts.ModularCodegen;
 }
 
 ModuleMap::~ModuleMap() {
   for (auto &M : Modules)
     delete M.getValue();
 }
 
 void ModuleMap::setTarget(const TargetInfo &Target) {

 
 Module *ModuleMap::lookupModuleQualified(StringRef Name, Module *Context) const{
   if (!Context)
     return findModule(Name);
   
   return Context->findSubmodule(Name);
 }
 
-std::pair<Module *, bool> 
-ModuleMap::findOrCreateModule(StringRef Name, Module *Parent, bool IsFramework,
+std::pair<Module *, bool> ModuleMap::findOrCreateModule(StringRef Name,
+                                                        Module *Parent,
+                                                        bool IsFramework,
                                                         bool IsExplicit) {
   // Try to find an existing module with this name.
   if (Module *Sub = lookupModuleQualified(Name, Parent))
     return std::make_pair(Sub, false);
   
   // Create a new module with this name.
-  Module *Result = new Module(Name, SourceLocation(), Parent,
-                              IsFramework, IsExplicit, NumCreatedModules++);
+  Module *Result = new Module(Name, SourceLocation(), Parent, IsFramework,
+                              IsExplicit, NumCreatedModules++);
   if (!Parent) {
     if (LangOpts.CurrentModule == Name)
       SourceModule = Result;
     Modules[Name] = Result;
   }
   return std::make_pair(Result, true);
 }
 
void ModuleMapParser::parseModuleDecl() {
   if (Attrs.IsSystem || IsSystem)
     ActiveModule->IsSystem = true;
   if (Attrs.IsExternC)
     ActiveModule->IsExternC = true;
   if (Attrs.NoUndeclaredIncludes ||
       (!ActiveModule->Parent && ModuleName == "Darwin"))
     ActiveModule->NoUndeclaredIncludes = true;
   ActiveModule->Directory = Directory;
+  ActiveModule->WithCodegen = L.getLangOpts().ModularCodegen;
 
   if (!ActiveModule->Parent) {
     StringRef MapFileName(ModuleMapFile->getName());
     if (MapFileName.endswith("module.private.modulemap") ||
         MapFileName.endswith("module_private.map")) {
       // Adding a top-level module from a private modulemap is likely a
       // user error; we check to see if there's another top-level module
       // defined in the non-private map in the same dir, and if so emit a

lib/Serialization/ASTReader.cpp

case llvm::BitstreamEntry::SubBlock:
         break;
 
       case SOURCE_MANAGER_BLOCK_ID:
         if (ReadSourceManagerBlock(F))
           return Failure;
         break;
 
       case SUBMODULE_BLOCK_ID:
-        if (ASTReadResult Result = ReadSubmoduleBlock(F, ClientLoadCapabilities))
+        if (ASTReadResult Result =
+                ReadSubmoduleBlock(F, ClientLoadCapabilities))
           return Result;
         break;
 
       case COMMENTS_BLOCK_ID: {
         BitstreamCursor C = Stream;
         if (Stream.SkipBlock() ||
             ReadBlockAbbrevs(C, COMMENTS_BLOCK_ID)) {
           Error("malformed comments block in AST file");
while (true) {
 
     case EAGERLY_DESERIALIZED_DECLS:
       // FIXME: Skip reading this record if our ASTConsumer doesn't care
       // about "interesting" decls (for instance, if we're building a module).
       for (unsigned I = 0, N = Record.size(); I != N; ++I)
         EagerlyDeserializedDecls.push_back(getGlobalDeclID(F, Record[I]));
       break;
 
+    case MODULAR_CODEGEN_DECLS:
+      // FIXME: Skip reading this record if our ASTConsumer doesn't care about
+      // them (ie: if we're not codegenerating this module).
+      if (F.Kind == MK_MainFile)
+        for (unsigned I = 0, N = Record.size(); I != N; ++I)
+          EagerlyDeserializedDecls.push_back(getGlobalDeclID(F, Record[I]));
+      break;
+
     case SPECIAL_TYPES:
       if (SpecialTypes.empty()) {
         for (unsigned I = 0, N = Record.size(); I != N; ++I)
           SpecialTypes.push_back(getGlobalTypeID(F, Record[I]));
         break;
       }
 
       if (SpecialTypes.size() != Record.size()) {
case SUBMODULE_DEFINITION: {
       bool IsFramework = Record[Idx++];
       bool IsExplicit = Record[Idx++];
       bool IsSystem = Record[Idx++];
       bool IsExternC = Record[Idx++];
       bool InferSubmodules = Record[Idx++];
       bool InferExplicitSubmodules = Record[Idx++];
       bool InferExportWildcard = Record[Idx++];
       bool ConfigMacrosExhaustive = Record[Idx++];
+      bool WithCodegen = Record[Idx++];
 
       Module *ParentModule = nullptr;
       if (Parent)
         ParentModule = getSubmodule(Parent);
 
       // Retrieve this (sub)module from the module map, creating it if
       // necessary.
-      CurrentModule = ModMap.findOrCreateModule(Name, ParentModule, IsFramework,
-                                                IsExplicit).first;
+      CurrentModule =
+          ModMap.findOrCreateModule(Name, ParentModule, IsFramework, IsExplicit)
+              .first;
 
       // FIXME: set the definition loc for CurrentModule, or call
       // ModMap.setInferredModuleAllowedBy()
 
       SubmoduleID GlobalIndex = GlobalID - NUM_PREDEF_SUBMODULE_IDS;
       if (GlobalIndex >= SubmodulesLoaded.size() ||
           SubmodulesLoaded[GlobalIndex]) {
         Error("too many submodules");
case SUBMODULE_DEFINITION: {
       CurrentModule->Signature = F.Signature;
       CurrentModule->IsFromModuleFile = true;
       CurrentModule->IsSystem = IsSystem || CurrentModule->IsSystem;
       CurrentModule->IsExternC = IsExternC;
       CurrentModule->InferSubmodules = InferSubmodules;
       CurrentModule->InferExplicitSubmodules = InferExplicitSubmodules;
       CurrentModule->InferExportWildcard = InferExportWildcard;
       CurrentModule->ConfigMacrosExhaustive = ConfigMacrosExhaustive;
+      CurrentModule->WithCodegen = WithCodegen;
       if (DeserializationListener)
         DeserializationListener->ModuleRead(GlobalID, CurrentModule);
 
       SubmodulesLoaded[GlobalIndex] = CurrentModule;
 
       // Clear out data that will be replaced by what is in the module file.
       CurrentModule->LinkLibraries.clear();
       CurrentModule->ConfigMacros.clear();
if (ModuleMgr.size() == 1) {
     StringRef ModuleName = llvm::sys::path::filename(MF.OriginalSourceFileName);
     StringRef FileName = llvm::sys::path::filename(MF.FileName);
     return ASTReader::ASTSourceDescriptor(ModuleName, MF.OriginalDir, FileName,
                                           MF.Signature);
   }
   return None;
 }
 
+ExternalASTSource::ExtKind ASTReader::hasExternalDefinitions(unsigned ID) {
+  const Module *M = getSubmodule(ID);
+  if (!M || !M->WithCodegen)
+    return EK_ReplyHazy;
+
+  ModuleFile *MF = ModuleMgr.lookup(M->getASTFile());
+  assert(MF); // ?
+  if (MF->Kind == ModuleKind::MK_MainFile)
+    return EK_Never;
+  return EK_Always;
+}
+
 Selector ASTReader::getLocalSelector(ModuleFile &M, unsigned LocalID) {
   return DecodeSelector(getGlobalSelectorID(M, LocalID));
 }
 
 Selector ASTReader::DecodeSelector(serialization::SelectorID ID) {
   if (ID == 0)
     return Selector();
 

lib/Serialization/ASTWriter.cpp

#define RECORD(X) EmitRecordID(X, #X, Stream, Record)
 
   // AST Top-Level Block.
   BLOCK(AST_BLOCK);
   RECORD(TYPE_OFFSET);
   RECORD(DECL_OFFSET);
   RECORD(IDENTIFIER_OFFSET);
   RECORD(IDENTIFIER_TABLE);
   RECORD(EAGERLY_DESERIALIZED_DECLS);
+  RECORD(MODULAR_CODEGEN_DECLS);
   RECORD(SPECIAL_TYPES);
   RECORD(STATISTICS);
   RECORD(TENTATIVE_DEFINITIONS);
   RECORD(SELECTOR_OFFSETS);
   RECORD(METHOD_POOL);
   RECORD(PP_COUNTER_VALUE);
   RECORD(SOURCE_LOCATION_OFFSETS);
   RECORD(SOURCE_LOCATION_PRELOADS);
void ASTWriter::WriteSubmodules(Module *WritingModule) {
   Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 1)); // IsFramework
   Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 1)); // IsExplicit
   Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 1)); // IsSystem
   Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 1)); // IsExternC
   Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 1)); // InferSubmodules...
   Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 1)); // InferExplicit...
   Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 1)); // InferExportWild...
   Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 1)); // ConfigMacrosExh...
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 2)); // WithCodegen
   Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Blob)); // Name
   unsigned DefinitionAbbrev = Stream.EmitAbbrev(std::move(Abbrev));
 
   Abbrev = std::make_shared<BitCodeAbbrev>();
   Abbrev->Add(BitCodeAbbrevOp(SUBMODULE_UMBRELLA_HEADER));
   Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Blob)); // Name
   unsigned UmbrellaAbbrev = Stream.EmitAbbrev(std::move(Abbrev));
 
while (!Q.empty()) {
     }
 
     // Emit the definition of the block.
     {
       RecordData::value_type Record[] = {
           SUBMODULE_DEFINITION, ID, ParentID, Mod->IsFramework, Mod->IsExplicit,
           Mod->IsSystem, Mod->IsExternC, Mod->InferSubmodules,
           Mod->InferExplicitSubmodules, Mod->InferExportWildcard,
-          Mod->ConfigMacrosExhaustive};
+          Mod->ConfigMacrosExhaustive, Mod->WithCodegen};
       Stream.EmitRecordWithBlob(DefinitionAbbrev, Record, Mod->Name);
     }
 
     // Emit the requirements.
     for (const auto &R : Mod->Requirements) {
       RecordData::value_type Record[] = {SUBMODULE_REQUIRES, R.second};
       Stream.EmitRecordWithBlob(RequiresAbbrev, Record, R.first);
     }
if (WritingModule)
     WriteSubmodules(WritingModule);
 
   Stream.EmitRecord(SPECIAL_TYPES, SpecialTypes);
 
   // Write the record containing external, unnamed definitions.
   if (!EagerlyDeserializedDecls.empty())
     Stream.EmitRecord(EAGERLY_DESERIALIZED_DECLS, EagerlyDeserializedDecls);
 
+  if (Context.getLangOpts().ModularCodegen)
+    Stream.EmitRecord(MODULAR_CODEGEN_DECLS, ModularCodegenDecls);
+
   // Write the record containing tentative definitions.
   if (!TentativeDefinitions.empty())
     Stream.EmitRecord(TENTATIVE_DEFINITIONS, TentativeDefinitions);
 
   // Write the record containing unused file scoped decls.
   if (!UnusedFileScopedDecls.empty())
     Stream.EmitRecord(UNUSED_FILESCOPED_DECLS, UnusedFileScopedDecls);
 

lib/Serialization/ASTWriterDecl.cpp

 /// code generation requirements, if other clients have a different notion of
 /// what is "required" then we may have to consider an alternate scheme where
 /// clients can iterate over the top-level decls and get information on them,
 /// without necessary deserializing them. We could explicitly require such
 /// clients to use a separate API call to "realize" the decl. This should be
 /// relatively painless since they would presumably only do it for top-level
 /// decls.
 static bool isRequiredDecl(const Decl *D, ASTContext &Context,
-                           bool WritingModule) {
+                           bool WritingModule, bool ModularCode) {
   // An ObjCMethodDecl is never considered as "required" because its
   // implementation container always is.
 
   // File scoped assembly or obj-c or OMP declare target implementation must be
   // seen.
   if (isa<FileScopeAsmDecl>(D) || isa<ObjCImplDecl>(D) ||
       D->hasAttr<OMPDeclareTargetDeclAttr>())
     return true;
 
   if (WritingModule && (isa<VarDecl>(D) || isa<ImportDecl>(D))) {
     // These declarations are part of the module initializer, and are emitted
     // if and when the module is imported, rather than being emitted eagerly.
     return false;
   }
 
-  return Context.DeclMustBeEmitted(D);
+  return Context.DeclMustBeEmitted(D, ModularCode);
 }
 
 void ASTWriter::WriteDecl(ASTContext &Context, Decl *D) {
   // Determine the ID for this declaration.
   serialization::DeclID ID;
   assert(!D->isFromASTFile() && "should not be emitting imported decl");
   serialization::DeclID &IDR = DeclIDs[D];
   if (IDR == 0)
void ASTWriter::WriteDecl(ASTContext &Context, Decl *D) {
   }
 
   SourceManager &SM = Context.getSourceManager();
   if (Loc.isValid() && SM.isLocalSourceLocation(Loc))
     associateDeclWithFile(D, ID);
 
   // Note declarations that should be deserialized eagerly so that we can add
   // them to a record in the AST file later.
-  if (isRequiredDecl(D, Context, WritingModule))
+  if (isRequiredDecl(D, Context, WritingModule, false))
     EagerlyDeserializedDecls.push_back(ID);
+  else if (Context.getLangOpts().ModularCodegen && WritingModule &&
+           isRequiredDecl(D, Context, true, true))
+    ModularCodegenDecls.push_back(ID);
 }
 
 void ASTRecordWriter::AddFunctionDefinition(const FunctionDecl *FD) {
   // Switch case IDs are per function body.
   Writer->ClearSwitchCaseIDs();
 
   assert(FD->doesThisDeclarationHaveABody());
   if (auto *CD = dyn_cast<CXXConstructorDecl>(FD)) {
     Record->push_back(CD->getNumCtorInitializers());
     if (CD->getNumCtorInitializers())
       AddCXXCtorInitializers(
           llvm::makeArrayRef(CD->init_begin(), CD->init_end()));
   }
   AddStmt(FD->getBody());
 }

test/Modules/Inputs/codegen/bar.h

+#include "foo.h"
+inline void bar() { foo(); }

test/Modules/Inputs/codegen/bar.modulemap

+module bar { header "bar.h" }

test/Modules/Inputs/codegen/foo.h

+void f1(int&);
+static void f2() { }
+inline void foo() {
+  static int i;
+  f1(i);
+  f2();
+}
+inline void foo2() {
+}
+void foo_ext() {}

test/Modules/Inputs/codegen/foo.modulemap

+module foo { header "foo.h" }

test/Modules/Inputs/codegen/use.cpp

+#include "bar.h"
+int main() { bar(); }

test/Modules/codegen.test

+RUN: rm -rf %t
+
+RUN: %clang_cc1 -fmodule-codegen -x c++ -fmodules -emit-module -fmodule-name=foo %S/Inputs/codegen/foo.modulemap -o %t/foo.pcm
+RUN: %clang_cc1 -fmodule-codegen -x c++ -fmodules -emit-module -fmodule-name=bar %S/Inputs/codegen/bar.modulemap -o %t/bar.pcm -fmodule-file=%t/foo.pcm
+
+RUN: %clang_cc1 -emit-llvm %t/foo.pcm -o - | FileCheck --check-prefix=FOO %s
+RUN: %clang_cc1 -emit-llvm %t/bar.pcm -o - -fmodule-file=%t/foo.pcm | FileCheck --check-prefix=BAR-CMN --check-prefix=BAR %s
+RUN: %clang_cc1 -fmodules -fmodule-file=%t/foo.pcm -fmodule-file=%t/bar.pcm %S/Inputs/codegen/use.cpp -emit-llvm -o - | FileCheck --check-prefix=USE-CMN --check-prefix=USE %s
+
+RUN: %clang_cc1 -O2 -disable-llvm-passes -emit-llvm %t/foo.pcm -o - | FileCheck --check-prefix=FOO %s
+RUN: %clang_cc1 -O2 -disable-llvm-passes -emit-llvm %t/bar.pcm -o - -fmodule-file=%t/foo.pcm | FileCheck --check-prefix=BAR-CMN --check-prefix=BAR-OPT %s
+RUN: %clang_cc1 -O2 -disable-llvm-passes -fmodules -fmodule-file=%t/foo.pcm -fmodule-file=%t/bar.pcm %S/Inputs/codegen/use.cpp -emit-llvm -o - | FileCheck --check-prefix=USE-CMN --check-prefix=USE-OPT %s
+
+FOO-NOT: comdat
+FOO: $_Z3foov = comdat any
+FOO: $_Z4foo2v = comdat any
+FOO: $_ZZ3foovE1i = comdat any
+FOO: @_ZZ3foovE1i = linkonce_odr global i32 0, comdat
+FOO-NOT: {{comdat|define|declare}}
+FOO: define void @_Z7foo_extv()
+FOO-NOT: {{define|declare}}
+FOO: define weak_odr void @_Z3foov() #{{[0-9]+}} comdat
+FOO-NOT: {{define|declare}}
+FOO: declare void @_Z2f1Ri(i32*
+FOO-NOT: {{define|declare}}
+
+FIXME: this internal function should be weak_odr, comdat, and with a new mangling
+FOO: define internal void @_ZL2f2v() #{{[0-9]+}}
+FOO-NOT: {{define|declare}}
+
+FOO: define weak_odr void @_Z4foo2v() #{{[0-9]+}} comdat
+FOO-NOT: {{define|declare}}
+
+
+BAR-CMN-NOT: comdat
+BAR-CMN: $_Z3barv = comdat any
+BAR-OPT: @_ZZ3foovE1i = linkonce_odr global i32 0, comdat
+BAR-CMN-NOT: {{comdat|define|declare}}
+BAR-CMN: define weak_odr void @_Z3barv() #{{[0-9]+}} comdat
+BAR-CMN-NOT: {{define|declare}}
+BAR: declare void @_Z3foov()
+Include all the available_externally definitions required for bar (foo -> f2)
+BAR-OPT: define available_externally void @_Z3foov()
+BAR-CMN-NOT: {{define|declare}}
+BAR-OPT: declare void @_Z2f1Ri(i32*
+BAR-OPT-NOT: {{define|declare}}
+BAR-OPT: define available_externally void @_ZL2f2v()
+BAR-OPT-NOT: {{define|declare}}
+
+
+USE-OPT: @_ZZ3foovE1i = linkonce_odr global i32 0, comdat
+USE-CMN-NOT: {{comdat|define|declare}}
+USE-CMN: define i32 @main()
+USE-CMN-NOT: {{define|declare}}
+USE: declare void @_Z3barv()
+Include all the available_externally definitions required for main (bar -> foo -> f2)
+USE-OPT: define available_externally void @_Z3barv()
+USE-CMN-NOT: {{define|declare}}
+USE-OPT: define available_externally void @_Z3foov()
+USE-OPT-NOT: {{define|declare}}
+USE-OPT: declare void @_Z2f1Ri(i32*
+USE-OPT-NOT: {{define|declare}}
+USE-OPT: define available_externally void @_ZL2f2v()
+USE-OPT-NOT: {{define|declare}}