Modular Codegen: Add/use a bit in serialized function definitions to track whether they are the subject of modular codegen

[dblaikie]

Summary

Some decls are created not where they are written, but in other module
files/users (implicit special members and function template implicit
specializations). To correctly identify them, use a bit next to the definition
to track the modular codegen property.

Possible improvement would be to store this bit on the Decl itself, rather than
a sidemap - happy to give that a whirl if it seems better. (actually it'd be
two bits, since 3 states are required - currently reflected by "not in the
map", "in the map with Always", "in the map with Never")

Discussed whether the module file bit could be omitted in favor of
reconstituting from the modular codegen decls list - best guess today is that
the efficiency improvement of not having to deserialize the whole list whenever
any function is queried by a module user is worth it for the small size
increase of this redundant (list + bit-on-def) representation.

[dblaikie]

• Simplify ModuleFile lookup

[dblaikie]

• Simplify ModuleFile lookup
• Build ModularCodegenDecls list from the same place the modular codegen bit is set on the decl
• Cleanup no-longer-needed changes to DeclMustBeEmitted/isRequiredDecl

[dblaikie]

sync/resolve

[dblaikie]

Ping

I've got a relatively small patch to add support for debug info types after this one & working on a patch on top of that to enable one/the other/both of these (to test their value independently/together). Might be more expedient to sit down & push through them all together side-by-side? Or not, not sure.

include/clang/AST/ExternalASTSource.h

public:
     const Module *getModuleOrNull() const { return ClangModule; }
   };
 
   /// Return a descriptor for the corresponding module, if one exists.
   virtual llvm::Optional<ASTSourceDescriptor> getSourceDescriptor(unsigned ID);
 
   enum ExtKind { EK_Always, EK_Never, EK_ReplyHazy };
 
-  virtual ExtKind hasExternalDefinitions(unsigned ID);
+  virtual ExtKind hasExternalDefinitions(const FunctionDecl *FD);
 
   /// \brief Finds all declarations lexically contained within the given
   /// DeclContext, after applying an optional filter predicate.
   ///
   /// \param IsKindWeWant a predicate function that returns true if the passed
   /// declaration kind is one we are looking for.
   ///
   /// The default implementation of this method is a no-op.

include/clang/Sema/MultiplexExternalSemaSource.h

public:
   /// \brief Resolve the offset of a set of C++ base specifiers in the decl
   /// stream into an array of specifiers.
   CXXBaseSpecifier *GetExternalCXXBaseSpecifiers(uint64_t Offset) override;
 
   /// \brief Resolve a handle to a list of ctor initializers into the list of
   /// initializers themselves.
   CXXCtorInitializer **GetExternalCXXCtorInitializers(uint64_t Offset) override;
 
-  ExtKind hasExternalDefinitions(unsigned ID) override;
+  ExtKind hasExternalDefinitions(const FunctionDecl *FD) override;
 
   /// \brief Find all declarations with the given name in the
   /// given context.
   bool FindExternalVisibleDeclsByName(const DeclContext *DC,
                                       DeclarationName Name) override;
 
   /// \brief Ensures that the table of all visible declarations inside this
   /// context is up to date.

include/clang/Serialization/ASTReader.h

private:
   ///
   /// In most cases, this string will be empty, because the predefines
   /// buffer computed to build the PCH file will be identical to the
   /// predefines buffer computed from the command line. However, when
   /// there are differences that the PCH reader can work around, this
   /// predefines buffer may contain additional definitions.
   std::string SuggestedPredefines;
 
+  std::map<const FunctionDecl*, ExternalASTSource::ExtKind> BodySource;
+
   /// \brief Reads a statement from the specified cursor.
   Stmt *ReadStmtFromStream(ModuleFile &F);
 
   struct InputFileInfo {
     std::string Filename;
     off_t StoredSize;
     time_t StoredTime;
     bool Overridden;
public:
   ModuleFile *getLocalModuleFile(ModuleFile &M, unsigned ID);
 
   /// \brief Get an ID for the given module file.
   unsigned getModuleFileID(ModuleFile *M);
 
   /// \brief Return a descriptor for the corresponding module.
   llvm::Optional<ASTSourceDescriptor> getSourceDescriptor(unsigned ID) override;
 
-  ExtKind hasExternalDefinitions(unsigned ID) override;
+  ExtKind hasExternalDefinitions(const FunctionDecl *FD) override;
 
   /// \brief Retrieve a selector from the given module with its local ID
   /// number.
   Selector getLocalSelector(ModuleFile &M, unsigned LocalID);
 
   Selector DecodeSelector(serialization::SelectorID Idx);
 
   Selector GetExternalSelector(serialization::SelectorID ID) override;

lib/AST/ASTContext.cpp

static GVALinkage adjustGVALinkageForAttributes(const ASTContext &Context,
   return L;
 }
 
 GVALinkage ASTContext::GetGVALinkageForFunction(const FunctionDecl *FD) const {
   auto L = adjustGVALinkageForAttributes(
       *this, basicGVALinkageForFunction(*this, FD), FD);
   auto EK = ExternalASTSource::EK_ReplyHazy;
   if (auto *Ext = getExternalSource())
-    EK = Ext->hasExternalDefinitions(FD->getOwningModuleID());
+    EK = Ext->hasExternalDefinitions(FD);
   switch (EK) {
   case ExternalASTSource::EK_Never:
     if (L == GVA_DiscardableODR)
       return GVA_StrongODR;
     break;
   case ExternalASTSource::EK_Always:
     return GVA_AvailableExternally;
   case ExternalASTSource::EK_ReplyHazy:

lib/AST/ExternalASTSource.cpp

 ExternalASTSource::~ExternalASTSource() { }
 
 llvm::Optional<ExternalASTSource::ASTSourceDescriptor>
 ExternalASTSource::getSourceDescriptor(unsigned ID) {
   return None;
 }
 
 ExternalASTSource::ExtKind
-ExternalASTSource::hasExternalDefinitions(unsigned ID) {
+ExternalASTSource::hasExternalDefinitions(const FunctionDecl *FD) {
   return EK_ReplyHazy;
 }
 
 ExternalASTSource::ASTSourceDescriptor::ASTSourceDescriptor(const Module &M)
   : Signature(M.Signature), ClangModule(&M) {
   if (M.Directory)
     Path = M.Directory->getName();
   if (auto *File = M.getASTFile())

lib/Sema/MultiplexExternalSemaSource.cpp

 MultiplexExternalSemaSource::GetExternalCXXCtorInitializers(uint64_t Offset) {
   for (auto *S : Sources)
     if (auto *R = S->GetExternalCXXCtorInitializers(Offset))
       return R;
   return nullptr;
 }
 
 ExternalASTSource::ExtKind
-MultiplexExternalSemaSource::hasExternalDefinitions(unsigned int ID) {
+MultiplexExternalSemaSource::hasExternalDefinitions(const FunctionDecl *FD) {
   for (const auto &S : Sources)
-    if (auto EK = S->hasExternalDefinitions(ID))
+    if (auto EK = S->hasExternalDefinitions(FD))
       if (EK != EK_ReplyHazy)
         return EK;
   return EK_ReplyHazy;
 }
 
 bool MultiplexExternalSemaSource::
 FindExternalVisibleDeclsByName(const DeclContext *DC, DeclarationName Name) {
   bool AnyDeclsFound = false;

lib/Serialization/ASTReader.cpp

if (ModuleMgr.size() == 1) {
     StringRef ModuleName = llvm::sys::path::filename(MF.OriginalSourceFileName);
     StringRef FileName = llvm::sys::path::filename(MF.FileName);
     return ASTReader::ASTSourceDescriptor(ModuleName, MF.OriginalDir, FileName,
                                           MF.Signature);
   }
   return None;
 }
 
-ExternalASTSource::ExtKind ASTReader::hasExternalDefinitions(unsigned ID) {
-  const Module *M = getSubmodule(ID);
-  if (!M || !M->WithCodegen)
+ExternalASTSource::ExtKind ASTReader::hasExternalDefinitions(const FunctionDecl *FD) {
+  auto I = BodySource.find(FD);
+  if (I == BodySource.end())
     return EK_ReplyHazy;
-
-  ModuleFile *MF = ModuleMgr.lookup(M->getASTFile());
-  assert(MF); // ?
-  if (MF->Kind == ModuleKind::MK_MainFile)
-    return EK_Never;
-  return EK_Always;
+  return I->second;
 }
 
 Selector ASTReader::getLocalSelector(ModuleFile &M, unsigned LocalID) {
   return DecodeSelector(getGlobalSelectorID(M, LocalID));
 }
 
 Selector ASTReader::DecodeSelector(serialization::SelectorID ID) {
   if (ID == 0)
void ASTReader::finishPendingActions() {
   // FIXME: We shouldn't require complete redeclaration chains here.
   for (PendingBodiesMap::iterator PB = PendingBodies.begin(),
                                PBEnd = PendingBodies.end();
        PB != PBEnd; ++PB) {
     if (FunctionDecl *FD = dyn_cast<FunctionDecl>(PB->first)) {
       // FIXME: Check for =delete/=default?
       // FIXME: Complain about ODR violations here?
       const FunctionDecl *Defn = nullptr;
-      if (!getContext().getLangOpts().Modules || !FD->hasBody(Defn))
+      if (!getContext().getLangOpts().Modules || !FD->hasBody(Defn)) {
         FD->setLazyBody(PB->second);
-      else
+      } else
         mergeDefinitionVisibility(const_cast<FunctionDecl*>(Defn), FD);
       continue;
     }
 
     ObjCMethodDecl *MD = cast<ObjCMethodDecl>(PB->first);
     if (!getContext().getLangOpts().Modules || !MD->hasBody())
       MD->setLazyBody(PB->second);
   }

lib/Serialization/ASTReaderDecl.cpp

return llvm::make_range(MergedRedeclIterator<DeclT>(D),
                           MergedRedeclIterator<DeclT>());
 }
 
 uint64_t ASTDeclReader::GetCurrentCursorOffset() {
   return Loc.F->DeclsCursor.GetCurrentBitNo() + Loc.F->GlobalBitOffset;
 }
 
 void ASTDeclReader::ReadFunctionDefinition(FunctionDecl *FD) {
+  if (Record.readInt()) {
+    Reader.BodySource[FD] =
+        Reader.getLocalBitOffset(GetCurrentCursorOffset()).F->Kind ==
+                ModuleKind::MK_MainFile
+            ? ExternalASTSource::EK_Never
+            : ExternalASTSource::EK_Always;
+  }
   if (auto *CD = dyn_cast<CXXConstructorDecl>(FD)) {
     CD->NumCtorInitializers = Record.readInt();
     if (CD->NumCtorInitializers)
       CD->CtorInitializers = ReadGlobalOffset();
   }
   // Store the offset of the body so we can lazily load it later.
   Reader.PendingBodies[FD] = GetCurrentCursorOffset();
   HasPendingBody = true;

lib/Serialization/ASTWriterDecl.cpp

else if (Context.getLangOpts().ModularCodegen && WritingModule &&
     ModularCodegenDecls.push_back(ID);
 }
 
 void ASTRecordWriter::AddFunctionDefinition(const FunctionDecl *FD) {
   // Switch case IDs are per function body.
   Writer->ClearSwitchCaseIDs();
 
   assert(FD->doesThisDeclarationHaveABody());
+  Record->push_back(Writer->Context->getLangOpts().ModularCodegen &&
+                    Writer->WritingModule);
   if (auto *CD = dyn_cast<CXXConstructorDecl>(FD)) {
     Record->push_back(CD->getNumCtorInitializers());
     if (CD->getNumCtorInitializers())
       AddCXXCtorInitializers(
           llvm::makeArrayRef(CD->init_begin(), CD->init_end()));
   }
   AddStmt(FD->getBody());
 }

test/Modules/Inputs/codegen/foo.h

 inline void f1(const char* fmt, ...) {
   __builtin_va_list args;
   __builtin_va_start(args, fmt);
 }
+
+struct non_trivial_dtor {
+  ~non_trivial_dtor();
+};
+
+struct implicit_dtor {
+  non_trivial_dtor d;
+};
+
+struct uninst_implicit_dtor {
+  non_trivial_dtor d;
+};
+
+inline void use_implicit_dtor() {
+  implicit_dtor d;
+}
+
+template<typename T>
+void inst() {
+}
+
+inline void inst_decl() {
+  // cause inst<int>'s declaration to be instantiated, without a definition.
+  (void)sizeof(&inst<int>);
+  inst<float>();
+}

test/Modules/Inputs/codegen/use.cpp

+#include "foo.h"
+void non_modular_use_of_implicit_dtor() {
+  implicit_dtor d1;
+  uninst_implicit_dtor d2;
+}
+void use_of_instantiated_declaration_without_definition() {
+  inst<int>();
+}

test/Modules/codegen.test

 RUN: rm -rf %t
 REQUIRES: x86-registered-target
 
 RUN: %clang_cc1 -triple=x86_64-linux-gnu -fmodules-codegen -x c++ -fmodules -emit-module -fmodule-name=foo %S/Inputs/codegen/foo.modulemap -o %t/foo.pcm
 
-RUN: %clang_cc1 -triple x86_64-linux-gnu -emit-llvm -o - %t/foo.pcm | FileCheck %s
+RUN: %clang_cc1 -triple x86_64-linux-gnu -emit-llvm -o - %t/foo.pcm | FileCheck --check-prefix=FOO --check-prefix=BOTH %s
+RUN: %clang_cc1 -triple x86_64-linux-gnu -emit-llvm -o - -fmodules -fmodule-file=%t/foo.pcm %S/Inputs/codegen/use.cpp | FileCheck --check-prefix=BOTH --check-prefix=USE %s
 
-CHECK: $_Z2f1PKcz = comdat any
-CHECK: define weak_odr void @_Z2f1PKcz(i8* %fmt, ...) #{{[0-9]+}} comdat
-CHECK:   call void @llvm.va_start(i8* %{{[a-zA-Z0-9]*}})
+FOO: $_Z2f1PKcz = comdat any
+FOO: $_ZN13implicit_dtorD1Ev = comdat any
+USE: $_Z4instIiEvv = comdat any
+FOO: $_ZN13implicit_dtorD2Ev = comdat any
+FOO: define weak_odr void @_Z2f1PKcz(i8* %fmt, ...) #{{[0-9]+}} comdat
+FOO:   call void @llvm.va_start(i8* %{{[a-zA-Z0-9]*}})
+
+Test that implicit special members (like this dtor) are emitted into both module (if they're used there) and user.
+FIXME: Proactively instantiate any valid implicit special members to emit them into the module object.
+
+FOO: define weak_odr void @_ZN13implicit_dtorD1Ev(%struct.implicit_dtor* %this) unnamed_addr #0 comdat align 2 {
+FOO: define weak_odr void @_Z4instIfEvv() #0 comdat {
+FOO: define weak_odr void @_ZN13implicit_dtorD2Ev(%struct.implicit_dtor* %this) unnamed_addr #0 comdat align 2 {
+
+USE: define linkonce_odr void @_ZN20uninst_implicit_dtorD1Ev(%struct.uninst_implicit_dtor* %this) unnamed_addr #0 comdat align 2 {
+USE: define linkonce_odr void @_Z4instIiEvv() #0 comdat {
+USE: define linkonce_odr void @_ZN20uninst_implicit_dtorD2Ev(%struct.uninst_implicit_dtor* %this) unnamed_addr #0 comdat align 2 {