[astimporter] Support importing CXXDependentScopeMemberExpr and FunctionTemplateDecl

[khazem]

Summary

This patch adds support for importing two different kinds of AST nodes. The patch also adds an AST Matcher for cxxDependentScopeMemberExpr to support the addition of three new unit tests.

This fixes an issue where Clang would error out with "Unsupported AST node..." when attempting to import programs containing these nodes.

[a.sidorin]

Hello Kareem,
Thank you for working on it! You can find my comments below.

include/clang/ASTMatchers/ASTMatchers.h

5459

I'd prefer to have a discussion with ASTMatcher developers (@aaron.ballman, @klimek) first. Maybe it is better to place this matcher into ASTImporterTest.cpp - as I understand, it will be used very rarely.

lib/AST/ASTImporter.cpp

3495

If DC is nullptr, we should return on the previous line - ImportDeclParts returns true in this case. We may turn it into an assertion, however.

3509

Seems like we don't attempt to lookup an existing declaration first. This may lead to appearance of multiple definitions of same decl. I'd also prefer to see a functional test for this in test/ASTMerge.

You can also refer to https://github.com/haoNoQ/clang/blob/summary-ipa-draft/lib/AST/ASTImporter.cpp#L3594 because some important parts (setting of lexical decl context, adding a decl to the imported map) are missed here.

[khazem]

Thanks for the comments, Aleksei!

I've merged some aspects of the code you pointed me to, although I had to change some of it because some of the function calls have changed since 2015. In particular, I'm checking for already-imported functions, and adding the decl to the imported map.

I'm not sure how to proceed with tests. I was looking at test/ASTMerge/class-template-partial-spec.cpp, would it be something like this?

But I'm not sure what error I should throw if we have already imported the function, there is no appropriate error in include/clang/Basic/DiagnosticASTKinds.td. Should I define a new error, e.g. "function %0 defined in multiple translation units"?

[a.sidorin]

Hello Kareem.

There are some tests in "class-template" dir that may serve as example. You don't need a warning for a function that was found: you should just return found declaration. There is already a warning for a declaration that is defined in multiple TUs. You can run these tests to see them.

[spyffe]

There are several missing imports here, as well as a few minor nits.
If the unit test cases aren't catching these, I'm a little concerned. We should be catching this.
Also we should definitely test that bodies of function templates (in particular, bodies that use the template arguments) get imported properly.

lib/AST/ASTImporter.cpp

2327

Would

for (Attr *A : From->atrs()) {

work in this case?

3564

You didn't import TemplatedFD before installing it in ToFunc. This code is broken, and we should make sure the unit tests know to catch these cases.

6482

You're installing Base and BaseType without importing them, as well as all the Locs.

include/clang/ASTMatchers/ASTMatchers.h

 /// \code
 ///   kernel<<<i,j>>>();
 /// \endcode
 const internal::VariadicDynCastAllOfMatcher<
   Stmt,
   CUDAKernelCallExpr> cudaKernelCallExpr;
 
 
+/// \brief Matches member access expressions where the member could not
+///        be resolved because the base member was dependent.
+///
+/// Example matches d.t, and also (&d)->t
+/// \code
+///   template <typename T> class C {T t; };
+///   template <typename T> void foo() {
+///     C<T> d;
+///     d.t = T();
+///     (&d)->t = T();
+///   }
+/// \endcode
+const internal::VariadicDynCastAllOfMatcher<Expr, CXXDependentScopeMemberExpr>
+  cxxDependentScopeMemberExpr;
+
+
 /// \brief Matches expressions that resolve to a null pointer constant, such as
 /// GNU's __null, C++11's nullptr, or C's NULL macro.
 ///
 /// Given:
 /// \code
 ///   void *v1 = NULL;
 ///   void *v2 = nullptr;
 ///   void *v3 = __null; // GNU extension

lib/AST/ASTImporter.cpp

     bool ImportDefinition(VarDecl *From, VarDecl *To,
                           ImportDefinitionKind Kind = IDK_Default);
     bool ImportDefinition(EnumDecl *From, EnumDecl *To,
                           ImportDefinitionKind Kind = IDK_Default);
     bool ImportDefinition(ObjCInterfaceDecl *From, ObjCInterfaceDecl *To,
                           ImportDefinitionKind Kind = IDK_Default);
     bool ImportDefinition(ObjCProtocolDecl *From, ObjCProtocolDecl *To,
                           ImportDefinitionKind Kind = IDK_Default);
+
+    void ImportAttributes(Decl *From, Decl *To);
+
     TemplateParameterList *ImportTemplateParameterList(
                                                  TemplateParameterList *Params);
     TemplateArgument ImportTemplateArgument(const TemplateArgument &From);
     TemplateArgumentLoc ImportTemplateArgumentLoc(
         const TemplateArgumentLoc &TALoc, bool &Error);
     bool ImportTemplateArguments(const TemplateArgument *FromArgs,
                                  unsigned NumFromArgs,
                                SmallVectorImpl<TemplateArgument> &ToArgs);
     bool IsStructuralMatch(RecordDecl *FromRecord, RecordDecl *ToRecord,
                            bool Complain = true);
     bool IsStructuralMatch(VarDecl *FromVar, VarDecl *ToVar,
                            bool Complain = true);
     bool IsStructuralMatch(EnumDecl *FromEnum, EnumDecl *ToRecord);
     bool IsStructuralMatch(EnumConstantDecl *FromEC, EnumConstantDecl *ToEC);
+    bool IsStructuralMatch(FunctionTemplateDecl *From,
+                           FunctionTemplateDecl *To);
     bool IsStructuralMatch(ClassTemplateDecl *From, ClassTemplateDecl *To);
     bool IsStructuralMatch(VarTemplateDecl *From, VarTemplateDecl *To);
     Decl *VisitDecl(Decl *D);
     Decl *VisitAccessSpecDecl(AccessSpecDecl *D);
     Decl *VisitStaticAssertDecl(StaticAssertDecl *D);
     Decl *VisitTranslationUnitDecl(TranslationUnitDecl *D);
     Decl *VisitNamespaceDecl(NamespaceDecl *D);
     Decl *VisitTypedefNameDecl(TypedefNameDecl *D, bool IsAlias);
     Decl *VisitTypedefDecl(TypedefDecl *D);
     Decl *VisitTypeAliasDecl(TypeAliasDecl *D);
     Decl *VisitLabelDecl(LabelDecl *D);
     Decl *VisitEnumDecl(EnumDecl *D);
     Decl *VisitRecordDecl(RecordDecl *D);
     Decl *VisitEnumConstantDecl(EnumConstantDecl *D);
     Decl *VisitFunctionDecl(FunctionDecl *D);
     Decl *VisitCXXMethodDecl(CXXMethodDecl *D);
     Decl *VisitCXXConstructorDecl(CXXConstructorDecl *D);
     Decl *VisitCXXDestructorDecl(CXXDestructorDecl *D);
     Decl *VisitCXXConversionDecl(CXXConversionDecl *D);
     Decl *VisitFieldDecl(FieldDecl *D);
     Decl *VisitIndirectFieldDecl(IndirectFieldDecl *D);
     Decl *VisitFriendDecl(FriendDecl *D);
+    Decl *VisitFunctionTemplateDecl(FunctionTemplateDecl *D);
     Decl *VisitObjCIvarDecl(ObjCIvarDecl *D);
     Decl *VisitVarDecl(VarDecl *D);
     Decl *VisitImplicitParamDecl(ImplicitParamDecl *D);
     Decl *VisitParmVarDecl(ParmVarDecl *D);
     Decl *VisitObjCMethodDecl(ObjCMethodDecl *D);
     Decl *VisitObjCTypeParamDecl(ObjCTypeParamDecl *D);
     Decl *VisitObjCCategoryDecl(ObjCCategoryDecl *D);
     Decl *VisitObjCProtocolDecl(ObjCProtocolDecl *D);

     Expr *VisitCXXNoexceptExpr(CXXNoexceptExpr *E);
     Expr *VisitCXXDefaultArgExpr(CXXDefaultArgExpr *E);
     Expr *VisitCXXScalarValueInitExpr(CXXScalarValueInitExpr *E);
     Expr *VisitCXXBindTemporaryExpr(CXXBindTemporaryExpr *E);
     Expr *VisitCXXTemporaryObjectExpr(CXXTemporaryObjectExpr *CE);
     Expr *VisitMaterializeTemporaryExpr(MaterializeTemporaryExpr *E);
     Expr *VisitCXXNewExpr(CXXNewExpr *CE);
     Expr *VisitCXXDeleteExpr(CXXDeleteExpr *E);
+    Expr *VisitCXXDependentScopeMemberExpr(CXXDependentScopeMemberExpr *E);
     Expr *VisitCXXConstructExpr(CXXConstructExpr *E);
     Expr *VisitCXXMemberCallExpr(CXXMemberCallExpr *E);
     Expr *VisitExprWithCleanups(ExprWithCleanups *EWC);
     Expr *VisitCXXThisExpr(CXXThisExpr *E);
     Expr *VisitCXXBoolLiteralExpr(CXXBoolLiteralExpr *E);
     Expr *VisitMemberExpr(MemberExpr *E);
     Expr *VisitCallExpr(CallExpr *E);
     Expr *VisitInitListExpr(InitListExpr *E);

   // FIXME: we might need to merge the number of positive or negative bits
   // if the enumerator lists don't match.
   To->completeDefinition(T, ToPromotionType,
                          From->getNumPositiveBits(),
                          From->getNumNegativeBits());
   return false;
 }
 
+void ASTNodeImporter::ImportAttributes(Decl *From, Decl *To) {
+  for (Decl::attr_iterator I = From->attr_begin(), E = From->attr_end(); I != E;
+       ++I) {
+    Attr *ToAttr = (*I)->clone(Importer.getToContext());
+    ToAttr->setRange(Importer.Import((*I)->getRange()));
+    To->addAttr(ToAttr);
+  }
+}
+
 TemplateParameterList *ASTNodeImporter::ImportTemplateParameterList(
                                                 TemplateParameterList *Params) {
   SmallVector<NamedDecl *, 4> ToParams(Params->size());
   if (ImportContainerChecked(*Params, ToParams))
     return nullptr;
   
   Expr *ToRequiresClause;
   if (Expr *const R = Params->getRequiresClause()) {

   const llvm::APSInt &FromVal = FromEC->getInitVal();
   const llvm::APSInt &ToVal = ToEC->getInitVal();
 
   return FromVal.isSigned() == ToVal.isSigned() &&
          FromVal.getBitWidth() == ToVal.getBitWidth() &&
          FromVal == ToVal;
 }
 
+bool ASTNodeImporter::IsStructuralMatch(FunctionTemplateDecl *From,
+                                        FunctionTemplateDecl *To) {
+  StructuralEquivalenceContext Ctx(Importer.getFromContext(),
+                                   Importer.getToContext(),
+                                   Importer.getNonEquivalentDecls(),
+                                   false, false);
+  return Ctx.IsStructurallyEquivalent(From, To);
+}
+
 bool ASTNodeImporter::IsStructuralMatch(ClassTemplateDecl *From,
                                         ClassTemplateDecl *To) {
   StructuralEquivalenceContext Ctx(Importer.getFromContext(),
                                    Importer.getToContext(),
                                    Importer.getNonEquivalentDecls());
   return Ctx.IsStructurallyEquivalent(From, To);  
 }
 

   FriendDecl::FriendUnion ToFU;
   if (NamedDecl *FriendD = D->getFriendDecl())
     ToFU = cast_or_null<NamedDecl>(Importer.Import(FriendD));
   else
     ToFU = Importer.Import(D->getFriendType());
   if (!ToFU)
     return nullptr;
 
   SmallVector<TemplateParameterList *, 1> ToTPLists(D->NumTPLists);
   TemplateParameterList **FromTPLists =
       D->getTrailingObjects<TemplateParameterList *>();
   for (unsigned I = 0; I < D->NumTPLists; I++) {
     TemplateParameterList *List = ImportTemplateParameterList(FromTPLists[I]);
     if (!List)
       return nullptr;
     ToTPLists[I] = List;
   }
 
   FriendDecl *FrD = FriendDecl::Create(Importer.getToContext(), DC,
                                        Importer.Import(D->getLocation()),
                                        ToFU, Importer.Import(D->getFriendLoc()),
                                        ToTPLists);
 
   Importer.Imported(D, FrD);
   RD->pushFriendDecl(FrD);
 
   FrD->setAccess(D->getAccess());
   FrD->setLexicalDeclContext(LexicalDC);
   LexicalDC->addDeclInternal(FrD);
   return FrD;
 }
 
+Decl *ASTNodeImporter::VisitFunctionTemplateDecl(FunctionTemplateDecl *D) {
+  DeclContext *DC, *LexicalDC;
+  DeclarationName Name;
+  SourceLocation Loc;
+  NamedDecl *ToD;
+
+  if (ImportDeclParts(D, DC, LexicalDC, Name, ToD, Loc))
+    return nullptr;
+  assert(DC && "Null DeclContext after importing decl parts");
+  if (ToD)
+      return ToD;
+
+  // Try to find a function in our own ("to") context with the same name, same
+  // type, and in the same context as the function we're importing.
+  if (!LexicalDC->isFunctionOrMethod()) {
+    unsigned IDNS = Decl::IDNS_Ordinary;
+    SmallVector<NamedDecl *, 2> FoundDecls;
+    DC->getRedeclContext()->localUncachedLookup(Name, FoundDecls);
+    for (unsigned I = 0, N = FoundDecls.size(); I != N; ++I) {
+      if (!FoundDecls[I]->isInIdentifierNamespace(IDNS))
+        continue;
+
+      if (FunctionTemplateDecl *FoundFunction =
+          dyn_cast<FunctionTemplateDecl>(FoundDecls[I])) {
+        if (FoundFunction->hasExternalFormalLinkage() &&
+            D->hasExternalFormalLinkage()) {
+          if (IsStructuralMatch(D, FoundFunction)) {
+            Importer.Imported(D, FoundFunction);
+            // FIXME: Actually try to merge the body and other attributes.
+            return FoundFunction;
+          }
+        }
+      }
+    }
+  }
+
+  TemplateParameterList *Params = ImportTemplateParameterList(
+                                    D->getTemplateParameters());
+  if (!Params)
+    return nullptr;
+
+  FunctionDecl *TemplatedFD = cast_or_null<FunctionDecl>(D->getTemplatedDecl());
+  if (!TemplatedFD)
+    return nullptr;
+
+  FunctionTemplateDecl *ToFunc = FunctionTemplateDecl::Create(
+      Importer.getToContext(), DC, Loc, Name, Params, TemplatedFD);
+
+  TemplatedFD->setDescribedFunctionTemplate(ToFunc);
+  ImportAttributes(D, ToFunc);
+  ToFunc->setAccess(D->getAccess());
+  ToFunc->setLexicalDeclContext(LexicalDC);
+  Importer.Imported(D, ToFunc);
+
+  LexicalDC->addDeclInternal(ToFunc);
+  return ToFunc;
+}
+
 Decl *ASTNodeImporter::VisitObjCIvarDecl(ObjCIvarDecl *D) {
   // Import the major distinguishing characteristics of an ivar.
   DeclContext *DC, *LexicalDC;
   DeclarationName Name;
   SourceLocation Loc;
   NamedDecl *ToD;
   if (ImportDeclParts(D, DC, LexicalDC, Name, ToD, Loc))
     return nullptr;

         E->isArrayForm(),
         E->isArrayFormAsWritten(),
         E->doesUsualArrayDeleteWantSize(),
         OperatorDeleteDecl,
         ToArg,
         Importer.Import(E->getLocStart()));
 }
 
+Expr *ASTNodeImporter::VisitCXXDependentScopeMemberExpr(
+    CXXDependentScopeMemberExpr *E) {
+  Expr *Base = E->getBase();
+  if (!Base)
+    return nullptr;
+
+  QualType BaseType = E->getBaseType();
+  if (BaseType.isNull())
+    return nullptr;
+
+  TemplateArgumentListInfo ToTAInfo;
+  TemplateArgumentListInfo *ResInfo = nullptr;
+  if (E->hasExplicitTemplateArgs()) {
+    for (const auto &FromLoc : E->template_arguments()) {
+      bool Error = false;
+      TemplateArgumentLoc ToTALoc = ImportTemplateArgumentLoc(FromLoc, Error);
+      if (Error)
+        return nullptr;
+      ToTAInfo.addArgument(ToTALoc);
+    }
+    ResInfo = &ToTAInfo;
+  }
+
+  return CXXDependentScopeMemberExpr::Create(Importer.getToContext(),
+                                             Base, BaseType,
+                                             E->isArrow(),
+                                             E->getOperatorLoc(),
+                                             E->getQualifierLoc(),
+                                             E->getTemplateKeywordLoc(),
+                                             E->getFirstQualifierFoundInScope(),
+                                             E->getMemberNameInfo(),
+                                             ResInfo);
+}
+
 Expr *ASTNodeImporter::VisitCXXConstructExpr(CXXConstructExpr *E) {
   QualType T = Importer.Import(E->getType());
   if (T.isNull())
     return nullptr;
 
   CXXConstructorDecl *ToCCD =
     dyn_cast_or_null<CXXConstructorDecl>(Importer.Import(E->getConstructor()));
   if (!ToCCD)

lib/ASTMatchers/Dynamic/Registry.cpp

RegistryMaps::RegistryMaps() {
   REGISTER_MATCHER(cxxCatchStmt);
   REGISTER_MATCHER(cxxConstCastExpr);
   REGISTER_MATCHER(cxxConstructExpr);
   REGISTER_MATCHER(cxxConstructorDecl);
   REGISTER_MATCHER(cxxConversionDecl);
   REGISTER_MATCHER(cxxCtorInitializer);
   REGISTER_MATCHER(cxxDefaultArgExpr);
   REGISTER_MATCHER(cxxDeleteExpr);
+  REGISTER_MATCHER(cxxDependentScopeMemberExpr);
   REGISTER_MATCHER(cxxDestructorDecl);
   REGISTER_MATCHER(cxxDynamicCastExpr);
   REGISTER_MATCHER(cxxForRangeStmt);
   REGISTER_MATCHER(cxxFunctionalCastExpr);
   REGISTER_MATCHER(cxxMemberCallExpr);
   REGISTER_MATCHER(cxxMethodDecl);
   REGISTER_MATCHER(cxxNewExpr);
   REGISTER_MATCHER(cxxNullPtrLiteralExpr);

unittests/AST/ASTImporterTest.cpp

EXPECT_TRUE(testImport("void declToImport() { typedef _Atomic(int) a_int; }",
                                has(
                                  declStmt(
                                    has(
                                      typedefDecl(
                                        has(atomicType()))))))))));
 }
 
 
+TEST(ImportDecl, ImportFunctionTemplateDecl) {
+  MatchVerifier<Decl> Verifier;
+  EXPECT_TRUE(
+        testImport(
+          "template <typename T> void declToImport() { };",
+          Lang_CXX, "", Lang_CXX, Verifier,
+          functionTemplateDecl()));
+}
+
+
+TEST(ImportExpr, ImportCXXDependentScopeMemberExpr) {
+  MatchVerifier<Decl> Verifier;
+  EXPECT_TRUE(
+        testImport(
+          "template <typename T> class C { T t; };"
+          "template <typename T> void declToImport() {"
+            "C<T> d;"
+            "d.t = T();"
+          "}",
+          Lang_CXX, "", Lang_CXX, Verifier,
+          functionTemplateDecl(
+            has(
+              functionDecl(
+                has(
+                  compoundStmt(
+                    has(
+                      binaryOperator(
+                        has(
+                          cxxDependentScopeMemberExpr()))))))))));
+  EXPECT_TRUE(
+        testImport(
+          "template <typename T> class C { T t; };"
+          "template <typename T> void declToImport() {"
+            "C<T> d;"
+            "(&d)->t = T();"
+          "}",
+          Lang_CXX, "", Lang_CXX, Verifier,
+          functionTemplateDecl(
+            has(
+              functionDecl(
+                has(
+                  compoundStmt(
+                    has(
+                      binaryOperator(
+                        has(
+                          cxxDependentScopeMemberExpr()))))))))));
+}
+
+
 } // end namespace ast_matchers
 } // end namespace clang