Traverse-ignore explicit template instantiations

[steveire]

Summary

Continue to dump and match on explicit template specializations, but
omit explicit instantiation declarations and definitions.

[aaron.ballman]

clang/lib/ASTMatchers/ASTMatchFinder.cpp

503

Please spell this type out rather than use auto.

506

Same here, though this could also be simplified to:

ScopedTraversal = (SK == TSK_ExplicitInstantiationDeclaration || SK == TSK_ExplicitInstantiationDefinition);

clang/unittests/AST/ASTTraverserTest.cpp

1092

Huh, do you have any idea if that's a bug? We have ClassTemplateSpecializationDecl and VarTemplateSpecializationDecl, but we have FunctionTemplateSpecializationInfo that doesn't generate an AST node and no mention of why in the comments that I've spotted yet.

clang/unittests/ASTMatchers/ASTMatchersTraversalTest.cpp

2280

Explicitly instantiating a function template in ignore mode returns false, but explicitly instantiating a class template returns true? Is this intentional or just fallout from the lack of explicit instantiation information in the AST for functions?

[steveire]

Uppate

[steveire]

clang/unittests/AST/ASTTraverserTest.cpp

1092

I don't know why, but this is part of the confusion in the test in the discussion below.

I can look into it after this is merged if you don't beat me to it.

clang/unittests/ASTMatchers/ASTMatchersTraversalTest.cpp

2280

I've added some more tests and comments to try to clarify this.

We should be able to match on the template arguments of explicit instantiations, but not the contents of the instantiations.

Lack of representation of explicit function instantiations makes the expected test results confusing, but hopefully the comments now clarify.

[steveire]

clang/lib/ASTMatchers/ASTMatchFinder.cpp

506

If ScopedTraversal is set to true above, this could wrongly set it to false.

[aaron.ballman]

LGTM aside from a tiny commenting request.

clang/lib/ASTMatchers/ASTMatchFinder.cpp

506

If ScopedTraversal is set to true above, this could wrongly set it to false.

Good catch!

clang/unittests/AST/ASTTraverserTest.cpp

1092

I don't know why, but this is part of the confusion in the test in the discussion below.

Okay, I kind of thought that might be the case, thank you for confirming.

clang/unittests/ASTMatchers/ASTMatchersTraversalTest.cpp

2280

Thanks for the new comment, that clarifies nicely! Can you add a full stop to the end of the comment though?

clang/include/clang/AST/ASTNodeTraverser.h

getNodeDelegate().AddChild([=] {
         Visit(A);
 
       if (const comments::FullComment *Comment =
               D->getASTContext().getLocalCommentForDeclUncached(D))
         Visit(Comment, Comment);
 
       // Decls within functions are visited by the body.
       if (!isa<FunctionDecl>(*D) && !isa<ObjCMethodDecl>(*D)) {
+        if (isa<ClassTemplateSpecializationDecl>(*D) && Traversal != TK_AsIs)
+          return;
         if (const auto *DC = dyn_cast<DeclContext>(D))
           dumpDeclContext(DC);
       }
     });
   }
 
   void Visit(const Stmt *Node, StringRef Label = {}) {
     getNodeDelegate().AddChild(Label, [=] {

clang/include/clang/ASTMatchers/ASTMatchersInternal.h

static_assert(std::is_base_of<Decl, T>::value ||
                       std::is_base_of<TypeLoc, T>::value,
                   "type not allowed for recursive matching");
     return matchesAncestorOf(DynTypedNode::create(Node), getASTContext(),
                              Matcher, Builder, MatchMode);
   }
 
   virtual ASTContext &getASTContext() const = 0;
 
-  virtual bool isMatchingInImplicitTemplateInstantiation() const = 0;
+  virtual bool IsMatchingInTemplateInstantiationNotSpelledInSource() const = 0;
 
 protected:
   virtual bool matchesChildOf(const DynTypedNode &Node, ASTContext &Ctx,
                               const DynTypedMatcher &Matcher,
                               BoundNodesTreeBuilder *Builder,
                               TraversalKind Traverse, BindKind Bind) = 0;
 
   virtual bool matchesDescendantOf(const DynTypedNode &Node, ASTContext &Ctx,

clang/lib/ASTMatchers/ASTMatchFinder.cpp

bool memoizedMatchesRecursively(const DynTypedNode &Node, ASTContext &Ctx,
     return CachedResult.ResultOfMatch;
   }
 
   // Matches children or descendants of 'Node' with 'BaseMatcher'.
   bool matchesRecursively(const DynTypedNode &Node,
                           const DynTypedMatcher &Matcher,
                           BoundNodesTreeBuilder *Builder, int MaxDepth,
                           TraversalKind Traversal, BindKind Bind) {
+    bool ScopedTraversal = TraversingTemplateInstantiationNotSpelledInSource;
+
+    if (const auto *CTSD = Node.get<ClassTemplateSpecializationDecl>()) {
+      int SK = CTSD->getSpecializationKind();
+      if (SK == TSK_ExplicitInstantiationDeclaration ||
+          SK == TSK_ExplicitInstantiationDefinition)
+        ScopedTraversal = true;
+    }
+
+    TemplateInstantiationNotSpelledInSourceScope RAII(this, ScopedTraversal);
+
     MatchChildASTVisitor Visitor(
       &Matcher, this, Builder, MaxDepth, Traversal, Bind);
     return Visitor.findMatch(Node);
   }
 
   bool classIsDerivedFrom(const CXXRecordDecl *Declaration,
                           const Matcher<NamedDecl> &Base,
                           BoundNodesTreeBuilder *Builder,
public:
   }
 
   // Implements ASTMatchFinder::getASTContext.
   ASTContext &getASTContext() const override { return *ActiveASTContext; }
 
   bool shouldVisitTemplateInstantiations() const { return true; }
   bool shouldVisitImplicitCode() const { return true; }
 
-  bool isMatchingInImplicitTemplateInstantiation() const override {
-    return TraversingImplicitTemplateInstantiation;
+  bool IsMatchingInTemplateInstantiationNotSpelledInSource() const override {
+    return TraversingTemplateInstantiationNotSpelledInSource;
   }
 
   bool TraverseTemplateInstantiations(ClassTemplateDecl *D) {
-    ImplicitTemplateInstantiationScope RAII(this, true);
+    TemplateInstantiationNotSpelledInSourceScope RAII(this, true);
     return RecursiveASTVisitor<MatchASTVisitor>::TraverseTemplateInstantiations(
         D);
   }
 
   bool TraverseTemplateInstantiations(VarTemplateDecl *D) {
-    ImplicitTemplateInstantiationScope RAII(this, true);
+    TemplateInstantiationNotSpelledInSourceScope RAII(this, true);
     return RecursiveASTVisitor<MatchASTVisitor>::TraverseTemplateInstantiations(
         D);
   }
 
   bool TraverseTemplateInstantiations(FunctionTemplateDecl *D) {
-    ImplicitTemplateInstantiationScope RAII(this, true);
+    TemplateInstantiationNotSpelledInSourceScope RAII(this, true);
     return RecursiveASTVisitor<MatchASTVisitor>::TraverseTemplateInstantiations(
         D);
   }
 
 private:
-  bool TraversingImplicitTemplateInstantiation = false;
+  bool TraversingTemplateInstantiationNotSpelledInSource = false;
 
-  struct ImplicitTemplateInstantiationScope {
-    ImplicitTemplateInstantiationScope(MatchASTVisitor *V, bool B)
-        : MV(V), MB(V->TraversingImplicitTemplateInstantiation) {
-      V->TraversingImplicitTemplateInstantiation = B;
+  struct TemplateInstantiationNotSpelledInSourceScope {
+    TemplateInstantiationNotSpelledInSourceScope(MatchASTVisitor *V, bool B)
+        : MV(V), MB(V->TraversingTemplateInstantiationNotSpelledInSource) {
+      V->TraversingTemplateInstantiationNotSpelledInSource = B;
     }
-    ~ImplicitTemplateInstantiationScope() {
-      MV->TraversingImplicitTemplateInstantiation = MB;
+    ~TemplateInstantiationNotSpelledInSourceScope() {
+      MV->TraversingTemplateInstantiationNotSpelledInSource = MB;
     }
 
   private:
     MatchASTVisitor *MV;
     bool MB;
   };
 
   class TimeBucketRegion {

clang/lib/ASTMatchers/ASTMatchersInternal.cpp

 bool DynTypedMatcher::matches(const DynTypedNode &DynNode,
                               ASTMatchFinder *Finder,
                               BoundNodesTreeBuilder *Builder) const {
   TraversalKindScope RAII(Finder->getASTContext(),
                           Implementation->TraversalKind());
 
   if (Finder->getASTContext().getParentMapContext().getTraversalKind() ==
           TK_IgnoreUnlessSpelledInSource &&
-      Finder->isMatchingInImplicitTemplateInstantiation())
+      Finder->IsMatchingInTemplateInstantiationNotSpelledInSource())
     return false;
 
   auto N =
       Finder->getASTContext().getParentMapContext().traverseIgnored(DynNode);
 
   if (RestrictKind.isBaseOf(N.getNodeKind()) &&
       Implementation->dynMatches(N, Finder, Builder)) {
     return true;
   }
   // Delete all bindings when a matcher does not match.
   // This prevents unexpected exposure of bound nodes in unmatches
   // branches of the match tree.
   Builder->removeBindings([](const BoundNodesMap &) { return true; });
   return false;
 }
 
 bool DynTypedMatcher::matchesNoKindCheck(const DynTypedNode &DynNode,
                                          ASTMatchFinder *Finder,
                                          BoundNodesTreeBuilder *Builder) const {
   TraversalKindScope raii(Finder->getASTContext(),
                           Implementation->TraversalKind());
 
   if (Finder->getASTContext().getParentMapContext().getTraversalKind() ==
           TK_IgnoreUnlessSpelledInSource &&
-      Finder->isMatchingInImplicitTemplateInstantiation())
+      Finder->IsMatchingInTemplateInstantiationNotSpelledInSource())
     return false;
 
   auto N =
       Finder->getASTContext().getParentMapContext().traverseIgnored(DynNode);
 
   assert(RestrictKind.isBaseOf(N.getNodeKind()));
   if (Implementation->dynMatches(N, Finder, Builder)) {
     return true;

clang/unittests/AST/ASTTraverserTest.cpp

 
 void instantiate()
 {
   TemplStruct<int> ti;
   TemplStruct<double> td;
   (void)timesTwo<int>(2);
   (void)timesTwo<double>(2);
 }
+
+template class TemplStruct<float>;
+
+extern template class TemplStruct<long>;
+
+template<> class TemplStruct<bool> {
+  TemplStruct() {}
+  ~TemplStruct() {}
+
+  void foo() {}
+private:
+  bool m_t;
+};
+
+// Explicit instantiation of template functions do not appear in the AST
+template float timesTwo(float);
+
+template<> bool timesTwo<bool>(bool) {
+  return true;
+}
 )cpp");
   {
     auto BN = ast_matchers::match(
         classTemplateDecl(hasName("TemplStruct")).bind("rec"),
         AST->getASTContext());
     EXPECT_EQ(BN.size(), 1u);
 
     EXPECT_EQ(dumpASTString(TK_IgnoreUnlessSpelledInSource,

 | |-CXXConstructorDecl 'TemplStruct'
 | | `-CompoundStmt
 | |-CXXDestructorDecl '~TemplStruct'
 | | `-CompoundStmt
 | |-AccessSpecDecl
 | |-FieldDecl 'm_t'
 | `-CXXConstructorDecl 'TemplStruct'
 |   `-ParmVarDecl ''
+|-ClassTemplateSpecializationDecl 'TemplStruct'
+| |-TemplateArgument type double
+| | `-BuiltinType
+| |-CXXRecordDecl 'TemplStruct'
+| |-CXXConstructorDecl 'TemplStruct'
+| | `-CompoundStmt
+| |-CXXDestructorDecl '~TemplStruct'
+| | `-CompoundStmt
+| |-AccessSpecDecl
+| |-FieldDecl 'm_t'
+| `-CXXConstructorDecl 'TemplStruct'
+|   `-ParmVarDecl ''
+|-ClassTemplateSpecializationDecl 'TemplStruct'
+| |-TemplateArgument type float
+| | `-BuiltinType
+| |-CXXRecordDecl 'TemplStruct'
+| |-CXXConstructorDecl 'TemplStruct'
+| | `-CompoundStmt
+| |-CXXDestructorDecl '~TemplStruct'
+| | `-CompoundStmt
+| |-AccessSpecDecl
+| `-FieldDecl 'm_t'
+|-ClassTemplateSpecializationDecl 'TemplStruct'
+| |-TemplateArgument type long
+| | `-BuiltinType
+| |-CXXRecordDecl 'TemplStruct'
+| |-CXXConstructorDecl 'TemplStruct'
+| |-CXXDestructorDecl '~TemplStruct'
+| |-AccessSpecDecl
+| `-FieldDecl 'm_t'
 `-ClassTemplateSpecializationDecl 'TemplStruct'
-  |-TemplateArgument type double
+  |-TemplateArgument type _Bool
   | `-BuiltinType
   |-CXXRecordDecl 'TemplStruct'
   |-CXXConstructorDecl 'TemplStruct'
   | `-CompoundStmt
   |-CXXDestructorDecl '~TemplStruct'
   | `-CompoundStmt
+  |-CXXMethodDecl 'foo'
+  | `-CompoundStmt
   |-AccessSpecDecl
-  |-FieldDecl 'm_t'
-  `-CXXConstructorDecl 'TemplStruct'
-    `-ParmVarDecl ''
+  `-FieldDecl 'm_t'
 )cpp");
   }
   {
     auto BN = ast_matchers::match(
         functionTemplateDecl(hasName("timesTwo")).bind("fn"),
         AST->getASTContext());
     EXPECT_EQ(BN.size(), 1u);
 

 | | `-BuiltinType
 | |-ParmVarDecl 'input'
 | `-CompoundStmt
 |   `-ReturnStmt
 |     `-BinaryOperator
 |       |-ImplicitCastExpr
 |       | `-DeclRefExpr 'input'
 |       `-IntegerLiteral
+|-FunctionDecl 'timesTwo'
+| |-TemplateArgument type double
+| | `-BuiltinType
+| |-ParmVarDecl 'input'
+| `-CompoundStmt
+|   `-ReturnStmt
+|     `-BinaryOperator
+|       |-ImplicitCastExpr
+|       | `-DeclRefExpr 'input'
+|       `-ImplicitCastExpr
+|         `-IntegerLiteral
+|-FunctionDecl 'timesTwo'
+| |-TemplateArgument type float
+| | `-BuiltinType
+| |-ParmVarDecl 'input'
+| `-CompoundStmt
+|   `-ReturnStmt
+|     `-BinaryOperator
+|       |-ImplicitCastExpr
+|       | `-DeclRefExpr 'input'
+|       `-ImplicitCastExpr
+|         `-IntegerLiteral
+|-FunctionDecl 'timesTwo'
+| |-TemplateArgument type _Bool
+| | `-BuiltinType
+| |-ParmVarDecl ''
+| `-CompoundStmt
+|   `-ReturnStmt
+|     `-CXXBoolLiteralExpr
 `-FunctionDecl 'timesTwo'
-  |-TemplateArgument type double
+  |-TemplateArgument type _Bool
   | `-BuiltinType
-  |-ParmVarDecl 'input'
-  `-CompoundStmt
-    `-ReturnStmt
-      `-BinaryOperator
-        |-ImplicitCastExpr
-        | `-DeclRefExpr 'input'
-        `-ImplicitCastExpr
-          `-IntegerLiteral
+  `-ParmVarDecl 'input'
+)cpp");
+  }
+  {
+    auto BN = ast_matchers::match(
+        classTemplateSpecializationDecl(
+            hasName("TemplStruct"),
+            hasTemplateArgument(
+                0, templateArgument(refersToType(asString("float")))),
+            hasParent(translationUnitDecl()))
+            .bind("rec"),
+        AST->getASTContext());
+    EXPECT_EQ(BN.size(), 1u);
+
+    EXPECT_EQ(dumpASTString(TK_IgnoreUnlessSpelledInSource,
+                            BN[0].getNodeAs<Decl>("rec")),
+              R"cpp(
+ClassTemplateSpecializationDecl 'TemplStruct'
+`-TemplateArgument type float
+  `-BuiltinType
+)cpp");
+
+    EXPECT_EQ(dumpASTString(TK_AsIs, BN[0].getNodeAs<Decl>("rec")),
+              R"cpp(
+ClassTemplateSpecializationDecl 'TemplStruct'
+|-TemplateArgument type float
+| `-BuiltinType
+|-CXXRecordDecl 'TemplStruct'
+|-CXXConstructorDecl 'TemplStruct'
+| `-CompoundStmt
+|-CXXDestructorDecl '~TemplStruct'
+| `-CompoundStmt
+|-AccessSpecDecl
+`-FieldDecl 'm_t'
 )cpp");
   }
 }
 
 } // namespace clang

clang/unittests/ASTMatchers/ASTMatchersTraversalTest.cpp

 void instantiate()
 {
   TemplStruct<int> ti;
   TemplStruct<double> td;
   (void)timesTwo<int>(2);
   (void)timesTwo<double>(2);
 }
 
+template class TemplStruct<float>;
+
+extern template class TemplStruct<long>;
+
+template<> class TemplStruct<bool> {
+  TemplStruct() {}
+  ~TemplStruct() {}
+
+  void boolSpecializationMethodOnly() {}
+private:
+  bool m_t;
+};
+
+template float timesTwo(float);
+template<> bool timesTwo<bool>(bool){
+  return true;
+}
 )cpp";
   {
     auto M = cxxRecordDecl(hasName("TemplStruct"),
                            has(fieldDecl(hasType(asString("int")))));
     EXPECT_TRUE(matches(Code, traverse(TK_AsIs, M)));
     EXPECT_FALSE(matches(Code, traverse(TK_IgnoreUnlessSpelledInSource, M)));
   }
   {
)cpp";
   }
   {
     auto M =
         functionDecl(hasName("timesTwo"),
                      hasParameter(0, parmVarDecl(hasType(asString("double")))));
     EXPECT_TRUE(matches(Code, traverse(TK_AsIs, M)));
     EXPECT_FALSE(matches(Code, traverse(TK_IgnoreUnlessSpelledInSource, M)));
   }
+  {
+    // Match on the integer literal in the explicit instantiation:
+    auto MDef =
+        functionDecl(hasName("timesTwo"),
+                     hasParameter(0, parmVarDecl(hasType(asString("float")))),
+                     hasDescendant(integerLiteral(equals(2))));
+    EXPECT_TRUE(matches(Code, traverse(TK_AsIs, MDef)));
+    EXPECT_FALSE(matches(Code, traverse(TK_IgnoreUnlessSpelledInSource, MDef)));
+
+    auto MTempl =
+        functionDecl(hasName("timesTwo"),
+                     hasTemplateArgument(0, refersToType(asString("float"))));
+    EXPECT_TRUE(matches(Code, traverse(TK_AsIs, MTempl)));
+    // TODO: If we could match on explicit instantiations of function templates,
+    // this would be EXPECT_TRUE
+    EXPECT_FALSE(
+        matches(Code, traverse(TK_IgnoreUnlessSpelledInSource, MTempl)));
+  }
+  {
+    auto M = functionDecl(hasName("timesTwo"),
+                          hasParameter(0, parmVarDecl(hasType(booleanType()))));
+    EXPECT_TRUE(matches(Code, traverse(TK_AsIs, M)));
+    EXPECT_TRUE(matches(Code, traverse(TK_IgnoreUnlessSpelledInSource, M)));
+  }
+  {
+    // Match on the field within the explicit instantiation
+    auto MRecord = cxxRecordDecl(hasName("TemplStruct"),
+                                 has(fieldDecl(hasType(asString("float")))));
+    EXPECT_TRUE(matches(Code, traverse(TK_AsIs, MRecord)));
+    EXPECT_FALSE(
+        matches(Code, traverse(TK_IgnoreUnlessSpelledInSource, MRecord)));
+
+    // Match on the explicit template instantiation itself:
+    auto MTempl = classTemplateSpecializationDecl(
+        hasName("TemplStruct"),
+        hasTemplateArgument(0,
+                            templateArgument(refersToType(asString("float")))));
+    EXPECT_TRUE(matches(Code, traverse(TK_AsIs, MTempl)));
+    EXPECT_TRUE(
+        matches(Code, traverse(TK_IgnoreUnlessSpelledInSource, MTempl)));
+  }
+  {
+    // The template argument is matchable, but the instantiation is not:
+    auto M = classTemplateSpecializationDecl(
+        hasName("TemplStruct"),
+        hasTemplateArgument(0,
+                            templateArgument(refersToType(asString("float")))),
+        has(cxxConstructorDecl(hasName("TemplStruct"))));
+    EXPECT_TRUE(matches(Code, traverse(TK_AsIs, M)));
+    EXPECT_FALSE(matches(Code, traverse(TK_IgnoreUnlessSpelledInSource, M)));
+  }
+  {
+    // The template argument is matchable, but the instantiation is not:
+    auto M = classTemplateSpecializationDecl(
+        hasName("TemplStruct"),
+        hasTemplateArgument(0,
+                            templateArgument(refersToType(asString("long")))),
+        has(cxxConstructorDecl(hasName("TemplStruct"))));
+    EXPECT_TRUE(matches(Code, traverse(TK_AsIs, M)));
+    EXPECT_FALSE(matches(Code, traverse(TK_IgnoreUnlessSpelledInSource, M)));
+  }
+  {
+    // Explicit specialization is written in source and it matches
+    auto M = classTemplateSpecializationDecl(
+        hasName("TemplStruct"),
+        hasTemplateArgument(0, templateArgument(refersToType(booleanType()))),
+        has(cxxConstructorDecl(hasName("TemplStruct"))),
+        has(cxxMethodDecl(hasName("boolSpecializationMethodOnly"))));
+    EXPECT_TRUE(matches(Code, traverse(TK_AsIs, M)));
+    EXPECT_TRUE(matches(Code, traverse(TK_IgnoreUnlessSpelledInSource, M)));
+  }
 }
 
 template <typename MatcherT>
 bool matcherTemplateWithBinding(StringRef Code, const MatcherT &M) {
   return matchAndVerifyResultTrue(
       Code, M.bind("matchedStmt"),
       std::make_unique<VerifyIdIsBoundTo<ReturnStmt>>("matchedStmt", 1));
 }
TEST(MatcherMemoize, HasDiffersFromHasDescendant) {
   // Test introduced after detecting a bug in memoization
   constexpr auto code = "void f() { throw 1+1; }";
   EXPECT_TRUE(notMatches(
     code,
     cxxThrowExpr(has(integerLiteral()))));
   EXPECT_TRUE(matches(
     code,
     cxxThrowExpr(hasDescendant(integerLiteral()))));
-  EXPECT_TRUE(notMatches(code, 
-    cxxThrowExpr(allOf(
-      hasDescendant(integerLiteral()),
+  EXPECT_TRUE(
+      notMatches(code, cxxThrowExpr(allOf(hasDescendant(integerLiteral()),
                                           has(integerLiteral())))));
 }
 TEST(HasAncestor, MatchesAllAncestors) {
   EXPECT_TRUE(matches(
     "template <typename T> struct C { static void f() { 42; } };"
       "void t() { C<int>::f(); }",
     integerLiteral(
       equals(42),
       allOf(