Add documentation illustrating use of IgnoreUnlessSpelledInSource

[steveire]

Update

[aaron.ballman]

Thank you for the documentation effort!

clang/docs/LibASTMatchersReference.html

531

I think this section should go above the Node Matchers section in the document so that Node Matchers stays directly above the table of node matchers.

535

Can you add <pre> tags around AsIs and IgnoreUnlessSpelledInSource to make it clear that these are sort of syntactic constructs rather than prose?

536–538

Rather than say the default mode is hard to use correctly, how about:

This mode requires writing AST matchers that explicitly traverse or ignore implicit nodes, such as parentheses surrounding an expression or expressions with cleanups. These implicit nodes are not always obvious from the syntax of the source code, and so this mode requires careful consideration and testing to get the desired behavior from an AST matcher.

547

familiar with the AST -> familiar with where implicit nodes appear in the AST

559

<pre> tags around traverse()

584

Should we consistently switch this style to:

struct B {

to save on some vertical whitespace (it doesn't impact all of the rows, but will help in some cases)?

719

How about switching the exclamation points into full stops (here and elsewhere)?

734

Backticks won't help here -- should probably use <pre> tags.

743–749

Some more potential places for vertical whitespace savings.

Actually, would it be too onerous to ask you to run all of the code snippets through clang-format with the LLVM style so that they're consistently formatted and trims some of the vertical whitespace?

787

Could use some <pre> tags around the code constructs.

[steveire]

Update

[steveire]

clang/docs/LibASTMatchersReference.html

734

pre is equivalent to three backticks. I added a span for use within paragraphs.

[aaron.ballman]

clang/docs/LibASTMatchersReference.html

91–94

I was trying to reword this to avoid making a judgmental statement that the default mode is hard to use correctly -- I'd strike this part of the paragraph. WDYT?

734

Thank you for translating my suggestion into the one I was trying to make. :-)

[steveire]

clang/docs/LibASTMatchersReference.html

91–94

Oops, that's what I was trying to do, but accidentally left the previous stuff behind. Gone now.

[steveire]

Update

[aaron.ballman]

LGTM, thank you for the new documentation!

clang/docs/LibASTMatchersReference.html

 td.doc {
   display: none;
   border-bottom: 1px solid black;
 }
 td.name:hover {
   color: blue;
   cursor: pointer;
 }
+span.mono { font-family: monospace; }
+
+.traverse_compare, .traverse_compare td, .traverse_compare th {
+  border: 1px solid black;
+  border-collapse: collapse;
+}
 </style>
 <script type="text/javascript">
 function toggle(id) {
   if (!id) return;
   row = document.getElementById(id);
   if (row.style.display != 'table-cell')
     row.style.display = 'table-cell';
   else

 recordDecl(hasDescendant(
     ifStmt(hasTrueExpression(
         expr(hasDescendant(
             ifStmt()))))))
 </pre>
 </p>
 
 <!-- ======================================================================= -->
+<h2 id="traverse-mode">Traverse Mode</h2>
+<!-- ======================================================================= -->
+
+<p>The default mode of operation of AST Matchers visits all nodes in the AST,
+even if they are not spelled in the source. This is
+<span class="mono">AsIs</span> mode.  This mode requires writing AST matchers
+that explicitly traverse or ignore implicit nodes, such as parentheses
+surrounding an expression or expressions with cleanups.  These implicit
+nodes are not always obvious from the syntax of the source code, and so this
+mode requires careful consideration and testing to get the desired behavior
+from an AST matcher.
+</p>
+
+<p>In addition, because template instantations are matched in the default mode,
+transformations can be accidentally made to template declarations. Finally,
+because implicit nodes are matched by default, transformations can be made on
+entirely incorrect places in the code.</p>
+
+<p>For these reasons, it is possible to ignore AST nodes which are not spelled
+in the source using the <span class="mono">IgnoreUnlessSpelledInSource</span>
+mode. This is likely to be far less error-prone for users who are not already
+very familiar with where implicit nodes appear in the AST. It is also likely
+to be less error-prone for experienced AST users, as difficult cases do not
+need to be encountered and matcher expressions adjusted for these cases.</p>
+
+<p>In clang-query, the mode can be changed with
+<pre>
+set traversal IgnoreUnlessSpelledInSource
+</pre>
+</p>
+This affects both matchers and AST dump output in results.
+
+<p>When using the C++ API such as in clang-tidy checks, the
+<span class="mono">traverse()</span> matcher is used to set the mode:
+<pre>
+Finder->addMatcher(traverse(TK_IgnoreUnlessSpelledInSource,
+  returnStmt(hasReturnArgument(integerLiteral(equals(0))))
+  ), this);
+</pre>
+</p>
+<p>The following table compares the <span class="mono">AsIs</span> mode with
+the <span class="mono">IgnoreUnlessSpelledInSource</span> mode:</p>
+
+<table class="traverse_compare">
+<tr>
+<th></th>
+<th><span class="mono">AsIs</span></th>
+<th><span class="mono">IgnoreUnlessSpelledInSource</span></th>
+</tr>
+<tr>
+  <td>AST dump of <span class="mono">func1</span>:
+<pre>
+struct B {
+  B(int);
+};
+
+B func1() { return 42; }
+</pre>
+
+  </td>
+  <td>
+C++98 dialect:
+<pre>
+FunctionDecl
+`-CompoundStmt
+  `-ReturnStmt
+    `-ExprWithCleanups
+      `-CXXConstructExpr
+        `-MaterializeTemporaryExpr
+          `-ImplicitCastExpr
+            `-ImplicitCastExpr
+              `-CXXConstructExpr
+                `-IntegerLiteral 'int' 42
+</pre>
+C++11, C++14 dialect:
+<pre>
+FunctionDecl
+`-CompoundStmt
+  `-ReturnStmt
+    `-ExprWithCleanups
+      `-CXXConstructExpr
+        `-MaterializeTemporaryExpr
+          `-ImplicitCastExpr
+            `-CXXConstructExpr
+              `-IntegerLiteral 'int' 42
+</pre>
+C++17, C++20 dialect:
+<pre>
+FunctionDecl
+`-CompoundStmt
+  `-ReturnStmt
+    `-ImplicitCastExpr
+      `-CXXConstructExpr
+        `-IntegerLiteral 'int' 42
+</pre>
+</td>
+  <td>
+All dialects:
+    <pre>
+FunctionDecl
+`-CompoundStmt
+  `-ReturnStmt
+    `-IntegerLiteral 'int' 42
+</pre></td>
+</tr>
+
+<tr>
+<td>Matcher for returned <span class="mono">42</span>:
+<pre>
+struct B {
+  B(int);
+};
+
+B func1() { return 42; }
+</pre>
+
+  </td>
+  <td>
+All dialects:
+<pre>
+returnStmt(hasReturnValue(
+    ignoringImplicit(
+        ignoringElidableConstructorCall(
+            ignoringImplicit(
+                cxxConstructExpr(hasArgument(0,
+                    ignoringImplicit(
+                        integerLiteral().bind("returnVal")
+                        )
+                    ))
+                )
+            )
+        )
+    ))
+</pre></td>
+  <td>
+All dialects:
+<pre>
+returnStmt(hasReturnValue(
+    integerLiteral().bind("returnVal")
+))
+</pre></td>
+</tr>
+<tr>
+<td>Match result for
+<pre>implicitCastExpr()</pre>
+given:
+<pre>
+struct B {
+  B(int);
+};
+
+B func1() { return 42; }
+</pre>
+
+</td>
+<td>
+Match found.</td>
+  <td>
+No match.</td>
+</tr>
+<tr>
+  <td>Match result for:
+<pre>
+cxxConstructorDecl(
+  isCopyConstructor()
+  ).bind("prepend_explicit")
+</pre>
+given:
+<pre>
+struct Other {};
+struct Copyable {
+  Other m_o;
+  Copyable();
+};
+</pre>
+</td>
+<td>
+Match found. Insertion produces incorrect output:
+<pre>
+struct Other {};
+struct explicit Copyable {
+  Other m_o;
+  Copyable();
+};
+</pre>
+</td>
+<td>
+No match found. Incorrect replacement not possible.
+</td>
+</tr>
+<tr>
+  <td>Replacement of <span class="mono">begin()</span>
+    with <span class="mono">cbegin()</span>:
+<pre>
+cxxMemberCallExpr(
+  on(ConstContainerExpr),
+  callee(cxxMethodDecl(hasName("begin")))
+  ).bind("replace_with_cbegin")
+</pre>
+given:
+<pre>
+void foo() {
+  const Container c;
+  c.begin();
+
+  for (auto i : c) {
+  }
+}
+</pre>
+</td>
+<td>
+2 matches found. Replacement produces incorrect output:
+<pre>
+void foo() {
+  const Container c;
+  c.cbegin();
+
+  for (auto i :.cbegin() c) {
+  }
+}
+</pre>
+</td>
+<td>
+1 match found. Replacement produces correct output:
+<pre>
+void foo() {
+  const Container c;
+  c.cbegin();
+
+  for (auto i : c) {
+  }
+}
+</pre>
+</td>
+</tr>
+<tr>
+  <td>Replacement of <span class="mono">int</span> member
+    with <span class="mono">safe_int</span>:
+<pre>
+fieldDecl(
+  hasType(asString("int"))
+  ).bind("use_safe_int")
+</pre>
+given:
+<pre>
+struct S {
+  int m_i;
+};
+
+template &lt;typename T&gt; struct TemplStruct {
+  TemplStruct() {}
+  ~TemplStruct() {}
+
+private:
+  T m_t;
+};
+
+void instantiate() { TemplStruct&lt;int&gt; ti; }
+</pre>
+</td>
+<td>
+2 matches found. Replacement produces incorrect output:
+<pre>
+struct S {
+  safe_int m_i;
+};
+
+template &lt;typename T&gt; struct TemplStruct {
+  TemplStruct() {}
+  ~TemplStruct() {}
+
+private:
+  safe_int m_t;
+};
+
+void instantiate() { TemplStruct&lt;int&gt; ti; }
+</pre>
+</td>
+<td>
+1 match found. Replacement produces correct output:
+<pre>
+struct S {
+  safe_int m_i;
+};
+
+template &lt;typename T&gt; struct TemplStruct {
+  TemplStruct() {}
+  ~TemplStruct() {}
+
+private:
+  T m_t;
+};
+
+void instantiate() { TemplStruct&lt;int&gt; ti; }
+</pre>
+</td>
+</tr>
+<tr>
+  <td>Add prefix to member initializer
+<pre>
+cxxCtorInitializer(
+  forField(fieldDecl())
+  ).bind("add_prefix")
+</pre>
+given:
+<pre>
+struct Simple {};
+
+struct Record {
+  Record() : i(42) {}
+private:
+  int i;
+  Simple s;
+};
+</pre>
+</td>
+<td>
+2 matches found. Replacement produces incorrect output:
+<pre>
+struct Simple {};
+
+struct Record {
+  m_Record() : m_i(42) {}
+private:
+  int i;
+  Simple s;
+};
+</pre>
+</td>
+<td>
+1 match found. Replacement produces correct output:
+<pre>
+struct Simple {};
+
+struct Record {
+  Record() : m_i(42) {}
+private:
+  int i;
+  Simple s;
+};
+</pre>
+</td>
+</tr>
+<tr>
+  <td>Ignored default arguments
+<pre>
+callExpr(
+  callee(functionDecl(
+    hasName("hasDefaultArg")
+    )),
+  argumentCountIs(1)
+  ).bind("add_prefix")
+</pre>
+given:
+<pre>
+void hasDefaultArg(int i, int j = 0) {}
+void callDefaultArg() { hasDefaultArg(42); }
+</pre>
+</td>
+<td>
+No match.
+</td>
+<td>
+1 match found.
+</td>
+</tr>
+<tr>
+  <td>Lambda fields
+<pre>
+fieldDecl(
+  hasType(asString("int"))
+  ).bind("make_safe")
+</pre>
+given:
+<pre>
+struct S {
+  int m_i;
+};
+
+void func() {
+  int a = 0;
+  int c = 0;
+
+  auto l = [a, b = c](int d) { int e = d; };
+  l(43);
+}
+</pre>
+</td>
+<td>
+2 matches found. Replacement produces incorrect output:
+<pre>
+struct S {
+  safe_int m_i;
+};
+
+void func() {
+  int a = 0;
+  int c = 0;
+
+  auto l = [safe_a, safe_b = c](int d) { int e = d; };
+  l(43);
+}
+</pre>
+</td>
+<td>
+1 match found. Replacement produces correct output:
+<pre>
+struct S {
+  safe_int m_i;
+};
+
+void func() {
+  int a = 0;
+  int c = 0;
+
+  auto l = [a, b = c](int d) { int e = d; };
+  l(43);
+}
+</pre>
+</td>
+
+</tr>
+</table>
+
+<!-- ======================================================================= -->
 <h2 id="decl-matchers">Node Matchers</h2>
 <!-- ======================================================================= -->
 
 <p>Node matchers are at the core of matcher expressions - they specify the type
 of node that is expected. Every match expression starts with a node matcher,
 which can then be further refined with a narrowing or traversal matcher. All
 traversal matchers take node matchers as their arguments.</p>
 

 <pre>
 // This binds the CXXRecordDecl to "id", as the decl() matcher will stay on
 // the same node.
 recordDecl(decl().bind("id"), hasName("::MyClass"))
 </pre>
 </p>
 
 <table>
 <tr style="text-align:left"><th>Return type</th><th>Name</th><th>Parameters</th></tr>
 <!-- START_DECL_MATCHERS -->
 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1CXXCtorInitializer.html">CXXCtorInitializer</a>&gt;</td><td class="name" onclick="toggle('cxxCtorInitializer0')"><a name="cxxCtorInitializer0Anchor">cxxCtorInitializer</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1CXXCtorInitializer.html">CXXCtorInitializer</a>&gt;...</td></tr>
 <tr><td colspan="4" class="doc" id="cxxCtorInitializer0"><pre>Matches constructor initializers.
 
 Examples matches i(42).
   class C {
     C() : i(42) {}
     int i;
   };
 </pre></td></tr>
 
 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1Decl.html">Decl</a>&gt;</td><td class="name" onclick="toggle('accessSpecDecl0')"><a name="accessSpecDecl0Anchor">accessSpecDecl</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1AccessSpecDecl.html">AccessSpecDecl</a>&gt;...</td></tr>
 <tr><td colspan="4" class="doc" id="accessSpecDecl0"><pre>Matches C++ access specifier declarations.
 
 Given
   class C {
   public:
     int a;
   };
 accessSpecDecl()
   matches 'public:'
 </pre></td></tr>
 
 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1Decl.html">Decl</a>&gt;</td><td class="name" onclick="toggle('blockDecl0')"><a name="blockDecl0Anchor">blockDecl</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1BlockDecl.html">BlockDecl</a>&gt;...</td></tr>
 <tr><td colspan="4" class="doc" id="blockDecl0"><pre>Matches block declarations.
 
 Example matches the declaration of the nameless block printing an input
 integer.
 
   myFunc(^(int p) {
     printf("%d", p);
   })
 </pre></td></tr>
 
 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1Decl.html">Decl</a>&gt;</td><td class="name" onclick="toggle('classTemplateDecl0')"><a name="classTemplateDecl0Anchor">classTemplateDecl</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1ClassTemplateDecl.html">ClassTemplateDecl</a>&gt;...</td></tr>
 <tr><td colspan="4" class="doc" id="classTemplateDecl0"><pre>Matches C++ class template declarations.
 
 Example matches Z
   template&lt;class T&gt; class Z {};
 </pre></td></tr>
 
 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1Decl.html">Decl</a>&gt;</td><td class="name" onclick="toggle('classTemplatePartialSpecializationDecl0')"><a name="classTemplatePartialSpecializationDecl0Anchor">classTemplatePartialSpecializationDecl</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1ClassTemplatePartialSpecializationDecl.html">ClassTemplatePartialSpecializationDecl</a>&gt;...</td></tr>
 <tr><td colspan="4" class="doc" id="classTemplatePartialSpecializationDecl0"><pre>Matches C++ class template partial specializations.
 
 Given
   template&lt;class T1, class T2, int I&gt;
   class A {};
 
   template&lt;class T, int I&gt;
   class A&lt;T, T*, I&gt; {};
 
   template&lt;&gt;
   class A&lt;int, int, 1&gt; {};
 classTemplatePartialSpecializationDecl()
   matches the specialization A&lt;T,T*,I&gt; but not A&lt;int,int,1&gt;
 </pre></td></tr>

 Given
   class X { int m; };
 fieldDecl()
   matches 'm'.
 </pre></td></tr>
 
 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1Decl.html">Decl</a>&gt;</td><td class="name" onclick="toggle('friendDecl0')"><a name="friendDecl0Anchor">friendDecl</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1FriendDecl.html">FriendDecl</a>&gt;...</td></tr>
 <tr><td colspan="4" class="doc" id="friendDecl0"><pre>Matches friend declarations.
 
 Given
   class X { friend void foo(); };
 friendDecl()
   matches 'friend void foo()'.
 </pre></td></tr>
 
 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1Decl.html">Decl</a>&gt;</td><td class="name" onclick="toggle('functionDecl0')"><a name="functionDecl0Anchor">functionDecl</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1FunctionDecl.html">FunctionDecl</a>&gt;...</td></tr>
 <tr><td colspan="4" class="doc" id="functionDecl0"><pre>Matches function declarations.
 
 Example matches f
   void f();
 </pre></td></tr>
 
 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1Decl.html">Decl</a>&gt;</td><td class="name" onclick="toggle('functionTemplateDecl0')"><a name="functionTemplateDecl0Anchor">functionTemplateDecl</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1FunctionTemplateDecl.html">FunctionTemplateDecl</a>&gt;...</td></tr>
 <tr><td colspan="4" class="doc" id="functionTemplateDecl0"><pre>Matches C++ function template declarations.
 
 Example matches f
   template&lt;class T&gt; void f(T t) {}
 </pre></td></tr>
 
 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1Decl.html">Decl</a>&gt;</td><td class="name" onclick="toggle('indirectFieldDecl0')"><a name="indirectFieldDecl0Anchor">indirectFieldDecl</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1IndirectFieldDecl.html">IndirectFieldDecl</a>&gt;...</td></tr>
 <tr><td colspan="4" class="doc" id="indirectFieldDecl0"><pre>Matches indirect field declarations.
 
 Given
   struct X { struct { int a; }; };
 indirectFieldDecl()
   matches 'a'.
 </pre></td></tr>
 
 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1Decl.html">Decl</a>&gt;</td><td class="name" onclick="toggle('labelDecl0')"><a name="labelDecl0Anchor">labelDecl</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1LabelDecl.html">LabelDecl</a>&gt;...</td></tr>
 <tr><td colspan="4" class="doc" id="labelDecl0"><pre>Matches a declaration of label.
 
 Given
Example matches X, S, the anonymous union type, i, and U;
   typedef int X;
   struct S {
     union {
       int i;
     } U;
   };
 </pre></td></tr>
 
 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1Decl.html">Decl</a>&gt;</td><td class="name" onclick="toggle('namespaceAliasDecl0')"><a name="namespaceAliasDecl0Anchor">namespaceAliasDecl</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1NamespaceAliasDecl.html">NamespaceAliasDecl</a>&gt;...</td></tr>
 <tr><td colspan="4" class="doc" id="namespaceAliasDecl0"><pre>Matches a declaration of a namespace alias.
 
 Given
   namespace test {}
   namespace alias = ::test;
 namespaceAliasDecl()
   matches "namespace alias" but not "namespace test"