[AST Matchers] Update dump_ast_matchers.py to query only class index page.

[jcking1034]

Summary

It may be possible to avoid relying on accessing many individual class pages,
by instead scanning the class index page at
https://clang.llvm.org/doxygen/classes.html. This updates the script to do so,
and includes updates to LibASTMatchersReference.html generated by the
modified script.

[ymandel]

Aaron -- I'm not the right reviewer for this. I've put you as the reviewer, but feel free to re-route if there's someone better. Thanks!

[aaron.ballman]

clang/docs/tools/dump_ast_matchers.py

13

Do we need to handle an exception here if there are network issues?

[jcking1034]

Catch and reraise exceptions with an appropriate message.

[jcking1034]

clang/docs/tools/dump_ast_matchers.py

13

Technically, if there is a network issue, any exceptions would cause the script to terminate. But I think you're right that we should handle this case more carefully.

[aaron.ballman]

LGTM, though I'll note I'm by no means a Python expert. I'm adding a few more AST matching folks in case they spot something I've missed, so please give this at least 24 hours before landing.

[sammccall]

Nice!

clang/docs/tools/dump_ast_matchers.py

48

While here I'd add a comment:

"""
Wrap a likely AST node name in a link to its clang docs.

We want to do this only if the page exists, in which case it will be referenced from the class index page.
"""

(You're only adding a bit of conceptual complexity here, but it looks like it's been built up over time...)

[jcking1034]

Add docstring to link_if_exists.

[jcking1034]

Rebase onto master

clang/docs/LibASTMatchersReference.html

 recordDecl(decl().bind("id"), hasName("::MyClass"))
 </pre>
 </p>
 
 <table>
 <tr style="text-align:left"><th>Return type</th><th>Name</th><th>Parameters</th></tr>
 <!-- START_DECL_MATCHERS -->
 
-<tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1CXXBaseSpecifier.html">CXXBaseSpecifier</a>&gt;</td><td class="name" onclick="toggle('cxxBaseSpecifier0')"><a name="cxxBaseSpecifier0Anchor">cxxBaseSpecifier</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1CXXBaseSpecifier.html">CXXBaseSpecifier</a>&gt;...</td></tr>
-<tr><td colspan="4" class="doc" id="cxxBaseSpecifier0"><pre>Matches class bases.
-
-Examples matches public virtual B.
-  class B {};
-  class C : public virtual B {};
-</pre></td></tr>
-
-
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1Attr.html">Attr</a>&gt;</td><td class="name" onclick="toggle('attr0')"><a name="attr0Anchor">attr</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1Attr.html">Attr</a>&gt;...</td></tr>
 <tr><td colspan="4" class="doc" id="attr0"><pre>Matches attributes.
 Attributes may be attached with a variety of different syntaxes (including
 keywords, C++11 attributes, GNU ``__attribute``` and MSVC `__declspec``,
 and ``#pragma``s). They may also be implicit.
 
 Given
   struct [[nodiscard]] Foo{};
   void bar(int * __attribute__((nonnull)) );
   __declspec(noinline) void baz();
 
   #pragma omp declare simd
   int min();
 attr()
   matches "nodiscard", "nonnull", "noinline", and the whole "#pragma" line.
 </pre></td></tr>
 
 
+<tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1CXXBaseSpecifier.html">CXXBaseSpecifier</a>&gt;</td><td class="name" onclick="toggle('cxxBaseSpecifier0')"><a name="cxxBaseSpecifier0Anchor">cxxBaseSpecifier</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1CXXBaseSpecifier.html">CXXBaseSpecifier</a>&gt;...</td></tr>
+<tr><td colspan="4" class="doc" id="cxxBaseSpecifier0"><pre>Matches class bases.
+
+Examples matches public virtual B.
+  class B {};
+  class C : public virtual B {};
+</pre></td></tr>
+
+
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1CXXCtorInitializer.html">CXXCtorInitializer</a>&gt;</td><td class="name" onclick="toggle('cxxCtorInitializer0')"><a name="cxxCtorInitializer0Anchor">cxxCtorInitializer</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1CXXCtorInitializer.html">CXXCtorInitializer</a>&gt;...</td></tr>
 <tr><td colspan="4" class="doc" id="cxxCtorInitializer0"><pre>Matches constructor initializers.
 
 Examples matches i(42).
   class C {
     C() : i(42) {}
     int i;
   };

 
 Given
   namespace X { int x; }
   using namespace X;
 usingDirectiveDecl()
   matches using namespace X </pre></td></tr>
 
 
+<tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1Decl.html">Decl</a>&gt;</td><td class="name" onclick="toggle('usingEnumDecl0')"><a name="usingEnumDecl0Anchor">usingEnumDecl</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1UsingEnumDecl.html">UsingEnumDecl</a>&gt;...</td></tr>
+<tr><td colspan="4" class="doc" id="usingEnumDecl0"><pre>Matches using-enum declarations.
+
+Given
+  namespace X { enum x {...}; }
+  using enum X::x;
+usingEnumDecl()
+  matches using enum X::x </pre></td></tr>
+
+
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1Decl.html">Decl</a>&gt;</td><td class="name" onclick="toggle('valueDecl0')"><a name="valueDecl0Anchor">valueDecl</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1ValueDecl.html">ValueDecl</a>&gt;...</td></tr>
 <tr><td colspan="4" class="doc" id="valueDecl0"><pre>Matches any value declaration.
 
 Example matches A, B, C and F
   enum X { A, B, C };
   void F();
 </pre></td></tr>
 
Given
   S(int) -&gt; S&lt;true&gt; // #5
   explicit S(double) -&gt; S&lt;false&gt; // #6
 cxxConstructorDecl(isExplicit()) will match #2 and #8, but not #1, #7 or #9.
 cxxConversionDecl(isExplicit()) will match #4, but not #3.
 cxxDeductionGuideDecl(isExplicit()) will match #6, but not #5.
 </pre></td></tr>
 
 
+<tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1CXXConstructorDecl.html">CXXConstructorDecl</a>&gt;</td><td class="name" onclick="toggle('isInheritingConstructor0')"><a name="isInheritingConstructor0Anchor">isInheritingConstructor</a></td><td></td></tr>
+<tr><td colspan="4" class="doc" id="isInheritingConstructor0"><pre></pre></td></tr>
+
+
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1CXXConstructorDecl.html">CXXConstructorDecl</a>&gt;</td><td class="name" onclick="toggle('isMoveConstructor0')"><a name="isMoveConstructor0Anchor">isMoveConstructor</a></td><td></td></tr>
 <tr><td colspan="4" class="doc" id="isMoveConstructor0"><pre>Matches constructor declarations that are move constructors.
 
 Given
   struct S {
     S(); // #1
     S(const S &amp;); // #2
     S(S &amp;&amp;); // #3

 f. If the matcher is used from clang-query, attr::Kind parameter should be
 passed as a quoted string. e.g., hasAttr("attr::CUDADevice").
 </pre></td></tr>
 
 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1Decl.html">Decl</a>&gt;</td><td class="name" onclick="toggle('isExpandedFromMacro0')"><a name="isExpandedFromMacro0Anchor">isExpandedFromMacro</a></td><td>std::string MacroName</td></tr>
 <tr><td colspan="4" class="doc" id="isExpandedFromMacro0"><pre>Matches statements that are (transitively) expanded from the named macro.
 Does not match if only part of the statement is expanded from that macro or
-if different parts of the the statement are expanded from different
+if different parts of the statement are expanded from different
 appearances of the macro.
 </pre></td></tr>
 
 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1Decl.html">Decl</a>&gt;</td><td class="name" onclick="toggle('isExpansionInFileMatching0')"><a name="isExpansionInFileMatching0Anchor">isExpansionInFileMatching</a></td><td>StringRef RegExp, Regex::RegexFlags Flags = NoFlags</td></tr>
 <tr><td colspan="4" class="doc" id="isExpansionInFileMatching0"><pre>Matches AST nodes that were expanded within files whose name is
 partially matching a given regex.
 

 classTemplateSpecializationDecl(templateArgumentCountIs(1))
   matches C&lt;int&gt;.
 </pre></td></tr>
 
 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1TypeLoc.html">TypeLoc</a>&gt;</td><td class="name" onclick="toggle('isExpandedFromMacro2')"><a name="isExpandedFromMacro2Anchor">isExpandedFromMacro</a></td><td>std::string MacroName</td></tr>
 <tr><td colspan="4" class="doc" id="isExpandedFromMacro2"><pre>Matches statements that are (transitively) expanded from the named macro.
 Does not match if only part of the statement is expanded from that macro or
-if different parts of the the statement are expanded from different
+if different parts of the statement are expanded from different
 appearances of the macro.
 </pre></td></tr>
 
 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1TypeLoc.html">TypeLoc</a>&gt;</td><td class="name" onclick="toggle('isExpansionInFileMatching2')"><a name="isExpansionInFileMatching2Anchor">isExpansionInFileMatching</a></td><td>StringRef RegExp, Regex::RegexFlags Flags = NoFlags</td></tr>
 <tr><td colspan="4" class="doc" id="isExpansionInFileMatching2"><pre>Matches AST nodes that were expanded within files whose name is
 partially matching a given regex.
 
Given
   auto b = 2.0;
 autoType(hasDeducedType(isInteger()))
   matches "auto a"
 
 Usable as: Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1AutoType.html">AutoType</a>&gt;
 </pre></td></tr>
 
 
+<tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1BaseUsingDecl.html">BaseUsingDecl</a>&gt;</td><td class="name" onclick="toggle('hasAnyUsingShadowDecl0')"><a name="hasAnyUsingShadowDecl0Anchor">hasAnyUsingShadowDecl</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1UsingShadowDecl.html">UsingShadowDecl</a>&gt; InnerMatcher</td></tr>
+<tr><td colspan="4" class="doc" id="hasAnyUsingShadowDecl0"><pre>Matches any using shadow declaration.
+
+Given
+  namespace X { void b(); }
+  using X::b;
+usingDecl(hasAnyUsingShadowDecl(hasName("b"))))
+  matches using X::b </pre></td></tr>
+
+
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1BinaryOperator.html">BinaryOperator</a>&gt;</td><td class="name" onclick="toggle('hasEitherOperand0')"><a name="hasEitherOperand0Anchor">hasEitherOperand</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1Expr.html">Expr</a>&gt; InnerMatcher</td></tr>
 <tr><td colspan="4" class="doc" id="hasEitherOperand0"><pre>Matches if either the left hand side or the right hand side of a
 binary operator matches.
 </pre></td></tr>
 
 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1BinaryOperator.html">BinaryOperator</a>&gt;</td><td class="name" onclick="toggle('hasLHS0')"><a name="hasLHS0Anchor">hasLHS</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1Expr.html">Expr</a>&gt; InnerMatcher</td></tr>
 <tr><td colspan="4" class="doc" id="hasLHS0"><pre>Matches the left hand side of binary operator expressions.

 
 Given
   namespace N {
     namespace M {
       class D {};
     }
   }
 
-cxxRecordDecl(hasDeclContext(namedDecl(hasName("M")))) matches the
+cxxRcordDecl(hasDeclContext(namedDecl(hasName("M")))) matches the
 declaration of class D.
 </pre></td></tr>
 
 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1DecltypeType.html">DecltypeType</a>&gt;</td><td class="name" onclick="toggle('hasUnderlyingType0')"><a name="hasUnderlyingType0Anchor">hasUnderlyingType</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1Type.html">Type</a>&gt;</td></tr>
 <tr><td colspan="4" class="doc" id="hasUnderlyingType0"><pre>Matches DecltypeType nodes to find out the underlying type.
 
 Given
Usable as: Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1AddrLabelExpr.html">AddrLabelExpr</a>&gt;, Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1CallExpr.html">CallExpr</a>&gt;,
   Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1EnumType.html">EnumType</a>&gt;, Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1InjectedClassNameType.html">InjectedClassNameType</a>&gt;, Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1LabelStmt.html">LabelStmt</a>&gt;,
   Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1MemberExpr.html">MemberExpr</a>&gt;, Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1QualType.html">QualType</a>&gt;, Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1RecordType.html">RecordType</a>&gt;,
   Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1TagType.html">TagType</a>&gt;, Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1TemplateSpecializationType.html">TemplateSpecializationType</a>&gt;,
   Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1TemplateTypeParmType.html">TemplateTypeParmType</a>&gt;, Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1TypedefType.html">TypedefType</a>&gt;,
   Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1UnresolvedUsingType.html">UnresolvedUsingType</a>&gt;
 </pre></td></tr>
 
 
-<tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1UsingDecl.html">UsingDecl</a>&gt;</td><td class="name" onclick="toggle('hasAnyUsingShadowDecl0')"><a name="hasAnyUsingShadowDecl0Anchor">hasAnyUsingShadowDecl</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1UsingShadowDecl.html">UsingShadowDecl</a>&gt; InnerMatcher</td></tr>
-<tr><td colspan="4" class="doc" id="hasAnyUsingShadowDecl0"><pre>Matches any using shadow declaration.
-
-Given
-  namespace X { void b(); }
-  using X::b;
-usingDecl(hasAnyUsingShadowDecl(hasName("b"))))
-  matches using X::b </pre></td></tr>
-
-
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1UsingShadowDecl.html">UsingShadowDecl</a>&gt;</td><td class="name" onclick="toggle('hasTargetDecl0')"><a name="hasTargetDecl0Anchor">hasTargetDecl</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1NamedDecl.html">NamedDecl</a>&gt; InnerMatcher</td></tr>
 <tr><td colspan="4" class="doc" id="hasTargetDecl0"><pre>Matches a using shadow declaration where the target declaration is
 matched by the given matcher.
 
 Given
   namespace X { int a; void b(); }
   using X::a;
   using X::b;

clang/docs/tools/dump_ast_matchers.py

 #!/usr/bin/env python
 # A tool to parse ASTMatchers.h and update the documentation in
 # ../LibASTMatchersReference.html automatically. Run from the
 # directory in which this file is located to update the docs.
 
 import collections
 import re
 try:
     from urllib.request import urlopen
 except ImportError:
     from urllib2 import urlopen
 
+CLASS_INDEX_PAGE_URL = 'https://clang.llvm.org/doxygen/classes.html'
+try:
+  CLASS_INDEX_PAGE = urlopen(CLASS_INDEX_PAGE_URL).read()
+except Exception as e:
+  raise Exception('Unable to get %s: %s' % (CLASS_INDEX_PAGE_URL, e))
+
 MATCHERS_FILE = '../../include/clang/ASTMatchers/ASTMatchers.h'
 
 # Each matcher is documented in one row of the form:
 #   result | name | argA
 # The subsequent row contains the documentation and is hidden by default,
 # becoming visible via javascript when the user clicks the matcher name.
 TD_TEMPLATE="""
 <tr><td>%(result)s</td><td class="name" onclick="toggle('%(id)s')"><a name="%(id)sAnchor">%(name)s</a></td><td>%(args)s</td></tr>

 # Cache for doxygen urls we have already verified.
 doxygen_probes = {}
 
 def esc(text):
   """Escape any html in the given text."""
   text = re.sub(r'&', '&amp;', text)
   text = re.sub(r'<', '&lt;', text)
   text = re.sub(r'>', '&gt;', text)
   def link_if_exists(m):
     name = m.group(1)
     url = 'https://clang.llvm.org/doxygen/classclang_1_1%s.html' % name
     if url not in doxygen_probes:
-      try:
-        print('Probing %s...' % url)
-        urlopen(url)
-        doxygen_probes[url] = True
-      except:
-        doxygen_probes[url] = False
+      search_str = 'href="classclang_1_1%s.html"' % name
+      doxygen_probes[url] = search_str in CLASS_INDEX_PAGE
+      if not doxygen_probes[url]:
+        print('Did not find %s in class index page' % name)
     if doxygen_probes[url]:
       return r'Matcher&lt;<a href="%s">%s</a>&gt;' % (url, name)
     else:
       return m.group(0)
   text = re.sub(
     r'Matcher&lt;([^\*&]+)&gt;', link_if_exists, text)
   return text