[clang][docs] Defensively turn off exception behavior in dump_ast_matchers.py

[t-rasmud]

Summary

This patch turns exceptions (that are being triggered in most cases) into defensive prints

[NoQ]

Yes, as far as I understand, the URL request is more of an "assert" thing; it confirms that there's a doxygen page for the class mentioned in the code, but I'd be very surprised if there are any actual classes that you can use in the C++ code for which there's no doxygen page.

This check was running into problems on, uh, certain corporate networks with unusual SSL certificate manipulations, and I think it makes sense to make it less mandatory. It's still checking all URLs by default whenever it can, so it'll start flagging broken URLs anyway if they ever appear, it just unblocks progress for folks who need or prefer it to be offline for some reason. Which is really helpful when it's like your first time running this script and you've no idea what to expect.

clang/docs/tools/dump_ast_matchers.py

61–64

It's probably better to be explicit about what we expect to happen here. This is just a string operation, it probably can't go wrong in many ways.

[t-rasmud]

Address feedback.

[ymandel]

nit: I would suggest splitting the update to the docs from the modification to the script itself, since they are unrelated.

[t-rasmud]

Splitting the update: Updates to LibASTMAtchersReference.html are now in https://reviews.llvm.org/D155304

[jkorous]

LGTM

clang/docs/LibASTMatchersReference.html

 
 Given
   while (true) { co_return; }
 coreturnStmt()
   matches 'co_return'
 </pre></td></tr>
 
 
+<tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1Stmt.html">Stmt</a>&gt;</td><td class="name" onclick="toggle('coroutineBodyStmt0')"><a name="coroutineBodyStmt0Anchor">coroutineBodyStmt</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1CoroutineBodyStmt.html">CoroutineBodyStmt</a>&gt;...</td></tr>
+<tr><td colspan="4" class="doc" id="coroutineBodyStmt0"><pre>Matches coroutine body statements.
+
+coroutineBodyStmt() matches the coroutine below
+  generator&lt;int&gt; gen() {
+    co_return;
+  }
+</pre></td></tr>
+
+
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1Stmt.html">Stmt</a>&gt;</td><td class="name" onclick="toggle('coyieldExpr0')"><a name="coyieldExpr0Anchor">coyieldExpr</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1CoyieldExpr.html">CoyieldExpr</a>&gt;...</td></tr>
 <tr><td colspan="4" class="doc" id="coyieldExpr0"><pre>Matches co_yield expressions.
 
 Given
   co_yield 1;
 coyieldExpr()
   matches 'co_yield 1'
 </pre></td></tr>
Given
   } catch (...) {
     // ...
   }
 cxxCatchStmt(isCatchAll()) matches catch(...) but not catch(int).
 </pre></td></tr>
 
 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1CXXConstructExpr.html">CXXConstructExpr</a>&gt;</td><td class="name" onclick="toggle('argumentCountAtLeast1')"><a name="argumentCountAtLeast1Anchor">argumentCountAtLeast</a></td><td>unsigned N</td></tr>
-<tr><td colspan="4" class="doc" id="argumentCountAtLeast1"><pre>Checks that a call expression or a constructor call expression has
-at least the specified number of arguments (including absent default arguments).
+<tr><td colspan="4" class="doc" id="argumentCountAtLeast1"><pre>Checks that a call expression or a constructor call expression has at least
+the specified number of arguments (including absent default arguments).
 
-Example matches f(0, 0) and g(0, 0, 0) (matcher = callExpr(argumentCountAtLeast(2)))
+Example matches f(0, 0) and g(0, 0, 0)
+(matcher = callExpr(argumentCountAtLeast(2)))
   void f(int x, int y);
   void g(int x, int y, int z);
   f(0, 0);
   g(0, 0, 0);
 </pre></td></tr>
 
 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1CXXConstructExpr.html">CXXConstructExpr</a>&gt;</td><td class="name" onclick="toggle('argumentCountIs1')"><a name="argumentCountIs1Anchor">argumentCountIs</a></td><td>unsigned N</td></tr>

 Example 2: matches s1 &lt; s2
            (matcher = cxxOperatorCallExpr(isComparisonOperator()))
   struct S { bool operator&lt;(const S&amp; other); };
   void x(S s1, S s2) { bool b1 = s1 &lt; s2; }
 </pre></td></tr>
 
 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1CXXUnresolvedConstructExpr.html">CXXUnresolvedConstructExpr</a>&gt;</td><td class="name" onclick="toggle('argumentCountAtLeast2')"><a name="argumentCountAtLeast2Anchor">argumentCountAtLeast</a></td><td>unsigned N</td></tr>
-<tr><td colspan="4" class="doc" id="argumentCountAtLeast2"><pre>Checks that a call expression or a constructor call expression has
-at least the specified number of arguments (including absent default arguments).
+<tr><td colspan="4" class="doc" id="argumentCountAtLeast2"><pre>Checks that a call expression or a constructor call expression has at least
+the specified number of arguments (including absent default arguments).
 
-Example matches f(0, 0) and g(0, 0, 0) (matcher = callExpr(argumentCountAtLeast(2)))
+Example matches f(0, 0) and g(0, 0, 0)
+(matcher = callExpr(argumentCountAtLeast(2)))
   void f(int x, int y);
   void g(int x, int y, int z);
   f(0, 0);
   g(0, 0, 0);
 </pre></td></tr>
 
 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1CXXUnresolvedConstructExpr.html">CXXUnresolvedConstructExpr</a>&gt;</td><td class="name" onclick="toggle('argumentCountIs2')"><a name="argumentCountIs2Anchor">argumentCountIs</a></td><td>unsigned N</td></tr>
 <tr><td colspan="4" class="doc" id="argumentCountIs2"><pre>Checks that a call expression or a constructor call expression has
 a specific number of arguments (including absent default arguments).
 
 Example matches f(0, 0) (matcher = callExpr(argumentCountIs(2)))
   void f(int x, int y);
   f(0, 0);
 </pre></td></tr>
 
 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1CallExpr.html">CallExpr</a>&gt;</td><td class="name" onclick="toggle('argumentCountAtLeast0')"><a name="argumentCountAtLeast0Anchor">argumentCountAtLeast</a></td><td>unsigned N</td></tr>
-<tr><td colspan="4" class="doc" id="argumentCountAtLeast0"><pre>Checks that a call expression or a constructor call expression has
-at least the specified number of arguments (including absent default arguments).
+<tr><td colspan="4" class="doc" id="argumentCountAtLeast0"><pre>Checks that a call expression or a constructor call expression has at least
+the specified number of arguments (including absent default arguments).
 
-Example matches f(0, 0) and g(0, 0, 0) (matcher = callExpr(argumentCountAtLeast(2)))
+Example matches f(0, 0) and g(0, 0, 0)
+(matcher = callExpr(argumentCountAtLeast(2)))
   void f(int x, int y);
   void g(int x, int y, int z);
   f(0, 0);
   g(0, 0, 0);
 </pre></td></tr>
 
 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1CallExpr.html">CallExpr</a>&gt;</td><td class="name" onclick="toggle('argumentCountIs0')"><a name="argumentCountIs0Anchor">argumentCountIs</a></td><td>unsigned N</td></tr>

 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1ObjCInterfaceDecl.html">ObjCInterfaceDecl</a>&gt;</td><td class="name" onclick="toggle('isSameOrDerivedFrom3')"><a name="isSameOrDerivedFrom3Anchor">isSameOrDerivedFrom</a></td><td>std::string BaseName</td></tr>
 <tr><td colspan="4" class="doc" id="isSameOrDerivedFrom3"><pre>Overloaded method as shortcut for
 isSameOrDerivedFrom(hasName(...)).
 </pre></td></tr>
 
 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1ObjCMessageExpr.html">ObjCMessageExpr</a>&gt;</td><td class="name" onclick="toggle('argumentCountAtLeast3')"><a name="argumentCountAtLeast3Anchor">argumentCountAtLeast</a></td><td>unsigned N</td></tr>
-<tr><td colspan="4" class="doc" id="argumentCountAtLeast3"><pre>Checks that a call expression or a constructor call expression has
-at least the specified number of arguments (including absent default arguments).
+<tr><td colspan="4" class="doc" id="argumentCountAtLeast3"><pre>Checks that a call expression or a constructor call expression has at least
+the specified number of arguments (including absent default arguments).
 
-Example matches f(0, 0) and g(0, 0, 0) (matcher = callExpr(argumentCountAtLeast(2)))
+Example matches f(0, 0) and g(0, 0, 0)
+(matcher = callExpr(argumentCountAtLeast(2)))
   void f(int x, int y);
   void g(int x, int y, int z);
   f(0, 0);
   g(0, 0, 0);
 </pre></td></tr>
 
 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1ObjCMessageExpr.html">ObjCMessageExpr</a>&gt;</td><td class="name" onclick="toggle('argumentCountIs3')"><a name="argumentCountIs3Anchor">argumentCountIs</a></td><td>unsigned N</td></tr>
memberExpr(hasObjectExpression(hasType(cxxRecordDecl(hasName("X")))))
   matches `x.m`, but not `m`; however,
 memberExpr(hasObjectExpression(hasType(pointsTo(
      cxxRecordDecl(hasName("X"))))))
   matches `m` (aka. `this-&gt;m`), but not `x.m`.
 </pre></td></tr>
 
 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1CXXForRangeStmt.html">CXXForRangeStmt</a>&gt;</td><td class="name" onclick="toggle('hasBody3')"><a name="hasBody3Anchor">hasBody</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1Stmt.html">Stmt</a>&gt; InnerMatcher</td></tr>
-<tr><td colspan="4" class="doc" id="hasBody3"><pre>Matches a 'for', 'while', 'do' statement or a function definition that has
-a given body. Note that in case of functions this matcher only matches the
-definition itself and not the other declarations of the same function.
+<tr><td colspan="4" class="doc" id="hasBody3"><pre>Matches a 'for', 'while', 'while' statement or a function or coroutine
+definition that has a given body. Note that in case of functions or
+coroutines this matcher only matches the definition itself and not the
+other declarations of the same function or coroutine.
 
 Given
   for (;;) {}
 forStmt(hasBody(compoundStmt()))
   matches 'for (;;) {}'
 with compoundStmt()
   matching '{}'
 
Given
   { {}; 1+2; }
 hasAnySubstatement(compoundStmt())
   matches '{ {}; 1+2; }'
 with compoundStmt()
   matching '{}'
 </pre></td></tr>
 
 
+<tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1CoroutineBodyStmt.html">CoroutineBodyStmt</a>&gt;</td><td class="name" onclick="toggle('hasBody5')"><a name="hasBody5Anchor">hasBody</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1Stmt.html">Stmt</a>&gt; InnerMatcher</td></tr>
+<tr><td colspan="4" class="doc" id="hasBody5"><pre>Matches a 'for', 'while', 'while' statement or a function or coroutine
+definition that has a given body. Note that in case of functions or
+coroutines this matcher only matches the definition itself and not the
+other declarations of the same function or coroutine.
+
+Given
+  for (;;) {}
+forStmt(hasBody(compoundStmt()))
+  matches 'for (;;) {}'
+with compoundStmt()
+  matching '{}'
+
+Given
+  void f();
+  void f() {}
+functionDecl(hasBody(compoundStmt()))
+  matches 'void f() {}'
+with compoundStmt()
+  matching '{}'
+  but does not match 'void f();'
+</pre></td></tr>
+
+
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1DecayedType.html">DecayedType</a>&gt;</td><td class="name" onclick="toggle('hasDecayedType0')"><a name="hasDecayedType0Anchor">hasDecayedType</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1QualType.html">QualType</a>&gt; InnerType</td></tr>
 <tr><td colspan="4" class="doc" id="hasDecayedType0"><pre>Matches the decayed type, whoes decayed type matches InnerMatcher
 </pre></td></tr>
 
 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1DeclRefExpr.html">DeclRefExpr</a>&gt;</td><td class="name" onclick="toggle('hasDeclaration11')"><a name="hasDeclaration11Anchor">hasDeclaration</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1Decl.html">Decl</a>&gt;  InnerMatcher</td></tr>
 <tr><td colspan="4" class="doc" id="hasDeclaration11"><pre>Matches a node if the declaration associated with that node
 matches the given matcher.

 The matcher:
   decompositionDecl(hasBinding(0,
   bindingDecl(hasName("f").bind("fBinding"))))
 matches the decomposition decl with 'f' bound to "fBinding".
 </pre></td></tr>
 
 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1DoStmt.html">DoStmt</a>&gt;</td><td class="name" onclick="toggle('hasBody0')"><a name="hasBody0Anchor">hasBody</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1Stmt.html">Stmt</a>&gt; InnerMatcher</td></tr>
-<tr><td colspan="4" class="doc" id="hasBody0"><pre>Matches a 'for', 'while', 'do' statement or a function definition that has
-a given body. Note that in case of functions this matcher only matches the
-definition itself and not the other declarations of the same function.
+<tr><td colspan="4" class="doc" id="hasBody0"><pre>Matches a 'for', 'while', 'while' statement or a function or coroutine
+definition that has a given body. Note that in case of functions or
+coroutines this matcher only matches the definition itself and not the
+other declarations of the same function or coroutine.
 
 Given
   for (;;) {}
 forStmt(hasBody(compoundStmt()))
   matches 'for (;;) {}'
 with compoundStmt()
   matching '{}'
 

 fieldDecl(hasInClassInitializer(integerLiteral(equals(2))))
   matches 'int a;' but not 'int b;'.
 fieldDecl(hasInClassInitializer(anything()))
   matches 'int a;' and 'int b;' but not 'int c;'.
 </pre></td></tr>
 
 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1ForStmt.html">ForStmt</a>&gt;</td><td class="name" onclick="toggle('hasBody1')"><a name="hasBody1Anchor">hasBody</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1Stmt.html">Stmt</a>&gt; InnerMatcher</td></tr>
-<tr><td colspan="4" class="doc" id="hasBody1"><pre>Matches a 'for', 'while', 'do' statement or a function definition that has
-a given body. Note that in case of functions this matcher only matches the
-definition itself and not the other declarations of the same function.
+<tr><td colspan="4" class="doc" id="hasBody1"><pre>Matches a 'for', 'while', 'while' statement or a function or coroutine
+definition that has a given body. Note that in case of functions or
+coroutines this matcher only matches the definition itself and not the
+other declarations of the same function or coroutine.
 
 Given
   for (;;) {}
 forStmt(hasBody(compoundStmt()))
   matches 'for (;;) {}'
 with compoundStmt()
   matching '{}'
 
classTemplateSpecializationDecl(hasAnyTemplateArgument(
   matches the specialization A&lt;int&gt;
 
 functionDecl(hasAnyTemplateArgument(refersToType(asString("int"))))
   matches the specialization f&lt;int&gt;
 </pre></td></tr>
 
 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1FunctionDecl.html">FunctionDecl</a>&gt;</td><td class="name" onclick="toggle('hasBody4')"><a name="hasBody4Anchor">hasBody</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1Stmt.html">Stmt</a>&gt; InnerMatcher</td></tr>
-<tr><td colspan="4" class="doc" id="hasBody4"><pre>Matches a 'for', 'while', 'do' statement or a function definition that has
-a given body. Note that in case of functions this matcher only matches the
-definition itself and not the other declarations of the same function.
+<tr><td colspan="4" class="doc" id="hasBody4"><pre>Matches a 'for', 'while', 'while' statement or a function or coroutine
+definition that has a given body. Note that in case of functions or
+coroutines this matcher only matches the definition itself and not the
+other declarations of the same function or coroutine.
 
 Given
   for (;;) {}
 forStmt(hasBody(compoundStmt()))
   matches 'for (;;) {}'
 with compoundStmt()
   matching '{}'
 
Given
   }
 variableArrayType(hasSizeExpr(ignoringImpCasts(declRefExpr(to(
   varDecl(hasName("b")))))))
   matches "int a[b]"
 </pre></td></tr>
 
 
 <tr><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1WhileStmt.html">WhileStmt</a>&gt;</td><td class="name" onclick="toggle('hasBody2')"><a name="hasBody2Anchor">hasBody</a></td><td>Matcher&lt;<a href="https://clang.llvm.org/doxygen/classclang_1_1Stmt.html">Stmt</a>&gt; InnerMatcher</td></tr>
-<tr><td colspan="4" class="doc" id="hasBody2"><pre>Matches a 'for', 'while', 'do' statement or a function definition that has
-a given body. Note that in case of functions this matcher only matches the
-definition itself and not the other declarations of the same function.
+<tr><td colspan="4" class="doc" id="hasBody2"><pre>Matches a 'for', 'while', 'while' statement or a function or coroutine
+definition that has a given body. Note that in case of functions or
+coroutines this matcher only matches the definition itself and not the
+other declarations of the same function or coroutine.
 
 Given
   for (;;) {}
 forStmt(hasBody(compoundStmt()))
   matches 'for (;;) {}'
 with compoundStmt()
   matching '{}'
 

clang/docs/tools/dump_ast_matchers.py

try:
     from urllib.request import urlopen
 except ImportError:
     from urllib2 import urlopen
 
 CLASS_INDEX_PAGE_URL = "https://clang.llvm.org/doxygen/classes.html"
 try:
     CLASS_INDEX_PAGE = urlopen(CLASS_INDEX_PAGE_URL).read().decode("utf-8")
 except Exception as e:
-    raise Exception("Unable to get %s: %s" % (CLASS_INDEX_PAGE_URL, e))
+    print("Unable to get %s: %s" % (CLASS_INDEX_PAGE_URL, e))
 
 MATCHERS_FILE = "../../include/clang/ASTMatchers/ASTMatchers.h"
 
 # Each matcher is documented in one row of the form:
 #   result | name | argA
 # The subsequent row contains the documentation and is hidden by default,
 # becoming visible via javascript when the user clicks the matcher name.
 TD_TEMPLATE = """
def link_if_exists(m):
 
         We want to do this only if the page exists, in which case it will be
         referenced from the class index page.
         """
         name = m.group(1)
         url = "https://clang.llvm.org/doxygen/classclang_1_1%s.html" % name
         if url not in doxygen_probes:
             search_str = 'href="classclang_1_1%s.html"' % name
+            try:
                 doxygen_probes[url] = search_str in CLASS_INDEX_PAGE
+            except:
+                doxygen_probes[url] = True
             if not doxygen_probes[url]:
                 print("Did not find %s in class index page" % name)
         if doxygen_probes[url]:
             return r'Matcher&lt;<a href="%s">%s</a>&gt;' % (url, name)
         else:
             return m.group(0)
 
     text = re.sub(r"Matcher&lt;([^\*&]+)&gt;", link_if_exists, text)
def act_on_decl(declaration, comment, allowed_types):
     """Parse the matcher out of the given declaration and comment.
 
     If 'allowed_types' is set, it contains a list of node types the matcher
     can match on, as extracted from the static type asserts in the matcher
     definition.
     """
     if declaration.strip():
 
-        if re.match(r"^\s?(#|namespace|using)", declaration):
+        if re.match(r"^\s?(#|namespace|using|template <typename NodeType> using|})", declaration):
             return
 
         # Node matchers are defined by writing:
         #   VariadicDynCastAllOfMatcher<ResultType, ArgumentType> name;
         m = re.match(
             r""".*Variadic(?:DynCast)?AllOfMatcher\s*<
                        \s*([^\s,]+)\s*(?:,
                        \s*([^\s>]+)\s*)?>