[AST Matchers] Update dump_ast_matchers.py to query only class index page.

[jcking1034]

Summary

It may be possible to avoid relying on accessing many individual class pages,
by instead scanning the class index page at
https://clang.llvm.org/doxygen/classes.html. This updates the script to do so,
and includes updates to LibASTMatchersReference.html generated by the
modified script.

[ymandel]

Aaron -- I'm not the right reviewer for this. I've put you as the reviewer, but feel free to re-route if there's someone better. Thanks!

[aaron.ballman]

clang/docs/tools/dump_ast_matchers.py

13

Do we need to handle an exception here if there are network issues?

[jcking1034]

Catch and reraise exceptions with an appropriate message.

[jcking1034]

clang/docs/tools/dump_ast_matchers.py

13

Technically, if there is a network issue, any exceptions would cause the script to terminate. But I think you're right that we should handle this case more carefully.

[aaron.ballman]

LGTM, though I'll note I'm by no means a Python expert. I'm adding a few more AST matching folks in case they spot something I've missed, so please give this at least 24 hours before landing.

[sammccall]

Nice!

clang/docs/tools/dump_ast_matchers.py

48–53

While here I'd add a comment:

"""
Wrap a likely AST node name in a link to its clang docs.

We want to do this only if the page exists, in which case it will be referenced from the class index page.
"""

(You're only adding a bit of conceptual complexity here, but it looks like it's been built up over time...)

[jcking1034]

Add docstring to link_if_exists.

[jcking1034]

Rebase onto master

clang/docs/tools/dump_ast_matchers.py

 #!/usr/bin/env python
 # A tool to parse ASTMatchers.h and update the documentation in
 # ../LibASTMatchersReference.html automatically. Run from the
 # directory in which this file is located to update the docs.
 
 import collections
 import re
 try:
     from urllib.request import urlopen
 except ImportError:
     from urllib2 import urlopen
 
+CLASS_INDEX_PAGE_URL = 'https://clang.llvm.org/doxygen/classes.html'
+try:
+  CLASS_INDEX_PAGE = urlopen(CLASS_INDEX_PAGE_URL).read()
+except Exception as e:
+  raise Exception('Unable to get %s: %s' % (CLASS_INDEX_PAGE_URL, e))
+
 MATCHERS_FILE = '../../include/clang/ASTMatchers/ASTMatchers.h'
 
 # Each matcher is documented in one row of the form:
 #   result | name | argA
 # The subsequent row contains the documentation and is hidden by default,
 # becoming visible via javascript when the user clicks the matcher name.
 TD_TEMPLATE="""
 <tr><td>%(result)s</td><td class="name" onclick="toggle('%(id)s')"><a name="%(id)sAnchor">%(name)s</a></td><td>%(args)s</td></tr>

 # Cache for doxygen urls we have already verified.
 doxygen_probes = {}
 
 def esc(text):
   """Escape any html in the given text."""
   text = re.sub(r'&', '&amp;', text)
   text = re.sub(r'<', '&lt;', text)
   text = re.sub(r'>', '&gt;', text)
   def link_if_exists(m):
+    """Wrap a likely AST node name in a link to its clang docs.
+
+       We want to do this only if the page exists, in which case it will be
+       referenced from the class index page.
+    """
     name = m.group(1)
     url = 'https://clang.llvm.org/doxygen/classclang_1_1%s.html' % name
     if url not in doxygen_probes:
-      try:
-        print('Probing %s...' % url)
-        urlopen(url)
-        doxygen_probes[url] = True
-      except:
-        doxygen_probes[url] = False
+      search_str = 'href="classclang_1_1%s.html"' % name
+      doxygen_probes[url] = search_str in CLASS_INDEX_PAGE
+      if not doxygen_probes[url]:
+        print('Did not find %s in class index page' % name)
     if doxygen_probes[url]:
       return r'Matcher&lt;<a href="%s">%s</a>&gt;' % (url, name)
     else:
       return m.group(0)
   text = re.sub(
     r'Matcher&lt;([^\*&]+)&gt;', link_if_exists, text)
   return text