[AST] Add generator for source location introspection

[steveire]

Summary

Generate a json file containing descriptions of AST classes and their
public accessors which return SourceLocation or SourceRange.

Use the JSON file to generate a C++ API and implementation for accessing
the source locations and method names for accessing them for a given AST
node.

This new API can be used to implement 'srcloc' output in clang-query:

http://ce.steveire.com/z/m_kTIo

In this first version of this feature, only the accessors for Stmt
classes are generated, not Decls, TypeLocs etc. Those can be added
after this change is reviewed, as this change is mostly about
infrastructure of these code generators.

[steveire]

Update

[aaron.ballman]

Do I understand correctly that the workflow is to use the new dumping tool to generate the needed JSON file that then gets used as input to generate_cxx_src_locs.py which creates NodeLocationIntrospection.cpp/.h that then gets used by clang-query (eventually)? So there are two levels of translation involved to get the final source code? If so, do you know what the performance/overhead for this looks like compared to a typical build? I'm trying to get an idea for whether this will have negative impacts on the build bots such that we may want to add an LLVM cmake configure option to control whether this work happens or not.

clang/lib/Tooling/DumpTool/APIData.h

2

Looks like a copy pasta error.

11

Might as well fix this lint warning.

22

per the usual naming rules.

26

Are these extensions going to add new members for those? If so, perhaps Locs and Rngs should have more descriptive names initially?

clang/lib/Tooling/DumpTool/ASTSrcLocProcessor.cpp

41

TIL what "clade" means, thank you for that new word. :-D

72–75

Similar question here about whether we should use less generic names or not.

126

128

135

136–137

155–156

158

clang/lib/Tooling/DumpTool/ASTSrcLocProcessor.h

11

This one as well.

18

I don't think this is being used, but you should include what you use (StringRef, unique_ptr)

30

Should this ctor be marked explicit?

32–35

clang/lib/Tooling/DumpTool/ClangSrcLocDump.cpp

44

Hmmm, do we want such a long name for this option? I was guessing that -o isn't appropriate because there may be multiple files output and so this isn't naming the output file but the output directory, but json-output-path is a mouthful for setting the output location. Or do you think users won't be using that option often enough for the length to matter?

71

74

106

Do you think we may need the ability for the user to pass other options to this driver invocation for things like -fms-compatibility or whatnot?

109

113

clang/lib/Tooling/DumpTool/generate_cxx_src_locs.py

77–87

Would this be a more clear equivalent?

clang/unittests/Introspection/IntrospectionTest.cpp

15

Might as well fix this lint warning.

41

65

[steveire]

Update

[steveire]

Do I understand correctly that the workflow is to use the new dumping tool to generate the needed JSON file that then gets used as input to generate_cxx_src_locs.py which creates NodeLocationIntrospection.cpp/.h that then gets used by clang-query (eventually)?

Yes, that's right. I've added the patch for the latter now at D93325.

So there are two levels of translation involved to get the final source code?

There reason for the separation of the generation of JSON from the generation of C++ (using the JSON) is that the JSON can also be used to generate bindings for other languages. In https://steveire.wordpress.com/2019/04/30/the-future-of-ast-matching-refactoring-tools-eurollvm-and-accu/ I demonstrated that by generating and using Javascript bindings, but I've also proven the concept with Python3 bindings and added a clang-tidy module allowing clang-tidy checks to be written in Python. That resolves https://bugs.llvm.org//show_bug.cgi?id=32739 at least partially (there may be things which can still only be done in C++).

If so, do you know what the performance/overhead for this looks like compared to a typical build? I'm trying to get an idea for whether this will have negative impacts on the build bots such that we may want to add an LLVM cmake configure option to control whether this work happens or not.

In this patch, the generation is already disabled for Debug builds because the JSON generation is slow in Debug builds (I think we discussed that in Belfast). The reason it is slow is that compiling AST/AST.h with a debug-build seems to be slow. The slowness is not caused by the AST matching, but seems to be spent parsing.

With a release build, generating the JSON takes 3 seconds on my laptop. Plenty of compilations in the llvm build take longer. Generating the c++ files with generate_cxx_src_locs.py takes 0.04 seconds.

I'm not opposed to adding a condition in cmake for it, in case someone wants to build it even in Debug mode.

clang/lib/Tooling/DumpTool/APIData.h

22

I changed it, but are you referring to this proposal? https://llvm.org/docs/Proposals/VariableNames.html

26

Just for an idea, in a follow-up I have

TypeSourceInfos
TypeLocs
TemplateArgumentLocs
DeclNames
NestedNames
ConstCharStarMethods
StringRefMethods
StdStringMethods

We can review those names in the follow-up.

clang/lib/Tooling/DumpTool/ClangSrcLocDump.cpp

44

Yes, this tool will only be called from the buildsystem. The intention is also that it doesn't generate any other files either. The JSON file should have all the content needed for clang-query srcloc, bindings defined and used in the llvm repo etc and any other use we find for this. If we want to enable third parties to do similar things, we would install the JSON file, not this tool.

106

I don't think the user needs to have that ability, no. If there's any platform-specific options needed, they would be added directly here.

109

error: ‘Compilation’ was not declared in this scope; did you mean ‘clang::driver::Compilation’? Is this really necessary?

113

error: ‘JobList’ does not name a type I don't get a suggestion from the compiler. Is this really necessary?

clang/lib/Tooling/DumpTool/generate_cxx_src_locs.py

77–87

tools/clang/include/clang/Tooling/NodeLocationIntrospection.h: In member function ‘bool clang::tooling::internal::RangeLessThan::operator()(const std::pair<clang::SourceRange, std::__cxx11::basic_string<char> >&, const std::pair<clang::SourceRange, std::__cxx11::basic_string<char> >&) const’:
tools/clang/include/clang/Tooling/NodeLocationIntrospection.h:31:21: error: cannot bind non-const lvalue reference of type ‘clang::SourceLocation&’ to an rvalue of type ‘clang::SourceLocation’
   31 |     return std::tie(SourceLocation(LHS.first.getBegin()), LHS.first.getEnd(),
      |                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/c++/9/bits/unique_ptr.h:37,
                 from /usr/include/c++/9/memory:80,
                 from llvm-project/llvm/include/llvm/Support/Casting.h:20,
                 from llvm-project/clang/include/clang/Basic/LLVM.h:21,
                 from llvm-project/clang/include/clang/Basic/DiagnosticIDs.h:17,
                 from llvm-project/clang/include/clang/Basic/Diagnostic.h:17,
                 from llvm-project/clang/include/clang/AST/NestedNameSpecifier.h:18,
                 from llvm-project/clang/include/clang/AST/Type.h:21,
                 from llvm-project/clang/include/clang/AST/CanonicalType.h:17,
                 from llvm-project/clang/include/clang/AST/ASTContext.h:19,
                 from llvm-project/clang/include/clang/AST/AST.h:17,
                 from tools/clang/lib/Tooling/generated/NodeLocationIntrospection.cpp:10:
/usr/include/c++/9/tuple:1611:19: note:   initializing argument 1 of ‘constexpr std::tuple<_Elements& ...> std::tie(_Elements& ...) [with _Elements = {clang::SourceLocation, clang::SourceLocation, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >}]’
 1611 |     tie(_Elements&... __args) noexcept
      |         ~~~~~~~~~~^~~~~~~~~~

I wasn't able to make it work.

clang/unittests/Introspection/IntrospectionTest.cpp

15

Funny, I thought clang-format would fix these things.

[steveire]

Update

[steveire]

clang/lib/Tooling/DumpTool/ASTSrcLocProcessor.cpp

41

It's a biology word, but I think it works well!

[steveire]

Update

[steveire]

Update

[steveire]

Update

[steveire]

@thakis FYI - I think the GN build would need to be adapted to this.

[steveire]

Update

[steveire]

Update

[steveire]

Update

[njames93]

Mostly mechanical changes requested here.

In D93164#2456247, @steveire wrote:

@thakis FYI - I think the GN build would need to be adapted to this.

FWIW the GN build has a bot that can typically update its gn files.

clang/include/clang/Tooling/NodeIntrospection.h

41

Is there a strong use case to return by value here?

55

Should this (and potential a few others) be moved to an implementation file.

56

It's probably more effective to use a SmallVector to reduce the needs for allocations here.

61

This is wasteful, just operate on rbegin/rend later and eliminate this call.

64

Is this more readable, IDK, but it sure as hell is more fun :)

66–67

clang/lib/Tooling/DumpTool/APIData.h

22

While there is a proposal in place, right now we should ensure we aren't deviating from the current system in patches. Unfortunately readability-identifier-naming has been disabled on clang directory due to excessive violations. May I suggest re-enabling it locally and then running clang-tidy-diff.py over the patch. should square most things up.

clang/lib/Tooling/DumpTool/ASTSrcLocProcessor.cpp

23–26

Why is this a variable, a templated function should do the same thing.
I imagine its something like this.

46

This seems dangerous, the MatchASTConsumer doesn't own its MatchFinder, so this is going to leak.

clang/lib/Tooling/DumpTool/ASTSrcLocProcessor.h

2

Can you add a modeline here, same goes for other headers.

15

Any reason for empty lines between includes?

clang/lib/Tooling/DumpTool/ClangSrcLocDump.cpp

81

87–89

[njames93]

Another point, is there any use case of this outside of clang-query. If not would it not be wise to move this infrastructure to clang-tools-extra/clang-query?

[steveire]

Update

[steveire]

Update

[steveire]

Update

[steveire]

clang/lib/Tooling/DumpTool/ASTSrcLocProcessor.cpp

23–26

That seems far more noisy. I've left it as a lambda and moved it to the point of use.

[steveire]

clang/include/clang/Tooling/NodeIntrospection.h

55

The implementation file is generated by the python script. Rather than hiding this in the python script, I think it's better to leave it here.

[steveire]

Update

[steveire]

Update

[njames93]

This is almost ready but a few more points need addressing.
Running clang-format over the inc file is pointless and just extends compilation time while adding an unnecessary dependency on clang-format.
The inc file should likely live in the include build directory, All tablegen files seem to live in there. You could either move the CMake code that generates it into the include directory, or alter the directory, this should do that if you want and its safer than replace as it would only change the last /lib/ detected.

# Replace the last lib component of the current binary directory with include
string(FIND ${CMAKE_CURRENT_BINARY_DIR} "/lib/" PATH_LIB_START REVERSE)
if(PATH_LIB_START EQUAL -1)
  message(FATAL_ERROR "Couldn't find lib component in binary directory")
endif()
math(EXPR PATH_LIB_END "${PATH_LIB_START}+5")
string(SUBSTRING ${CMAKE_CURRENT_BINARY_DIR} 0 ${PATH_LIB_START} PATH_HEAD)
string(SUBSTRING ${CMAKE_CURRENT_BINARY_DIR} ${PATH_LIB_END} -1 PATH_TAIL)
string(CONCAT BINARY_INCLUDE_DIR ${PATH_HEAD} "/include/" ${PATH_TAIL})

After moving it to the Include output folder, In the cpp file you would need #include "clang/Tooling/NodeIntrospection.inc".
This would also remove a lot of those commands in lib/tooling/CMakeLists.txt.
Tablegen has a command line option --write-if-changed It may be wise to also include that in your generator script instead of using copy-if-different in the aforementioned CMakeLists.txt.

clang/include/clang/Tooling/NodeIntrospection.h

17–18

These should be quoted includes

clang/lib/Tooling/CMakeLists.txt

29

It may be wise to use the COMMENT argument to let the users know that it's building the ASTNodeAPI.json.

56

Likewise a comment to say building NodeIntrospection.inc.

128

This shouldn't appear in the source list.

clang/lib/Tooling/DumpTool/APIData.h

11

Header guard should be LLVM_CLANG_LIB_TOOLING_DUMPTOOL_APIDATA_H

clang/lib/Tooling/NodeIntrospection.cpp

14–16

Quoted includes and the NodeIntrospection.h include is the MainFileInclude so should appear first.

[steveire]

Update

[steveire]

Update

[steveire]

Update

[steveire]

clang/lib/Tooling/CMakeLists.txt

128

We need to tell CMake the dependency.

[njames93]

nit: A few reformat hints

[steveire]

@thakis Presumably you'll have to update the GN build now.

[thakis]

A few more high-level questions:

• What's the point of the intermediary json file? Why not generate the final c++ directly? (As far as I can tell, this wasn't discussed during the review yet)
• Do we need to generate code for this at all? Could this be done via xmacros or tablegen?

Having a bespoke custom python -> json -> python -> c++ pipeline here seems like it's fairly different from how the rest of clang does things, and it seems like it duplicates some of the existing tooling we have here.

(Having said that, I'm no code owner here -- @rsmith is. Maybe he has an opinion.)

Lower-level: Did you see all the comments on https://reviews.llvm.org/rGd627a27d264b47eda3f15f086ff419dfe053ebf7 ? This relanded with them unaddressed. Please address them in a follow-up. (Sorry for leaving the comments on the commit instead of the review!)

clang/lib/Tooling/CMakeLists.txt

31

Putting this in the root of the build dir seems a bit untidy. I think CMAKE_CURRENT_BINARY_DIR is what we usually use for generated files.

74

...like used here. Why generated/? Everything in CMAKE_CURRENT_BINARY_DIR is generated. (Compare to find . -name '*.inc')

Also, why not make the python script write the file only if changed instead of making a copy here? (like llvm-tblgen does)

[dmgreen]

Hello. Does this work when the default target triple isn't native? This seems to be trying to compile clang sources with the just built clang - something that I don't think is always possible. I'm seeing errors like fatal error: 'cstddef' file not found, and failing to link the new IntrospectionTests, with undefined references to NodeIntrospection::GetLocations, as a full toolchain is not setup. I don't believe the buildbots are working right now, so it's difficult to see if any other systems have similar problems.

Also some of the files here have been added with a University of Illinois Open Source License. They should presumably be using the newer Apache license now.

[steveire]

In D93164#2624690, @thakis wrote:

A few more high-level questions:

• What's the point of the intermediary json file? Why not generate the final c++ directly? (As far as I can tell, this wasn't discussed during the review yet)

It came up in review earlier: https://reviews.llvm.org/D93164#2456181

• Do we need to generate code for this at all? Could this be done via xmacros or tablegen?

Can you say more? Would this require generating the declarations in include/clang/AST? It sounds like a large maintenance burden, but maybe I'm missing something.

Having a bespoke custom python -> json -> python -> c++ pipeline here seems like it's fairly different from how the rest of clang does things, and it seems like it duplicates some of the existing tooling we have here.

(Having said that, I'm no code owner here -- @rsmith is. Maybe he has an opinion.)

Lower-level: Did you see all the comments on https://reviews.llvm.org/rGd627a27d264b47eda3f15f086ff419dfe053ebf7 ? This relanded with them unaddressed. Please address them in a follow-up. (Sorry for leaving the comments on the commit instead of the review!)

Done, thanks!

[steveire]

In D93164#2624760, @dmgreen wrote:

Hello. Does this work when the default target triple isn't native? This seems to be trying to compile clang sources with the just built clang - something that I don't think is always possible. I'm seeing errors like fatal error: 'cstddef' file not found, and failing to link the new IntrospectionTests, with undefined references to NodeIntrospection::GetLocations, as a full toolchain is not setup. I don't believe the buildbots are working right now, so it's difficult to see if any other systems have similar problems.

Can you report if running cmake . -DCLANG_TOOLING_BUILD_AST_INTROSPECTION=OFF allows you to complete the build?

Is there some way I can check that the default target triple isn't native in the cmake code? Then I can set that option automatically.

Also some of the files here have been added with a University of Illinois Open Source License. They should presumably be using the newer Apache license now.

Fixed, thanks!

[nikic]

Reverted in https://github.com/llvm/llvm-project/commit/e0f70a8a979f5b843e90a0a20442ca79e2507208 due to build failure.

[steveire]

In D93164#2624798, @nikic wrote:

Reverted in https://github.com/llvm/llvm-project/commit/e0f70a8a979f5b843e90a0a20442ca79e2507208 due to build failure.

Was the problem there just the shebang line?

[steveire]

In D93164#2624798, @nikic wrote:

Reverted in https://github.com/llvm/llvm-project/commit/e0f70a8a979f5b843e90a0a20442ca79e2507208 due to build failure.

@nikic How do I run a test build on that machine? Or can you diagnose the problem instead?

[nikic]

@steveire When running the command manually I get:

/root/llvm-compile-time-tracker/llvm-project/clang/lib/Tooling/DumpTool/generate_cxx_src_locs.py --json-input-path /root/llvm-compile-time-tracker/llvm-project-build/ASTNodeAPI.json --output-file generated/NodeIntrospection.inc --empty-implementation 0
-bash: /root/llvm-compile-time-tracker/llvm-project/clang/lib/Tooling/DumpTool/generate_cxx_src_locs.py: /usr/bin/python: bad interpreter: No such file or directory

So probably this doesn't work on any system that only has Python 3.

As the .py file is invoked directly, I assume that ${PYTHON_EXECUTABLE} is empty. Maybe you were looking for ${Python3_EXECUTABLE}?

[arichardson]

clang/lib/Tooling/DumpTool/generate_cxx_src_locs.py

2

Maybe this should be #!/use/bin/env python (or python3) instead?

[njames93]

It may be wise to alter this so there is no need for the python script.
How about altering the dump tool to support outputting both json files and the cpp code needed for node introspection.
Maybe have arguments to the tool --json-output-path and --introspection-output-path.
If both are specified write both, if none specified error out.

[thakis]

ast-dump-tool is still somewhere in lib/ instead of in tools/ in the reland as far as I can tell.

[nathanchance]

I am seeing a spew of errors after 19740652c4c4329e2b9e77f96e5e31c360b4e8bb (what appears to be the latest version of this patch):

$ cmake \
-G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER=$(command -v clang) \
-DCMAKE_CXX_COMPILER=$(command -v clang++) \
-DLLVM_CCACHE_BUILD=ON \
-DLLVM_ENABLE_PROJECTS=clang \
../llvm
...

$ ninja run-ast-api-dump-tool
...
In file included from /home/nathan/src/llvm-project/build/tools/clang/lib/Tooling/ASTTU.cpp:2:
In file included from /home/nathan/src/llvm-project/llvm/../clang/include/clang/AST/AST.h:17:
In file included from /home/nathan/src/llvm-project/llvm/../clang/include/clang/AST/ASTContext.h:19:
In file included from /home/nathan/src/llvm-project/llvm/../clang/include/clang/AST/CanonicalType.h:17:
In file included from /home/nathan/src/llvm-project/llvm/../clang/include/clang/AST/Type.h:20:
In file included from /home/nathan/src/llvm-project/llvm/../clang/include/clang/AST/DependenceFlags.h:11:
In file included from /home/nathan/src/llvm-project/llvm/../clang/include/clang/Basic/BitmaskEnum.h:18:
In file included from /home/nathan/src/llvm-project/llvm/include/llvm/ADT/BitmaskEnum.h:16:
In file included from /home/nathan/src/llvm-project/llvm/include/llvm/Support/MathExtras.h:21:
/usr/include/c++/10.2.0/cstdint:52:11: error: no member named 'int_fast8_t' in the global namespace
  using ::int_fast8_t;
        ~~^
/usr/include/c++/10.2.0/cstdint:53:11: error: no member named 'int_fast16_t' in the global namespace; did you mean '__int_least16_t'?
  using ::int_fast16_t;
        ~~^
/usr/include/bits/types.h:54:19: note: '__int_least16_t' declared here
typedef __int16_t __int_least16_t;
                  ^
In file included from /home/nathan/src/llvm-project/build/tools/clang/lib/Tooling/ASTTU.cpp:2:
In file included from /home/nathan/src/llvm-project/llvm/../clang/include/clang/AST/AST.h:17:
In file included from /home/nathan/src/llvm-project/llvm/../clang/include/clang/AST/ASTContext.h:19:
In file included from /home/nathan/src/llvm-project/llvm/../clang/include/clang/AST/CanonicalType.h:17:
In file included from /home/nathan/src/llvm-project/llvm/../clang/include/clang/AST/Type.h:20:
In file included from /home/nathan/src/llvm-project/llvm/../clang/include/clang/AST/DependenceFlags.h:11:
In file included from /home/nathan/src/llvm-project/llvm/../clang/include/clang/Basic/BitmaskEnum.h:18:
In file included from /home/nathan/src/llvm-project/llvm/include/llvm/ADT/BitmaskEnum.h:16:
In file included from /home/nathan/src/llvm-project/llvm/include/llvm/Support/MathExtras.h:21:
/usr/include/c++/10.2.0/cstdint:54:11: error: no member named 'int_fast32_t' in the global namespace; did you mean '__int_least32_t'?
  using ::int_fast32_t;
        ~~^
/usr/include/bits/types.h:56:19: note: '__int_least32_t' declared here
typedef __int32_t __int_least32_t;
                  ^
In file included from /home/nathan/src/llvm-project/build/tools/clang/lib/Tooling/ASTTU.cpp:2:
In file included from /home/nathan/src/llvm-project/llvm/../clang/include/clang/AST/AST.h:17:
In file included from /home/nathan/src/llvm-project/llvm/../clang/include/clang/AST/ASTContext.h:19:
In file included from /home/nathan/src/llvm-project/llvm/../clang/include/clang/AST/CanonicalType.h:17:
In file included from /home/nathan/src/llvm-project/llvm/../clang/include/clang/AST/Type.h:20:
In file included from /home/nathan/src/llvm-project/llvm/../clang/include/clang/AST/DependenceFlags.h:11:
In file included from /home/nathan/src/llvm-project/llvm/../clang/include/clang/Basic/BitmaskEnum.h:18:
In file included from /home/nathan/src/llvm-project/llvm/include/llvm/ADT/BitmaskEnum.h:16:
In file included from /home/nathan/src/llvm-project/llvm/include/llvm/Support/MathExtras.h:21:
/usr/include/c++/10.2.0/cstdint:55:11: error: no member named 'int_fast64_t' in the global namespace; did you mean '__int_least64_t'?
  using ::int_fast64_t;
        ~~^
/usr/include/bits/types.h:58:19: note: '__int_least64_t' declared here
typedef __int64_t __int_least64_t;
                  ^
...

[steveire]

In D93164#2630203, @nathanchance wrote:

I am seeing a spew of errors after 19740652c4c4329e2b9e77f96e5e31c360b4e8bb (what appears to be the latest version of this patch):

Hi @nathanchance Does it make your build fail? I have pushed a fix. Can you update and try again?

[nathanchance]

In D93164#2630534, @steveire wrote:

In D93164#2630203, @nathanchance wrote:

I am seeing a spew of errors after 19740652c4c4329e2b9e77f96e5e31c360b4e8bb (what appears to be the latest version of this patch):

Hi @nathanchance Does it make your build fail? I have pushed a fix. Can you update and try again?

Does not look like my build ever errored out (exit code was 0 even without the fix you pushed).

It does look like the errors are hidden now though (mostly but that is fine enough for me):

[1373/1373] ASTNodeAPI.json
20 errors generated.

Thanks for the quick response!

[mgorny]

This change breaks cross-compilation now, as it tries running an executable built for the target system:

FAILED: tools/clang/lib/Tooling/ASTNodeAPI.json 
cd /home/mgorny/llvm-project/build.arm64/tools/clang/lib/Tooling && /home/mgorny/llvm-project/build.arm64/bin/clang-ast-dump --skip-processing=0 --astheader=/home/mgorny/llvm-project/build.arm64/tools/clang/lib/Tooling/ASTTU.cpp -I /home/mgorny/llvm-project/build.arm64/lib/clang/13.0.0/include -I /home/mgorny/llvm-project/llvm/../clang/include -I /home/mgorny/llvm-project/build.arm64/tools/clang/include -I /home/mgorny/llvm-project/build.arm64/include -I /home/mgorny/llvm-project/llvm/include -I /sysroot/arm64/usr/include/c++/v1 -I /usr/lib/clang/11.0.1/include -I /sysroot/arm64/usr/include --json-output-path /home/mgorny/llvm-project/build.arm64/tools/clang/lib/Tooling/ASTNodeAPI.json
/bin/sh: /home/mgorny/llvm-project/build.arm64/bin/clang-ast-dump: Exec format error

I guess you can look at TableGens how to correctly generate and use host executables.

[lancethepants]

In D93164#2653067, @mgorny wrote:

This change breaks cross-compilation now, as it tries running an executable built for the target system:

FAILED: tools/clang/lib/Tooling/ASTNodeAPI.json 
cd /home/mgorny/llvm-project/build.arm64/tools/clang/lib/Tooling && /home/mgorny/llvm-project/build.arm64/bin/clang-ast-dump --skip-processing=0 --astheader=/home/mgorny/llvm-project/build.arm64/tools/clang/lib/Tooling/ASTTU.cpp -I /home/mgorny/llvm-project/build.arm64/lib/clang/13.0.0/include -I /home/mgorny/llvm-project/llvm/../clang/include -I /home/mgorny/llvm-project/build.arm64/tools/clang/include -I /home/mgorny/llvm-project/build.arm64/include -I /home/mgorny/llvm-project/llvm/include -I /sysroot/arm64/usr/include/c++/v1 -I /usr/lib/clang/11.0.1/include -I /sysroot/arm64/usr/include --json-output-path /home/mgorny/llvm-project/build.arm64/tools/clang/lib/Tooling/ASTNodeAPI.json
/bin/sh: /home/mgorny/llvm-project/build.arm64/bin/clang-ast-dump: Exec format error

I guess you can look at TableGens how to correctly generate and use host executables.

I am now running into this error now that clang13 has released. I cannot figure a way around this. I do not see any cmake options similar to the tablegens that allow you to specify the binary for the build system. Am I missing something or is clang13 just broken for cross compilation?

[smeenai]

clang/lib/Tooling/CMakeLists.txt

29–30

I'm looking at this commit in the context of https://bugs.llvm.org/show_bug.cgi?id=52106. Why do we not run the generator when targeting Windows or Apple platforms? Note that WIN32 and APPLE reflect the target platform, not the platform you're building on.

[steveire]

clang/lib/Tooling/CMakeLists.txt

29–30

I was not able to make the change pass the LLVM CI system on those platforms.

[andrew-wja]

In D93164#3048130, @lancethepants wrote:

In D93164#2653067, @mgorny wrote:

This change breaks cross-compilation now, as it tries running an executable built for the target system:

FAILED: tools/clang/lib/Tooling/ASTNodeAPI.json 
cd /home/mgorny/llvm-project/build.arm64/tools/clang/lib/Tooling && /home/mgorny/llvm-project/build.arm64/bin/clang-ast-dump --skip-processing=0 --astheader=/home/mgorny/llvm-project/build.arm64/tools/clang/lib/Tooling/ASTTU.cpp -I /home/mgorny/llvm-project/build.arm64/lib/clang/13.0.0/include -I /home/mgorny/llvm-project/llvm/../clang/include -I /home/mgorny/llvm-project/build.arm64/tools/clang/include -I /home/mgorny/llvm-project/build.arm64/include -I /home/mgorny/llvm-project/llvm/include -I /sysroot/arm64/usr/include/c++/v1 -I /usr/lib/clang/11.0.1/include -I /sysroot/arm64/usr/include --json-output-path /home/mgorny/llvm-project/build.arm64/tools/clang/lib/Tooling/ASTNodeAPI.json
/bin/sh: /home/mgorny/llvm-project/build.arm64/bin/clang-ast-dump: Exec format error

I guess you can look at TableGens how to correctly generate and use host executables.

I am now running into this error now that clang13 has released. I cannot figure a way around this. I do not see any cmake options similar to the tablegens that allow you to specify the binary for the build system. Am I missing something or is clang13 just broken for cross compilation?

Another confirmation that this change has broken cross compilation. I have tried setting both -DCLANG_TOOLING_BUILD_AST_INTROSPECTION=OFF and -DCMAKE_CROSSCOMPILING=ON separately and in combination. This means that LLVM cannot be cross compiled for AArch64, which is a pretty serious problem!

[lancethepants]

In D93164#3250945, @andrew-wja wrote:

In D93164#3048130, @lancethepants wrote:

In D93164#2653067, @mgorny wrote:

This change breaks cross-compilation now, as it tries running an executable built for the target system:

FAILED: tools/clang/lib/Tooling/ASTNodeAPI.json 
cd /home/mgorny/llvm-project/build.arm64/tools/clang/lib/Tooling && /home/mgorny/llvm-project/build.arm64/bin/clang-ast-dump --skip-processing=0 --astheader=/home/mgorny/llvm-project/build.arm64/tools/clang/lib/Tooling/ASTTU.cpp -I /home/mgorny/llvm-project/build.arm64/lib/clang/13.0.0/include -I /home/mgorny/llvm-project/llvm/../clang/include -I /home/mgorny/llvm-project/build.arm64/tools/clang/include -I /home/mgorny/llvm-project/build.arm64/include -I /home/mgorny/llvm-project/llvm/include -I /sysroot/arm64/usr/include/c++/v1 -I /usr/lib/clang/11.0.1/include -I /sysroot/arm64/usr/include --json-output-path /home/mgorny/llvm-project/build.arm64/tools/clang/lib/Tooling/ASTNodeAPI.json
/bin/sh: /home/mgorny/llvm-project/build.arm64/bin/clang-ast-dump: Exec format error

I guess you can look at TableGens how to correctly generate and use host executables.

I am now running into this error now that clang13 has released. I cannot figure a way around this. I do not see any cmake options similar to the tablegens that allow you to specify the binary for the build system. Am I missing something or is clang13 just broken for cross compilation?

Another confirmation that this change has broken cross compilation. I have tried setting both -DCLANG_TOOLING_BUILD_AST_INTROSPECTION=OFF and -DCMAKE_CROSSCOMPILING=ON separately and in combination. This means that LLVM cannot be cross compiled for AArch64, which is a pretty serious problem!

So I did find a solution. Maybe somewhere else in the forums or on irc. Can't remember for sure.

-DCMAKE_SYSTEM_NAME="Linux"
Add this is well to your cmake invocation. For some reason this triggers in cmake that we are cross-compiling. Why -DCMAKE_CROSSCOMPILING=ON is insufficient I have no idea why. I'm compiling on linux for linux, didn't think I'd need it. Kind of unintuitive, but that should hopefully get you going.

[smeenai]

In D93164#3252509, @lancethepants wrote:

In D93164#3250945, @andrew-wja wrote:

In D93164#3048130, @lancethepants wrote:

In D93164#2653067, @mgorny wrote:

This change breaks cross-compilation now, as it tries running an executable built for the target system:

FAILED: tools/clang/lib/Tooling/ASTNodeAPI.json 
cd /home/mgorny/llvm-project/build.arm64/tools/clang/lib/Tooling && /home/mgorny/llvm-project/build.arm64/bin/clang-ast-dump --skip-processing=0 --astheader=/home/mgorny/llvm-project/build.arm64/tools/clang/lib/Tooling/ASTTU.cpp -I /home/mgorny/llvm-project/build.arm64/lib/clang/13.0.0/include -I /home/mgorny/llvm-project/llvm/../clang/include -I /home/mgorny/llvm-project/build.arm64/tools/clang/include -I /home/mgorny/llvm-project/build.arm64/include -I /home/mgorny/llvm-project/llvm/include -I /sysroot/arm64/usr/include/c++/v1 -I /usr/lib/clang/11.0.1/include -I /sysroot/arm64/usr/include --json-output-path /home/mgorny/llvm-project/build.arm64/tools/clang/lib/Tooling/ASTNodeAPI.json
/bin/sh: /home/mgorny/llvm-project/build.arm64/bin/clang-ast-dump: Exec format error

I guess you can look at TableGens how to correctly generate and use host executables.

I am now running into this error now that clang13 has released. I cannot figure a way around this. I do not see any cmake options similar to the tablegens that allow you to specify the binary for the build system. Am I missing something or is clang13 just broken for cross compilation?

Another confirmation that this change has broken cross compilation. I have tried setting both -DCLANG_TOOLING_BUILD_AST_INTROSPECTION=OFF and -DCMAKE_CROSSCOMPILING=ON separately and in combination. This means that LLVM cannot be cross compiled for AArch64, which is a pretty serious problem!

So I did find a solution. Maybe somewhere else in the forums or on irc. Can't remember for sure.

-DCMAKE_SYSTEM_NAME="Linux"
Add this is well to your cmake invocation. For some reason this triggers in cmake that we are cross-compiling. Why -DCMAKE_CROSSCOMPILING=ON is insufficient I have no idea why. I'm compiling on linux for linux, didn't think I'd need it. Kind of unintuitive, but that should hopefully get you going.

I believe the relevant bit of CMake logic is https://gitlab.kitware.com/cmake/cmake/-/blob/a2e42a577b08dc65ba801e18e5c8be163df83455/Modules/CMakeDetermineSystem.cmake#L136-158. If CMAKE_SYSTEM_NAME isn't set, it'll set CMAKE_CROSSCOMPILING to FALSE internally.

[andrew-wja]

In D93164#3252509, @lancethepants wrote:

In D93164#3250945, @andrew-wja wrote:

In D93164#3048130, @lancethepants wrote:

In D93164#2653067, @mgorny wrote:

This change breaks cross-compilation now, as it tries running an executable built for the target system:

FAILED: tools/clang/lib/Tooling/ASTNodeAPI.json 
cd /home/mgorny/llvm-project/build.arm64/tools/clang/lib/Tooling && /home/mgorny/llvm-project/build.arm64/bin/clang-ast-dump --skip-processing=0 --astheader=/home/mgorny/llvm-project/build.arm64/tools/clang/lib/Tooling/ASTTU.cpp -I /home/mgorny/llvm-project/build.arm64/lib/clang/13.0.0/include -I /home/mgorny/llvm-project/llvm/../clang/include -I /home/mgorny/llvm-project/build.arm64/tools/clang/include -I /home/mgorny/llvm-project/build.arm64/include -I /home/mgorny/llvm-project/llvm/include -I /sysroot/arm64/usr/include/c++/v1 -I /usr/lib/clang/11.0.1/include -I /sysroot/arm64/usr/include --json-output-path /home/mgorny/llvm-project/build.arm64/tools/clang/lib/Tooling/ASTNodeAPI.json
/bin/sh: /home/mgorny/llvm-project/build.arm64/bin/clang-ast-dump: Exec format error

I guess you can look at TableGens how to correctly generate and use host executables.

I am now running into this error now that clang13 has released. I cannot figure a way around this. I do not see any cmake options similar to the tablegens that allow you to specify the binary for the build system. Am I missing something or is clang13 just broken for cross compilation?

Another confirmation that this change has broken cross compilation. I have tried setting both -DCLANG_TOOLING_BUILD_AST_INTROSPECTION=OFF and -DCMAKE_CROSSCOMPILING=ON separately and in combination. This means that LLVM cannot be cross compiled for AArch64, which is a pretty serious problem!

So I did find a solution. Maybe somewhere else in the forums or on irc. Can't remember for sure.

-DCMAKE_SYSTEM_NAME="Linux"
Add this is well to your cmake invocation. For some reason this triggers in cmake that we are cross-compiling. Why -DCMAKE_CROSSCOMPILING=ON is insufficient I have no idea why. I'm compiling on linux for linux, didn't think I'd need it. Kind of unintuitive, but that should hopefully get you going.

@lancethepants @smeenai thanks for the pointers! Unfortunately, it still doesn't work for me after passing -DCMAKE_SYSTEM_NAME="Linux". Passing that option did change the CMake output, so it's definitely recognized at least (the messages about which sanitizer tests will run now call the platform "Linux" as opposed to referring to the platform as default). I am also compiling on Linux for Linux, I'm just trying to build an AArch64 toolchain which I can copy to an SD card for testing on an AArch64 board. @lancethepants could you check your CMake version where you got this to work? I'm using CMake 3.22.1 and trying to build llvmorg-13.0.0

The full invocation I'm using is the following:

cmake -G Ninja \
  -DLLVM_TABLEGEN=${TOP}/${CLANG_TBLGEN_DIR}/bin/llvm-tblgen \
  -DCLANG_TABLEGEN=${TOP}/${CLANG_TBLGEN_DIR}/bin/clang-tblgen \
  -DCMAKE_SYSTEM_NAME="Linux" \
  -DCMAKE_CROSSCOMPILING=ON \
  -DCLANG_TOOLING_BUILD_AST_INTROSPECTION=OFF \
  -DCMAKE_C_COMPILER=aarch64-linux-gnu-gcc \
  -DCMAKE_CXX_COMPILER=aarch64-linux-gnu-g++ \
  -DCMAKE_BUILD_TYPE=MinSizeRel \
  -DCMAKE_INSTALL_PREFIX=${TOP}/${ROOT_DIR} \
  -DLLVM_DEFAULT_TARGET_TRIPLE=aarch64-linux-gnu \
  -DLLVM_NATIVE_ARCH="X86" \
  -DLLVM_TARGET_ARCH="AArch64" \
  -DLLVM_TARGETS_TO_BUILD="AArch64" \
  -DLLVM_ENABLE_PROJECTS="clang;compiler-rt;lld;" \
  -DLLVM_ENABLE_TERMINFO=OFF \
  -DLLVM_ENABLE_BINDINGS=OFF \
  -DLLVM_BUILD_EXAMPLES=OFF \
  -DLLVM_BUILD_TESTS=OFF \
  -DLLVM_BUILD_BENCHMARKS=OFF \
  ${TOP}/${SRC_DIR}/llvm/

[lancethepants]

@lancethepants @smeenai thanks for the pointers! Unfortunately, it still doesn't work for me after passing -DCMAKE_SYSTEM_NAME="Linux". Passing that option did change the CMake output, so it's definitely recognized at least (the messages about which sanitizer tests will run now call the platform "Linux" as opposed to referring to the platform as default). I am also compiling on Linux for Linux, I'm just trying to build an AArch64 toolchain which I can copy to an SD card for testing on an AArch64 board. @lancethepants could you check your CMake version where you got this to work? I'm using CMake 3.22.1 and trying to build llvmorg-13.0.0

The full invocation I'm using is the following:

cmake -G Ninja \
  -DLLVM_TABLEGEN=${TOP}/${CLANG_TBLGEN_DIR}/bin/llvm-tblgen \
  -DCLANG_TABLEGEN=${TOP}/${CLANG_TBLGEN_DIR}/bin/clang-tblgen \
  -DCMAKE_SYSTEM_NAME="Linux" \
  -DCMAKE_CROSSCOMPILING=ON \
  -DCLANG_TOOLING_BUILD_AST_INTROSPECTION=OFF \
  -DCMAKE_C_COMPILER=aarch64-linux-gnu-gcc \
  -DCMAKE_CXX_COMPILER=aarch64-linux-gnu-g++ \
  -DCMAKE_BUILD_TYPE=MinSizeRel \
  -DCMAKE_INSTALL_PREFIX=${TOP}/${ROOT_DIR} \
  -DLLVM_DEFAULT_TARGET_TRIPLE=aarch64-linux-gnu \
  -DLLVM_NATIVE_ARCH="X86" \
  -DLLVM_TARGET_ARCH="AArch64" \
  -DLLVM_TARGETS_TO_BUILD="AArch64" \
  -DLLVM_ENABLE_PROJECTS="clang;compiler-rt;lld;" \
  -DLLVM_ENABLE_TERMINFO=OFF \
  -DLLVM_ENABLE_BINDINGS=OFF \
  -DLLVM_BUILD_EXAMPLES=OFF \
  -DLLVM_BUILD_TESTS=OFF \
  -DLLVM_BUILD_BENCHMARKS=OFF \
  ${TOP}/${SRC_DIR}/llvm/

Are you stil encountering the same issue?
This is my cmake invocation.
https://github.com/lancethepants/tomatoware/blob/master/scripts/buildroot.sh#L808-L843

I typically update cmake to the most recent whenver I get around to updating my project. The latest I've used is 3.21.4. Looks like you're using the very newest, so I doubt it's the difference between my version and your version.

clang/include/clang/Tooling/NodeIntrospection.h

+//===- NodeIntrospection.h ------------------------------------*- C++ -*---===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+//  This file contains the implementation of the NodeIntrospection.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_TOOLING_NODEINTROSPECTION_H
+#define LLVM_CLANG_TOOLING_NODEINTROSPECTION_H
+
+#include "clang/AST/ASTTypeTraits.h"
+#include "clang/AST/DeclarationName.h"
+
+#include <memory>
+#include <set>
+
+namespace clang {
+
+class Stmt;
+
+namespace tooling {
+
+class LocationCall {
+public:
+  enum LocationCallFlags { NoFlags, ReturnsPointer, IsCast };
+  LocationCall(std::shared_ptr<LocationCall> on, std::string name,
+               LocationCallFlags flags = NoFlags)
+      : m_on(on), m_name(name), m_flags(flags) {}
+  LocationCall(std::shared_ptr<LocationCall> on, std::string name,
+               std::vector<std::string> const &args,
+               LocationCallFlags flags = NoFlags)
+      : m_on(on), m_name(name), m_flags(flags) {}
+
+  LocationCall *on() const { return m_on.get(); }
+  StringRef name() const { return m_name; }
+  std::vector<std::string> const &args() const { return m_args; }
+  bool returnsPointer() const { return m_flags & ReturnsPointer; }
+  bool isCast() const { return m_flags & IsCast; }
+
+private:
+  std::shared_ptr<LocationCall> m_on;
+  std::string m_name;
+  std::vector<std::string> m_args;
+  LocationCallFlags m_flags;
+};
+
+class LocationCallFormatterCpp {
+public:
+  static std::string format(LocationCall *Call);
+};
+
+namespace internal {
+struct RangeLessThan {
+  bool operator()(
+      std::pair<SourceRange, std::shared_ptr<LocationCall>> const &LHS,
+      std::pair<SourceRange, std::shared_ptr<LocationCall>> const &RHS) const;
+};
+} // namespace internal
+
+template <typename T, typename U, typename Comp = std::less<std::pair<T, U>>>
+using UniqueMultiMap = std::set<std::pair<T, U>, Comp>;
+
+using SourceLocationMap =
+    UniqueMultiMap<SourceLocation, std::shared_ptr<LocationCall>>;
+using SourceRangeMap =
+    UniqueMultiMap<SourceRange, std::shared_ptr<LocationCall>,
+                   internal::RangeLessThan>;
+
+struct NodeLocationAccessors {
+  SourceLocationMap LocationAccessors;
+  SourceRangeMap RangeAccessors;
+};
+
+namespace NodeIntrospection {
+NodeLocationAccessors GetLocations(clang::Stmt const *Object);
+NodeLocationAccessors GetLocations(clang::DynTypedNode const &Node);
+} // namespace NodeIntrospection
+} // namespace tooling
+} // namespace clang
+#endif

clang/lib/Tooling/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   Option
   FrontendOpenMP
   Support
   )
 
 add_subdirectory(Core)
 add_subdirectory(Inclusions)
 add_subdirectory(Refactoring)
 add_subdirectory(ASTDiff)
+add_subdirectory(DumpTool)
 add_subdirectory(Syntax)
 add_subdirectory(DependencyScanning)
 add_subdirectory(Transformer)
 
+find_package(Python3 COMPONENTS Interpreter)
+
+# Replace the last lib component of the current binary directory with include
+string(FIND ${CMAKE_CURRENT_BINARY_DIR} "/lib/" PATH_LIB_START REVERSE)
+if(PATH_LIB_START EQUAL -1)
+  message(FATAL_ERROR "Couldn't find lib component in binary directory")
+endif()
+math(EXPR PATH_LIB_END "${PATH_LIB_START}+5")
+string(SUBSTRING ${CMAKE_CURRENT_BINARY_DIR} 0 ${PATH_LIB_START} PATH_HEAD)
+string(SUBSTRING ${CMAKE_CURRENT_BINARY_DIR} ${PATH_LIB_END} -1 PATH_TAIL)
+string(CONCAT BINARY_INCLUDE_DIR ${PATH_HEAD} "/include/clang/" ${PATH_TAIL})
+
+if (NOT Python3_EXECUTABLE
+    OR WIN32
+    OR APPLE
+    OR GENERATOR_IS_MULTI_CONFIG
+    OR NOT LLVM_NATIVE_ARCH IN_LIST LLVM_TARGETS_TO_BUILD
+    )
+  file(GENERATE OUTPUT ${BINARY_INCLUDE_DIR}/NodeIntrospection.inc
+    CONTENT "
+namespace clang {
+namespace tooling {
+
+NodeLocationAccessors NodeIntrospection::GetLocations(clang::Stmt const *) {
+  return {};
+}
+NodeLocationAccessors
+NodeIntrospection::GetLocations(clang::DynTypedNode const &) {
+  return {};
+}
+} // namespace tooling
+} // namespace clang
+"
+    )
+else()
+  # The generation of ASTNodeAPI.json takes a long time in a
+  # Debug build due to parsing AST.h. Disable the processing
+  # but setting CLANG_TOOLING_BUILD_AST_INTROSPECTION as an
+  # internal hidden setting to override.
+  # When the processing is disabled, a trivial/empty JSON
+  # file is generated by clang-ast-dump and generate_cxx_src_locs.py
+  # generates the same API, but with a trivial implementation.
+  option(CLANG_TOOLING_BUILD_AST_INTROSPECTION "Enable AST introspection" TRUE)
+
+  set(skip_expensive_processing $<OR:$<CONFIG:Debug>,$<NOT:$<BOOL:${CLANG_TOOLING_BUILD_AST_INTROSPECTION}>>>)
+
+  file(GENERATE OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/ASTTU.cpp
+    CONTENT "
+#include <clang/AST/AST.h>
+")
+
+  add_custom_command(
+      COMMENT Generate ASTNodeAPI.json
+      OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/ASTNodeAPI.json
+      DEPENDS clang-ast-dump clang-resource-headers
+      COMMAND
+      $<TARGET_FILE:clang-ast-dump>
+        # Skip this in debug mode because parsing AST.h is too slow
+        --skip-processing=${skip_expensive_processing}
+        --astheader=${CMAKE_CURRENT_BINARY_DIR}/ASTTU.cpp
+        -I ${CMAKE_BINARY_DIR}/lib/clang/${LLVM_VERSION_MAJOR}.${LLVM_VERSION_MINOR}.${LLVM_VERSION_PATCH}/include
+        -I ${CMAKE_SOURCE_DIR}/../clang/include
+        -I ${CMAKE_BINARY_DIR}/tools/clang/include/
+        -I ${CMAKE_BINARY_DIR}/include
+        -I ${CMAKE_SOURCE_DIR}/include
+        --json-output-path ${CMAKE_CURRENT_BINARY_DIR}/ASTNodeAPI.json
+  )
+
+  add_custom_target(run-ast-api-dump-tool
+      DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/ASTNodeAPI.json
+  )
+
+  add_custom_command(
+      COMMENT Generate NodeIntrospection.inc
+      OUTPUT ${BINARY_INCLUDE_DIR}/NodeIntrospection.inc
+      DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/ASTNodeAPI.json ${CMAKE_CURRENT_SOURCE_DIR}/DumpTool/generate_cxx_src_locs.py
+      COMMAND
+      ${CMAKE_COMMAND} -E make_directory
+        ${CMAKE_CURRENT_BINARY_DIR}/generated/
+      COMMAND
+      ${Python3_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/DumpTool/generate_cxx_src_locs.py
+        --json-input-path ${CMAKE_CURRENT_BINARY_DIR}/ASTNodeAPI.json
+        --output-file NodeIntrospection.inc
+        --empty-implementation ${skip_expensive_processing}
+      COMMAND
+      ${CMAKE_COMMAND} -E copy_if_different
+        ${CMAKE_CURRENT_BINARY_DIR}/NodeIntrospection.inc
+        ${BINARY_INCLUDE_DIR}/NodeIntrospection.inc
+  )
+
+  add_custom_target(run-ast-api-generate-tool
+      DEPENDS
+      ${BINARY_INCLUDE_DIR}/NodeIntrospection.inc
+  )
+endif()
+
 add_clang_library(clangTooling
   AllTUsExecution.cpp
   ArgumentsAdjusters.cpp
   CommonOptionsParser.cpp
   CompilationDatabase.cpp
   Execution.cpp
   ExpandResponseFilesCompilationDatabase.cpp
   FileMatchTrie.cpp
   FixIt.cpp
   GuessTargetAndModeCompilationDatabase.cpp
   InterpolatingCompilationDatabase.cpp
   JSONCompilationDatabase.cpp
   Refactoring.cpp
   RefactoringCallbacks.cpp
   StandaloneExecution.cpp
+  NodeIntrospection.cpp
+  ${BINARY_INCLUDE_DIR}/NodeIntrospection.inc
   Tooling.cpp
 
   DEPENDS
   ClangDriverOptions
   omp_gen
 
   LINK_LIBS
   clangAST

clang/lib/Tooling/DumpTool/APIData.h

+//===- APIData.h ---------------------------------------------*- C++ -*----===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_LIB_TOOLING_DUMPTOOL_APIDATA_H
+#define LLVM_CLANG_LIB_TOOLING_DUMPTOOL_APIDATA_H
+
+#include <string>
+#include <vector>
+
+namespace clang {
+namespace tooling {
+
+struct ClassData {
+
+  bool isEmpty() const {
+    return ASTClassLocations.empty() && ASTClassRanges.empty();
+  }
+
+  std::vector<std::string> ASTClassLocations;
+  std::vector<std::string> ASTClassRanges;
+  // TODO: Extend this with locations available via typelocs etc.
+};
+
+} // namespace tooling
+} // namespace clang
+
+#endif

clang/lib/Tooling/DumpTool/ASTSrcLocProcessor.h

+//===- ASTSrcLocProcessor.h ---------------------------------*- C++ -*-----===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_TOOLING_DUMPTOOL_ASTSRCLOCPROCESSOR_H
+#define LLVM_CLANG_TOOLING_DUMPTOOL_ASTSRCLOCPROCESSOR_H
+
+#include "APIData.h"
+#include "clang/ASTMatchers/ASTMatchFinder.h"
+#include "llvm/ADT/StringRef.h"
+#include <memory>
+#include <string>
+#include <vector>
+
+namespace clang {
+
+class CompilerInstance;
+
+namespace tooling {
+
+class ASTSrcLocProcessor : public ast_matchers::MatchFinder::MatchCallback {
+public:
+  explicit ASTSrcLocProcessor(StringRef JsonPath);
+
+  std::unique_ptr<ASTConsumer> createASTConsumer(CompilerInstance &Compiler,
+                                                 StringRef File);
+
+  void generate();
+
+private:
+  void run(const ast_matchers::MatchFinder::MatchResult &Result) override;
+
+  llvm::StringMap<StringRef> ClassInheritance;
+  llvm::StringMap<std::vector<StringRef>> ClassesInClade;
+  llvm::StringMap<ClassData> ClassEntries;
+
+  std::string JsonPath;
+  std::unique_ptr<clang::ast_matchers::MatchFinder> Finder;
+};
+
+} // namespace tooling
+} // namespace clang
+
+#endif

clang/lib/Tooling/DumpTool/ASTSrcLocProcessor.cpp

+//===- ASTSrcLocProcessor.cpp --------------------------------*- C++ -*----===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "ASTSrcLocProcessor.h"
+
+#include "clang/Frontend/CompilerInstance.h"
+#include "llvm/Support/JSON.h"
+
+using namespace clang::tooling;
+using namespace llvm;
+using namespace clang::ast_matchers;
+
+ASTSrcLocProcessor::ASTSrcLocProcessor(StringRef JsonPath)
+    : JsonPath(JsonPath) {
+
+  MatchFinder::MatchFinderOptions FinderOptions;
+
+  Finder = std::make_unique<MatchFinder>(std::move(FinderOptions));
+  Finder->addMatcher(
+      cxxRecordDecl(
+          isDefinition(),
+          isSameOrDerivedFrom(
+              // TODO: Extend this with other clades
+              namedDecl(hasName("clang::Stmt")).bind("nodeClade")),
+          optionally(isDerivedFrom(cxxRecordDecl().bind("derivedFrom"))))
+          .bind("className"),
+      this);
+}
+
+std::unique_ptr<clang::ASTConsumer>
+ASTSrcLocProcessor::createASTConsumer(clang::CompilerInstance &Compiler,
+                                      StringRef File) {
+  return Finder->newASTConsumer();
+}
+
+llvm::json::Object toJSON(llvm::StringMap<std::vector<StringRef>> const &Obj) {
+  using llvm::json::toJSON;
+
+  llvm::json::Object JsonObj;
+  for (const auto &Item : Obj) {
+    JsonObj[Item.first()] = Item.second;
+  }
+  return JsonObj;
+}
+
+llvm::json::Object toJSON(llvm::StringMap<StringRef> const &Obj) {
+  using llvm::json::toJSON;
+
+  llvm::json::Object JsonObj;
+  for (const auto &Item : Obj) {
+    JsonObj[Item.first()] = Item.second;
+  }
+  return JsonObj;
+}
+
+llvm::json::Object toJSON(ClassData const &Obj) {
+  llvm::json::Object JsonObj;
+
+  if (!Obj.ASTClassLocations.empty())
+    JsonObj["sourceLocations"] = Obj.ASTClassLocations;
+  if (!Obj.ASTClassRanges.empty())
+    JsonObj["sourceRanges"] = Obj.ASTClassRanges;
+  return JsonObj;
+}
+
+llvm::json::Object toJSON(llvm::StringMap<ClassData> const &Obj) {
+  using llvm::json::toJSON;
+
+  llvm::json::Object JsonObj;
+  for (const auto &Item : Obj) {
+    if (!Item.second.isEmpty())
+      JsonObj[Item.first()] = ::toJSON(Item.second);
+  }
+  return JsonObj;
+}
+
+void WriteJSON(std::string JsonPath,
+               llvm::StringMap<StringRef> const &ClassInheritance,
+               llvm::StringMap<std::vector<StringRef>> const &ClassesInClade,
+               llvm::StringMap<ClassData> const &ClassEntries) {
+  llvm::json::Object JsonObj;
+
+  using llvm::json::toJSON;
+
+  JsonObj["classInheritance"] = ::toJSON(ClassInheritance);
+  JsonObj["classesInClade"] = ::toJSON(ClassesInClade);
+  JsonObj["classEntries"] = ::toJSON(ClassEntries);
+
+  std::error_code EC;
+  llvm::raw_fd_ostream JsonOut(JsonPath, EC, llvm::sys::fs::F_Text);
+  if (EC)
+    return;
+
+  llvm::json::Value JsonVal(std::move(JsonObj));
+  JsonOut << formatv("{0:2}", JsonVal);
+}
+
+void ASTSrcLocProcessor::generate() {
+  WriteJSON(JsonPath, ClassInheritance, ClassesInClade, ClassEntries);
+}
+
+std::vector<std::string>
+CaptureMethods(std::string TypeString, const clang::CXXRecordDecl *ASTClass,
+               const MatchFinder::MatchResult &Result) {
+
+  auto publicAccessor = [](auto... InnerMatcher) {
+    return cxxMethodDecl(isPublic(), parameterCountIs(0), isConst(),
+                         InnerMatcher...);
+  };
+
+  auto BoundNodesVec =
+      match(findAll(publicAccessor(ofClass(equalsNode(ASTClass)),
+                                   returns(asString(TypeString)))
+                        .bind("classMethod")),
+            *ASTClass, *Result.Context);
+
+  std::vector<std::string> Methods;
+  for (const auto &BN : BoundNodesVec) {
+    if (const auto *Node = BN.getNodeAs<clang::NamedDecl>("classMethod")) {
+      // Only record the getBeginLoc etc on Stmt etc, because it will call
+      // more-derived implementations pseudo-virtually.
+      if ((ASTClass->getName() != "Stmt" && ASTClass->getName() != "Decl") &&
+          (Node->getName() == "getBeginLoc" || Node->getName() == "getEndLoc" ||
+           Node->getName() == "getSourceRange")) {
+        continue;
+      }
+      // Only record the getExprLoc on Expr, because it will call
+      // more-derived implementations pseudo-virtually.
+      if (ASTClass->getName() != "Expr" && Node->getName() == "getExprLoc") {
+        continue;
+      }
+      Methods.push_back(Node->getName().str());
+    }
+  }
+  return Methods;
+}
+
+void ASTSrcLocProcessor::run(const MatchFinder::MatchResult &Result) {
+
+  if (const auto *ASTClass =
+          Result.Nodes.getNodeAs<clang::CXXRecordDecl>("className")) {
+
+    StringRef ClassName = ASTClass->getName();
+
+    ClassData CD;
+
+    const auto *NodeClade =
+        Result.Nodes.getNodeAs<clang::CXXRecordDecl>("nodeClade");
+    StringRef CladeName = NodeClade->getName();
+
+    if (const auto *DerivedFrom =
+            Result.Nodes.getNodeAs<clang::CXXRecordDecl>("derivedFrom"))
+      ClassInheritance[ClassName] = DerivedFrom->getName();
+
+    CD.ASTClassLocations =
+        CaptureMethods("class clang::SourceLocation", ASTClass, Result);
+    CD.ASTClassRanges =
+        CaptureMethods("class clang::SourceRange", ASTClass, Result);
+
+    if (!CD.isEmpty()) {
+      ClassEntries[ClassName] = CD;
+      ClassesInClade[CladeName].push_back(ClassName);
+    }
+  }
+}

clang/lib/Tooling/DumpTool/CMakeLists.txt

+
+add_clang_executable(clang-ast-dump
+  ASTSrcLocProcessor.cpp
+  ClangSrcLocDump.cpp
+)
+
+target_link_libraries(clang-ast-dump
+  PRIVATE
+  clangAST
+  clangASTMatchers
+  clangBasic
+  clangDriver
+  clangFrontend
+  clangSerialization
+  clangToolingCore
+)

clang/lib/Tooling/DumpTool/ClangSrcLocDump.cpp

+//===- ClangSrcLocDump.cpp ------------------------------------*- C++ -*---===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Basic/Diagnostic.h"
+#include "clang/Driver/Compilation.h"
+#include "clang/Driver/Driver.h"
+#include "clang/Driver/Job.h"
+#include "clang/Driver/Options.h"
+#include "clang/Driver/Tool.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Frontend/TextDiagnosticPrinter.h"
+#include "clang/Tooling/Tooling.h"
+#include "llvm/Option/ArgList.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/Host.h"
+#include "llvm/Support/JSON.h"
+
+#include "ASTSrcLocProcessor.h"
+
+using namespace clang::tooling;
+using namespace clang;
+using namespace llvm;
+
+static cl::list<std::string> IncludeDirectories(
+    "I", cl::desc("Include directories to use while compiling"),
+    cl::value_desc("directory"), cl::Required, cl::OneOrMore, cl::Prefix);
+
+static cl::opt<std::string>
+    AstHeaderFile("astheader", cl::desc("AST header to parse API from"),
+                  cl::Required, cl::value_desc("AST header file"));
+
+static cl::opt<bool>
+    SkipProcessing("skip-processing",
+                   cl::desc("Avoid processing the AST header file"),
+                   cl::Required, cl::value_desc("bool"));
+
+static cl::opt<std::string> JsonOutputPath("json-output-path",
+                                           cl::desc("json output path"),
+                                           cl::Required,
+                                           cl::value_desc("path"));
+
+class ASTSrcLocGenerationAction : public clang::ASTFrontendAction {
+public:
+  ASTSrcLocGenerationAction() : Processor(JsonOutputPath) {}
+
+  ~ASTSrcLocGenerationAction() { Processor.generate(); }
+
+  std::unique_ptr<clang::ASTConsumer>
+  CreateASTConsumer(clang::CompilerInstance &Compiler,
+                    llvm::StringRef File) override {
+    return Processor.createASTConsumer(Compiler, File);
+  }
+
+private:
+  ASTSrcLocProcessor Processor;
+};
+
+int main(int argc, const char **argv) {
+
+  cl::ParseCommandLineOptions(argc, argv);
+
+  if (SkipProcessing) {
+    std::error_code EC;
+    llvm::raw_fd_ostream JsonOut(JsonOutputPath, EC, llvm::sys::fs::F_Text);
+    if (EC)
+      return 1;
+    JsonOut << formatv("{0:2}", llvm::json::Value(llvm::json::Object()));
+    return 0;
+  }
+
+  std::vector<std::string> Args;
+  Args.push_back("-cc1");
+
+  llvm::transform(IncludeDirectories, std::back_inserter(Args),
+                  [](const std::string &IncDir) { return "-I" + IncDir; });
+
+  Args.push_back("-fsyntax-only");
+  Args.push_back(AstHeaderFile);
+
+  std::vector<const char *> Argv(Args.size(), nullptr);
+  llvm::transform(Args, Argv.begin(),
+                  [](const std::string &Arg) { return Arg.c_str(); });
+
+  IntrusiveRefCntPtr<DiagnosticOptions> DiagOpts = new DiagnosticOptions();
+  unsigned MissingArgIndex, MissingArgCount;
+  auto Opts = driver::getDriverOptTable();
+  auto ParsedArgs = Opts.ParseArgs(llvm::makeArrayRef(Argv).slice(1),
+                                   MissingArgIndex, MissingArgCount);
+  ParseDiagnosticArgs(*DiagOpts, ParsedArgs);
+  TextDiagnosticPrinter DiagnosticPrinter(llvm::errs(), &*DiagOpts);
+  DiagnosticsEngine Diagnostics(
+      IntrusiveRefCntPtr<DiagnosticIDs>(new DiagnosticIDs()), &*DiagOpts,
+      &DiagnosticPrinter, false);
+
+  FileManager Files(FileSystemOptions(), vfs::getRealFileSystem());
+
+  auto Driver = std::make_unique<driver::Driver>(
+      "clang", llvm::sys::getDefaultTargetTriple(), Diagnostics,
+      "ast-api-dump-tool", &Files.getVirtualFileSystem());
+
+  auto Comp = Driver->BuildCompilation(llvm::makeArrayRef(Argv));
+  if (!Comp)
+    return 1;
+
+  const auto &Jobs = Comp->getJobs();
+  if (Jobs.size() != 1 || !isa<driver::Command>(*Jobs.begin())) {
+    SmallString<256> error_msg;
+    llvm::raw_svector_ostream error_stream(error_msg);
+    Jobs.Print(error_stream, "; ", true);
+    return 1;
+  }
+
+  const auto &Cmd = cast<driver::Command>(*Jobs.begin());
+  const llvm::opt::ArgStringList &CC1Args = Cmd.getArguments();
+
+  auto Invocation = std::make_unique<CompilerInvocation>();
+  CompilerInvocation::CreateFromArgs(*Invocation, CC1Args, Diagnostics);
+
+  CompilerInstance Compiler(std::make_shared<clang::PCHContainerOperations>());
+  Compiler.setInvocation(std::move(Invocation));
+
+  Compiler.createDiagnostics(&DiagnosticPrinter, false);
+  if (!Compiler.hasDiagnostics())
+    return 1;
+
+  Compiler.createSourceManager(Files);
+
+  ASTSrcLocGenerationAction ScopedToolAction;
+  Compiler.ExecuteAction(ScopedToolAction);
+
+  Files.clearStatCache();
+
+  return 0;
+}

clang/lib/Tooling/DumpTool/generate_cxx_src_locs.py

+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+
+import os
+import sys
+import json
+
+import argparse
+
+class Generator(object):
+
+    implementationContent = ''
+
+    def GeneratePrologue(self):
+
+        self.implementationContent += \
+            """
+/*===- Generated file -------------------------------------------*- C++ -*-===*\
+|*                                                                            *|
+|* Introspection of available AST node SourceLocations                        *|
+|*                                                                            *|
+|* Automatically generated file, do not edit!                                 *|
+|*                                                                            *|
+\*===----------------------------------------------------------------------===*/
+
+namespace clang {
+namespace tooling {
+
+using LocationAndString = SourceLocationMap::value_type;
+using RangeAndString = SourceRangeMap::value_type;
+"""
+
+    def GenerateBaseGetLocationsDeclaration(self, CladeName):
+        self.implementationContent += \
+            """
+void GetLocationsImpl(std::shared_ptr<LocationCall> const& Prefix,
+    clang::{0} const *Object, SourceLocationMap &Locs,
+    SourceRangeMap &Rngs);
+""".format(CladeName)
+
+    def GenerateSrcLocMethod(self, ClassName, ClassData):
+
+        self.implementationContent += \
+            """
+static void GetLocations{0}(std::shared_ptr<LocationCall> const& Prefix,
+    clang::{0} const &Object,
+    SourceLocationMap &Locs, SourceRangeMap &Rngs)
+{{
+""".format(ClassName)
+
+        if 'sourceLocations' in ClassData:
+            for locName in ClassData['sourceLocations']:
+                self.implementationContent += \
+                    """
+  Locs.insert(LocationAndString(Object.{0}(),
+    std::make_shared<LocationCall>(Prefix, "{0}")));
+""".format(locName)
+
+            self.implementationContent += '\n'
+
+        if 'sourceRanges' in ClassData:
+            for rngName in ClassData['sourceRanges']:
+                self.implementationContent += \
+                    """
+  Rngs.insert(RangeAndString(Object.{0}(),
+    std::make_shared<LocationCall>(Prefix, "{0}")));
+""".format(rngName)
+
+            self.implementationContent += '\n'
+
+        self.implementationContent += '}\n'
+
+    def GenerateFiles(self, OutputFile):
+        with open(os.path.join(os.getcwd(),
+                  OutputFile), 'w') as f:
+            f.write(self.implementationContent)
+
+    def GenerateBaseGetLocationsFunction(self, ASTClassNames, CladeName):
+
+        MethodReturnType = 'NodeLocationAccessors'
+
+        Signature = \
+            'GetLocations(clang::{0} const *Object)'.format(CladeName)
+        ImplSignature = \
+            """
+GetLocationsImpl(std::shared_ptr<LocationCall> const& Prefix,
+    clang::{0} const *Object, SourceLocationMap &Locs,
+    SourceRangeMap &Rngs)
+""".format(CladeName)
+
+        self.implementationContent += \
+            'void {0} {{ GetLocations{1}(Prefix, *Object, Locs, Rngs);'.format(
+                ImplSignature,
+                CladeName)
+
+        for ASTClassName in ASTClassNames:
+            if ASTClassName != CladeName:
+                self.implementationContent += \
+                    """
+if (auto Derived = llvm::dyn_cast<clang::{0}>(Object)) {{
+  GetLocations{0}(Prefix, *Derived, Locs, Rngs);
+}}
+""".format(ASTClassName)
+
+        self.implementationContent += '}'
+
+        self.implementationContent += \
+            """
+{0} NodeIntrospection::{1} {{
+  NodeLocationAccessors Result;
+  std::shared_ptr<LocationCall> Prefix;
+
+  GetLocationsImpl(Prefix, Object, Result.LocationAccessors,
+                   Result.RangeAccessors);
+""".format(MethodReturnType,
+                Signature)
+
+        self.implementationContent += 'return Result; }'
+
+    def GenerateDynNodeVisitor(self, CladeNames):
+        MethodReturnType = 'NodeLocationAccessors'
+
+        Signature = \
+            'GetLocations(clang::DynTypedNode const &Node)'
+
+        self.implementationContent += MethodReturnType \
+            + ' NodeIntrospection::' + Signature + '{'
+
+        for CladeName in CladeNames:
+            self.implementationContent += \
+                """
+    if (const auto *N = Node.get<{0}>())
+      return GetLocations(const_cast<{0} *>(N));""".format(CladeName)
+
+        self.implementationContent += '\nreturn {}; }'
+
+    def GenerateEpilogue(self):
+
+        self.implementationContent += '''
+  }
+}
+'''
+
+def main():
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--json-input-path',
+                      help='Read API description from FILE', metavar='FILE')
+    parser.add_argument('--output-file', help='Generate output in FILEPATH',
+                      metavar='FILEPATH')
+    parser.add_argument('--empty-implementation',
+                      help='Generate empty implementation',
+                      action="store", type=int)
+
+    options = parser.parse_args()
+
+    if options.empty_implementation:
+        with open(os.path.join(os.getcwd(),
+                  options.output_file), 'w') as f:
+            f.write("""
+namespace clang {
+namespace tooling {
+
+NodeLocationAccessors NodeIntrospection::GetLocations(clang::Stmt const *) {
+  return {};
+}
+NodeLocationAccessors
+NodeIntrospection::GetLocations(clang::DynTypedNode const &) {
+  return {};
+}
+} // namespace tooling
+} // namespace clang
+    """)
+        sys.exit(0)
+
+    with open(options.json_input_path) as f:
+        jsonData = json.load(f)
+
+    g = Generator()
+
+    g.GeneratePrologue()
+
+    if 'classesInClade' in jsonData:
+        for (CladeName, ClassNameData) in jsonData['classesInClade'].items():
+            g.GenerateBaseGetLocationsDeclaration(CladeName)
+
+        for (ClassName, ClassAccessors) in jsonData['classEntries'].items():
+            if ClassAccessors:
+                g.GenerateSrcLocMethod(ClassName, ClassAccessors)
+
+        for (CladeName, ClassNameData) in jsonData['classesInClade'].items():
+            g.GenerateBaseGetLocationsFunction(ClassNameData, CladeName)
+
+        g.GenerateDynNodeVisitor(jsonData['classesInClade'].keys())
+
+    g.GenerateEpilogue()
+
+    g.GenerateFiles(options.output_file)
+
+if __name__ == '__main__':
+    main()

clang/lib/Tooling/NodeIntrospection.cpp

+//===- NodeIntrospection.h -----------------------------------*- C++ -*----===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+//  This file contains the implementation of the NodeIntrospection.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Tooling/NodeIntrospection.h"
+
+#include "clang/AST/AST.h"
+
+namespace clang {
+
+namespace tooling {
+
+std::string LocationCallFormatterCpp::format(LocationCall *Call) {
+  SmallVector<LocationCall *> vec;
+  while (Call) {
+    vec.push_back(Call);
+    Call = Call->on();
+  }
+  std::string result;
+  for (auto *VecCall : llvm::reverse(llvm::makeArrayRef(vec).drop_front())) {
+    result +=
+        (VecCall->name() + "()" + (VecCall->returnsPointer() ? "->" : "."))
+            .str();
+  }
+  result += (vec.back()->name() + "()").str();
+  return result;
+}
+
+namespace internal {
+bool RangeLessThan::operator()(
+    std::pair<SourceRange, std::shared_ptr<LocationCall>> const &LHS,
+    std::pair<SourceRange, std::shared_ptr<LocationCall>> const &RHS) const {
+  if (!LHS.first.isValid() || !RHS.first.isValid())
+    return false;
+
+  if (LHS.first.getBegin() < RHS.first.getBegin())
+    return true;
+  else if (LHS.first.getBegin() != RHS.first.getBegin())
+    return false;
+
+  if (LHS.first.getEnd() < RHS.first.getEnd())
+    return true;
+  else if (LHS.first.getEnd() != RHS.first.getEnd())
+    return false;
+
+  return LHS.second->name() < RHS.second->name();
+}
+} // namespace internal
+
+} // namespace tooling
+} // namespace clang
+
+#include "clang/Tooling/NodeIntrospection.inc"

clang/unittests/CMakeLists.txt

 if(CLANG_ENABLE_STATIC_ANALYZER)
   add_subdirectory(Analysis)
   add_subdirectory(StaticAnalyzer)
 endif()
 add_subdirectory(ASTMatchers)
 add_subdirectory(AST)
 add_subdirectory(CrossTU)
 add_subdirectory(Tooling)
+add_subdirectory(Introspection)
 add_subdirectory(Format)
 add_subdirectory(Frontend)
 add_subdirectory(Rewrite)
 add_subdirectory(Sema)
 add_subdirectory(CodeGen)
 # FIXME: libclang unit tests are disabled on Windows due
 # to failures, mostly in libclang.VirtualFileOverlay_*.
 if(NOT WIN32 AND CLANG_TOOL_LIBCLANG_BUILD) 
   add_subdirectory(libclang)
 endif()
 add_subdirectory(DirectoryWatcher)
 add_subdirectory(Rename)
 add_subdirectory(Index)
 add_subdirectory(Serialization)

clang/unittests/Introspection/CMakeLists.txt

+set(LLVM_LINK_COMPONENTS
+  FrontendOpenMP
+  Support
+  )
+
+add_clang_unittest(IntrospectionTests
+  IntrospectionTest.cpp
+  )
+
+clang_target_link_libraries(IntrospectionTests
+  PRIVATE
+  clangAST
+  clangASTMatchers
+  clangTooling
+  clangBasic
+  clangSerialization
+  clangFrontend
+  )
+target_compile_definitions(IntrospectionTests PRIVATE
+  SKIP_INTROSPECTION_GENERATION=$<OR:$<CONFIG:Debug>,$<NOT:$<BOOL:${CLANG_TOOLING_BUILD_AST_INTROSPECTION}>>>
+)
+target_link_libraries(IntrospectionTests
+  PRIVATE
+  LLVMTestingSupport
+)

clang/unittests/Introspection/IntrospectionTest.cpp

+//===- unittest/Introspection/IntrospectionTest.cpp ----------*- C++ -*---===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Tests for AST location API introspection.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/AST/ASTContext.h"
+#include "clang/ASTMatchers/ASTMatchFinder.h"
+#include "clang/ASTMatchers/ASTMatchers.h"
+#include "clang/Tooling/NodeIntrospection.h"
+#include "clang/Tooling/Tooling.h"
+#include "gmock/gmock-matchers.h"
+#include "gmock/gmock.h"
+#include "gtest/gtest.h"
+
+using namespace clang;
+using namespace clang::ast_matchers;
+using namespace clang::tooling;
+
+using ::testing::UnorderedElementsAre;
+using ::testing::Pair;
+
+#if SKIP_INTROSPECTION_GENERATION
+
+TEST(Introspection, NonFatalAPI) {
+  auto AST = buildASTFromCode("void foo() {} void bar() { foo(); }", "foo.cpp",
+                              std::make_shared<PCHContainerOperations>());
+  auto &Ctx = AST->getASTContext();
+  auto &TU = *Ctx.getTranslationUnitDecl();
+
+  auto BoundNodes = ast_matchers::match(
+      decl(hasDescendant(
+          callExpr(callee(functionDecl(hasName("foo")))).bind("fooCall"))),
+      TU, Ctx);
+
+  EXPECT_EQ(BoundNodes.size(), 1u);
+
+  auto *FooCall = BoundNodes[0].getNodeAs<CallExpr>("fooCall");
+
+  auto result = NodeIntrospection::GetLocations(FooCall);
+
+  EXPECT_EQ(result.LocationAccessors.size(), 0u);
+  EXPECT_EQ(result.RangeAccessors.size(), 0u);
+}
+
+#else
+
+TEST(Introspection, SourceLocations) {
+  auto AST = buildASTFromCode("void foo() {} void bar() { foo(); }", "foo.cpp",
+                              std::make_shared<PCHContainerOperations>());
+  auto &Ctx = AST->getASTContext();
+  auto &TU = *Ctx.getTranslationUnitDecl();
+
+  auto BoundNodes = ast_matchers::match(
+      decl(hasDescendant(
+          callExpr(callee(functionDecl(hasName("foo")))).bind("fooCall"))),
+      TU, Ctx);
+
+  EXPECT_EQ(BoundNodes.size(), 1u);
+
+  auto *FooCall = BoundNodes[0].getNodeAs<CallExpr>("fooCall");
+
+  auto result = NodeIntrospection::GetLocations(FooCall);
+
+  std::map<std::string, SourceLocation> ExpectedLocations;
+  llvm::transform(result.LocationAccessors,
+                  std::inserter(ExpectedLocations, ExpectedLocations.end()),
+                  [](const auto &Accessor) {
+                    return std::make_pair(
+                        LocationCallFormatterCpp::format(Accessor.second.get()),
+                        Accessor.first);
+                  });
+
+  EXPECT_THAT(ExpectedLocations,
+              UnorderedElementsAre(
+                  Pair("getBeginLoc()", FooCall->getBeginLoc()),
+                  Pair("getEndLoc()", FooCall->getEndLoc()),
+                  Pair("getExprLoc()", FooCall->getExprLoc()),
+                  Pair("getRParenLoc()", FooCall->getRParenLoc())));
+
+  std::map<std::string, SourceRange> ExpectedRanges;
+  llvm::transform(result.RangeAccessors,
+                  std::inserter(ExpectedRanges, ExpectedRanges.end()),
+                  [](const auto &Accessor) {
+                    return std::make_pair(
+                        LocationCallFormatterCpp::format(Accessor.second.get()),
+                        Accessor.first);
+                  });
+
+  EXPECT_THAT(ExpectedRanges,
+              UnorderedElementsAre(
+                  Pair("getSourceRange()", FooCall->getSourceRange())));
+}
+#endif

llvm/utils/gn/secondary/clang/lib/Tooling/BUILD.gn

+# FIXME: The cmake build runs DumpTool:clang-ast-dump to generate a json
+# file and feeds it into this step in non-debug builds or if an option is set.
+action("node_introspection_inc") {
+  script = "DumpTool/generate_cxx_src_locs.py"
+  outputs = [ "$target_gen_dir/clang/Tooling/NodeIntrospection.inc" ]
+  args = [
+    "--empty-implementation=1",
+    "--output-file=" + rebase_path(outputs[0], root_build_dir),
+  ]
+}
+
 static_library("Tooling") {
   output_name = "clangTooling"
   configs += [ "//llvm/utils/gn/build:clang_code" ]
   deps = [
+    ":node_introspection_inc",
     "//clang/include/clang/Driver:Options",
     "//clang/lib/AST",
     "//clang/lib/ASTMatchers",
     "//clang/lib/Basic",
     "//clang/lib/Driver",
     "//clang/lib/Format",
     "//clang/lib/Frontend",
     "//clang/lib/Lex",
     "//clang/lib/Rewrite",
     "//clang/lib/Tooling/Core",
   ]
+  include_dirs = [ target_gen_dir ]
   sources = [
     "AllTUsExecution.cpp",
     "ArgumentsAdjusters.cpp",
     "CommonOptionsParser.cpp",
     "CompilationDatabase.cpp",
     "Execution.cpp",
     "ExpandResponseFilesCompilationDatabase.cpp",
     "FileMatchTrie.cpp",
     "FixIt.cpp",
     "GuessTargetAndModeCompilationDatabase.cpp",
     "InterpolatingCompilationDatabase.cpp",
     "JSONCompilationDatabase.cpp",
+    "NodeIntrospection.cpp",
     "Refactoring.cpp",
     "RefactoringCallbacks.cpp",
     "StandaloneExecution.cpp",
     "Tooling.cpp",
   ]
 }

llvm/utils/gn/secondary/clang/lib/Tooling/DumpTool/BUILD.gn

+executable("clang-ast-dump") {
+  configs += [ "//llvm/utils/gn/build:clang_code" ]
+  deps = [
+    "//clang/lib/AST",
+    "//clang/lib/ASTMatchers",
+    "//clang/lib/Basic",
+    "//clang/lib/Driver",
+    "//clang/lib/Format",
+    "//clang/lib/Frontend",
+    "//clang/lib/Lex",
+    "//clang/lib/Rewrite",
+    "//clang/lib/Serialization",
+    "//clang/lib/Tooling/Core",
+  ]
+
+  sources = [
+    "ASTSrcLocProcessor.cpp",
+    "ClangSrcLocDump.cpp",
+  ]
+}

llvm/utils/gn/secondary/clang/unittests/BUILD.gn

 import("//clang/lib/StaticAnalyzer/Frontend/enable.gni")
 
 group("unittests") {
   deps = [
     "AST:ASTTests",
     "ASTMatchers:ASTMatchersTests",
     "ASTMatchers/Dynamic:DynamicASTMatchersTests",
     "Basic:BasicTests",
     "CodeGen:ClangCodeGenTests",
     "CrossTU:CrossTUTests",
     "Driver:ClangDriverTests",
     "Format:FormatTests",
     "Frontend:FrontendTests",
     "Index:IndexTests",
+    "Introspection:IntrospectionTests",
     "Lex:LexTests",
     "Rename:ClangRenameTests",
     "Rewrite:RewriteTests",
     "Sema:SemaTests",
     "Serialization:SerializationTests",
     "Tooling:ToolingTests",
     "Tooling/Syntax:SyntaxTests",
   ]

llvm/utils/gn/secondary/clang/unittests/Introspection/BUILD.gn

+import("//llvm/utils/unittest/unittest.gni")
+
+unittest("IntrospectionTests") {
+  configs += [ "//llvm/utils/gn/build:clang_code" ]
+  deps = [
+    "//clang/lib/AST",
+    "//clang/lib/ASTMatchers",
+    "//clang/lib/Basic",
+    "//clang/lib/Frontend",
+    "//clang/lib/Serialization",
+    "//clang/lib/Tooling",
+    "//llvm/lib/Support",
+    "//llvm/lib/Testing/Support",
+  ]
+
+
+  defines = [ "SKIP_INTROSPECTION_GENERATION" ]
+
+  sources = [ "IntrospectionTest.cpp" ]
+}