[analyzer] On-demand parsing capability for CTU

[gamesh411]

Summary

Add an option to enable on-demand parsing of needed ASTs during CTU analysis.
Two options are introduced. CTUOnDemandParsing enables the feature, and
CTUOnDemandParsingDatabase specifies the path to a compilation database, which
has all the necessary information to generate the ASTs.

[martong]

clang/lib/CrossTU/CrossTranslationUnit.cpp

407

CI.getPCHContainerOperations() is probably not needed, because we are not going to exercise the ASTReader, so this could be a nullptr (default param value).

[martong]

clang/lib/StaticAnalyzer/Core/CallEvent.cpp

578 ↗

(On Diff #248408)

left here some debugging

[balazske]

clang/include/clang/CrossTU/CrossTranslationUnit.h

231

class would look better here? (The descendants are class too.)

234

Probably it is good to mention in the documentation that the function is used with a string read from a file and the type of the file determines the meaning of the Identifier here (the calling code does have no direct knowledge about it).

clang/lib/CrossTU/CrossTranslationUnit.cpp

454

matches

493

Members of ASTOnDemandLoader would be better in a single group in the source file.

[martong]

clang/lib/CrossTU/CrossTranslationUnit.cpp

364

Could we check somehow if CTUDir is indeed an absolute path? If not then probably we should bail out with an error. Though I am not sure if there is an easy and portable way in llvm:: to check for beeing an absolute path.

369

There is no variable named as OnDemandParsingCommands, perhaps you made a rename but forgot to rename in the comments?

378

There is no variable named as ASTSourcePath, perhaps you made a rename but forgot to rename in the comments?

407

If CI.getPCHContainerOperations() is not needed then we can remove the CI member of ASTOnDemandLoader.

437

Perhaps we should make sure that Identifier is indeed an absolute path and if not then we should emit an error. If a user do not use CodeChecker to generate the ExternalDefMapping it may contain relative paths. See e.g.: https://clang.llvm.org/docs/analyzer/user-docs/CrossTranslationUnit.html#manual-ctu-analysis

451

Could we have a lit test for this case? I.e having a compile_commands.json with two ambiguous entries for the same file and then we should expect the compiler diag?
(Note, I am asking this because it is not immediate for me from the code that this will ever fire.)

clang/test/Analysis/ctu-main.c

53

Is this hunk related?

clang/test/Analysis/ctu-on-demand-parsing.c

3

Why do we need sed here? I don't see any \ in the compile commands json. Is it because of Windows builds? Could you please explain in a comment maybe in the previous line.

[gamesh411]

Refactored the test
Copying the compile_commands.json in case of test for C files
results in the extdef mapping tool finding the correct compiler
flags for the C to be imported.

[gamesh411]

In D75665#1907914, @martong wrote:

The warning below suggests that we parse the ctu-other.c file (and presumably every file) as a C++ file, even if it should be parsed as a C file. Perhaps the invocation of the parser is missing some setting? Also, could this be the reason why on-the-fly and pch driven ctu gives different statistics?

warning: treating 'c' input as 'c++' when in C++ mode, this behavior is deprecated [-Wdeprecated]
/mnt/disks/ssd0/agent/workspace/amd64_debian_testing_clang8/clang/test/Analysis/Inputs/ctu-other.c:47:26: error: 'DataType' cannot be defined in a parameter type
int structInProto(struct DataType {int a;int b; } * d) {
                         ^
1 error generated.
Error while processing /mnt/disks/ssd0/agent/workspace/amd64_debian_testing_clang8/clang/test/Analysis/Inputs/ctu-other.c.

I have investigated the issue, and the compiler flags were looked up using the heuristic implemented in tooling.
This heuristic looks for the suitable compilation database in an upward ascending fashion inside the directory tree startin from the input source file.
By copying both the source file and the compilation database to the test directory this heuristic does the right thing now. (up until now the found compile_commands.json was the one used for llvm-project itself, and picked up a c++ specific compilation).

This issue is solved, however the ASTImporter still cannot import the inline definition of the struct, and emits an error.
Right now I am debugging the master branch that uses the AST-dumps wheter I also encounter this error inside the ASTImporter (just to see if I am operating on sane assumptions).

[gamesh411]

In D75665#1907938, @martong wrote:

As the CTUDir string prefix is only needed in case of the old approach, I have refactored the external definition mapping storage to NOT include that.

Both ctu-main.c and ctu-on-demand-parsing.c use the -analyzer-config ctu-dir=%t/ctudir2 setting. So it is not clear for me why we don't need the prefix. I believe we still need that directory setting otherwise where could we find the externalDefMap file?

You are right, the ctudir2 prefix is not needed as the %t substitution is test-file-level unique.

[martong]

In D75665#1971440, @gamesh411 wrote:

In D75665#1907914, @martong wrote:

The warning below suggests that we parse the ctu-other.c file (and presumably every file) as a C++ file, even if it should be parsed as a C file. Perhaps the invocation of the parser is missing some setting? Also, could this be the reason why on-the-fly and pch driven ctu gives different statistics?

warning: treating 'c' input as 'c++' when in C++ mode, this behavior is deprecated [-Wdeprecated]
/mnt/disks/ssd0/agent/workspace/amd64_debian_testing_clang8/clang/test/Analysis/Inputs/ctu-other.c:47:26: error: 'DataType' cannot be defined in a parameter type
int structInProto(struct DataType {int a;int b; } * d) {
                         ^
1 error generated.
Error while processing /mnt/disks/ssd0/agent/workspace/amd64_debian_testing_clang8/clang/test/Analysis/Inputs/ctu-other.c.

I have investigated the issue, and the compiler flags were looked up using the heuristic implemented in tooling.
This heuristic looks for the suitable compilation database in an upward ascending fashion inside the directory tree startin from the input source file.
By copying both the source file and the compilation database to the test directory this heuristic does the right thing now. (up until now the found compile_commands.json was the one used for llvm-project itself, and picked up a c++ specific compilation).

This issue is solved, however the ASTImporter still cannot import the inline definition of the struct, and emits an error.
Right now I am debugging the master branch that uses the AST-dumps wheter I also encounter this error inside the ASTImporter (just to see if I am operating on sane assumptions).

I've been looking into this and the ASTImporter indeed does not (cannot) import the definition of structInProto(). But that's just fine, that is why we have a branching for the call expression (TRUE/FALSE) below in ctu-main.c . All other functions whose definition were imported gives only the TRUE branch.

clang_analyzer_eval(structInProto(&d) == 0); // expected-warning{{TRUE}} expected-warning{{FALSE}}

[gamesh411]

Reorganize test code to overcome testing infrastructure limitations

[martong]

The tests are getting better, thanks Endre!

clang/test/Analysis/ctu-on-demand-parsing.c

6

Why do we need the -Wno-visibility flag?

7

If we execute clang_extdef_map then we don't need Inputs/ctu-other.c[pp].externalDefMap.on-the-fly.txt files. Do we?

[gamesh411]

Answered most review comments. Thanks for the reviewers @balazske, @martong so far.
The question of absolute path policy is still up for debate.

clang/lib/CrossTU/CrossTranslationUnit.cpp

364

Actually this does not need to be absolute. The naming of the variable should be changed to PrefixedPath or similar. I will do that for now.

364

It not necessarily an absolute path, only prefixed by CTUDir. There is no clear policy as to where to use strictly absolute paths and where to accept relative paths. IMHO this should be considered, and I am most certainly open for a discussion. In the meantime, I will just rename the variable to PrefixedPath.

407

As for the automatic lifetime of Tool, I am inclined to say this is not the case. I have run my clang-build with sanitizers (undefined and memory), and no bad access or UB was reported. If someone has more domain knowledge about tooling and the lifetime issues that could arise, I would be thrilled to be enlightened.

407

As for the need to pass PCHContainerOperaitons:
I think you are right, and I will check if this modifies the behaviour of the import in any way. I am removing this for now.

437

The policy of absolute and relative paths should be discussed. See my previous answer concerning AbsPath variable above. BTW llvm::sys::path::is_absolute is a solution for asserting this as I see it.

clang/test/Analysis/ctu-main.c

53

Seems unrelated, not sure how it got included. Fixing it.

clang/test/Analysis/ctu-on-demand-parsing.c

3

I have done this by example. Some CodeGen lit tests use sed the same way I did here. Needed for Windows compatibility maybe? (note that %t could substitute into a path containing backslashes on Windows) However, on Windows platforms, the availability of sed is not guaranteed either. The backslashes are used as escape chars in the JSON, hence the need to double-escape them. No comments are provided on those test also... I will add something tho (and also in the cpp test).

[gamesh411]

clang/test/Analysis/ctu-on-demand-parsing.c

6

Needed because the in-expression definition of a struct (which we cannot import) also warrants a compiler warning, and this comes up 2 times even (once when generating the extdef-mapping, second time when we analyze the file). Its easier to handle this warning in the compilation database, as that removes both occurrences of the warning.

7

Cannot remove as of yet, because they have .ast suffixed entries. In order to remove them the whole logic of what should be inside the index (externalDefMap.txt) for on-demand and non-on-demand cases should be discussed.

[gamesh411]

Implement review suggestions

[gamesh411]

clang/lib/CrossTU/CrossTranslationUnit.cpp

451

Good idea, I will create a standalone test case for this!

[gamesh411]

Rebase and update the number of analyzer options

[gamesh411]

Add ambiguous compilation database test case

[gamesh411]

Update user documentation

[martong]

Thanks Endre for the docs! I checked the CI job also, seemed okay, so, I think we are ready! Nice work! Let's do the commit!

clang/docs/analyzer/user-docs/CrossTranslationUnit.rst

26

Maybe just simply write This tool uses a compilation command database ...
(Same below with the on-demand version.)

[xazax.hun]

clang/include/clang/CrossTU/CrossTranslationUnit.h

52

The two enum values refer to compilation database and compile command database. I'd prefer to use the same wording in both values.

237

I am not sure if this is good design.
Here, if the meaning of the Identifier depends on the subclass, the caller of this method always needs to be aware of the dynamic type of the object. What is the point of a common base class if we always need to know the dynamic type?

Looking at the code this does not look bad. But it might be a code smell.

291

Why is this no longer const?

clang/lib/CrossTU/CrossTranslationUnit.cpp

118

Unfortunately, this is a very common case for a large number of projects that supports multiple targets (e.g. 32 and 64 bits). Is there a plan to mitigate this problem?

[martong]

clang/include/clang/CrossTU/CrossTranslationUnit.h

52

+1

237

The way how we process the extDefMapping file is identical in both cases. That's an index, keyed with the USRs of functions and then we get back a value. And the way how we use that value is different. In the PCH case that holds the path for the .ast file, in the ODM case that is the name of the source file which we must find in the compile db. So, I think the process of getting the AST for a USR requires the polymorphic behavior from the loaders.

We discussed other alternatives with Endre. We were thinking that maybe the extDefMapping file should be identical in both cases. But then we would need to add the .ast postfixes for the entries in the PCH case. And we cannot just do that, because we may not know if what is the correct postfix. The user may have generated .pch files instead. Also, we don't want to compel any Clang user to use CodeChecker (CC will always create .ast files). CTU should be running fine by manually executing the independent steps.

clang/lib/CrossTU/CrossTranslationUnit.cpp

118

I don't think we could do that. We need to merge ASTs of those TUs that are linkable. Otherwise the merge will be unsuccessful because of certain mismatches in Decls (structural eq will fail).
Even in CodeChecker we are planning to issue a hard error in these ambiguous cases, even with the PCH method too. (The error would be suppressible by the user of course, but if that is suppressed then we take no responsibility.)

[xazax.hun]

Ok, looks good to me.

The minor nit regarding the naming is easy to fix before commit. The design question I had is not a blocker, my suggested alternative can be implemented later (if desired) in a backward-compatible way from the user's point of view.

clang/include/clang/CrossTU/CrossTranslationUnit.h

237

Let me rephrase my concerns a bit. Do we really need a polymorphic ASTLoader to be present for the whole analysis? Wouldn't it make more sense to always do the same thing, i.e. if we are given a pch file load it, if we are given a source file, parse it? This way we would not be restricted to on-demand or two pass ctu analysis, but we could do any combination of the two.

[whisperity]

clang/docs/analyzer/user-docs/CrossTranslationUnit.rst

21

I think it's PCH-based with a -.

25

symbols' USR names

26

clang tool

26

And perhaps add some cross-references to what a compilation database is, etc. These things are also documented within Clang's doc tree.

31

Instead of a prefix, create an overview of PCH-based analysis, and add this and the next one as a 3rd-level heading?

112–115

Are these flags documented somewhere?

208–209

Same here for comments from above.

312–316

These lines could be removed, I think, they don't add anything to the presentation.

317–318

Due to the lack of colours, perhaps you could use ls -F which suffixes each directory with a /: reports/.

368

as a runner script the driver for this feature

369

expect it to work only ...

(to emphasise the limitation)

clang/lib/CrossTU/CrossTranslationUnit.cpp

118

More clever logging of link commands and creating separate CDBs for each target or each binary could help this cause, but that needs infrastructural capabilities from both the build system and the analysis driver (i.e. CodeChecker).

[martong]

clang/include/clang/CrossTU/CrossTranslationUnit.h

237

Well yeah, we could do that, it is a good idea, thanks! We will consider this in the future. I like in this idea that the command line options to Clang would be simplified. But then we must be transparent and show/log the user which method we are using.

[whisperity]

clang/docs/analyzer/user-docs/CrossTranslationUnit.rst

376

What's this line? Is this added by Phab, or is this a weird merge conflict / parsing error of the patch?

[gamesh411]

Fix issues in documentation

[gamesh411]

@whisperity Thanks for the review :3

clang/docs/analyzer/user-docs/CrossTranslationUnit.rst

112–115

As with ctu-dir, these flags are only documented in analyzer options. However, it would indeed be beneficial to list CTU related flags here in the user documentation. I will create a new revision for that specifically.

312–316

Good point, i remove it from here and from the above example as well.

[gamesh411]

Reword enum value

[gamesh411]

Fix index error enum declaration

[gamesh411]

I am landing this. Thanks for the reviews @martong @balazske @xazax.hun @whisperity !

[thakis]

clang/lib/CrossTU/CMakeLists.txt

13 ↗

(On Diff #260238)

We've been very careful to make clang (the compiler binary) only depend on clangToolingCore and not on the much bigger clangTooling library. Since clang depends on the CrossTU library, this breaks that. Can you reorganize things so that we don't need a dependency from clang-the-compiler-binary on clangTooling?

[thakis]

Looks like this also breaks tests on Windows and Mac, eg http://45.33.8.238/win/13902/step_7.txt

It probably makes sense to revert while you investigate?

[gribozavr2]

Sorry, this change broke a couple bots that are in the official CI, at least http://lab.llvm.org:8011/builders/llvm-clang-win-x-aarch64/builds/7566 and http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-scei-ps4-ubuntu-fast/builds/66618. I'm reverting this change -- feel free to reland after fixing the issue.

[gribozavr2]

Reverted in https://github.com/llvm/llvm-project/commit/96717125e852d1c6ddf41c22dd2d556f4f5aa34d.

[gamesh411]

Non-linux buildbots were utterly broken :( . Trying to fix them...

Thanks for reverting this for me :)
Also I will investigate the possibility to not depend on Tooling.

[gamesh411]

Remove platform constraint from test file

[gamesh411]

[NFC] Fix arcanist double commit revisioning

[gamesh411]

clang/lib/CrossTU/CMakeLists.txt

13 ↗

(On Diff #260238)

I have gone, and investigated where the main functionality of on-demand AST loading comes from. (Only after wishfully thinking changing the dependency from clangTooling to clangToolingCore would work, which of course didnt't )

It seems that this functionality used the major components of clangTooling, namely JSONCompilationDatabase::loadFromFile, ClangTool ctor/dtor, ClangTool::buildAST.
In order to circumvent the dependency and the reimplementation of handling compilation databases, a call to an external tool could be used, or the clangTooling library would need to be refactored in a way which is not a very tasteful refactoring. IMHO. I would like to know which of the aforementioned way is deemed to be best.

[gamesh411]

Remove arch target from another invocation

[gamesh411]

Implement a solution without a dependency on clangTooling

[balazske]

clang/lib/CrossTU/CrossTranslationUnit.cpp

363–364

Is here a lambda necessary? I think it must not be over-used. In this case we can construct the file path as the first step and then use it to load the file, without using lambda call.

555

Is here a special reason to use auto && type (and not auto & or std::string &)? Probably this code is faulty: The CmdPart is moved out and then destructed, before the content of it (where the c_str points to) is used?

[whisperity]

Please check the summary of the patch, it seems to contain old information as well.

clang/docs/analyzer/user-docs/CrossTranslationUnit.rst

211–213

This is obsolete information.

clang/include/clang/CrossTU/CrossTranslationUnit.h

244

Is the explicit needed here?

clang/include/clang/StaticAnalyzer/Core/AnalyzerOptions.def

394–395

"valid compilation database" is this line still what we wish to say? A YAML is used, so most likely we need a "valid invocation list" or something.

399–404

Maybe it would be worthwhile to add a dummy example like main.cpp: g++ hello-world.cpp which has the proper format into this help.

clang/lib/CrossTU/CrossTranslationUnit.cpp

21–24

The idea here was to not depend on libtooling but these lines are still here!

32

<> -> "" and order

117–118

still using terms "compilation database", is this intended?

555

@balazske This is a generic lambda where auto&& is the universal reference. It is equivalent to writing template <typename T> f(T&& param). It obeys the usual point of instantiation and reference cascading rules, if a const std::string& is given as T, const std::string& && will become a single &.

581

Are the headers from which these types come from included?

clang/test/Analysis/Inputs/ctu-other.c

34–38

Possibly unrelated change?

clang/test/Analysis/ctu-on-demand-parsing.c

2–16

Does the LIT syntax allow for better organising these commands? This is now a bit hard to read

Something like this would help:

// RUN: rm -rf %t && mkdir -p %t
// RUN: cp "%s" "%t/ctu-on-demand-parsing.c" && cp "%S/Inputs/ctu-other.c" "%t/ctu-other.c"
// [empty]
// [comment]
// RUN: [json magic]
// RUN: [yaml magic]
// [empty]
// RUN: extdef map
// [empty]
// RUN: cc1

[whisperity]

(Maybe this will make Phab not show "✅ Accepted"...)

[martong]

Hi Endre, just checked the latest update. Overall looks good to me, but found some nits.

clang/include/clang/CrossTU/CrossTranslationUnit.h

252

Should we rename this member to ....PathTo....InvocationList... ?

clang/lib/CrossTU/CrossTranslationUnit.cpp

553

Could we avoid this transfer if InvocationList was storing const char * values instead of std::strings? Why can't we store char*s in InvocationList?

565

typo: filed -> field

clang/test/Analysis/ctu-on-demand-parsing.c

7

Perhaps we could document here that the compile_commands.json is needed ONLY for %clang_extdef_map.

clang/test/Analysis/ctu-on-demand-parsing.cpp

10

I know this is just a nit, but this is very hard to read. Do you think we could break this up (and other long lines) into multiple lines but within the same RUN directive?

[gamesh411]

Update functional changes, documentation update incoming

[gamesh411]

The remaining documentation and test changes are also underway.

clang/include/clang/CrossTU/CrossTranslationUnit.h

237

I will implement this separation by suffixes in the next change.

244

This constructor started out with having a single param only. I agree, that explicit is no longer necessary.

252

Renamed to InvocationListFilePath.

291

Most usages of the CI require a non-const ref (this is due to lacking API support for getting analyzer options), and I found it more convenient to just take a non-const ref, instead of const_casting it away later. Not sure myself, but I think I will roll with this now (and I should definitely check the API of CompilerInstance).

clang/lib/CrossTU/CrossTranslationUnit.cpp

117–118

renamed

118

I have solved the problem of supporting multiple architectures by offloading the responsibility to the user (to the tool driving the analysis). The invocation list file is now supposed to contain only invocations with the same arch/target and this should also be the same as the arch/target of the main TU analyzed. If there is still ambiguity after this separation, this error should nevertheless be emitted ( IMHO ).

363–364

Refactored to use path concatenation.

553

We certainly could, I think there won't be null-characters in paths so storing char* s would not lose generality. However, I think the memory management done by std::string is worth it, as it is clear where the character-data lives in memory.

555

Results of my research: In theory, the STL could do something fancy here if it wanted, but it is not the case in reality. If you want move-operations with algorithms (the ones not explicitly mentioning move in their names) you have to be explicit about it: move-iterators-fluent-cpp.

581

Included the llvm/Support/YAMLParser.h

clang/test/Analysis/Inputs/ctu-other.c

34–38

Funny enough, this is very relevant. Dumping the AST does not errors out on this construction, however, the parser in on-demand mode does. This asm syntax is however supported by both. To document this, I have added a TODO as well.

[whisperity]

clang/lib/CrossTU/CrossTranslationUnit.cpp

24

Duped include in L34.

[gamesh411]

Fix documentation and commit message

[martong]

LGTM!

[gamesh411]

Extend index file format
Update documentation

[gamesh411]

Fix test case, and reorder warning

[whisperity]

clang/lib/CrossTU/CrossTranslationUnit.cpp

119–120

These two lines could be formatted together.

clang/unittests/CrossTU/CrossTranslationUnitTest.cpp

269

Just for good measure, I would like a test case on the "invocation list is ambiguous" error. (So in case we in the future figure out how to disambiguate, we can remove that test case!)

[gamesh411]

use consumeError in test

[gamesh411]

add ambiguous invocation list test case

[gamesh411]

fix typo

[gamesh411]

add ambiguity checking during invocation list parsing

[thakis]

This breaks check-clang on mac: http://45.33.8.238/mac/15258/step_7.txt

Please take a look, and revert for now if it takes a while to fix.

[gamesh411]

In D75665#2084889, @thakis wrote:

This breaks check-clang on mac: http://45.33.8.238/mac/15258/step_7.txt

Please take a look, and revert for now if it takes a while to fix.

Thanks for reporting this, i think the output is valuable for debugging. I have restricted the test case to linux for now.

clang/docs/analyzer/user-docs/CrossTranslationUnit.rst

 =====================================
 Cross Translation Unit (CTU) Analysis
 =====================================
 
 Normally, static analysis works in the boundary of one translation unit (TU).
-However, with additional steps and configuration we can enable the analysis to inline the definition of a function from another TU.
+However, with additional steps and configuration we can enable the analysis to inline the definition of a function from
+another TU.
 
 .. contents::
    :local:
 
-Manual CTU Analysis
--------------------
+Overview
+________
+CTU analysis can be used in a variety of ways. The importing of external TU definitions can work with pre-dumped PCH
+files or generating the necessary AST structure on-demand, during the analysis of the main TU. Driving the static
+analysis can also be implemented in multiple ways. The most direct way is to specify the necessary commandline options
+of the Clang frontend manually (and generate the prerequisite dependencies of the specific import method by hand). This
+process can be automated by other tools, like `CodeChecker <https://github.com/Ericsson/codechecker>`_ and scan-build-py
+(preference for the former).
+
+PCH-based analysis
+__________________
+The analysis needs the PCH dumps of all the translations units used in the project.
+These can be generated by the Clang Frontend itself, and must be arranged in a specific way in the filesystem.
+The index, which maps symbols' USR names to PCH dumps containing them must also be generated by the
+`clang-extdef-mapping`. Entries in the index *must* have an `.ast` suffix if the goal
+is to use PCH-based analysis, as the lack of that extension signals that the entry is to be used as a source-file, and parsed on-demand.
+This tool uses a :doc:`compilation database <../../JSONCompilationDatabase>` to
+determine the compilation flags used.
+The analysis invocation must be provided with the directory which contains the dumps and the mapping files.
+
 
+Manual CTU Analysis
+###################
 Let's consider these source files in our minimal example:
 
 .. code-block:: cpp
 
   // main.cpp
   int foo();
 
   int main() {
[
     {
       "directory": "/path/to/your/project",
       "command": "clang++ -c main.cpp -o main.o",
       "file": "main.cpp"
     }
   ]
 
 We'd like to analyze `main.cpp` and discover the division by zero bug.
-In order to be able to inline the definition of `foo` from `foo.cpp` first we have to generate the `AST` (or `PCH`) file of `foo.cpp`:
+In order to be able to inline the definition of `foo` from `foo.cpp` first we have to generate the `AST` (or `PCH`) file
+of `foo.cpp`:
 
 .. code-block:: bash
 
   $ pwd $ /path/to/your/project
   $ clang++ -emit-ast -o foo.cpp.ast foo.cpp
   $ # Check that the .ast file is generated:
   $ ls
   compile_commands.json  foo.cpp.ast  foo.cpp  main.cpp
   $
 
-The next step is to create a CTU index file which holds the `USR` name and location of external definitions in the source files:
+The next step is to create a CTU index file which holds the `USR` name and location of external definitions in the
+source files:
 
 .. code-block:: bash
 
   $ clang-extdef-mapping -p . foo.cpp
   c:@F@foo# /path/to/your/project/foo.cpp
   $ clang-extdef-mapping -p . foo.cpp > externalDefMap.txt
 
 We have to modify `externalDefMap.txt` to contain the name of the `.ast` files instead of the source files:

 
 Now everything is available for the CTU analysis.
 We have to feed Clang with CTU specific extra arguments:
 
 .. code-block:: bash
 
   $ pwd
   /path/to/your/project
-  $ clang++ --analyze -Xclang -analyzer-config -Xclang experimental-enable-naive-ctu-analysis=true -Xclang -analyzer-config -Xclang ctu-dir=. -Xclang -analyzer-output=plist-multi-file main.cpp
+  $ clang++ --analyze \
+      -Xclang -analyzer-config -Xclang experimental-enable-naive-ctu-analysis=true \
+      -Xclang -analyzer-config -Xclang ctu-dir=. \
+      -Xclang -analyzer-output=plist-multi-file \
+      main.cpp
   main.cpp:5:12: warning: Division by zero
     return 3 / foo();
            ~~^~~~~~~
   1 warning generated.
   $ # The plist file with the result is generated.
-  $ ls
+  $ ls -F
   compile_commands.json  externalDefMap.txt  foo.ast  foo.cpp  foo.cpp.ast  main.cpp  main.plist
   $
 
-This manual procedure is error-prone and not scalable, therefore to analyze real projects it is recommended to use `CodeChecker` or `scan-build-py`.
+This manual procedure is error-prone and not scalable, therefore to analyze real projects it is recommended to use
+`CodeChecker` or `scan-build-py`.
 
 Automated CTU Analysis with CodeChecker
----------------------------------------
+#######################################
 The `CodeChecker <https://github.com/Ericsson/codechecker>`_ project fully supports automated CTU analysis with Clang.
 Once we have set up the `PATH` environment variable and we activated the python `venv` then it is all it takes:
 
 .. code-block:: bash
 
   $ CodeChecker analyze --ctu compile_commands.json -o reports
-  [INFO 2019-07-16 17:21] - Pre-analysis started.
-  [INFO 2019-07-16 17:21] - Collecting data for ctu analysis.
-  [INFO 2019-07-16 17:21] - [1/2] foo.cpp
-  [INFO 2019-07-16 17:21] - [2/2] main.cpp
-  [INFO 2019-07-16 17:21] - Pre-analysis finished.
-  [INFO 2019-07-16 17:21] - Starting static analysis ...
-  [INFO 2019-07-16 17:21] - [1/2] clangsa analyzed foo.cpp successfully.
-  [INFO 2019-07-16 17:21] - [2/2] clangsa analyzed main.cpp successfully.
-  [INFO 2019-07-16 17:21] - ----==== Summary ====----
-  [INFO 2019-07-16 17:21] - Successfully analyzed
-  [INFO 2019-07-16 17:21] -   clangsa: 2
-  [INFO 2019-07-16 17:21] - Total analyzed compilation commands: 2
-  [INFO 2019-07-16 17:21] - ----=================----
-  [INFO 2019-07-16 17:21] - Analysis finished.
-  [INFO 2019-07-16 17:21] - To view results in the terminal use the "CodeChecker parse" command.
-  [INFO 2019-07-16 17:21] - To store results use the "CodeChecker store" command.
-  [INFO 2019-07-16 17:21] - See --help and the user guide for further options about parsing and storing the reports.
-  [INFO 2019-07-16 17:21] - ----=================----
-  [INFO 2019-07-16 17:21] - Analysis length: 0.659618854523 sec.
-  $ ls
-  compile_commands.json  foo.cpp  foo.cpp.ast  main.cpp  reports
+  $ ls -F
+  compile_commands.json  foo.cpp  foo.cpp.ast  main.cpp  reports/
   $ tree reports
   reports
   ├── compile_cmd.json
   ├── compiler_info.json
   ├── foo.cpp_53f6fbf7ab7ec9931301524b551959e2.plist
   ├── main.cpp_23db3d8df52ff0812e6e5a03071c8337.plist
   ├── metadata.json
   └── unique_compile_commands.json

 Or we can use `CodeChecker parse -e html` to export the results into HTML format:
 
 .. code-block:: bash
 
   $ CodeChecker parse -e html -o html_out reports
   $ firefox html_out/index.html
 
 Automated CTU Analysis with scan-build-py (don't do it)
--------------------------------------------------------
-We actively develop CTU with CodeChecker as a "runner" script, `scan-build-py` is not actively developed for CTU.
-`scan-build-py` has various errors and issues, expect it to work with the very basic projects only.
+#############################################################
+We actively develop CTU with CodeChecker as the driver for this feature, `scan-build-py` is not actively developed for CTU.
+`scan-build-py` has various errors and issues, expect it to work only with the very basic projects only.
 
 Example usage of scan-build-py:
 
 .. code-block:: bash
 
   $ /your/path/to/llvm-project/clang/tools/scan-build-py/bin/analyze-build --ctu
   analyze-build: Run 'scan-view /tmp/scan-build-2019-07-17-17-53-33-810365-7fqgWk' to examine bug reports.
   $ /your/path/to/llvm-project/clang/tools/scan-view/bin/scan-view /tmp/scan-build-2019-07-17-17-53-33-810365-7fqgWk
   Starting scan-view at: http://127.0.0.1:8181
     Use Ctrl-C to exit.
   [6336:6431:0717/175357.633914:ERROR:browser_process_sub_thread.cc(209)] Waited 5 ms for network service
   Opening in existing browser session.
   ^C
   $
+
+On-demand analysis
+__________________
+The analysis produces the necessary AST structure of external TUs during analysis. This requires the
+exact compiler invocations for each TU, which can be generated by hand, or by tools driving the analyzer.
+The compiler invocation is a shell command that could be used to compile the TU-s main source file.
+The mapping from absolute source file paths of a TU to lists of compilation command segments used to
+compile said TU are given in YAML format referred to as `invocation list`, and must be passed as an
+analyer-config argument.
+The index, which maps function USR names to source files containing them must also be generated by the
+`clang-extdef-mapping`. Entries in the index must *not* have an `.ast` suffix if the goal
+is to use On-demand analysis, as that extension signals that the entry is to be used as an PCH-dump.
+The mapping of external definitions implicitly uses a
+:doc:`compilation database <../../JSONCompilationDatabase>` to determine the compilation flags used.
+The analysis invocation must be provided with the directory which contains the mapping
+files, and the `invocation list` which is used to determine compiler flags.
+
+
+Manual CTU Analysis
+###################
+
+Let's consider these source files in our minimal example:
+
+.. code-block:: cpp
+
+  // main.cpp
+  int foo();
+
+  int main() {
+    return 3 / foo();
+  }
+
+.. code-block:: cpp
+
+  // foo.cpp
+  int foo() {
+    return 0;
+  }
+
+The compilation database:
+
+.. code-block:: bash
+
+  [
+    {
+      "directory": "/path/to/your/project",
+      "command": "clang++ -c foo.cpp -o foo.o",
+      "file": "foo.cpp"
+    },
+    {
+      "directory": "/path/to/your/project",
+      "command": "clang++ -c main.cpp -o main.o",
+      "file": "main.cpp"
+    }
+  ]
+
+The `invocation list`:
+
+.. code-block:: bash
+  
+  "/path/to/your/project/foo.cpp": 
+    - "clang++"
+    - "-c"
+    - "/path/to/your/project/foo.cpp"
+    - "-o"
+    - "/path/to/your/project/foo.o"
+
+  "/path/to/your/project/main.cpp": 
+    - "clang++"
+    - "-c"
+    - "/path/to/your/project/main.cpp"
+    - "-o"
+    - "/path/to/your/project/main.o"
+
+We'd like to analyze `main.cpp` and discover the division by zero bug.
+As we are using On-demand mode, we only need to create a CTU index file which holds the `USR` name and location of
+external definitions in the source files:
+
+.. code-block:: bash
+
+  $ clang-extdef-mapping -p . foo.cpp
+  c:@F@foo# /path/to/your/project/foo.cpp
+  $ clang-extdef-mapping -p . foo.cpp > externalDefMap.txt
+
+Now everything is available for the CTU analysis.
+We have to feed Clang with CTU specific extra arguments:
+
+.. code-block:: bash
+
+  $ pwd
+  /path/to/your/project
+  $ clang++ --analyze \
+      -Xclang -analyzer-config -Xclang experimental-enable-naive-ctu-analysis=true \
+      -Xclang -analyzer-config -Xclang ctu-dir=. \
+      -Xclang -analyzer-config -Xclang ctu-invocation-list=invocations.yaml \
+      -Xclang -analyzer-output=plist-multi-file \
+      main.cpp
+  main.cpp:5:12: warning: Division by zero
+    return 3 / foo();
+           ~~^~~~~~~
+  1 warning generated.
+  $ # The plist file with the result is generated.
+  $ ls -F
+  compile_commands.json  externalDefMap.txt  foo.cpp  main.cpp  main.plist
+  $
+
+This manual procedure is error-prone and not scalable, therefore to analyze real projects it is recommended to use
+`CodeChecker` or `scan-build-py`.
+
+Automated CTU Analysis with CodeChecker
+#######################################
+The `CodeChecker <https://github.com/Ericsson/codechecker>`_ project fully supports automated CTU analysis with Clang.
+Once we have set up the `PATH` environment variable and we activated the python `venv` then it is all it takes:
+
+.. code-block:: bash
+
+  $ CodeChecker analyze --ctu --ctu-ast-loading-mode on-demand compile_commands.json -o reports
+  $ ls -F
+  compile_commands.json  foo.cpp main.cpp  reports/
+  $ tree reports
+  reports
+  ├── compile_cmd.json
+  ├── compiler_info.json
+  ├── foo.cpp_53f6fbf7ab7ec9931301524b551959e2.plist
+  ├── main.cpp_23db3d8df52ff0812e6e5a03071c8337.plist
+  ├── metadata.json
+  └── unique_compile_commands.json
+
+  0 directories, 6 files
+  $
+
+The `plist` files contain the results of the analysis, which may be viewed with the regular analysis tools.
+E.g. one may use `CodeChecker parse` to view the results in command line:
+
+.. code-block:: bash
+
+  $ CodeChecker parse reports
+  [HIGH] /home/egbomrt/ctu_mini_raw_project/main.cpp:5:12: Division by zero [core.DivideZero]
+    return 3 / foo();
+             ^
+
+  Found 1 defect(s) in main.cpp
+
+
+  ----==== Summary ====----
+  -----------------------
+  Filename | Report count
+  -----------------------
+  main.cpp |            1
+  -----------------------
+  -----------------------
+  Severity | Report count
+  -----------------------
+  HIGH     |            1
+  -----------------------
+  ----=================----
+  Total number of reports: 1
+  ----=================----
+
+Or we can use `CodeChecker parse -e html` to export the results into HTML format:
+
+.. code-block:: bash
+
+  $ CodeChecker parse -e html -o html_out reports
+  $ firefox html_out/index.html
+
+Automated CTU Analysis with scan-build-py (don't do it)
+#######################################################
+We actively develop CTU with CodeChecker as the driver for feature, `scan-build-py` is not actively developed for CTU.
+`scan-build-py` has various errors and issues, expect it to work only with the very basic projects only.
+
+Currently On-demand analysis is not supported with `scan-build-py`.
+

clang/include/clang/CrossTU/CrossTranslationUnit.h

 #define LLVM_CLANG_CROSSTU_CROSSTRANSLATIONUNIT_H
 
 #include "clang/AST/ASTImporterSharedState.h"
 #include "clang/Basic/LLVM.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/StringMap.h"
 #include "llvm/Support/Error.h"
+#include "llvm/Support/Path.h"
 
 namespace clang {
 class CompilerInstance;
 class ASTContext;
 class ASTImporter;
 class ASTUnit;
 class DeclContext;
 class FunctionDecl;
enum class index_error_code {
   multiple_definitions,
   missing_definition,
   failed_import,
   failed_to_get_external_ast,
   failed_to_generate_usr,
   triple_mismatch,
   lang_mismatch,
   lang_dialect_mismatch,
-  load_threshold_reached
+  load_threshold_reached,
+  invocation_list_ambiguous,
+  invocation_list_file_not_found,
+  invocation_list_empty,
+  invocation_list_wrong_format,
+  invocation_list_lookup_unsuccessful
 };
 
 class IndexError : public llvm::ErrorInfo<IndexError> {
 public:
   static char ID;
   IndexError(index_error_code C) : Code(C), LineNo(0) {}
   IndexError(index_error_code C, std::string FileName, int LineNo = 0)
       : Code(C), FileName(std::move(FileName)), LineNo(LineNo) {}
private:
   index_error_code Code;
   std::string FileName;
   int LineNo;
   std::string TripleToName;
   std::string TripleFromName;
 };
 
 /// This function parses an index file that determines which
-///        translation unit contains which definition.
+/// translation unit contains which definition. The IndexPath is not prefixed
+/// with CTUDir, so an absolute path is expected for consistent results.
 ///
 /// The index file format is the following:
 /// each line consists of an USR and a filepath separated by a space.
 ///
 /// \return Returns a map where the USR is the key and the filepath is the value
 ///         or an error.
 llvm::Expected<llvm::StringMap<std::string>>
-parseCrossTUIndex(StringRef IndexPath, StringRef CrossTUDir);
+parseCrossTUIndex(StringRef IndexPath);
 
 std::string createCrossTUIndexString(const llvm::StringMap<std::string> &Index);
 
+using InvocationListTy = llvm::StringMap<llvm::SmallVector<std::string, 32>>;
+/// Parse the YAML formatted invocation list file content \p FileContent.
+/// The format is expected to be a mapping from from absolute source file
+/// paths in the filesystem to a list of command-line parts, which
+/// constitute the invocation needed to compile that file. That invocation
+/// will be used to produce the AST of the TU.
+llvm::Expected<InvocationListTy> parseInvocationList(
+    StringRef FileContent,
+    llvm::sys::path::Style PathStyle = llvm::sys::path::Style::posix);
+
 // Returns true if the variable or any field of a record variable is const.
 bool containsConst(const VarDecl *VD, const ASTContext &ACtx);
 
 /// This class is used for tools that requires cross translation
 ///        unit capability.
 ///
-/// This class can load definitions from external AST files.
+/// This class can load definitions from external AST sources.
 /// The loaded definition will be merged back to the original AST using the
 /// AST Importer.
 /// In order to use this class, an index file is required that describes
 /// the locations of the AST files for each definition.
 ///
 /// Note that this class also implements caching.
 class CrossTranslationUnitContext {
 public:
   CrossTranslationUnitContext(CompilerInstance &CI);
   ~CrossTranslationUnitContext();
 
   /// This function loads a function or variable definition from an
   ///        external AST file and merges it into the original AST.
   ///
   /// This method should only be used on functions that have no definitions or
   /// variables that have no initializer in
   /// the current translation unit. A function definition with the same
   /// declaration will be looked up in the index file which should be in the
   /// \p CrossTUDir directory, called \p IndexName. In case the declaration is
-  /// found in the index the corresponding AST file will be loaded and the
+  /// found in the index the corresponding AST will be loaded and the
   /// definition will be merged into the original AST using the AST Importer.
   ///
   /// \return The declaration with the definition will be returned.
   /// If no suitable definition is found in the index file or multiple
   /// definitions found error will be returned.
   ///
   /// Note that the AST files should also be in the \p CrossTUDir.
   llvm::Expected<const FunctionDecl *>
   getCrossTUDefinition(const FunctionDecl *FD, StringRef CrossTUDir,
                        StringRef IndexName, bool DisplayCTUProgress = false);
   llvm::Expected<const VarDecl *>
   getCrossTUDefinition(const VarDecl *VD, StringRef CrossTUDir,
                        StringRef IndexName, bool DisplayCTUProgress = false);
 
   /// This function loads a definition from an external AST file.
   ///
   /// A definition with the same declaration will be looked up in the
   /// index file which should be in the \p CrossTUDir directory, called
   /// \p IndexName. In case the declaration is found in the index the
-  /// corresponding AST file will be loaded. If the number of TUs imported
+  /// corresponding AST will be loaded. If the number of TUs imported
   /// reaches \p CTULoadTreshold, no loading is performed.
   ///
   /// \return Returns a pointer to the ASTUnit that contains the definition of
   /// the looked up name or an Error.
   /// The returned pointer is never a nullptr.
   ///
   /// Note that the AST files should also be in the \p CrossTUDir.
   llvm::Expected<ASTUnit *> loadExternalAST(StringRef LookupName,
private:
   /// and the ASTUnit for the From context.
   /// This map is used by getImportedFromSourceLocation to lookup a FileID and
   /// its Preprocessor when knowing only the FileID in the 'To' context. The
   /// FileID could be imported by any of multiple 'From' ASTImporter objects.
   /// we do not want to loop over all ASTImporter's to find the one that
   /// imported the FileID.
   ImportedFileIDMap ImportedFileIDs;
 
-  /// Functor for loading ASTUnits from AST-dump files.
-  class ASTFileLoader {
+  using LoadResultTy = llvm::Expected<std::unique_ptr<ASTUnit>>;
+
+  /// Loads ASTUnits from AST-dumps or source-files.
+  class ASTLoader {
   public:
-    ASTFileLoader(const CompilerInstance &CI);
-    std::unique_ptr<ASTUnit> operator()(StringRef ASTFilePath);
+    ASTLoader(CompilerInstance &CI, StringRef CTUDir,
+              StringRef InvocationListFilePath);
+
+    /// Load the ASTUnit by its identifier found in the index file. If the
+    /// indentifier is suffixed with '.ast' it is considered a dump. Otherwise
+    /// it is treated as source-file, and on-demand parsed. Relative paths are
+    /// prefixed with CTUDir.
+    LoadResultTy load(StringRef Identifier);
+
+    /// Lazily initialize the invocation list information, which is needed for
+    /// on-demand parsing.
+    llvm::Error lazyInitInvocationList();
 
   private:
-    const CompilerInstance &CI;
+    /// The style used for storage and lookup of filesystem paths.
+    /// Defaults to posix.
+    const llvm::sys::path::Style PathStyle = llvm::sys::path::Style::posix;
+
+    /// Loads an AST from a pch-dump.
+    LoadResultTy loadFromDump(StringRef Identifier);
+    /// Loads an AST from a source-file.
+    LoadResultTy loadFromSource(StringRef Identifier);
+
+    CompilerInstance &CI;
+    StringRef CTUDir;
+    /// The path to the file containing the invocation list, which is in YAML
+    /// format, and contains a mapping from source files to compiler invocations
+    /// that produce the AST used for analysis.
+    StringRef InvocationListFilePath;
+    /// In case of on-demand parsing, the invocations for parsing the source
+    /// files is stored.
+    llvm::Optional<InvocationListTy> InvocationList;
   };
 
   /// Maintain number of AST loads and check for reaching the load limit.
   class ASTLoadGuard {
   public:
     ASTLoadGuard(unsigned Limit) : Limit(Limit) {}
 
     /// Indicates, whether a new load operation is permitted, it is within the
private:
     /// The limit (threshold) value for number of loaded ASTs.
     const unsigned Limit;
   };
 
   /// Storage and load of ASTUnits, cached access, and providing searchability
   /// are the concerns of ASTUnitStorage class.
   class ASTUnitStorage {
   public:
-    ASTUnitStorage(const CompilerInstance &CI);
+    ASTUnitStorage(CompilerInstance &CI);
     /// Loads an ASTUnit for a function.
     ///
     /// \param FunctionName USR name of the function.
     /// \param CrossTUDir Path to the directory used to store CTU related files.
     /// \param IndexName Name of the file inside \p CrossTUDir which maps
     /// function USR names to file paths. These files contain the corresponding
     /// AST-dumps.
     /// \param DisplayCTUProgress Display a message about loading new ASTs.
private:
     using NonOwningMapTy = BaseMapTy<clang::ASTUnit *>;
 
     OwningMapTy FileASTUnitMap;
     NonOwningMapTy NameASTUnitMap;
 
     using IndexMapTy = BaseMapTy<std::string>;
     IndexMapTy NameFileMap;
 
-    ASTFileLoader FileAccessor;
+    /// Loads the AST based on the identifier found in the index.
+    ASTLoader Loader;
 
-    /// Limit the number of loaded ASTs. Used to limit the  memory usage of the
-    /// CrossTranslationUnitContext.
-    /// The ASTUnitStorage has the knowledge about if the AST to load is
-    /// actually loaded or returned from cache. This information is needed to
-    /// maintain the counter.
+    /// Limit the number of loaded ASTs. It is used to limit the  memory usage
+    /// of the CrossTranslationUnitContext. The ASTUnitStorage has the
+    /// information whether the AST to load is actually loaded or returned from
+    /// cache. This information is needed to maintain the counter.
     ASTLoadGuard LoadGuard;
   };
 
   ASTUnitStorage ASTStorage;
-
 };
 
 } // namespace cross_tu
 } // namespace clang
 
 #endif // LLVM_CLANG_CROSSTU_CROSSTRANSLATIONUNIT_H

clang/include/clang/StaticAnalyzer/Core/AnalyzerOptions.def

 //===----------------------------------------------------------------------===//
 // String analyzer options.
 //===----------------------------------------------------------------------===//
 
 ANALYZER_OPTION(StringRef, CTUDir, "ctu-dir",
                 "The directory containing the CTU related files.", "")
 
 ANALYZER_OPTION(StringRef, CTUIndexName, "ctu-index-name",
-                "the name of the file containing the CTU index of definitions.",
+                "The name of the file containing the CTU index of definitions. "
+                "The index file maps USR-names to identifiers. An identifier "
+                "can end with an '.ast' suffix, indicating the indentifier is "
+                "a path to a pch-dump. Otherwise the identifier is regarded as "
+                "path to a source file which is parsed on-demand. Relative "
+                "paths are prefixed with ctu-dir, absolute paths are used "
+                "unmodified during lookup.",
                 "externalDefMap.txt")
 
 ANALYZER_OPTION(
+    StringRef, CTUInvocationList, "ctu-invocation-list",
+    "The path to the YAML format file containing a mapping from source file "
+    "paths to command-line invocations represented as a list of arguments. "
+    "This invocation is used produce the source-file's AST in case on-demand "
+    "loading is performed. Example file-content: "
+    "{/main.cpp: [clang++, /main.cpp], other.cpp: [clang++, /other.cpp]}",
+    "invocations.yaml")
+
+ANALYZER_OPTION(
     StringRef, ModelPath, "model-path",
     "The analyzer can inline an alternative implementation written in C at the "
     "call site if the called function's body is not available. This is a path "
     "where to look for those alternative implementations (called models).",
     "")
 
 ANALYZER_OPTION(
     StringRef, CXXMemberInliningMode, "c++-inlining",
     "Controls which C++ member functions will be considered for inlining. "
     "Value: \"constructors\", \"destructors\", \"methods\".",
     "destructors")
 
 ANALYZER_OPTION(

clang/lib/CrossTU/CrossTranslationUnit.cpp

 #include "clang/AST/ASTImporter.h"
 #include "clang/AST/Decl.h"
 #include "clang/Basic/TargetInfo.h"
 #include "clang/CrossTU/CrossTUDiagnostic.h"
 #include "clang/Frontend/ASTUnit.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/TextDiagnosticPrinter.h"
 #include "clang/Index/USRGeneration.h"
-#include "llvm/ADT/Triple.h"
+#include "llvm/ADT/Optional.h"
 #include "llvm/ADT/Statistic.h"
+#include "llvm/ADT/Triple.h"
+#include "llvm/Option/ArgList.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/ManagedStatic.h"
 #include "llvm/Support/Path.h"
+#include "llvm/Support/YAMLParser.h"
 #include "llvm/Support/raw_ostream.h"
+#include <algorithm>
 #include <fstream>
 #include <sstream>
 
 namespace clang {
 namespace cross_tu {
 
 namespace {
 
 #define DEBUG_TYPE "CrossTranslationUnit"
 STATISTIC(NumGetCTUCalled, "The # of getCTUDefinition function called");
std::string message(int Condition) const override {
     case index_error_code::triple_mismatch:
       return "Triple mismatch";
     case index_error_code::lang_mismatch:
       return "Language mismatch";
     case index_error_code::lang_dialect_mismatch:
       return "Language dialect mismatch";
     case index_error_code::load_threshold_reached:
       return "Load threshold reached";
+    case index_error_code::invocation_list_ambiguous:
+      return "Invocation list file contains multiple references to the same "
+             "source file.";
+    case index_error_code::invocation_list_file_not_found:
+      return "Invocation list file is not found.";
+    case index_error_code::invocation_list_empty:
+      return "Invocation list file is empty.";
+    case index_error_code::invocation_list_wrong_format:
+      return "Invocation list file is in wrong format.";
+    case index_error_code::invocation_list_lookup_unsuccessful:
+      return "Invocation list file does not contain the requested source file.";
     }
     llvm_unreachable("Unrecognized index_error_code.");
   }
 };
 
 static llvm::ManagedStatic<IndexErrorCategory> Category;
 } // end anonymous namespace
 
 char IndexError::ID;
 
 void IndexError::log(raw_ostream &OS) const {
   OS << Category->message(static_cast<int>(Code)) << '\n';
 }
 
 std::error_code IndexError::convertToErrorCode() const {
   return std::error_code(static_cast<int>(Code), *Category);
 }
 
 llvm::Expected<llvm::StringMap<std::string>>
-parseCrossTUIndex(StringRef IndexPath, StringRef CrossTUDir) {
+parseCrossTUIndex(StringRef IndexPath) {
   std::ifstream ExternalMapFile{std::string(IndexPath)};
   if (!ExternalMapFile)
     return llvm::make_error<IndexError>(index_error_code::missing_index_file,
                                         IndexPath.str());
 
   llvm::StringMap<std::string> Result;
   std::string Line;
   unsigned LineNo = 1;
   while (std::getline(ExternalMapFile, Line)) {
-    const size_t Pos = Line.find(" ");
-    if (Pos > 0 && Pos != std::string::npos) {
-      StringRef LineRef{Line};
-      StringRef LookupName = LineRef.substr(0, Pos);
-      if (Result.count(LookupName))
+    const size_t Delimiter = LineRef.find(" ");
+    if (Delimiter > 0 && Delimiter != std::string::npos) {
+      StringRef LookupName = LineRef.substr(0, Delimiter);
+
+      // Store paths with posix-style directory separator.
+      SmallVector<char, 32> FilePath;
+      llvm::Twine{LineRef.substr(Delimiter + 1)}.toVector(FilePath);
+      llvm::sys::path::native(FilePath, llvm::sys::path::Style::posix);
+
+      bool InsertionOccured;
+      std::tie(std::ignore, InsertionOccured) =
+          Result.try_emplace(LookupName, FilePath.begin(), FilePath.end());
+      if (!InsertionOccured)
         return llvm::make_error<IndexError>(
             index_error_code::multiple_definitions, IndexPath.str(), LineNo);
-      StringRef FileName = LineRef.substr(Pos + 1);
-      SmallString<256> FilePath = CrossTUDir;
-      llvm::sys::path::append(FilePath, FileName);
-      Result[LookupName] = std::string(FilePath);
     } else
       return llvm::make_error<IndexError>(
           index_error_code::invalid_index_format, IndexPath.str(), LineNo);
-    LineNo++;
+    ++LineNo;
   }
   return Result;
 }
 
 std::string
 createCrossTUIndexString(const llvm::StringMap<std::string> &Index) {
   std::ostringstream Result;
   for (const auto &E : Index)
void CrossTranslationUnitContext::emitCrossTUDiagnostics(const IndexError &IE) {
   case index_error_code::triple_mismatch:
     Context.getDiagnostics().Report(diag::warn_ctu_incompat_triple)
         << IE.getFileName() << IE.getTripleToName() << IE.getTripleFromName();
     break;
   default:
     break;
   }
 }
 
-CrossTranslationUnitContext::ASTFileLoader::ASTFileLoader(
-    const CompilerInstance &CI)
-    : CI(CI) {}
-
-std::unique_ptr<ASTUnit>
-CrossTranslationUnitContext::ASTFileLoader::operator()(StringRef ASTFilePath) {
-  // Load AST from ast-dump.
-  IntrusiveRefCntPtr<DiagnosticOptions> DiagOpts = new DiagnosticOptions();
-  TextDiagnosticPrinter *DiagClient =
-      new TextDiagnosticPrinter(llvm::errs(), &*DiagOpts);
-  IntrusiveRefCntPtr<DiagnosticIDs> DiagID(new DiagnosticIDs());
-  IntrusiveRefCntPtr<DiagnosticsEngine> Diags(
-      new DiagnosticsEngine(DiagID, &*DiagOpts, DiagClient));
-
-  return ASTUnit::LoadFromASTFile(
-      std::string(ASTFilePath), CI.getPCHContainerOperations()->getRawReader(),
-      ASTUnit::LoadEverything, Diags, CI.getFileSystemOpts());
-}
-
 CrossTranslationUnitContext::ASTUnitStorage::ASTUnitStorage(
-    const CompilerInstance &CI)
-    : FileAccessor(CI), LoadGuard(const_cast<CompilerInstance &>(CI)
-                                      .getAnalyzerOpts()
-                                      ->CTUImportThreshold) {}
+    CompilerInstance &CI)
+    : Loader(CI, CI.getAnalyzerOpts()->CTUDir,
+             CI.getAnalyzerOpts()->CTUInvocationList),
+      LoadGuard(CI.getAnalyzerOpts()->CTUImportThreshold) {}
 
 llvm::Expected<ASTUnit *>
 CrossTranslationUnitContext::ASTUnitStorage::getASTUnitForFile(
     StringRef FileName, bool DisplayCTUProgress) {
   // Try the cache first.
   auto ASTCacheEntry = FileASTUnitMap.find(FileName);
   if (ASTCacheEntry == FileASTUnitMap.end()) {
 
     // Do not load if the limit is reached.
     if (!LoadGuard) {
       ++NumASTLoadThresholdReached;
       return llvm::make_error<IndexError>(
           index_error_code::load_threshold_reached);
     }
 
-    // Load the ASTUnit from the pre-dumped AST file specified by ASTFileName.
-    std::unique_ptr<ASTUnit> LoadedUnit = FileAccessor(FileName);
+    auto LoadAttempt = Loader.load(FileName);
+
+    if (!LoadAttempt)
+      return LoadAttempt.takeError();
+
+    std::unique_ptr<ASTUnit> LoadedUnit = std::move(LoadAttempt.get());
 
     // Need the raw pointer and the unique_ptr as well.
     ASTUnit *Unit = LoadedUnit.get();
 
     // Update the cache.
     FileASTUnitMap[FileName] = std::move(LoadedUnit);
 
     LoadGuard.indicateLoadSuccess();
 
     if (DisplayCTUProgress)
       llvm::errs() << "CTU loaded AST file: " << FileName << "\n";
 
     return Unit;
 
   } else {
     // Found in the cache.
     return ASTCacheEntry->second.get();
   }
 }
 
 llvm::Expected<ASTUnit *>
 CrossTranslationUnitContext::ASTUnitStorage::getASTUnitForFunction(
     StringRef FunctionName, StringRef CrossTUDir, StringRef IndexName,
     bool DisplayCTUProgress) {
   // Try the cache first.
   auto ASTCacheEntry = NameASTUnitMap.find(FunctionName);
if (ASTCacheEntry == NameASTUnitMap.end()) {
 
     // Search in the index for the filename where the definition of FuncitonName
     // resides.
     if (llvm::Expected<ASTUnit *> FoundForFile =
             getASTUnitForFile(NameFileMap[FunctionName], DisplayCTUProgress)) {
 
       // Update the cache.
       NameASTUnitMap[FunctionName] = *FoundForFile;
       return *FoundForFile;
 
     } else {
       return FoundForFile.takeError();
     }
   } else {
     // Found in the cache.
     return ASTCacheEntry->second;
   }
 }
 
 llvm::Expected<std::string>
 CrossTranslationUnitContext::ASTUnitStorage::getFileForFunction(
     StringRef FunctionName, StringRef CrossTUDir, StringRef IndexName) {
   if (llvm::Error IndexLoadError = ensureCTUIndexLoaded(CrossTUDir, IndexName))
     return std::move(IndexLoadError);
   return NameFileMap[FunctionName];
 }
 
 llvm::Error CrossTranslationUnitContext::ASTUnitStorage::ensureCTUIndexLoaded(
     StringRef CrossTUDir, StringRef IndexName) {
   // Dont initialize if the map is filled.
   if (!NameFileMap.empty())
     return llvm::Error::success();
 
   // Get the absolute path to the index file.
   SmallString<256> IndexFile = CrossTUDir;
   if (llvm::sys::path::is_absolute(IndexName))
     IndexFile = IndexName;
   else
     llvm::sys::path::append(IndexFile, IndexName);
 
-  if (auto IndexMapping = parseCrossTUIndex(IndexFile, CrossTUDir)) {
+  if (auto IndexMapping = parseCrossTUIndex(IndexFile)) {
     // Initialize member map.
     NameFileMap = *IndexMapping;
     return llvm::Error::success();
   } else {
     // Error while parsing CrossTU index file.
     return IndexMapping.takeError();
   };
 }
 
 llvm::Expected<ASTUnit *> CrossTranslationUnitContext::loadExternalAST(
     StringRef LookupName, StringRef CrossTUDir, StringRef IndexName,
     bool DisplayCTUProgress) {
   // FIXME: The current implementation only supports loading decls with
   //        a lookup name from a single translation unit. If multiple
   //        translation units contains decls with the same lookup name an
   //        error will be returned.
 
   // Try to get the value from the heavily cached storage.
   llvm::Expected<ASTUnit *> Unit = ASTStorage.getASTUnitForFunction(
       LookupName, CrossTUDir, IndexName, DisplayCTUProgress);
 
   if (!Unit)
     return Unit.takeError();
 
   // Check whether the backing pointer of the Expected is a nullptr.
   if (!*Unit)
     return llvm::make_error<IndexError>(
         index_error_code::failed_to_get_external_ast);
 
   return Unit;
 }
 
+CrossTranslationUnitContext::ASTLoader::ASTLoader(
+    CompilerInstance &CI, StringRef CTUDir, StringRef InvocationListFilePath)
+    : CI(CI), CTUDir(CTUDir), InvocationListFilePath(InvocationListFilePath) {}
+
+CrossTranslationUnitContext::LoadResultTy
+CrossTranslationUnitContext::ASTLoader::load(StringRef Identifier) {
+  llvm::SmallString<256> Path;
+  if (llvm::sys::path::is_absolute(Identifier, PathStyle)) {
+    Path = Identifier;
+  } else {
+    Path = CTUDir;
+    llvm::sys::path::append(Path, PathStyle, Identifier);
+  }
+
+  // The path is stored in the InvocationList member in posix style. To
+  // successfully lookup an entry based on filepath, it must be converted.
+  llvm::sys::path::native(Path, PathStyle);
+
+  // Normalize by removing relative path components.
+  llvm::sys::path::remove_dots(Path, /*remove_dot_dot*/ true, PathStyle);
+
+  if (Path.endswith(".ast"))
+    return loadFromDump(Path);
+  else
+    return loadFromSource(Path);
+}
+
+CrossTranslationUnitContext::LoadResultTy
+CrossTranslationUnitContext::ASTLoader::loadFromDump(StringRef ASTDumpPath) {
+  IntrusiveRefCntPtr<DiagnosticOptions> DiagOpts = new DiagnosticOptions();
+  TextDiagnosticPrinter *DiagClient =
+      new TextDiagnosticPrinter(llvm::errs(), &*DiagOpts);
+  IntrusiveRefCntPtr<DiagnosticIDs> DiagID(new DiagnosticIDs());
+  IntrusiveRefCntPtr<DiagnosticsEngine> Diags(
+      new DiagnosticsEngine(DiagID, &*DiagOpts, DiagClient));
+  return ASTUnit::LoadFromASTFile(
+      std::string(ASTDumpPath.str()),
+      CI.getPCHContainerOperations()->getRawReader(), ASTUnit::LoadEverything,
+      Diags, CI.getFileSystemOpts());
+}
+
+/// Load the AST from a source-file, which is supposed to be located inside the
+/// YAML formatted invocation list file under the filesystem path specified by
+/// \p InvocationList. The invocation list should contain absolute paths.
+/// \p SourceFilePath is the absolute path of the source file that contains the
+/// function definition the analysis is looking for. The Index is built by the
+/// \p clang-extdef-mapping tool, which is also supposed to be generating
+/// absolute paths.
+///
+/// Proper diagnostic emission requires absolute paths, so even if a future
+/// change introduces the handling of relative paths, this must be taken into
+/// consideration.
+CrossTranslationUnitContext::LoadResultTy
+CrossTranslationUnitContext::ASTLoader::loadFromSource(
+    StringRef SourceFilePath) {
+
+  if (llvm::Error InitError = lazyInitInvocationList())
+    return std::move(InitError);
+  assert(InvocationList);
+
+  auto Invocation = InvocationList->find(SourceFilePath);
+  if (Invocation == InvocationList->end())
+    return llvm::make_error<IndexError>(
+        index_error_code::invocation_list_lookup_unsuccessful);
+
+  const InvocationListTy::mapped_type &InvocationCommand = Invocation->second;
+
+  SmallVector<const char *, 32> CommandLineArgs(InvocationCommand.size());
+  std::transform(InvocationCommand.begin(), InvocationCommand.end(),
+                 CommandLineArgs.begin(),
+                 [](auto &&CmdPart) { return CmdPart.c_str(); });
+
+  IntrusiveRefCntPtr<DiagnosticOptions> DiagOpts{&CI.getDiagnosticOpts()};
+  auto *DiagClient = new ForwardingDiagnosticConsumer{CI.getDiagnosticClient()};
+  IntrusiveRefCntPtr<DiagnosticIDs> DiagID{
+      CI.getDiagnostics().getDiagnosticIDs()};
+  IntrusiveRefCntPtr<DiagnosticsEngine> Diags(
+      new DiagnosticsEngine{DiagID, &*DiagOpts, DiagClient});
+
+  return std::unique_ptr<ASTUnit>(ASTUnit::LoadFromCommandLine(
+      CommandLineArgs.begin(), (CommandLineArgs.end()),
+      CI.getPCHContainerOperations(), Diags,
+      CI.getHeaderSearchOpts().ResourceDir));
+}
+
+llvm::Expected<InvocationListTy>
+parseInvocationList(StringRef FileContent, llvm::sys::path::Style PathStyle) {
+  InvocationListTy InvocationList;
+
+  /// LLVM YAML parser is used to extract information from invocation list file.
+  llvm::SourceMgr SM;
+  llvm::yaml::Stream InvocationFile(FileContent, SM);
+
+  /// Only the first document is processed.
+  llvm::yaml::document_iterator FirstInvocationFile = InvocationFile.begin();
+
+  /// There has to be at least one document available.
+  if (FirstInvocationFile == InvocationFile.end())
+    return llvm::make_error<IndexError>(
+        index_error_code::invocation_list_empty);
+
+  llvm::yaml::Node *DocumentRoot = FirstInvocationFile->getRoot();
+  if (!DocumentRoot)
+    return llvm::make_error<IndexError>(
+        index_error_code::invocation_list_wrong_format);
+
+  /// According to the format specified the document must be a mapping, where
+  /// the keys are paths to source files, and values are sequences of invocation
+  /// parts.
+  auto *Mappings = dyn_cast<llvm::yaml::MappingNode>(DocumentRoot);
+  if (!Mappings)
+    return llvm::make_error<IndexError>(
+        index_error_code::invocation_list_wrong_format);
+
+  for (auto &NextMapping : *Mappings) {
+    /// The keys should be strings, which represent a source-file path.
+    auto *Key = dyn_cast<llvm::yaml::ScalarNode>(NextMapping.getKey());
+    if (!Key)
+      return llvm::make_error<IndexError>(
+          index_error_code::invocation_list_wrong_format);
+
+    /// The values should be sequences of strings, each representing a part of
+    /// the invocation.
+    auto *Args = dyn_cast<llvm::yaml::SequenceNode>(NextMapping.getValue());
+    if (!Args)
+      return llvm::make_error<IndexError>(
+          index_error_code::invocation_list_wrong_format);
+
+    SmallVector<char, 32> ValueStorage;
+    StringRef SourcePath = Key->getValue(ValueStorage);
+
+    // Store paths with PathStyle directory separator.
+    SmallVector<char, 32> NativeSourcePath;
+    llvm::Twine{SourcePath}.toVector(NativeSourcePath);
+    llvm::sys::path::native(NativeSourcePath, PathStyle);
+
+    StringRef InvocationKey{NativeSourcePath.begin(), NativeSourcePath.size()};
+
+    for (auto &Arg : *Args) {
+      auto *CmdString = dyn_cast<llvm::yaml::ScalarNode>(&Arg);
+      if (!CmdString)
+        return llvm::make_error<IndexError>(
+            index_error_code::invocation_list_wrong_format);
+      /// Every conversion starts with an empty working storage, as it is not
+      /// clear if this is a requirement of the YAML parser.
+      ValueStorage.clear();
+      InvocationList[InvocationKey].emplace_back(
+          CmdString->getValue(ValueStorage));
+    }
+
+    if (InvocationList[InvocationKey].empty())
+      return llvm::make_error<IndexError>(
+          index_error_code::invocation_list_wrong_format);
+  }
+
+  return InvocationList;
+}
+
+llvm::Error CrossTranslationUnitContext::ASTLoader::lazyInitInvocationList() {
+  /// Lazily initialize the invocation list member used for on-demand parsing.
+  if (InvocationList)
+    return llvm::Error::success();
+
+  llvm::ErrorOr<std::unique_ptr<llvm::MemoryBuffer>> FileContent =
+      llvm::MemoryBuffer::getFile(InvocationListFilePath);
+  if (!FileContent)
+    return llvm::make_error<IndexError>(
+        index_error_code::invocation_list_file_not_found);
+  std::unique_ptr<llvm::MemoryBuffer> ContentBuffer = std::move(*FileContent);
+  assert(ContentBuffer && "If no error was produced after loading, the pointer "
+                          "should not be nullptr.");
+
+  llvm::Expected<InvocationListTy> ExpectedInvocationList =
+      parseInvocationList(ContentBuffer->getBuffer(), PathStyle);
+
+  if (!ExpectedInvocationList)
+    return ExpectedInvocationList.takeError();
+
+  InvocationList = *ExpectedInvocationList;
+
+  return llvm::Error::success();
+}
+
 template <typename T>
 llvm::Expected<const T *>
 CrossTranslationUnitContext::importDefinitionImpl(const T *D, ASTUnit *Unit) {
   assert(hasBodyOrInit(D) && "Decls to be imported should have body or init.");
 
   assert(&D->getASTContext() == &Unit->getASTContext() &&
          "ASTContext of Decl and the unit should match.");
   ASTImporter &Importer = getOrCreateASTImporter(Unit);

clang/test/Analysis/Inputs/ctu-other.c

struct S {
   int a;
 };
 int g(struct S *ctx) {
   MYMACRO(ctx);
   return 0;
 }
 
 // Test that asm import does not fail.
+// TODO: Support the GNU extension asm keyword as well.
+// Example using the GNU extension: asm("mov $42, %0" : "=r"(res));
 int inlineAsm() {
   int res;
-  asm("mov $42, %0"
+  __asm__("mov $42, %0"
           : "=r"(res));
   return res;
 }
 
 // Implicit function.
 int identImplicit(int in) {
   return in;
 }
 
 // ASTImporter doesn't support this construct.
 int structInProto(struct DataType {int a;int b; } * d) {
   return 0;
 }

clang/test/Analysis/Inputs/ctu-other.c.externalDefMap.ast-dump.txt

 c:@F@inlineAsm ctu-other.c.ast
 c:@F@g ctu-other.c.ast
 c:@F@f ctu-other.c.ast
 c:@F@enumCheck ctu-other.c.ast
 c:@F@identImplicit ctu-other.c.ast
 c:@F@structInProto ctu-other.c.ast

clang/test/Analysis/Inputs/ctu-other.cpp.externalDefMap.ast-dump.txt

 c:@N@chns@F@chf1#I# ctu-other.cpp.ast
 c:@N@myns@N@embed_ns@F@fens#I# ctu-other.cpp.ast
 c:@F@g#I# ctu-other.cpp.ast
 c:@S@mycls@F@fscl#I#S ctu-other.cpp.ast
 c:@S@mycls@F@fcl#I# ctu-other.cpp.ast
 c:@S@mycls@F@fvcl#I# ctu-other.cpp.ast
 c:@N@myns@S@embed_cls@F@fecl#I# ctu-other.cpp.ast
 c:@S@mycls@S@embed_cls2@F@fecl2#I# ctu-other.cpp.ast
 c:@S@derived@F@fvcl#I# ctu-other.cpp.ast
 c:@F@f#I# ctu-other.cpp.ast
 c:@N@myns@F@fns#I# ctu-other.cpp.ast
 c:@F@h#I# ctu-other.cpp.ast
 c:@F@h_chain#I# ctu-chain.cpp.ast
 c:@N@chns@S@chcls@F@chf4#I# ctu-chain.cpp.ast
 c:@N@chns@F@chf2#I# ctu-chain.cpp.ast
 c:@F@fun_using_anon_struct#I# ctu-other.cpp.ast
 c:@F@other_macro_diag#I# ctu-other.cpp.ast
 c:@extInt ctu-other.cpp.ast
 c:@N@intns@extInt ctu-other.cpp.ast
 c:@extS ctu-other.cpp.ast
 c:@S@A@a ctu-other.cpp.ast
 c:@extSC ctu-other.cpp.ast
 c:@S@ST@sc ctu-other.cpp.ast
 c:@extSCN ctu-other.cpp.ast
 c:@extSubSCN ctu-other.cpp.ast
 c:@extSCC ctu-other.cpp.ast
 c:@extU ctu-other.cpp.ast
 c:@S@TestAnonUnionUSR@Test ctu-other.cpp.ast
 c:@F@testImportOfIncompleteDefaultParmDuringImport#I# ctu-other.cpp.ast
 c:@F@testImportOfDelegateConstructor#I# ctu-other.cpp.ast
 No newline at end of file

clang/test/Analysis/analyzer-config.c

 // CHECK-NEXT: core.CallAndMessage:NilReceiver = true
 // CHECK-NEXT: core.CallAndMessage:ParameterCount = true
 // CHECK-NEXT: core.CallAndMessage:UndefReceiver = true
 // CHECK-NEXT: cplusplus.Move:WarnOn = KnownsAndLocals
 // CHECK-NEXT: crosscheck-with-z3 = false
 // CHECK-NEXT: ctu-dir = ""
 // CHECK-NEXT: ctu-import-threshold = 100
 // CHECK-NEXT: ctu-index-name = externalDefMap.txt
+// CHECK-NEXT: ctu-invocation-list = invocations.yaml
 // CHECK-NEXT: deadcode.DeadStores:ShowFixIts = false
 // CHECK-NEXT: deadcode.DeadStores:WarnForDeadNestedAssignments = true
 // CHECK-NEXT: debug.AnalysisOrder:* = false
 // CHECK-NEXT: debug.AnalysisOrder:Bind = false
 // CHECK-NEXT: debug.AnalysisOrder:EndAnalysis = false
 // CHECK-NEXT: debug.AnalysisOrder:EndFunction = false
 // CHECK-NEXT: debug.AnalysisOrder:LiveSymbols = false
 // CHECK-NEXT: debug.AnalysisOrder:NewAllocator = false

clang/test/Analysis/ctu-different-triples.cpp

 // RUN: rm -rf %t && mkdir %t
 // RUN: mkdir -p %t/ctudir
 // RUN: %clang_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
 // RUN:   -emit-pch -o %t/ctudir/ctu-other.cpp.ast %S/Inputs/ctu-other.cpp
-// RUN: cp %S/Inputs/ctu-other.cpp.externalDefMap.txt %t/ctudir/externalDefMap.txt
+// RUN: cp %S/Inputs/ctu-other.cpp.externalDefMap.ast-dump.txt %t/ctudir/externalDefMap.txt
 // RUN: %clang_analyze_cc1 -std=c++14 -triple powerpc64-montavista-linux-gnu \
 // RUN:   -analyzer-checker=core,debug.ExprInspection \
 // RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
 // RUN:   -analyzer-config ctu-dir=%t/ctudir \
 // RUN:   -Werror=ctu \
 // RUN:   -verify %s
 
 // We expect an error in this file, but without a location.
 // expected-error-re@./ctu-different-triples.cpp:*{{imported AST from {{.*}} had been generated for a different target, current: powerpc64-montavista-linux-gnu, imported: x86_64-pc-linux-gnu}}
 
 int f(int);
 
 int main() {
   return f(5);
 }

clang/test/Analysis/ctu-main.c

 // RUN: rm -rf %t && mkdir %t
 // RUN: mkdir -p %t/ctudir2
 // RUN: %clang_cc1 -triple x86_64-pc-linux-gnu \
 // RUN:   -emit-pch -o %t/ctudir2/ctu-other.c.ast %S/Inputs/ctu-other.c
-// RUN: cp %S/Inputs/ctu-other.c.externalDefMap.txt %t/ctudir2/externalDefMap.txt
+// RUN: cp %S/Inputs/ctu-other.c.externalDefMap.ast-dump.txt %t/ctudir2/externalDefMap.txt
 // RUN: %clang_cc1 -triple x86_64-pc-linux-gnu -fsyntax-only -std=c89 -analyze \
 // RUN:   -analyzer-checker=core,debug.ExprInspection \
 // RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
 // RUN:   -analyzer-config ctu-dir=%t/ctudir2 \
 // RUN:   -verify %s
 
 void clang_analyzer_eval(int);
 
void testMacro(void) {
   g(0); // expected-warning@Inputs/ctu-other.c:29 {{Access to field 'a' results in a dereference of a null pointer (loaded from variable 'ctx')}}
 }
 
 // The external function prototype is incomplete.
 // warning:implicit functions are prohibited by c99
 void testImplicit() {
   int res = identImplicit(6);   // external implicit functions are not inlined
   clang_analyzer_eval(res == 6); // expected-warning{{TRUE}}
+  // Call something with uninitialized from the same function in which the implicit was called.
+  // This is necessary to reproduce a special bug in NoStoreFuncVisitor.
+  int uninitialized;
+  h(uninitialized); // expected-warning{{1st function call argument is an uninitialized value}}
 }
 
 // Tests the import of functions that have a struct parameter
 // defined in its prototype.
 struct DataType {
   int a;
   int b;
 };
 int structInProto(struct DataType *d);
 void testStructDefInArgument() {
   struct DataType d;
   d.a = 1;
   d.b = 0;
   clang_analyzer_eval(structInProto(&d) == 0); // expected-warning{{TRUE}} expected-warning{{FALSE}}
 }

clang/test/Analysis/ctu-main.cpp

 // RUN: rm -rf %t && mkdir %t
 // RUN: mkdir -p %t/ctudir
 // RUN: %clang_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
 // RUN:   -emit-pch -o %t/ctudir/ctu-other.cpp.ast %S/Inputs/ctu-other.cpp
 // RUN: %clang_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
 // RUN:   -emit-pch -o %t/ctudir/ctu-chain.cpp.ast %S/Inputs/ctu-chain.cpp
-// RUN: cp %S/Inputs/ctu-other.cpp.externalDefMap.txt %t/ctudir/externalDefMap.txt
+// RUN: cp %S/Inputs/ctu-other.cpp.externalDefMap.ast-dump.txt %t/ctudir/externalDefMap.txt
 // RUN: %clang_analyze_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
 // RUN:   -analyzer-checker=core,debug.ExprInspection \
 // RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
 // RUN:   -analyzer-config ctu-dir=%t/ctudir \
 // RUN:   -verify %s
 // RUN: %clang_analyze_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
 // RUN:   -analyzer-checker=core,debug.ExprInspection \
 // RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \

clang/test/Analysis/ctu-on-demand-parsing.c

+// RUN: rm -rf %t
+// RUN: mkdir -p %t
+// RUN: cp "%s" "%t/ctu-on-demand-parsing.c"
+// RUN: cp "%S/Inputs/ctu-other.c" "%t/ctu-other.c"
+//
+// Path substitutions on Windows platform could contain backslashes. These are escaped in the json file.
+// compile_commands.json is only needed for extdef_mapping, not for the analysis itself.
+// RUN: echo '[{"directory":"%t","command":"gcc -std=c89 -Wno-visibility ctu-other.c","file":"ctu-other.c"}]' | sed -e 's/\\/\\\\/g' > %t/compile_commands.json
+//
+// RUN: echo '"%t/ctu-other.c": ["gcc", "-std=c89", "-Wno-visibility", "ctu-other.c"]' | sed -e 's/\\/\\\\/g' > %t/invocations.yaml
+//
+// RUN: cd "%t" && %clang_extdef_map "%t/ctu-other.c" > externalDefMap.txt
+//
+// RUN: cd "%t" && %clang_cc1 -fsyntax-only -std=c89 -analyze \
+// RUN:   -analyzer-checker=core,debug.ExprInspection \
+// RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
+// RUN:   -analyzer-config ctu-dir=. \
+// RUN:   -analyzer-config ctu-invocation-list=invocations.yaml \
+// RUN:   -verify ctu-on-demand-parsing.c
+
+void clang_analyzer_eval(int);
+
+// Test typedef and global variable in function.
+typedef struct {
+  int a;
+  int b;
+} FooBar;
+extern FooBar fb;
+int f(int);
+void testGlobalVariable() {
+  clang_analyzer_eval(f(5) == 1); // expected-warning{{TRUE}}
+}
+
+// Test enums.
+int enumCheck(void);
+enum A { x,
+         y,
+         z };
+void testEnum() {
+  clang_analyzer_eval(x == 0);            // expected-warning{{TRUE}}
+  clang_analyzer_eval(enumCheck() == 42); // expected-warning{{TRUE}}
+}
+
+// Test that asm import does not fail.
+int inlineAsm();
+int testInlineAsm() { return inlineAsm(); }
+
+// Test reporting error in a macro.
+struct S;
+int g(struct S *);
+void testMacro(void) {
+  g(0);
+  // expected-warning@ctu-other.c:29 {{Access to field 'a' results in a dereference of a null pointer (loaded from variable 'ctx')}}
+}
+
+// The external function prototype is incomplete.
+// warning:implicit functions are prohibited by c99
+void testImplicit() {
+  int res = identImplicit(6);    // external implicit functions are not inlined
+  clang_analyzer_eval(res == 6); // expected-warning{{TRUE}}
+  // Call something with uninitialized from the same function in which the
+  // implicit was called. This is necessary to reproduce a special bug in
+  // NoStoreFuncVisitor.
+  int uninitialized;
+  h(uninitialized); // expected-warning{{1st function call argument is an uninitialized value}}
+}
+
+// Tests the import of functions that have a struct parameter
+// defined in its prototype.
+struct DataType {
+  int a;
+  int b;
+};
+int structInProto(struct DataType *d);
+void testStructDefInArgument() {
+  struct DataType d;
+  d.a = 1;
+  d.b = 0;
+  clang_analyzer_eval(structInProto(&d) == 0); // expected-warning{{TRUE}} expected-warning{{FALSE}}
+}

clang/test/Analysis/ctu-on-demand-parsing.cpp

+// RUN: rm -rf %t
+// RUN: mkdir -p %t/Inputs
+// RUN: cp %s %t/ctu-on-demand-parsing.cpp
+// RUN: cp %S/ctu-hdr.h %t/ctu-hdr.h
+// RUN: cp %S/Inputs/ctu-chain.cpp %t/Inputs/ctu-chain.cpp
+// RUN: cp %S/Inputs/ctu-other.cpp %t/Inputs/ctu-other.cpp
+//
+// Path substitutions on Windows platform could contain backslashes. These are escaped in the json file.
+// compile_commands.json is only needed for the extdef_mapping, not for the analysis itself.
+// RUN: echo '[{"directory":"%t/Inputs","command":"clang++ ctu-chain.cpp","file":"ctu-chain.cpp"},{"directory":"%t/Inputs","command":"clang++ ctu-other.cpp","file":"ctu-other.cpp"}]' | sed -e 's/\\/\\\\/g' > %t/compile_commands.json
+//
+// RUN: echo '{"%t/Inputs/ctu-chain.cpp": ["g++", "%t/Inputs/ctu-chain.cpp"], "%t/Inputs/ctu-other.cpp": ["g++", "%t/Inputs/ctu-other.cpp"]}' | sed -e 's/\\/\\\\/g' > %t/invocations.yaml
+//
+// RUN: cd "%t" && %clang_extdef_map Inputs/ctu-chain.cpp Inputs/ctu-other.cpp > externalDefMap.txt
+//
+// RUN: cd "%t" && %clang_analyze_cc1 \
+// RUN:   -analyzer-checker=core,debug.ExprInspection \
+// RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
+// RUN:   -analyzer-config ctu-dir=. \
+// RUN:   -analyzer-config ctu-invocation-list=invocations.yaml \
+// RUN:   -verify ctu-on-demand-parsing.cpp
+// RUN: cd "%t" && %clang_analyze_cc1 \
+// RUN:   -analyzer-checker=core,debug.ExprInspection \
+// RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
+// RUN:   -analyzer-config ctu-dir=. \
+// RUN:   -analyzer-config ctu-invocation-list=invocations.yaml \
+// RUN:   -analyzer-config display-ctu-progress=true ctu-on-demand-parsing.cpp 2>&1 | FileCheck %t/ctu-on-demand-parsing.cpp
+
+// CHECK: CTU loaded AST file: {{.*}}ctu-other.cpp
+// CHECK: CTU loaded AST file: {{.*}}ctu-chain.cpp
+
+#include "ctu-hdr.h"
+
+void clang_analyzer_eval(int);
+
+int f(int);
+int g(int);
+int h(int);
+
+int callback_to_main(int x) { return x + 1; }
+
+namespace myns {
+int fns(int x);
+
+namespace embed_ns {
+int fens(int x);
+}
+
+class embed_cls {
+public:
+  int fecl(int x);
+};
+} // namespace myns
+
+class mycls {
+public:
+  int fcl(int x);
+  virtual int fvcl(int x);
+  static int fscl(int x);
+
+  class embed_cls2 {
+  public:
+    int fecl2(int x);
+  };
+};
+
+class derived : public mycls {
+public:
+  virtual int fvcl(int x) override;
+};
+
+namespace chns {
+int chf1(int x);
+}
+
+int fun_using_anon_struct(int);
+int other_macro_diag(int);
+
+void test_virtual_functions(mycls *obj) {
+  // The dynamic type is known.
+  clang_analyzer_eval(mycls().fvcl(1) == 8);   // expected-warning{{TRUE}}
+  clang_analyzer_eval(derived().fvcl(1) == 9); // expected-warning{{TRUE}}
+  // We cannot decide about the dynamic type.
+  clang_analyzer_eval(obj->fvcl(1) == 8); // expected-warning{{FALSE}} expected-warning{{TRUE}}
+  clang_analyzer_eval(obj->fvcl(1) == 9); // expected-warning{{FALSE}} expected-warning{{TRUE}}
+}
+
+int main() {
+  clang_analyzer_eval(f(3) == 2); // expected-warning{{TRUE}}
+  clang_analyzer_eval(f(4) == 3); // expected-warning{{TRUE}}
+  clang_analyzer_eval(f(5) == 3); // expected-warning{{FALSE}}
+  clang_analyzer_eval(g(4) == 6); // expected-warning{{TRUE}}
+  clang_analyzer_eval(h(2) == 8); // expected-warning{{TRUE}}
+
+  clang_analyzer_eval(myns::fns(2) == 9);                   // expected-warning{{TRUE}}
+  clang_analyzer_eval(myns::embed_ns::fens(2) == -1);       // expected-warning{{TRUE}}
+  clang_analyzer_eval(mycls().fcl(1) == 6);                 // expected-warning{{TRUE}}
+  clang_analyzer_eval(mycls::fscl(1) == 7);                 // expected-warning{{TRUE}}
+  clang_analyzer_eval(myns::embed_cls().fecl(1) == -6);     // expected-warning{{TRUE}}
+  clang_analyzer_eval(mycls::embed_cls2().fecl2(0) == -11); // expected-warning{{TRUE}}
+
+  clang_analyzer_eval(chns::chf1(4) == 12);           // expected-warning{{TRUE}}
+  clang_analyzer_eval(fun_using_anon_struct(8) == 8); // expected-warning{{TRUE}}
+
+  clang_analyzer_eval(other_macro_diag(1) == 1); // expected-warning{{TRUE}}
+  // expected-warning@Inputs/ctu-other.cpp:93{{REACHABLE}}
+  MACRODIAG(); // expected-warning{{REACHABLE}}
+}

clang/test/Analysis/ctu-unknown-parts-in-triples.cpp

 // We do not expect any error when one part of the triple is unknown, but other
 // known parts are equal.
 
 // RUN: rm -rf %t && mkdir %t
 // RUN: mkdir -p %t/ctudir
 // RUN: %clang_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
 // RUN:   -emit-pch -o %t/ctudir/ctu-other.cpp.ast %S/Inputs/ctu-other.cpp
-// RUN: cp %S/Inputs/ctu-other.cpp.externalDefMap.txt %t/ctudir/externalDefMap.txt
+// RUN: cp %S/Inputs/ctu-other.cpp.externalDefMap.ast-dump.txt %t/ctudir/externalDefMap.txt
 // RUN: %clang_analyze_cc1 -std=c++14 -triple x86_64-unknown-linux-gnu \
 // RUN:   -analyzer-checker=core,debug.ExprInspection \
 // RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
 // RUN:   -analyzer-config ctu-dir=%t/ctudir \
 // RUN:   -Werror=ctu \
 // RUN:   -verify %s
 
 // expected-no-diagnostics
 
 int f(int);
 
 int main() {
   return f(5);
 }

clang/unittests/CrossTU/CrossTranslationUnitTest.cpp

 //===- unittest/Tooling/CrossTranslationUnitTest.cpp - Tooling unit tests -===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "clang/CrossTU/CrossTranslationUnit.h"
-#include "clang/Frontend/CompilerInstance.h"
 #include "clang/AST/ASTConsumer.h"
+#include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/FrontendAction.h"
 #include "clang/Tooling/Tooling.h"
+#include "llvm/ADT/Optional.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/ToolOutputFile.h"
 #include "gtest/gtest.h"
 #include <cassert>
 
 namespace clang {
 namespace cross_tu {
TEST(CrossTranslationUnit, IndexFormatCanBeParsed) {
   llvm::SmallString<256> IndexFileName;
   ASSERT_FALSE(llvm::sys::fs::createTemporaryFile("index", "txt", IndexFD,
                                                   IndexFileName));
   llvm::ToolOutputFile IndexFile(IndexFileName, IndexFD);
   IndexFile.os() << IndexText;
   IndexFile.os().flush();
   EXPECT_TRUE(llvm::sys::fs::exists(IndexFileName));
   llvm::Expected<llvm::StringMap<std::string>> IndexOrErr =
-      parseCrossTUIndex(IndexFileName, "");
+      parseCrossTUIndex(IndexFileName);
   EXPECT_TRUE((bool)IndexOrErr);
   llvm::StringMap<std::string> ParsedIndex = IndexOrErr.get();
   for (const auto &E : Index) {
     EXPECT_TRUE(ParsedIndex.count(E.getKey()));
     EXPECT_EQ(ParsedIndex[E.getKey()], E.getValue());
   }
   for (const auto &E : ParsedIndex)
     EXPECT_TRUE(Index.count(E.getKey()));
 }
 
-TEST(CrossTranslationUnit, CTUDirIsHandledCorrectly) {
-  llvm::StringMap<std::string> Index;
-  Index["a"] = "/b/c/d";
-  std::string IndexText = createCrossTUIndexString(Index);
+TEST(CrossTranslationUnit, EmptyInvocationListIsNotValid) {
+  auto Input = "";
 
-  int IndexFD;
-  llvm::SmallString<256> IndexFileName;
-  ASSERT_FALSE(llvm::sys::fs::createTemporaryFile("index", "txt", IndexFD,
-                                                  IndexFileName));
-  llvm::ToolOutputFile IndexFile(IndexFileName, IndexFD);
-  IndexFile.os() << IndexText;
-  IndexFile.os().flush();
-  EXPECT_TRUE(llvm::sys::fs::exists(IndexFileName));
-  llvm::Expected<llvm::StringMap<std::string>> IndexOrErr =
-      parseCrossTUIndex(IndexFileName, "/ctudir");
-  EXPECT_TRUE((bool)IndexOrErr);
-  llvm::StringMap<std::string> ParsedIndex = IndexOrErr.get();
-  EXPECT_EQ(ParsedIndex["a"], "/ctudir/b/c/d");
+  llvm::Expected<InvocationListTy> Result = parseInvocationList(Input);
+  EXPECT_FALSE(static_cast<bool>(Result));
+  bool IsWrongFromatError = false;
+  llvm::handleAllErrors(Result.takeError(), [&](IndexError &Err) {
+    IsWrongFromatError =
+        Err.getCode() == index_error_code::invocation_list_wrong_format;
+  });
+  EXPECT_TRUE(IsWrongFromatError);
+}
+
+TEST(CrossTranslationUnit, AmbiguousInvocationListIsDetected) {
+  // The same source file occurs twice (for two different architecture) in
+  // this test case. The disambiguation is the responsibility of the user.
+  auto Input = R"(
+  /tmp/main.cpp:
+    - clang++
+    - -c
+    - -m32
+    - -o
+    - main32.o
+    - /tmp/main.cpp
+  /tmp/main.cpp:
+    - clang++
+    - -c
+    - -m64
+    - -o
+    - main64.o
+    - /tmp/main.cpp
+  )";
+
+  llvm::Expected<InvocationListTy> Result = parseInvocationList(Input);
+  EXPECT_FALSE(static_cast<bool>(Result));
+  bool IsAmbiguousError = false;
+  llvm::handleAllErrors(Result.takeError(), [&](IndexError &Err) {
+    IsAmbiguousError =
+        Err.getCode() == index_error_code::invocation_list_ambiguous;
+  });
+  EXPECT_TRUE(IsAmbiguousError);
+}
+
+TEST(CrossTranslationUnit, SingleInvocationCanBeParsed) {
+  auto Input = R"(
+  /tmp/main.cpp:
+    - clang++
+    - /tmp/main.cpp
+  )";
+  llvm::Expected<InvocationListTy> Result = parseInvocationList(Input);
+  EXPECT_TRUE(static_cast<bool>(Result));
+
+  EXPECT_EQ(Result->size(), 1);
+
+  auto It = Result->find("/tmp/main.cpp");
+  EXPECT_TRUE(It != Result->end());
+  EXPECT_EQ(It->getValue()[0], "clang++");
+  EXPECT_EQ(It->getValue()[1], "/tmp/main.cpp");
+}
+
+TEST(CrossTranslationUnit, MultipleInvocationsCanBeParsed) {
+  auto Input = R"(
+  /tmp/main.cpp:
+    - clang++
+    - /tmp/other.o
+    - /tmp/main.cpp
+  /tmp/other.cpp:
+    - g++
+    - -c
+    - -o
+    - /tmp/other.o
+    - /tmp/other.cpp
+  )";
+  llvm::Expected<InvocationListTy> Result = parseInvocationList(Input);
+  EXPECT_TRUE(static_cast<bool>(Result));
+
+  EXPECT_EQ(Result->size(), 2);
+
+  auto It = Result->find("/tmp/main.cpp");
+  EXPECT_TRUE(It != Result->end());
+  EXPECT_EQ(It->getKey(), "/tmp/main.cpp");
+  EXPECT_EQ(It->getValue()[0], "clang++");
+  EXPECT_EQ(It->getValue()[1], "/tmp/other.o");
+  EXPECT_EQ(It->getValue()[2], "/tmp/main.cpp");
+
+  It = Result->find("/tmp/other.cpp");
+  EXPECT_TRUE(It != Result->end());
+  EXPECT_EQ(It->getValue()[0], "g++");
+  EXPECT_EQ(It->getValue()[1], "-c");
+  EXPECT_EQ(It->getValue()[2], "-o");
+  EXPECT_EQ(It->getValue()[3], "/tmp/other.o");
+  EXPECT_EQ(It->getValue()[4], "/tmp/other.cpp");
 }
 
 } // end namespace cross_tu
 } // end namespace clang