[analyzer] On-demand parsing capability for CTU

[gamesh411]

Summary

Add an option to enable on-demand parsing of needed ASTs during CTU analysis.
Two options are introduced. CTUOnDemandParsing enables the feature, and
CTUOnDemandParsingDatabase specifies the path to a compilation database, which
has all the necessary information to generate the ASTs.

[martong]

clang/lib/CrossTU/CrossTranslationUnit.cpp

402

CI.getPCHContainerOperations() is probably not needed, because we are not going to exercise the ASTReader, so this could be a nullptr (default param value).

[martong]

clang/lib/StaticAnalyzer/Core/CallEvent.cpp

577

left here some debugging

[balazske]

clang/include/clang/CrossTU/CrossTranslationUnit.h

221

class would look better here? (The descendants are class too.)

224

Probably it is good to mention in the documentation that the function is used with a string read from a file and the type of the file determines the meaning of the Identifier here (the calling code does have no direct knowledge about it).

clang/lib/CrossTU/CrossTranslationUnit.cpp

449

matches

515

Members of ASTOnDemandLoader would be better in a single group in the source file.

[martong]

clang/lib/CrossTU/CrossTranslationUnit.cpp

358

Could we check somehow if CTUDir is indeed an absolute path? If not then probably we should bail out with an error. Though I am not sure if there is an easy and portable way in llvm:: to check for beeing an absolute path.

364

There is no variable named as OnDemandParsingCommands, perhaps you made a rename but forgot to rename in the comments?

373

There is no variable named as ASTSourcePath, perhaps you made a rename but forgot to rename in the comments?

402

If CI.getPCHContainerOperations() is not needed then we can remove the CI member of ASTOnDemandLoader.

432

Perhaps we should make sure that Identifier is indeed an absolute path and if not then we should emit an error. If a user do not use CodeChecker to generate the ExternalDefMapping it may contain relative paths. See e.g.: https://clang.llvm.org/docs/analyzer/user-docs/CrossTranslationUnit.html#manual-ctu-analysis

446

Could we have a lit test for this case? I.e having a compile_commands.json with two ambiguous entries for the same file and then we should expect the compiler diag?
(Note, I am asking this because it is not immediate for me from the code that this will ever fire.)

clang/test/Analysis/ctu-main.c

53

Is this hunk related?

clang/test/Analysis/ctu-on-demand-parsing.c

3

Why do we need sed here? I don't see any \ in the compile commands json. Is it because of Windows builds? Could you please explain in a comment maybe in the previous line.

[gamesh411]

Refactored the test
Copying the compile_commands.json in case of test for C files
results in the extdef mapping tool finding the correct compiler
flags for the C to be imported.

[gamesh411]

In D75665#1907914, @martong wrote:

The warning below suggests that we parse the ctu-other.c file (and presumably every file) as a C++ file, even if it should be parsed as a C file. Perhaps the invocation of the parser is missing some setting? Also, could this be the reason why on-the-fly and pch driven ctu gives different statistics?

warning: treating 'c' input as 'c++' when in C++ mode, this behavior is deprecated [-Wdeprecated]
/mnt/disks/ssd0/agent/workspace/amd64_debian_testing_clang8/clang/test/Analysis/Inputs/ctu-other.c:47:26: error: 'DataType' cannot be defined in a parameter type
int structInProto(struct DataType {int a;int b; } * d) {
                         ^
1 error generated.
Error while processing /mnt/disks/ssd0/agent/workspace/amd64_debian_testing_clang8/clang/test/Analysis/Inputs/ctu-other.c.

I have investigated the issue, and the compiler flags were looked up using the heuristic implemented in tooling.
This heuristic looks for the suitable compilation database in an upward ascending fashion inside the directory tree startin from the input source file.
By copying both the source file and the compilation database to the test directory this heuristic does the right thing now. (up until now the found compile_commands.json was the one used for llvm-project itself, and picked up a c++ specific compilation).

This issue is solved, however the ASTImporter still cannot import the inline definition of the struct, and emits an error.
Right now I am debugging the master branch that uses the AST-dumps wheter I also encounter this error inside the ASTImporter (just to see if I am operating on sane assumptions).

[gamesh411]

In D75665#1907938, @martong wrote:

As the CTUDir string prefix is only needed in case of the old approach, I have refactored the external definition mapping storage to NOT include that.

Both ctu-main.c and ctu-on-demand-parsing.c use the -analyzer-config ctu-dir=%t/ctudir2 setting. So it is not clear for me why we don't need the prefix. I believe we still need that directory setting otherwise where could we find the externalDefMap file?

You are right, the ctudir2 prefix is not needed as the %t substitution is test-file-level unique.

[martong]

In D75665#1971440, @gamesh411 wrote:

In D75665#1907914, @martong wrote:

The warning below suggests that we parse the ctu-other.c file (and presumably every file) as a C++ file, even if it should be parsed as a C file. Perhaps the invocation of the parser is missing some setting? Also, could this be the reason why on-the-fly and pch driven ctu gives different statistics?

warning: treating 'c' input as 'c++' when in C++ mode, this behavior is deprecated [-Wdeprecated]
/mnt/disks/ssd0/agent/workspace/amd64_debian_testing_clang8/clang/test/Analysis/Inputs/ctu-other.c:47:26: error: 'DataType' cannot be defined in a parameter type
int structInProto(struct DataType {int a;int b; } * d) {
                         ^
1 error generated.
Error while processing /mnt/disks/ssd0/agent/workspace/amd64_debian_testing_clang8/clang/test/Analysis/Inputs/ctu-other.c.

I have investigated the issue, and the compiler flags were looked up using the heuristic implemented in tooling.
This heuristic looks for the suitable compilation database in an upward ascending fashion inside the directory tree startin from the input source file.
By copying both the source file and the compilation database to the test directory this heuristic does the right thing now. (up until now the found compile_commands.json was the one used for llvm-project itself, and picked up a c++ specific compilation).

This issue is solved, however the ASTImporter still cannot import the inline definition of the struct, and emits an error.
Right now I am debugging the master branch that uses the AST-dumps wheter I also encounter this error inside the ASTImporter (just to see if I am operating on sane assumptions).

I've been looking into this and the ASTImporter indeed does not (cannot) import the definition of structInProto(). But that's just fine, that is why we have a branching for the call expression (TRUE/FALSE) below in ctu-main.c . All other functions whose definition were imported gives only the TRUE branch.

clang_analyzer_eval(structInProto(&d) == 0); // expected-warning{{TRUE}} expected-warning{{FALSE}}

[gamesh411]

Reorganize test code to overcome testing infrastructure limitations

[martong]

The tests are getting better, thanks Endre!

clang/test/Analysis/ctu-on-demand-parsing.c

6

Why do we need the -Wno-visibility flag?

7

If we execute clang_extdef_map then we don't need Inputs/ctu-other.c[pp].externalDefMap.on-the-fly.txt files. Do we?

[gamesh411]

Answered most review comments. Thanks for the reviewers @balazske, @martong so far.
The question of absolute path policy is still up for debate.

clang/lib/CrossTU/CrossTranslationUnit.cpp

358

Actually this does not need to be absolute. The naming of the variable should be changed to PrefixedPath or similar. I will do that for now.

358

It not necessarily an absolute path, only prefixed by CTUDir. There is no clear policy as to where to use strictly absolute paths and where to accept relative paths. IMHO this should be considered, and I am most certainly open for a discussion. In the meantime, I will just rename the variable to PrefixedPath.

402

As for the automatic lifetime of Tool, I am inclined to say this is not the case. I have run my clang-build with sanitizers (undefined and memory), and no bad access or UB was reported. If someone has more domain knowledge about tooling and the lifetime issues that could arise, I would be thrilled to be enlightened.

402

As for the need to pass PCHContainerOperaitons:
I think you are right, and I will check if this modifies the behaviour of the import in any way. I am removing this for now.

432

The policy of absolute and relative paths should be discussed. See my previous answer concerning AbsPath variable above. BTW llvm::sys::path::is_absolute is a solution for asserting this as I see it.

clang/test/Analysis/ctu-main.c

53

Seems unrelated, not sure how it got included. Fixing it.

clang/test/Analysis/ctu-on-demand-parsing.c

3

I have done this by example. Some CodeGen lit tests use sed the same way I did here. Needed for Windows compatibility maybe? (note that %t could substitute into a path containing backslashes on Windows) However, on Windows platforms, the availability of sed is not guaranteed either. The backslashes are used as escape chars in the JSON, hence the need to double-escape them. No comments are provided on those test also... I will add something tho (and also in the cpp test).

[gamesh411]

clang/test/Analysis/ctu-on-demand-parsing.c

6

Needed because the in-expression definition of a struct (which we cannot import) also warrants a compiler warning, and this comes up 2 times even (once when generating the extdef-mapping, second time when we analyze the file). Its easier to handle this warning in the compilation database, as that removes both occurrences of the warning.

7

Cannot remove as of yet, because they have .ast suffixed entries. In order to remove them the whole logic of what should be inside the index (externalDefMap.txt) for on-demand and non-on-demand cases should be discussed.

[gamesh411]

Implement review suggestions

[gamesh411]

clang/lib/CrossTU/CrossTranslationUnit.cpp

446

Good idea, I will create a standalone test case for this!

[gamesh411]

Rebase and update the number of analyzer options

[gamesh411]

Add ambiguous compilation database test case

[gamesh411]

Update user documentation

[martong]

Thanks Endre for the docs! I checked the CI job also, seemed okay, so, I think we are ready! Nice work! Let's do the commit!

clang/docs/analyzer/user-docs/CrossTranslationUnit.rst

26

Maybe just simply write This tool uses a compilation command database ...
(Same below with the on-demand version.)

[xazax.hun]

clang/include/clang/CrossTU/CrossTranslationUnit.h

56

The two enum values refer to compilation database and compile command database. I'd prefer to use the same wording in both values.

227

I am not sure if this is good design.
Here, if the meaning of the Identifier depends on the subclass, the caller of this method always needs to be aware of the dynamic type of the object. What is the point of a common base class if we always need to know the dynamic type?

Looking at the code this does not look bad. But it might be a code smell.

285

Why is this no longer const?

clang/lib/CrossTU/CrossTranslationUnit.cpp

120

Unfortunately, this is a very common case for a large number of projects that supports multiple targets (e.g. 32 and 64 bits). Is there a plan to mitigate this problem?

[martong]

clang/include/clang/CrossTU/CrossTranslationUnit.h

56

+1

227

The way how we process the extDefMapping file is identical in both cases. That's an index, keyed with the USRs of functions and then we get back a value. And the way how we use that value is different. In the PCH case that holds the path for the .ast file, in the ODM case that is the name of the source file which we must find in the compile db. So, I think the process of getting the AST for a USR requires the polymorphic behavior from the loaders.

We discussed other alternatives with Endre. We were thinking that maybe the extDefMapping file should be identical in both cases. But then we would need to add the .ast postfixes for the entries in the PCH case. And we cannot just do that, because we may not know if what is the correct postfix. The user may have generated .pch files instead. Also, we don't want to compel any Clang user to use CodeChecker (CC will always create .ast files). CTU should be running fine by manually executing the independent steps.

clang/lib/CrossTU/CrossTranslationUnit.cpp

120

I don't think we could do that. We need to merge ASTs of those TUs that are linkable. Otherwise the merge will be unsuccessful because of certain mismatches in Decls (structural eq will fail).
Even in CodeChecker we are planning to issue a hard error in these ambiguous cases, even with the PCH method too. (The error would be suppressible by the user of course, but if that is suppressed then we take no responsibility.)

[xazax.hun]

Ok, looks good to me.

The minor nit regarding the naming is easy to fix before commit. The design question I had is not a blocker, my suggested alternative can be implemented later (if desired) in a backward-compatible way from the user's point of view.

clang/include/clang/CrossTU/CrossTranslationUnit.h

227

Let me rephrase my concerns a bit. Do we really need a polymorphic ASTLoader to be present for the whole analysis? Wouldn't it make more sense to always do the same thing, i.e. if we are given a pch file load it, if we are given a source file, parse it? This way we would not be restricted to on-demand or two pass ctu analysis, but we could do any combination of the two.

[whisperity]

clang/docs/analyzer/user-docs/CrossTranslationUnit.rst

21

I think it's PCH-based with a -.

25

symbols' USR names

26

clang tool

26

And perhaps add some cross-references to what a compilation database is, etc. These things are also documented within Clang's doc tree.

31

Instead of a prefix, create an overview of PCH-based analysis, and add this and the next one as a 3rd-level heading?

110–113

Are these flags documented somewhere?

207–208

Same here for comments from above.

311–315

These lines could be removed, I think, they don't add anything to the presentation.

316–317

Due to the lack of colours, perhaps you could use ls -F which suffixes each directory with a /: reports/.

367

as a runner script the driver for this feature

368

expect it to work only ...

(to emphasise the limitation)

clang/lib/CrossTU/CrossTranslationUnit.cpp

120

More clever logging of link commands and creating separate CDBs for each target or each binary could help this cause, but that needs infrastructural capabilities from both the build system and the analysis driver (i.e. CodeChecker).

[martong]

clang/include/clang/CrossTU/CrossTranslationUnit.h

227

Well yeah, we could do that, it is a good idea, thanks! We will consider this in the future. I like in this idea that the command line options to Clang would be simplified. But then we must be transparent and show/log the user which method we are using.

[whisperity]

clang/docs/analyzer/user-docs/CrossTranslationUnit.rst

353

What's this line? Is this added by Phab, or is this a weird merge conflict / parsing error of the patch?

[gamesh411]

Fix issues in documentation

[gamesh411]

@whisperity Thanks for the review :3

clang/docs/analyzer/user-docs/CrossTranslationUnit.rst

110–113

As with ctu-dir, these flags are only documented in analyzer options. However, it would indeed be beneficial to list CTU related flags here in the user documentation. I will create a new revision for that specifically.

311–315

Good point, i remove it from here and from the above example as well.

[gamesh411]

Reword enum value

[gamesh411]

Fix index error enum declaration

[gamesh411]

I am landing this. Thanks for the reviews @martong @balazske @xazax.hun @whisperity !

[thakis]

clang/lib/CrossTU/CMakeLists.txt

13

We've been very careful to make clang (the compiler binary) only depend on clangToolingCore and not on the much bigger clangTooling library. Since clang depends on the CrossTU library, this breaks that. Can you reorganize things so that we don't need a dependency from clang-the-compiler-binary on clangTooling?

[thakis]

Looks like this also breaks tests on Windows and Mac, eg http://45.33.8.238/win/13902/step_7.txt

It probably makes sense to revert while you investigate?

[gribozavr2]

Sorry, this change broke a couple bots that are in the official CI, at least http://lab.llvm.org:8011/builders/llvm-clang-win-x-aarch64/builds/7566 and http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-scei-ps4-ubuntu-fast/builds/66618. I'm reverting this change -- feel free to reland after fixing the issue.

[gribozavr2]

Reverted in https://github.com/llvm/llvm-project/commit/96717125e852d1c6ddf41c22dd2d556f4f5aa34d.

[gamesh411]

Non-linux buildbots were utterly broken :( . Trying to fix them...

Thanks for reverting this for me :)
Also I will investigate the possibility to not depend on Tooling.

[gamesh411]

Remove platform constraint from test file

[gamesh411]

[NFC] Fix arcanist double commit revisioning

[gamesh411]

clang/lib/CrossTU/CMakeLists.txt

13

I have gone, and investigated where the main functionality of on-demand AST loading comes from. (Only after wishfully thinking changing the dependency from clangTooling to clangToolingCore would work, which of course didnt't )

It seems that this functionality used the major components of clangTooling, namely JSONCompilationDatabase::loadFromFile, ClangTool ctor/dtor, ClangTool::buildAST.
In order to circumvent the dependency and the reimplementation of handling compilation databases, a call to an external tool could be used, or the clangTooling library would need to be refactored in a way which is not a very tasteful refactoring. IMHO. I would like to know which of the aforementioned way is deemed to be best.

[gamesh411]

Remove arch target from another invocation

[gamesh411]

Implement a solution without a dependency on clangTooling

[balazske]

clang/lib/CrossTU/CrossTranslationUnit.cpp

358–377

Is here a lambda necessary? I think it must not be over-used. In this case we can construct the file path as the first step and then use it to load the file, without using lambda call.

581

Is here a special reason to use auto && type (and not auto & or std::string &)? Probably this code is faulty: The CmdPart is moved out and then destructed, before the content of it (where the c_str points to) is used?

[whisperity]

Please check the summary of the patch, it seems to contain old information as well.

clang/docs/analyzer/user-docs/CrossTranslationUnit.rst

210–212

This is obsolete information.

clang/include/clang/CrossTU/CrossTranslationUnit.h

234

Is the explicit needed here?

clang/include/clang/StaticAnalyzer/Core/AnalyzerOptions.def

388–389

"valid compilation database" is this line still what we wish to say? A YAML is used, so most likely we need a "valid invocation list" or something.

393–398

Maybe it would be worthwhile to add a dummy example like main.cpp: g++ hello-world.cpp which has the proper format into this help.

clang/lib/CrossTU/CrossTranslationUnit.cpp

21–25

The idea here was to not depend on libtooling but these lines are still here!

32

<> -> "" and order

119–120

still using terms "compilation database", is this intended?

581

@balazske This is a generic lambda where auto&& is the universal reference. It is equivalent to writing template <typename T> f(T&& param). It obeys the usual point of instantiation and reference cascading rules, if a const std::string& is given as T, const std::string& && will become a single &.

607

Are the headers from which these types come from included?

clang/test/Analysis/Inputs/ctu-other.c

34–38

Possibly unrelated change?

clang/test/Analysis/ctu-on-demand-parsing.c

2–16

Does the LIT syntax allow for better organising these commands? This is now a bit hard to read

Something like this would help:

// RUN: rm -rf %t && mkdir -p %t
// RUN: cp "%s" "%t/ctu-on-demand-parsing.c" && cp "%S/Inputs/ctu-other.c" "%t/ctu-other.c"
// [empty]
// [comment]
// RUN: [json magic]
// RUN: [yaml magic]
// [empty]
// RUN: extdef map
// [empty]
// RUN: cc1

[whisperity]

(Maybe this will make Phab not show "✅ Accepted"...)

[martong]

Hi Endre, just checked the latest update. Overall looks good to me, but found some nits.

clang/include/clang/CrossTU/CrossTranslationUnit.h

260

Should we rename this member to ....PathTo....InvocationList... ?

clang/lib/CrossTU/CrossTranslationUnit.cpp

579

Could we avoid this transfer if InvocationList was storing const char * values instead of std::strings? Why can't we store char*s in InvocationList?

591

typo: filed -> field

clang/test/Analysis/ctu-on-demand-parsing.c

7

Perhaps we could document here that the compile_commands.json is needed ONLY for %clang_extdef_map.

clang/test/Analysis/ctu-on-demand-parsing.cpp

10

I know this is just a nit, but this is very hard to read. Do you think we could break this up (and other long lines) into multiple lines but within the same RUN directive?

[gamesh411]

Update functional changes, documentation update incoming

[gamesh411]

The remaining documentation and test changes are also underway.

clang/include/clang/CrossTU/CrossTranslationUnit.h

227

I will implement this separation by suffixes in the next change.

234

This constructor started out with having a single param only. I agree, that explicit is no longer necessary.

260

Renamed to InvocationListFilePath.

285

Most usages of the CI require a non-const ref (this is due to lacking API support for getting analyzer options), and I found it more convenient to just take a non-const ref, instead of const_casting it away later. Not sure myself, but I think I will roll with this now (and I should definitely check the API of CompilerInstance).

clang/lib/CrossTU/CrossTranslationUnit.cpp

119–120

renamed

120

I have solved the problem of supporting multiple architectures by offloading the responsibility to the user (to the tool driving the analysis). The invocation list file is now supposed to contain only invocations with the same arch/target and this should also be the same as the arch/target of the main TU analyzed. If there is still ambiguity after this separation, this error should nevertheless be emitted ( IMHO ).

358–377

Refactored to use path concatenation.

579

We certainly could, I think there won't be null-characters in paths so storing char* s would not lose generality. However, I think the memory management done by std::string is worth it, as it is clear where the character-data lives in memory.

581

Results of my research: In theory, the STL could do something fancy here if it wanted, but it is not the case in reality. If you want move-operations with algorithms (the ones not explicitly mentioning move in their names) you have to be explicit about it: move-iterators-fluent-cpp.

607

Included the llvm/Support/YAMLParser.h

clang/test/Analysis/Inputs/ctu-other.c

34–38

Funny enough, this is very relevant. Dumping the AST does not errors out on this construction, however, the parser in on-demand mode does. This asm syntax is however supported by both. To document this, I have added a TODO as well.

[whisperity]

clang/lib/CrossTU/CrossTranslationUnit.cpp

26

Duped include in L34.

[gamesh411]

Fix documentation and commit message

[martong]

LGTM!

[gamesh411]

Extend index file format
Update documentation

[gamesh411]

Fix test case, and reorder warning

[whisperity]

clang/lib/CrossTU/CrossTranslationUnit.cpp

121–122

These two lines could be formatted together.

clang/unittests/CrossTU/CrossTranslationUnitTest.cpp

176–177

Just for good measure, I would like a test case on the "invocation list is ambiguous" error. (So in case we in the future figure out how to disambiguate, we can remove that test case!)

[gamesh411]

use consumeError in test

[gamesh411]

add ambiguous invocation list test case

[gamesh411]

fix typo

[gamesh411]

add ambiguity checking during invocation list parsing

[thakis]

This breaks check-clang on mac: http://45.33.8.238/mac/15258/step_7.txt

Please take a look, and revert for now if it takes a while to fix.

[gamesh411]

In D75665#2084889, @thakis wrote:

This breaks check-clang on mac: http://45.33.8.238/mac/15258/step_7.txt

Please take a look, and revert for now if it takes a while to fix.

Thanks for reporting this, i think the output is valuable for debugging. I have restricted the test case to linux for now.

clang/docs/analyzer/user-docs/CrossTranslationUnit.rst

 =====================================
 Cross Translation Unit (CTU) Analysis
 =====================================
 
 Normally, static analysis works in the boundary of one translation unit (TU).
-However, with additional steps and configuration we can enable the analysis to inline the definition of a function from another TU.
+However, with additional steps and configuration we can enable the analysis to inline the definition of a function from
+another TU.
 
 .. contents::
    :local:
 
-Manual CTU Analysis
--------------------
+Overview
+________
+CTU analysis can be used in a variety of ways. The importing of external TU definitions can work with pre-dumped PCH
+files or generating the necessary AST structure on-demand, during the analysis of the main TU. Driving the static
+analysis can also be implemented in multiple ways. The most direct way is to specify the necessary commandline options
+of the Clang frontend manually (and generate the prerequisite dependencies of the specific import method by hand). This
+process can be automated by other tools, like `CodeChecker <https://github.com/Ericsson/codechecker>`_ and scan-build-py
+(preference for the former).
+
+PCH-based analysis
+__________________
+The analysis needs the PCH dumps of all the translations units used in the project.
+These can be generated by the Clang Frontend itself, and must be arranged in a specific way in the filesystem.
+The index, which maps symbols' USR names to PCH dumps containing them must also be generated by the
+`clang-extdef-mapping`. This tool uses a :doc:`compilation database <../../JSONCompilationDatabase>` to
+determine the compilation flags used.
+The analysis invocation must be provided with the directory which contains the dumps and the mapping files.
+
 
+Manual CTU Analysis
+###################
 Let's consider these source files in our minimal example:
 
 .. code-block:: cpp
 
   // main.cpp
   int foo();
 
   int main() {
[
     {
       "directory": "/path/to/your/project",
       "command": "clang++ -c main.cpp -o main.o",
       "file": "main.cpp"
     }
   ]
 
 We'd like to analyze `main.cpp` and discover the division by zero bug.
-In order to be able to inline the definition of `foo` from `foo.cpp` first we have to generate the `AST` (or `PCH`) file of `foo.cpp`:
+In order to be able to inline the definition of `foo` from `foo.cpp` first we have to generate the `AST` (or `PCH`) file
+of `foo.cpp`:
 
 .. code-block:: bash
 
   $ pwd $ /path/to/your/project
   $ clang++ -emit-ast -o foo.cpp.ast foo.cpp
   $ # Check that the .ast file is generated:
   $ ls
   compile_commands.json  foo.cpp.ast  foo.cpp  main.cpp
   $
 
-The next step is to create a CTU index file which holds the `USR` name and location of external definitions in the source files:
+The next step is to create a CTU index file which holds the `USR` name and location of external definitions in the
+source files:
 
 .. code-block:: bash
 
   $ clang-extdef-mapping -p . foo.cpp
   c:@F@foo# /path/to/your/project/foo.cpp
   $ clang-extdef-mapping -p . foo.cpp > externalDefMap.txt
 
 We have to modify `externalDefMap.txt` to contain the name of the `.ast` files instead of the source files:

 
 Now everything is available for the CTU analysis.
 We have to feed Clang with CTU specific extra arguments:
 
 .. code-block:: bash
 
   $ pwd
   /path/to/your/project
-  $ clang++ --analyze -Xclang -analyzer-config -Xclang experimental-enable-naive-ctu-analysis=true -Xclang -analyzer-config -Xclang ctu-dir=. -Xclang -analyzer-output=plist-multi-file main.cpp
+  $ clang++ --analyze \
+      -Xclang -analyzer-config -Xclang experimental-enable-naive-ctu-analysis=true \
+      -Xclang -analyzer-config -Xclang ctu-dir=. \
+      -Xclang -analyzer-config -Xclang ctu-on-demand-parsing=false \
+      -Xclang -analyzer-output=plist-multi-file \
+      main.cpp
   main.cpp:5:12: warning: Division by zero
     return 3 / foo();
            ~~^~~~~~~
   1 warning generated.
   $ # The plist file with the result is generated.
-  $ ls
+  $ ls -F
   compile_commands.json  externalDefMap.txt  foo.ast  foo.cpp  foo.cpp.ast  main.cpp  main.plist
   $
 
-This manual procedure is error-prone and not scalable, therefore to analyze real projects it is recommended to use `CodeChecker` or `scan-build-py`.
+This manual procedure is error-prone and not scalable, therefore to analyze real projects it is recommended to use
+`CodeChecker` or `scan-build-py`.
 
 Automated CTU Analysis with CodeChecker
----------------------------------------
+#######################################
 The `CodeChecker <https://github.com/Ericsson/codechecker>`_ project fully supports automated CTU analysis with Clang.
 Once we have set up the `PATH` environment variable and we activated the python `venv` then it is all it takes:
 
 .. code-block:: bash
 
   $ CodeChecker analyze --ctu compile_commands.json -o reports
-  [INFO 2019-07-16 17:21] - Pre-analysis started.
-  [INFO 2019-07-16 17:21] - Collecting data for ctu analysis.
-  [INFO 2019-07-16 17:21] - [1/2] foo.cpp
-  [INFO 2019-07-16 17:21] - [2/2] main.cpp
-  [INFO 2019-07-16 17:21] - Pre-analysis finished.
-  [INFO 2019-07-16 17:21] - Starting static analysis ...
-  [INFO 2019-07-16 17:21] - [1/2] clangsa analyzed foo.cpp successfully.
-  [INFO 2019-07-16 17:21] - [2/2] clangsa analyzed main.cpp successfully.
-  [INFO 2019-07-16 17:21] - ----==== Summary ====----
-  [INFO 2019-07-16 17:21] - Successfully analyzed
-  [INFO 2019-07-16 17:21] -   clangsa: 2
-  [INFO 2019-07-16 17:21] - Total analyzed compilation commands: 2
-  [INFO 2019-07-16 17:21] - ----=================----
-  [INFO 2019-07-16 17:21] - Analysis finished.
-  [INFO 2019-07-16 17:21] - To view results in the terminal use the "CodeChecker parse" command.
-  [INFO 2019-07-16 17:21] - To store results use the "CodeChecker store" command.
-  [INFO 2019-07-16 17:21] - See --help and the user guide for further options about parsing and storing the reports.
-  [INFO 2019-07-16 17:21] - ----=================----
-  [INFO 2019-07-16 17:21] - Analysis length: 0.659618854523 sec.
-  $ ls
-  compile_commands.json  foo.cpp  foo.cpp.ast  main.cpp  reports
+  $ ls -F
+  compile_commands.json  foo.cpp  foo.cpp.ast  main.cpp  reports/
   $ tree reports
   reports
   ├── compile_cmd.json
   ├── compiler_info.json
   ├── foo.cpp_53f6fbf7ab7ec9931301524b551959e2.plist
   ├── main.cpp_23db3d8df52ff0812e6e5a03071c8337.plist
   ├── metadata.json
   └── unique_compile_commands.json

 Or we can use `CodeChecker parse -e html` to export the results into HTML format:
 
 .. code-block:: bash
 
   $ CodeChecker parse -e html -o html_out reports
   $ firefox html_out/index.html
 
 Automated CTU Analysis with scan-build-py (don't do it)
--------------------------------------------------------
-We actively develop CTU with CodeChecker as a "runner" script, `scan-build-py` is not actively developed for CTU.
-`scan-build-py` has various errors and issues, expect it to work with the very basic projects only.
+#############################################################
+We actively develop CTU with CodeChecker as the driver for this feature, `scan-build-py` is not actively developed for CTU.
+`scan-build-py` has various errors and issues, expect it to work only with the very basic projects only.
 
 Example usage of scan-build-py:
 
 .. code-block:: bash
 
   $ /your/path/to/llvm-project/clang/tools/scan-build-py/bin/analyze-build --ctu
   analyze-build: Run 'scan-view /tmp/scan-build-2019-07-17-17-53-33-810365-7fqgWk' to examine bug reports.
   $ /your/path/to/llvm-project/clang/tools/scan-view/bin/scan-view /tmp/scan-build-2019-07-17-17-53-33-810365-7fqgWk
   Starting scan-view at: http://127.0.0.1:8181
     Use Ctrl-C to exit.
   [6336:6431:0717/175357.633914:ERROR:browser_process_sub_thread.cc(209)] Waited 5 ms for network service
   Opening in existing browser session.
   ^C
   $
+
+On-demand analysis
+__________________
+The analysis produces the necessary AST structure of external TUs during analysis. This requires the
+compilation database in order to determine the exact compiler invocation used for each TU.
+The index, which maps function USR names to source files containing them must also be generated by the
+`clang-extdef-mapping`. The mapping of external definitions implicitly uses a
+:doc:`compilation database <../../JSONCompilationDatabase>` to determine the compilation flags used.
+Preferably the same compilation database should be used when generating the external definitions, and
+during analysis.  The analysis invocation must be provided with the directory which contains the mapping
+files, and the compilation database which is used to determine compiler flags.
+
+
+Manual CTU Analysis
+###################
+
+Let's consider these source files in our minimal example:
+
+.. code-block:: cpp
+
+  // main.cpp
+  int foo();
+
+  int main() {
+    return 3 / foo();
+  }
+
+.. code-block:: cpp
+
+  // foo.cpp
+  int foo() {
+    return 0;
+  }
+
+And a compilation database:
+
+.. code-block:: bash
+
+  [
+    {
+      "directory": "/path/to/your/project",
+      "command": "clang++ -c foo.cpp -o foo.o",
+      "file": "foo.cpp"
+    },
+    {
+      "directory": "/path/to/your/project",
+      "command": "clang++ -c main.cpp -o main.o",
+      "file": "main.cpp"
+    }
+  ]
+
+We'd like to analyze `main.cpp` and discover the division by zero bug.
+As we are using On-demand mode, we only need to create a CTU index file which holds the `USR` name and location of
+external definitions in the source files:
+
+.. code-block:: bash
+
+  $ clang-extdef-mapping -p . foo.cpp
+  c:@F@foo# /path/to/your/project/foo.cpp
+  $ clang-extdef-mapping -p . foo.cpp > externalDefMap.txt
+
+Now everything is available for the CTU analysis.
+We have to feed Clang with CTU specific extra arguments:
+
+.. code-block:: bash
+
+  $ pwd
+  /path/to/your/project
+  $ clang++ --analyze \
+      -Xclang -analyzer-config -Xclang experimental-enable-naive-ctu-analysis=true \
+      -Xclang -analyzer-config -Xclang ctu-dir=. \
+      -Xclang -analyzer-config -Xclang ctu-on-demand-parsing=true \
+      -Xclang -analyzer-config -Xclang ctu-on-demand-parsing-database=compile_commands.json \
+      -Xclang -analyzer-output=plist-multi-file \
+      main.cpp
+  main.cpp:5:12: warning: Division by zero
+    return 3 / foo();
+           ~~^~~~~~~
+  1 warning generated.
+  $ # The plist file with the result is generated.
+  $ ls -F
+  compile_commands.json  externalDefMap.txt  foo.cpp  main.cpp  main.plist
+  $
+
+This manual procedure is error-prone and not scalable, therefore to analyze real projects it is recommended to use
+`CodeChecker` or `scan-build-py`.
+
+Automated CTU Analysis with CodeChecker
+#######################################
+The `CodeChecker <https://github.com/Ericsson/codechecker>`_ project fully supports automated CTU analysis with Clang.
+Once we have set up the `PATH` environment variable and we activated the python `venv` then it is all it takes:
+
+.. code-block:: bash
+
+  $ CodeChecker analyze --ctu --ctu-on-demand compile_commands.json -o reports
+  $ ls -F
+  compile_commands.json  foo.cpp main.cpp  reports/
+  $ tree reports
+  reports
+  ├── compile_cmd.json
+  ├── compiler_info.json
+  ├── foo.cpp_53f6fbf7ab7ec9931301524b551959e2.plist
+  ├── main.cpp_23db3d8df52ff0812e6e5a03071c8337.plist
+  ├── metadata.json
+  └── unique_compile_commands.json
+
+  0 directories, 6 files
+  $
+
+The `plist` files contain the results of the analysis, which may be viewed with the regular analysis tools.
+E.g. one may use `CodeChecker parse` to view the results in command line:
+
+.. code-block:: bash
+
+  $ CodeChecker parse reports
+  [HIGH] /home/egbomrt/ctu_mini_raw_project/main.cpp:5:12: Division by zero [core.DivideZero]
+    return 3 / foo();
+             ^
+
+  Found 1 defect(s) in main.cpp
+
+
+  ----==== Summary ====----
+  -----------------------
+  Filename | Report count
+  -----------------------
+  main.cpp |            1
+  -----------------------
+  -----------------------
+  Severity | Report count
+  -----------------------
+  HIGH     |            1
+  -----------------------
+  ----=================----
+  Total number of reports: 1
+  ----=================----
+
+Or we can use `CodeChecker parse -e html` to export the results into HTML format:
+
+.. code-block:: bash
+
+  $ CodeChecker parse -e html -o html_out reports
+  $ firefox html_out/index.html
+
+Automated CTU Analysis with scan-build-py (don't do it)
+#######################################################
+We actively develop CTU with CodeChecker as the driver for feature, `scan-build-py` is not actively developed for CTU.
+`scan-build-py` has various errors and issues, expect it to work only with the very basic projects only.
+
+Currently On-demand analysis is not supported with `scan-build-py`.
+

clang/include/clang/CrossTU/CrossTranslationUnit.h

 #define LLVM_CLANG_CROSSTU_CROSSTRANSLATIONUNIT_H
 
 #include "clang/AST/ASTImporterSharedState.h"
 #include "clang/Basic/LLVM.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/StringMap.h"
 #include "llvm/Support/Error.h"
 
 namespace clang {
 class CompilerInstance;
 class ASTContext;
 class ASTImporter;
 class ASTUnit;
 class DeclContext;
 class FunctionDecl;
 class VarDecl;
 class NamedDecl;
 class TranslationUnitDecl;
 
+namespace tooling {
+class JSONCompilationDatabase;
+}
+
 namespace cross_tu {
 
 enum class index_error_code {
   unspecified = 1,
   missing_index_file,
   invalid_index_format,
   multiple_definitions,
   missing_definition,
   failed_import,
+  failed_to_load_compilation_database,
   failed_to_get_external_ast,
   failed_to_generate_usr,
   triple_mismatch,
   lang_mismatch,
   lang_dialect_mismatch,
-  load_threshold_reached
+  load_threshold_reached,
+  ambiguous_compile_commands_database
 };
 
 class IndexError : public llvm::ErrorInfo<IndexError> {
 public:
   static char ID;
   IndexError(index_error_code C) : Code(C), LineNo(0) {}
   IndexError(index_error_code C, std::string FileName, int LineNo = 0)
       : Code(C), FileName(std::move(FileName)), LineNo(LineNo) {}
private:
   index_error_code Code;
   std::string FileName;
   int LineNo;
   std::string TripleToName;
   std::string TripleFromName;
 };
 
 /// This function parses an index file that determines which
-///        translation unit contains which definition.
+/// translation unit contains which definition. The IndexPath is not prefixed
+/// with CTUDir, so an absolute path is expected for consistent results.
 ///
 /// The index file format is the following:
 /// each line consists of an USR and a filepath separated by a space.
 ///
 /// \return Returns a map where the USR is the key and the filepath is the value
 ///         or an error.
 llvm::Expected<llvm::StringMap<std::string>>
-parseCrossTUIndex(StringRef IndexPath, StringRef CrossTUDir);
+parseCrossTUIndex(StringRef IndexPath);
 
 std::string createCrossTUIndexString(const llvm::StringMap<std::string> &Index);
 
 // Returns true if the variable or any field of a record variable is const.
 bool containsConst(const VarDecl *VD, const ASTContext &ACtx);
 
 /// This class is used for tools that requires cross translation
 ///        unit capability.
private:
   /// and the ASTUnit for the From context.
   /// This map is used by getImportedFromSourceLocation to lookup a FileID and
   /// its Preprocessor when knowing only the FileID in the 'To' context. The
   /// FileID could be imported by any of multiple 'From' ASTImporter objects.
   /// we do not want to loop over all ASTImporter's to find the one that
   /// imported the FileID.
   ImportedFileIDMap ImportedFileIDs;
 
-  /// Functor for loading ASTUnits from AST-dump files.
-  class ASTFileLoader {
+  using LoadResultTy = llvm::Expected<std::unique_ptr<ASTUnit>>;
+
+  class ASTLoader {
+  public:
+    /// Load the ASTUnit by an identifier. Subclasses should determine what this
+    /// would be. The function is used with a string read from the CTU index,
+    /// and the method used for loading determines the semantic meaning of
+    /// Identifier.
+    virtual LoadResultTy load(StringRef Identifier) = 0;
+    virtual ~ASTLoader() = default;
+  };
+
+  /// Implementation for loading ASTUnits from AST-dump files.
+  class ASTFileLoader : public ASTLoader {
+  public:
+    explicit ASTFileLoader(CompilerInstance &CI, StringRef CTUDir);
+
+    /// ASTFileLoader uses a the path of the dump file as Identifier.
+    LoadResultTy load(StringRef Identifier) override;
+
+  private:
+    CompilerInstance &CI;
+    StringRef CTUDir;
+  };
+
+  /// Implementation for loading ASTUnits by parsing them on-demand.
+  class ASTOnDemandLoader : public ASTLoader {
   public:
-    ASTFileLoader(const CompilerInstance &CI);
-    std::unique_ptr<ASTUnit> operator()(StringRef ASTFilePath);
+    ASTOnDemandLoader(StringRef OnDemandParsingDatabase);
+
+    /// ASTOnDemandLoader uses the path of the source file to be parsed as
+    /// Identifier.
+    LoadResultTy load(StringRef Identifier) override;
+
+    llvm::Error lazyInitCompileCommands();
 
   private:
-    const CompilerInstance &CI;
+    StringRef OnDemandParsingDatabase;
+    /// In case of on-demand parsing, the compilation database is parsed and
+    /// stored.
+    std::unique_ptr<tooling::JSONCompilationDatabase> CompileCommands;
   };
 
   /// Maintain number of AST loads and check for reaching the load limit.
   class ASTLoadGuard {
   public:
     ASTLoadGuard(unsigned Limit) : Limit(Limit) {}
 
     /// Indicates, whether a new load operation is permitted, it is within the
     /// threshold.
     operator bool() const { return Count < Limit; }
 
     /// Tell that a new AST was loaded successfully.
     void indicateLoadSuccess() { ++Count; }
 
   private:
     /// The number of ASTs actually imported.
     unsigned Count{0u};
     /// The limit (threshold) value for number of loaded ASTs.
     const unsigned Limit;
   };
 
   /// Storage and load of ASTUnits, cached access, and providing searchability
   /// are the concerns of ASTUnitStorage class.
   class ASTUnitStorage {
   public:
-    ASTUnitStorage(const CompilerInstance &CI);
+    ASTUnitStorage(CompilerInstance &CI);
     /// Loads an ASTUnit for a function.
     ///
     /// \param FunctionName USR name of the function.
     /// \param CrossTUDir Path to the directory used to store CTU related files.
     /// \param IndexName Name of the file inside \p CrossTUDir which maps
     /// function USR names to file paths. These files contain the corresponding
     /// AST-dumps.
     /// \param DisplayCTUProgress Display a message about loading new ASTs.
private:
     using NonOwningMapTy = BaseMapTy<clang::ASTUnit *>;
 
     OwningMapTy FileASTUnitMap;
     NonOwningMapTy NameASTUnitMap;
 
     using IndexMapTy = BaseMapTy<std::string>;
     IndexMapTy NameFileMap;
 
-    ASTFileLoader FileAccessor;
+    std::unique_ptr<ASTLoader> Loader;
 
-    /// Limit the number of loaded ASTs. Used to limit the  memory usage of the
-    /// CrossTranslationUnitContext.
-    /// The ASTUnitStorage has the knowledge about if the AST to load is
-    /// actually loaded or returned from cache. This information is needed to
-    /// maintain the counter.
+    /// Limit the number of loaded ASTs. It is used to limit the  memory usage
+    /// of the CrossTranslationUnitContext. The ASTUnitStorage has the
+    /// information whether the AST to load is actually loaded or returned from
+    /// cache. This information is needed to maintain the counter.
     ASTLoadGuard LoadGuard;
   };
 
   ASTUnitStorage ASTStorage;
-
 };
 
 } // namespace cross_tu
 } // namespace clang
 
 #endif // LLVM_CLANG_CROSSTU_CROSSTRANSLATIONUNIT_H

clang/include/clang/StaticAnalyzer/Core/AnalyzerOptions.def

 
 ANALYZER_OPTION(StringRef, CTUDir, "ctu-dir",
                 "The directory containing the CTU related files.", "")
 
 ANALYZER_OPTION(StringRef, CTUIndexName, "ctu-index-name",
                 "the name of the file containing the CTU index of definitions.",
                 "externalDefMap.txt")
 
+ANALYZER_OPTION(bool, CTUOnDemandParsing, "ctu-on-demand-parsing",
+                "Whether to parse function definitions from external TUs in "
+                "an on-demand manner during analysis. When using on-demand "
+                "parsing there is no need for pre-dumping ASTs. External "
+                "definition mapping is still needed, and a valid compilation "
+                "database with compile commands for the external TUs is also "
+                "necessary. Disabled by default.",
+                false)
+
+ANALYZER_OPTION(StringRef, CTUOnDemandParsingDatabase,
+                "ctu-on-demand-parsing-database",
+                "The path to the compilation database used for on-demand "
+                "parsing of ASTs during CTU analysis.",
+                "compile_commands.json")
+
 ANALYZER_OPTION(
     StringRef, ModelPath, "model-path",
     "The analyzer can inline an alternative implementation written in C at the "
     "call site if the called function's body is not available. This is a path "
     "where to look for those alternative implementations (called models).",
     "")
 
 ANALYZER_OPTION(

clang/lib/CrossTU/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   Support
   )
 
 add_clang_library(clangCrossTU
   CrossTranslationUnit.cpp
 
   LINK_LIBS
   clangAST
   clangBasic
   clangFrontend
   clangIndex
+  clangTooling
+  clangSerialization
   )

clang/lib/CrossTU/CrossTranslationUnit.cpp

 #include "clang/AST/ASTImporter.h"
 #include "clang/AST/Decl.h"
 #include "clang/Basic/TargetInfo.h"
 #include "clang/CrossTU/CrossTUDiagnostic.h"
 #include "clang/Frontend/ASTUnit.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/TextDiagnosticPrinter.h"
 #include "clang/Index/USRGeneration.h"
-#include "llvm/ADT/Triple.h"
+#include "clang/Tooling/JSONCompilationDatabase.h"
+#include "clang/Tooling/Tooling.h"
+#include "llvm/ADT/Optional.h"
 #include "llvm/ADT/Statistic.h"
+#include "llvm/ADT/Triple.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/ManagedStatic.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/raw_ostream.h"
+#include <algorithm>
 #include <fstream>
 #include <sstream>
 
 namespace clang {
 namespace cross_tu {
 
 namespace {
 
 #define DEBUG_TYPE "CrossTranslationUnit"
 STATISTIC(NumGetCTUCalled, "The # of getCTUDefinition function called");
std::string message(int Condition) const override {
     case index_error_code::multiple_definitions:
       return "Multiple definitions in the index file.";
     case index_error_code::missing_definition:
       return "Missing definition from the index file.";
     case index_error_code::failed_import:
       return "Failed to import the definition.";
     case index_error_code::failed_to_get_external_ast:
       return "Failed to load external AST source.";
+    case index_error_code::failed_to_load_compilation_database:
+      return "Failed to load compilation database.";
     case index_error_code::failed_to_generate_usr:
       return "Failed to generate USR.";
     case index_error_code::triple_mismatch:
       return "Triple mismatch";
     case index_error_code::lang_mismatch:
       return "Language mismatch";
     case index_error_code::lang_dialect_mismatch:
       return "Language dialect mismatch";
     case index_error_code::load_threshold_reached:
       return "Load threshold reached";
+    case index_error_code::ambiguous_compilation_database:
+      return "Compilation database contains multiple references to the same "
+             "source file.";
     }
     llvm_unreachable("Unrecognized index_error_code.");
   }
 };
 
 static llvm::ManagedStatic<IndexErrorCategory> Category;
 } // end anonymous namespace
 
 char IndexError::ID;
 
 void IndexError::log(raw_ostream &OS) const {
   OS << Category->message(static_cast<int>(Code)) << '\n';
 }
 
 std::error_code IndexError::convertToErrorCode() const {
   return std::error_code(static_cast<int>(Code), *Category);
 }
 
 llvm::Expected<llvm::StringMap<std::string>>
-parseCrossTUIndex(StringRef IndexPath, StringRef CrossTUDir) {
+parseCrossTUIndex(StringRef IndexPath) {
   std::ifstream ExternalMapFile{std::string(IndexPath)};
   if (!ExternalMapFile)
     return llvm::make_error<IndexError>(index_error_code::missing_index_file,
                                         IndexPath.str());
 
   llvm::StringMap<std::string> Result;
   std::string Line;
   unsigned LineNo = 1;
   while (std::getline(ExternalMapFile, Line)) {
     const size_t Pos = Line.find(" ");
     if (Pos > 0 && Pos != std::string::npos) {
       StringRef LineRef{Line};
       StringRef LookupName = LineRef.substr(0, Pos);
       if (Result.count(LookupName))
         return llvm::make_error<IndexError>(
             index_error_code::multiple_definitions, IndexPath.str(), LineNo);
       StringRef FileName = LineRef.substr(Pos + 1);
-      SmallString<256> FilePath = CrossTUDir;
-      llvm::sys::path::append(FilePath, FileName);
-      Result[LookupName] = std::string(FilePath);
+      Result[LookupName] = FileName.str();
     } else
       return llvm::make_error<IndexError>(
           index_error_code::invalid_index_format, IndexPath.str(), LineNo);
     LineNo++;
   }
   return Result;
 }
 
case index_error_code::triple_mismatch:
     Context.getDiagnostics().Report(diag::warn_ctu_incompat_triple)
         << IE.getFileName() << IE.getTripleToName() << IE.getTripleFromName();
     break;
   default:
     break;
   }
 }
 
-CrossTranslationUnitContext::ASTFileLoader::ASTFileLoader(
-    const CompilerInstance &CI)
-    : CI(CI) {}
+CrossTranslationUnitContext::ASTFileLoader::ASTFileLoader(CompilerInstance &CI,
+                                                          StringRef CTUDir)
+    : CI(CI), CTUDir(CTUDir) {}
 
-std::unique_ptr<ASTUnit>
-CrossTranslationUnitContext::ASTFileLoader::operator()(StringRef ASTFilePath) {
+CrossTranslationUnitContext::LoadResultTy
+CrossTranslationUnitContext::ASTFileLoader::load(StringRef Identifier) {
   // Load AST from ast-dump.
+
+  auto LoadFromFile = [this](StringRef Path) {
     IntrusiveRefCntPtr<DiagnosticOptions> DiagOpts = new DiagnosticOptions();
     TextDiagnosticPrinter *DiagClient =
         new TextDiagnosticPrinter(llvm::errs(), &*DiagOpts);
     IntrusiveRefCntPtr<DiagnosticIDs> DiagID(new DiagnosticIDs());
     IntrusiveRefCntPtr<DiagnosticsEngine> Diags(
         new DiagnosticsEngine(DiagID, &*DiagOpts, DiagClient));
-
     return ASTUnit::LoadFromASTFile(
-      std::string(ASTFilePath), CI.getPCHContainerOperations()->getRawReader(),
+        std::string(Path.str()), CI.getPCHContainerOperations()->getRawReader(),
         ASTUnit::LoadEverything, Diags, CI.getFileSystemOpts());
+  };
+
+  if (llvm::sys::path::is_absolute(Identifier))
+    return LoadFromFile(Identifier);
+
+  llvm::SmallString<256> PrefixedPath = CTUDir;
+  llvm::sys::path::append(PrefixedPath, Identifier);
+
+  return LoadFromFile(PrefixedPath);
 }
 
 CrossTranslationUnitContext::ASTUnitStorage::ASTUnitStorage(
-    const CompilerInstance &CI)
-    : FileAccessor(CI), LoadGuard(const_cast<CompilerInstance &>(CI)
-                                      .getAnalyzerOpts()
-                                      ->CTUImportThreshold) {}
+    CompilerInstance &CI)
+    : LoadGuard(CI.getAnalyzerOpts()->CTUImportThreshold) {
+
+  AnalyzerOptionsRef Opts = CI.getAnalyzerOpts();
+  if (Opts->CTUOnDemandParsing)
+    Loader =
+        std::make_unique<ASTOnDemandLoader>(Opts->CTUOnDemandParsingDatabase);
+  else
+    Loader = std::make_unique<ASTFileLoader>(CI, Opts->CTUDir);
+}
 
 llvm::Expected<ASTUnit *>
 CrossTranslationUnitContext::ASTUnitStorage::getASTUnitForFile(
     StringRef FileName, bool DisplayCTUProgress) {
   // Try the cache first.
   auto ASTCacheEntry = FileASTUnitMap.find(FileName);
   if (ASTCacheEntry == FileASTUnitMap.end()) {
 
     // Do not load if the limit is reached.
     if (!LoadGuard) {
       ++NumASTLoadThresholdReached;
       return llvm::make_error<IndexError>(
           index_error_code::load_threshold_reached);
     }
 
-    // Load the ASTUnit from the pre-dumped AST file specified by ASTFileName.
-    std::unique_ptr<ASTUnit> LoadedUnit = FileAccessor(FileName);
+    auto LoadAttempt = Loader->load(FileName);
+
+    if (!LoadAttempt)
+      return LoadAttempt.takeError();
+
+    std::unique_ptr<ASTUnit> LoadedUnit = std::move(LoadAttempt.get());
 
     // Need the raw pointer and the unique_ptr as well.
     ASTUnit *Unit = LoadedUnit.get();
 
     // Update the cache.
     FileASTUnitMap[FileName] = std::move(LoadedUnit);
 
     LoadGuard.indicateLoadSuccess();
 
     if (DisplayCTUProgress)
       llvm::errs() << "CTU loaded AST file: " << FileName << "\n";
 
     return Unit;
 
   } else {
     // Found in the cache.
     return ASTCacheEntry->second.get();
   }
 }
 
 llvm::Expected<ASTUnit *>
 CrossTranslationUnitContext::ASTUnitStorage::getASTUnitForFunction(
     StringRef FunctionName, StringRef CrossTUDir, StringRef IndexName,
     bool DisplayCTUProgress) {
   // Try the cache first.
   auto ASTCacheEntry = NameASTUnitMap.find(FunctionName);
   if (ASTCacheEntry == NameASTUnitMap.end()) {
     // Load the ASTUnit from the pre-dumped AST file specified by ASTFileName.
 
     // Ensure that the Index is loaded, as we need to search in it.
     if (llvm::Error IndexLoadError =
             ensureCTUIndexLoaded(CrossTUDir, IndexName))
       return std::move(IndexLoadError);
 
     // Check if there is and entry in the index for the function.
     if (!NameFileMap.count(FunctionName)) {
       ++NumNotInOtherTU;
       return llvm::make_error<IndexError>(index_error_code::missing_definition);
     }
 
     // Search in the index for the filename where the definition of FuncitonName
     // resides.
     if (llvm::Expected<ASTUnit *> FoundForFile =
             getASTUnitForFile(NameFileMap[FunctionName], DisplayCTUProgress)) {
 
       // Update the cache.
llvm::Error CrossTranslationUnitContext::ASTUnitStorage::ensureCTUIndexLoaded(
 
   // Get the absolute path to the index file.
   SmallString<256> IndexFile = CrossTUDir;
   if (llvm::sys::path::is_absolute(IndexName))
     IndexFile = IndexName;
   else
     llvm::sys::path::append(IndexFile, IndexName);
 
-  if (auto IndexMapping = parseCrossTUIndex(IndexFile, CrossTUDir)) {
+  if (auto IndexMapping = parseCrossTUIndex(IndexFile)) {
     // Initialize member map.
     NameFileMap = *IndexMapping;
     return llvm::Error::success();
   } else {
     // Error while parsing CrossTU index file.
     return IndexMapping.takeError();
   };
 }
 
+CrossTranslationUnitContext::ASTOnDemandLoader::ASTOnDemandLoader(
+    StringRef OnDemandParsingDatabase)
+    : OnDemandParsingDatabase(OnDemandParsingDatabase) {}
+
 llvm::Expected<ASTUnit *> CrossTranslationUnitContext::loadExternalAST(
     StringRef LookupName, StringRef CrossTUDir, StringRef IndexName,
     bool DisplayCTUProgress) {
   // FIXME: The current implementation only supports loading decls with
   //        a lookup name from a single translation unit. If multiple
   //        translation units contains decls with the same lookup name an
   //        error will be returned.
 
   // Try to get the value from the heavily cached storage.
   llvm::Expected<ASTUnit *> Unit = ASTStorage.getASTUnitForFunction(
       LookupName, CrossTUDir, IndexName, DisplayCTUProgress);
 
   if (!Unit)
     return Unit.takeError();
 
   // Check whether the backing pointer of the Expected is a nullptr.
   if (!*Unit)
     return llvm::make_error<IndexError>(
         index_error_code::failed_to_get_external_ast);
 
   return Unit;
 }
 
+/// Load the AST from a source-file, which is supposed to be located inside the
+/// compilation database \p CompileCommands. The compilation database
+/// can contain the path of the file under the key "file" as an absolute path,
+/// or as a relative path. When emitting diagnostics, plist files may contain
+/// references to a location in a TU, that is different from the main TU. In
+/// such cases, the file path emitted by the DiagnosticEngine is based on how
+/// the exact invocation is assembled inside the ClangTool, which performs the
+/// building of the ASTs. In order to ensure absolute paths inside the
+/// diagnostics, we use the ArgumentsAdjuster API of ClangTool to make sure that
+/// the invocation inside ClangTool is always made with an absolute path. \p
+/// Identifier is assumed to be the lookup-name of the file, which comes from
+/// the Index. The Index is built by the \p clang-extdef-mapping tool, which is
+/// supposed to generate absolute paths.
+///
+/// We must have absolute paths inside the plist, because otherwise we would
+/// not be able to parse the bug, because we could not find the files with
+/// relative paths. The directory of one entry in the compilation db may be
+/// different from the directory where the plist is interpreted.
+///
+/// Note that as the ClangTool is instantiated with a lookup-vector, which
+/// contains a single entry; the supposedly absolute path of the source file.
+/// So, the ArgumentAdjuster will only be used on the single corresponding
+/// invocation. This guarantees that even if two files match in name, but
+/// differ in location, only the correct one's invocation will be handled. This
+/// is due to the fact that the lookup is done correctly inside the
+/// OnDemandParsingDatabase, so it works for already absolute paths given under
+/// the "file" entry of the compilation database, but also if a relative path is
+/// given. In such a case, the lookup uses the "directory" entry as well to
+/// identify the correct file.
+CrossTranslationUnitContext::LoadResultTy
+CrossTranslationUnitContext::ASTOnDemandLoader::load(StringRef Identifier) {
+
+  if (auto InitError = lazyInitCompileCommands())
+    return std::move(InitError);
+
+  using namespace tooling;
+
+  SmallVector<std::string, 1> Files;
+  Files.push_back(std::string(Identifier));
+  ClangTool Tool(*CompileCommands, Files);
+
+  /// Lambda filter designed to find the source file argument inside an
+  /// invocation used to build the ASTs, and replace it with its absolute path
+  /// equivalent.
+  auto SourcePathNormalizer = [Identifier](const CommandLineArguments &Args,
+                                           StringRef FileName) {
+    /// Match the argument to the absolute path by checking whether it is a
+    /// postfix.
+    auto IsPostfixOfLookup = [Identifier](const std::string &Arg) {
+      return Identifier.rfind(Arg) != llvm::StringRef::npos;
+    };
+
+    /// Commandline arguments are modified, and the API dictates the return of
+    /// a new instance, so copy the original.
+    CommandLineArguments Result{Args};
+
+    /// Search for the source file argument. Start from the end as a heuristic,
+    /// as most invocations tend to contain the source file argument in their
+    /// latter half. Only the first match is replaced.
+    auto SourceFilePath =
+        std::find_if(Result.rbegin(), Result.rend(), IsPostfixOfLookup);
+
+    /// If source file argument could not been found, return the original
+    /// CommandlineArgumentsInstance.
+    if (SourceFilePath == Result.rend())
+      return Result;
+
+    /// Overwrite the argument with the \p ASTSourcePath, as it is assumed to
+    /// be the absolute path of the file.
+    *SourceFilePath = Identifier.str();
+
+    return Result;
+  };
+
+  Tool.appendArgumentsAdjuster(std::move(SourcePathNormalizer));
+
+  std::vector<std::unique_ptr<ASTUnit>> ASTs;
+  Tool.buildASTs(ASTs);
+
+  /// There is an assumption that the compilation database does not contain
+  /// multiple entries for the same source file.
+  if (ASTs.size() > 1)
+    return llvm::make_error<IndexError>(
+        index_error_code::ambiguous_compilation_database);
+
+  /// Ideally there is exactly one entry in the compilation database that
+  /// matches the source file.
+  if (ASTs.size() != 1)
+    return llvm::make_error<IndexError>(
+        index_error_code::failed_to_get_external_ast);
+
+  ASTs[0]->enableSourceFileDiagnostics();
+  return std::move(ASTs[0]);
+}
+
+llvm::Error
+CrossTranslationUnitContext::ASTOnDemandLoader::lazyInitCompileCommands() {
+  // Lazily initialize the compilation database.
+
+  if (CompileCommands)
+    return llvm::Error::success();
+
+  std::string LoadError;
+  CompileCommands = tooling::JSONCompilationDatabase::loadFromFile(
+      OnDemandParsingDatabase, LoadError,
+      tooling::JSONCommandLineSyntax::AutoDetect);
+  return CompileCommands ? llvm::Error::success()
+                         : llvm::make_error<IndexError>(
+                               index_error_code::failed_to_get_external_ast);
+}
+
 template <typename T>
 llvm::Expected<const T *>
 CrossTranslationUnitContext::importDefinitionImpl(const T *D, ASTUnit *Unit) {
   assert(hasBodyOrInit(D) && "Decls to be imported should have body or init.");
 
   assert(&D->getASTContext() == &Unit->getASTContext() &&
          "ASTContext of Decl and the unit should match.");
   ASTImporter &Importer = getOrCreateASTImporter(Unit);

clang/lib/Frontend/CompilerInvocation.cpp

#undef ANALYZER_OPTION_DEPENDS_ON_USER_MODE
   if (AnOpts.ShouldTrackConditionsDebug && !AnOpts.ShouldTrackConditions)
     Diags->Report(diag::err_analyzer_config_invalid_input)
         << "track-conditions-debug" << "'track-conditions' to also be enabled";
 
   if (!AnOpts.CTUDir.empty() && !llvm::sys::fs::is_directory(AnOpts.CTUDir))
     Diags->Report(diag::err_analyzer_config_invalid_input) << "ctu-dir"
                                                            << "a filename";
 
+  if (AnOpts.CTUOnDemandParsing &&
+      !llvm::sys::fs::exists(AnOpts.CTUOnDemandParsingDatabase))
+    Diags->Report(diag::err_analyzer_config_invalid_input)
+        << "ctu-on-demand-parsing-database"
+        << "a filename";
+
   if (!AnOpts.ModelPath.empty() &&
       !llvm::sys::fs::is_directory(AnOpts.ModelPath))
     Diags->Report(diag::err_analyzer_config_invalid_input) << "model-path"
                                                            << "a filename";
 }
 
 static bool ParseMigratorArgs(MigratorOptions &Opts, ArgList &Args) {
   Opts.NoNSAllocReallocError = Args.hasArg(OPT_migrator_no_nsalloc_error);

clang/lib/StaticAnalyzer/Core/CallEvent.cpp

RuntimeDefinition AnyFunctionCall::getRuntimeDefinition() const {
   AnalyzerOptions &Opts = Engine.getAnalysisManager().options;
 
   // Try to get CTU definition only if CTUDir is provided.
   if (!Opts.IsNaiveCTUEnabled)
     return {};
 
   cross_tu::CrossTranslationUnitContext &CTUCtx =
       *Engine.getCrossTranslationUnitContext();
+
   llvm::Expected<const FunctionDecl *> CTUDeclOrError =
       CTUCtx.getCrossTUDefinition(FD, Opts.CTUDir, Opts.CTUIndexName,
                                   Opts.DisplayCTUProgress);
 
   if (!CTUDeclOrError) {
     handleAllErrors(CTUDeclOrError.takeError(),
                     [&](const cross_tu::IndexError &IE) {
                       CTUCtx.emitCrossTUDiagnostics(IE);
                     });

clang/test/Analysis/Inputs/ctu-other.c

struct S {
   int a;
 };
 int g(struct S *ctx) {
   MYMACRO(ctx);
   return 0;
 }
 
 // Test that asm import does not fail.
+// TODO: Support the GNU extension asm keyword as well.
+// Example using the GNU extension: asm("mov $42, %0" : "=r"(res));
 int inlineAsm() {
   int res;
-  asm("mov $42, %0"
+  __asm__("mov $42, %0"
       : "=r"(res));
   return res;
 }
 
 // Implicit function.
 int identImplicit(int in) {
   return in;
 }
 
 // ASTImporter doesn't support this construct.
 int structInProto(struct DataType {int a;int b; } * d) {
   return 0;
 }

clang/test/Analysis/Inputs/ctu-other.c.externalDefMap.ast-dump.txt

 c:@F@inlineAsm ctu-other.c.ast
 c:@F@g ctu-other.c.ast
 c:@F@f ctu-other.c.ast
 c:@F@enumCheck ctu-other.c.ast
 c:@F@identImplicit ctu-other.c.ast
 c:@F@structInProto ctu-other.c.ast

clang/test/Analysis/Inputs/ctu-other.cpp.externalDefMap.ast-dump.txt

 c:@N@chns@F@chf1#I# ctu-other.cpp.ast
 c:@N@myns@N@embed_ns@F@fens#I# ctu-other.cpp.ast
 c:@F@g#I# ctu-other.cpp.ast
 c:@S@mycls@F@fscl#I#S ctu-other.cpp.ast
 c:@S@mycls@F@fcl#I# ctu-other.cpp.ast
 c:@S@mycls@F@fvcl#I# ctu-other.cpp.ast
 c:@N@myns@S@embed_cls@F@fecl#I# ctu-other.cpp.ast
 c:@S@mycls@S@embed_cls2@F@fecl2#I# ctu-other.cpp.ast
 c:@S@derived@F@fvcl#I# ctu-other.cpp.ast
 c:@F@f#I# ctu-other.cpp.ast
 c:@N@myns@F@fns#I# ctu-other.cpp.ast
 c:@F@h#I# ctu-other.cpp.ast
 c:@F@h_chain#I# ctu-chain.cpp.ast
 c:@N@chns@S@chcls@F@chf4#I# ctu-chain.cpp.ast
 c:@N@chns@F@chf2#I# ctu-chain.cpp.ast
 c:@F@fun_using_anon_struct#I# ctu-other.cpp.ast
 c:@F@other_macro_diag#I# ctu-other.cpp.ast
 c:@extInt ctu-other.cpp.ast
 c:@N@intns@extInt ctu-other.cpp.ast
 c:@extS ctu-other.cpp.ast
 c:@S@A@a ctu-other.cpp.ast
 c:@extSC ctu-other.cpp.ast
 c:@S@ST@sc ctu-other.cpp.ast
 c:@extSCN ctu-other.cpp.ast
 c:@extSubSCN ctu-other.cpp.ast
 c:@extSCC ctu-other.cpp.ast
 c:@extU ctu-other.cpp.ast
 c:@S@TestAnonUnionUSR@Test ctu-other.cpp.ast
 c:@F@testImportOfIncompleteDefaultParmDuringImport#I# ctu-other.cpp.ast
 c:@F@testImportOfDelegateConstructor#I# ctu-other.cpp.ast
 No newline at end of file

clang/test/Analysis/analyzer-config.c

 // CHECK-NEXT: cfg-rich-constructors = true
 // CHECK-NEXT: cfg-scopes = false
 // CHECK-NEXT: cfg-temporary-dtors = true
 // CHECK-NEXT: cplusplus.Move:WarnOn = KnownsAndLocals
 // CHECK-NEXT: crosscheck-with-z3 = false
 // CHECK-NEXT: ctu-dir = ""
 // CHECK-NEXT: ctu-import-threshold = 100
 // CHECK-NEXT: ctu-index-name = externalDefMap.txt
+// CHECK-NEXT: ctu-on-demand-parsing = false
+// CHECK-NEXT: ctu-on-demand-parsing-database = compile_commands.json
 // CHECK-NEXT: deadcode.DeadStores:ShowFixIts = false
 // CHECK-NEXT: deadcode.DeadStores:WarnForDeadNestedAssignments = true
 // CHECK-NEXT: debug.AnalysisOrder:* = false
 // CHECK-NEXT: debug.AnalysisOrder:Bind = false
 // CHECK-NEXT: debug.AnalysisOrder:EndAnalysis = false
 // CHECK-NEXT: debug.AnalysisOrder:EndFunction = false
 // CHECK-NEXT: debug.AnalysisOrder:LiveSymbols = false
 // CHECK-NEXT: debug.AnalysisOrder:NewAllocator = false

 // CHECK-NEXT: suppress-inlined-defensive-checks = true
 // CHECK-NEXT: suppress-null-return-paths = true
 // CHECK-NEXT: track-conditions = true
 // CHECK-NEXT: track-conditions-debug = false
 // CHECK-NEXT: unix.DynamicMemoryModeling:Optimistic = false
 // CHECK-NEXT: unroll-loops = false
 // CHECK-NEXT: widen-loops = false
 // CHECK-NEXT: [stats]
-// CHECK-NEXT: num-entries = 103
+// CHECK-NEXT: num-entries = 105

clang/test/Analysis/ctu-different-triples.cpp

 // RUN: rm -rf %t && mkdir %t
 // RUN: mkdir -p %t/ctudir
 // RUN: %clang_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
 // RUN:   -emit-pch -o %t/ctudir/ctu-other.cpp.ast %S/Inputs/ctu-other.cpp
-// RUN: cp %S/Inputs/ctu-other.cpp.externalDefMap.txt %t/ctudir/externalDefMap.txt
+// RUN: cp %S/Inputs/ctu-other.cpp.externalDefMap.ast-dump.txt %t/ctudir/externalDefMap.txt
 // RUN: %clang_analyze_cc1 -std=c++14 -triple powerpc64-montavista-linux-gnu \
 // RUN:   -analyzer-checker=core,debug.ExprInspection \
 // RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
 // RUN:   -analyzer-config ctu-dir=%t/ctudir \
 // RUN:   -Werror=ctu \
 // RUN:   -verify %s
 
 // We expect an error in this file, but without a location.
 // expected-error-re@./ctu-different-triples.cpp:*{{imported AST from {{.*}} had been generated for a different target, current: powerpc64-montavista-linux-gnu, imported: x86_64-pc-linux-gnu}}
 
 int f(int);
 
 int main() {
   return f(5);
 }

clang/test/Analysis/ctu-main.c

 // RUN: rm -rf %t && mkdir %t
 // RUN: mkdir -p %t/ctudir2
 // RUN: %clang_cc1 -triple x86_64-pc-linux-gnu \
 // RUN:   -emit-pch -o %t/ctudir2/ctu-other.c.ast %S/Inputs/ctu-other.c
-// RUN: cp %S/Inputs/ctu-other.c.externalDefMap.txt %t/ctudir2/externalDefMap.txt
+// RUN: cp %S/Inputs/ctu-other.c.externalDefMap.ast-dump.txt %t/ctudir2/externalDefMap.txt
 // RUN: %clang_cc1 -triple x86_64-pc-linux-gnu -fsyntax-only -std=c89 -analyze \
 // RUN:   -analyzer-checker=core,debug.ExprInspection \
 // RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
 // RUN:   -analyzer-config ctu-dir=%t/ctudir2 \
 // RUN:   -verify %s
 
 void clang_analyzer_eval(int);
 
void testMacro(void) {
   g(0); // expected-warning@Inputs/ctu-other.c:29 {{Access to field 'a' results in a dereference of a null pointer (loaded from variable 'ctx')}}
 }
 
 // The external function prototype is incomplete.
 // warning:implicit functions are prohibited by c99
 void testImplicit() {
   int res = identImplicit(6);   // external implicit functions are not inlined
   clang_analyzer_eval(res == 6); // expected-warning{{TRUE}}
+  // Call something with uninitialized from the same function in which the implicit was called.
+  // This is necessary to reproduce a special bug in NoStoreFuncVisitor.
+  int uninitialized;
+  h(uninitialized); // expected-warning{{1st function call argument is an uninitialized value}}
 }
 
 // Tests the import of functions that have a struct parameter
 // defined in its prototype.
 struct DataType {
   int a;
   int b;
 };
 int structInProto(struct DataType *d);
 void testStructDefInArgument() {
   struct DataType d;
   d.a = 1;
   d.b = 0;
   clang_analyzer_eval(structInProto(&d) == 0); // expected-warning{{TRUE}} expected-warning{{FALSE}}
 }

clang/test/Analysis/ctu-main.cpp

 // RUN: rm -rf %t && mkdir %t
 // RUN: mkdir -p %t/ctudir
 // RUN: %clang_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
 // RUN:   -emit-pch -o %t/ctudir/ctu-other.cpp.ast %S/Inputs/ctu-other.cpp
 // RUN: %clang_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
 // RUN:   -emit-pch -o %t/ctudir/ctu-chain.cpp.ast %S/Inputs/ctu-chain.cpp
-// RUN: cp %S/Inputs/ctu-other.cpp.externalDefMap.txt %t/ctudir/externalDefMap.txt
+// RUN: cp %S/Inputs/ctu-other.cpp.externalDefMap.ast-dump.txt %t/ctudir/externalDefMap.txt
 // RUN: %clang_analyze_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
 // RUN:   -analyzer-checker=core,debug.ExprInspection \
 // RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
 // RUN:   -analyzer-config ctu-dir=%t/ctudir \
 // RUN:   -verify %s
 // RUN: %clang_analyze_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
 // RUN:   -analyzer-checker=core,debug.ExprInspection \
 // RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \

clang/test/Analysis/ctu-on-demand-parsing-ambigous-compilation-database.c

+// RUN: rm -rf %t
+// RUN: mkdir -p %t
+// RUN: cp "%s" "%t/ctu-on-demand-parsing-ambiguous-compilation-database.c"
+// RUN: cp "%S/Inputs/ctu-other.c" "%t/ctu-other.c"
+// Path substitutions on Windows platform could contain backslashes. These are escaped in the json file.
+// Note there is a duplicate entry for 'ctu-other.c'.
+// RUN: echo '[{"directory":"%t","command":"gcc -c -std=c89 -Wno-visibility ctu-other.c","file":"ctu-other.c"},{"directory":"%t","command":"gcc -c -std=c89 -Wno-visibility ctu-other.c","file":"ctu-other.c"}]' | sed -e 's/\\/\\\\/g' > %t/compile_commands.json
+// RUN: cd "%t" && %clang_extdef_map ctu-other.c > externalDefMap.txt
+// The exit code of the analysis is 1 if the import error occurs
+// RUN: cd "%t" && not %clang_cc1 -triple x86_64-pc-linux-gnu -fsyntax-only -std=c89 -analyze \
+// RUN:   -analyzer-checker=core,debug.ExprInspection \
+// RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
+// RUN:   -analyzer-config ctu-dir=. \
+// RUN:   -analyzer-config ctu-on-demand-parsing=true \
+// RUN:   ctu-on-demand-parsing-ambiguous-compilation-database.c 2>&1 | FileCheck %t/ctu-on-demand-parsing-ambiguous-compilation-database.c
+
+// CHECK: {{.*}}multiple definitions are found for the same key in index
+
+// 'int f(int)' is defined in ctu-other.c
+int f(int);
+void testAmbiguousImport() {
+  f(0);
+}

clang/test/Analysis/ctu-on-demand-parsing.c

+// RUN: rm -rf %t
+// RUN: mkdir -p %t
+// RUN: cp "%s" "%t/ctu-on-demand-parsing.c"
+// RUN: cp "%S/Inputs/ctu-other.c" "%t/ctu-other.c"
+// Path substitutions on Windows platform could contain backslashes. These are escaped in the json file.
+// RUN: echo '[{"directory":"%t","command":"gcc -c -std=c89 -Wno-visibility ctu-other.c","file":"ctu-other.c"}]' | sed -e 's/\\/\\\\/g' > %t/compile_commands.json
+// RUN: cd "%t" && %clang_extdef_map ctu-other.c > externalDefMap.txt
+// RUN: cd "%t" && %clang_cc1 -triple x86_64-pc-linux-gnu -fsyntax-only -std=c89 -analyze \
+// RUN:   -analyzer-checker=core,debug.ExprInspection \
+// RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
+// RUN:   -analyzer-config ctu-dir=. \
+// RUN:   -analyzer-config ctu-on-demand-parsing=true \
+// RUN:   -verify ctu-on-demand-parsing.c
+
+void clang_analyzer_eval(int);
+
+// Test typedef and global variable in function.
+typedef struct {
+  int a;
+  int b;
+} FooBar;
+extern FooBar fb;
+int f(int);
+void testGlobalVariable() {
+  clang_analyzer_eval(f(5) == 1); // expected-warning{{TRUE}}
+}
+
+// Test enums.
+int enumCheck(void);
+enum A { x, y, z };
+void testEnum() {
+  clang_analyzer_eval(x == 0);            // expected-warning{{TRUE}}
+  clang_analyzer_eval(enumCheck() == 42); // expected-warning{{TRUE}}
+}
+
+// Test that asm import does not fail.
+int inlineAsm();
+int testInlineAsm() { return inlineAsm(); }
+
+// Test reporting error in a macro.
+struct S;
+int g(struct S *);
+void testMacro(void) {
+  g(0);
+  // expected-warning@ctu-other.c:29 {{Access to field 'a' results in a dereference of a null pointer (loaded from variable 'ctx')}}
+}
+
+// The external function prototype is incomplete.
+// warning:implicit functions are prohibited by c99
+void testImplicit() {
+  int res = identImplicit(6);    // external implicit functions are not inlined
+  clang_analyzer_eval(res == 6); // expected-warning{{TRUE}}
+  // Call something with uninitialized from the same function in which the
+  // implicit was called. This is necessary to reproduce a special bug in
+  // NoStoreFuncVisitor.
+  int uninitialized;
+  h(uninitialized); // expected-warning{{1st function call argument is an uninitialized value}}
+}
+
+// Tests the import of functions that have a struct parameter
+// defined in its prototype.
+struct DataType {
+  int a;
+  int b;
+};
+int structInProto(struct DataType *d);
+void testStructDefInArgument() {
+  struct DataType d;
+  d.a = 1;
+  d.b = 0;
+  clang_analyzer_eval(structInProto(&d) == 0); // expected-warning{{TRUE}} expected-warning{{FALSE}}
+}

clang/test/Analysis/ctu-on-demand-parsing.cpp

+// RUN: rm -rf %t
+// RUN: mkdir -p %t/ctudir
+// RUN: cp %s %t/ctu-on-demand-parsing.cpp
+// RUN: cp %S/ctu-hdr.h %t/ctu-hdr.h
+// RUN: cp %S/Inputs/ctu-chain.cpp %t/ctudir/ctu-chain.cpp
+// RUN: cp %S/Inputs/ctu-other.cpp %t/ctudir/ctu-other.cpp
+// Path substitutions on Windows platform could contain backslashes. These are escaped in the json file.
+// RUN: echo '[{"directory":"%t/ctudir","command":"clang++ -c ctu-chain.cpp","file":"ctu-chain.cpp"},{"directory":"%t/ctudir","command":"clang++ -c ctu-other.cpp","file":"ctu-other.cpp"}]' | sed -e 's/\\/\\\\/g' > %t/compile_commands.json
+// RUN: cd "%t/ctudir" && %clang_extdef_map ctu-chain.cpp ctu-other.cpp > externalDefMap.txt
+// RUN: cd "%t" && %clang_analyze_cc1 -triple x86_64-pc-linux-gnu \
+// RUN:   -analyzer-checker=core,debug.ExprInspection \
+// RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
+// RUN:   -analyzer-config ctu-dir=ctudir \
+// RUN:   -analyzer-config ctu-on-demand-parsing=true \
+// RUN:   -verify ctu-on-demand-parsing.cpp
+// RUN: cd "%t" && %clang_analyze_cc1 -triple x86_64-pc-linux-gnu \
+// RUN:   -analyzer-checker=core,debug.ExprInspection \
+// RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
+// RUN:   -analyzer-config ctu-dir="%t/ctudir" \
+// RUN:   -analyzer-config ctu-on-demand-parsing=true \
+// RUN:   -analyzer-config display-ctu-progress=true 2>&1 ctu-on-demand-parsing.cpp | FileCheck %t/ctu-on-demand-parsing.cpp
+
+// CHECK: CTU loaded AST file: {{.*}}ctu-other.cpp
+// CHECK: CTU loaded AST file: {{.*}}ctu-chain.cpp
+
+#include "ctu-hdr.h"
+
+void clang_analyzer_eval(int);
+
+int f(int);
+int g(int);
+int h(int);
+
+int callback_to_main(int x) { return x + 1; }
+
+namespace myns {
+int fns(int x);
+
+namespace embed_ns {
+int fens(int x);
+}
+
+class embed_cls {
+public:
+  int fecl(int x);
+};
+} // namespace myns
+
+class mycls {
+public:
+  int fcl(int x);
+  virtual int fvcl(int x);
+  static int fscl(int x);
+
+  class embed_cls2 {
+  public:
+    int fecl2(int x);
+  };
+};
+
+class derived : public mycls {
+public:
+  virtual int fvcl(int x) override;
+};
+
+namespace chns {
+int chf1(int x);
+}
+
+int fun_using_anon_struct(int);
+int other_macro_diag(int);
+
+void test_virtual_functions(mycls *obj) {
+  // The dynamic type is known.
+  clang_analyzer_eval(mycls().fvcl(1) == 8);   // expected-warning{{TRUE}}
+  clang_analyzer_eval(derived().fvcl(1) == 9); // expected-warning{{TRUE}}
+  // We cannot decide about the dynamic type.
+  clang_analyzer_eval(obj->fvcl(1) == 8); // expected-warning{{FALSE}} expected-warning{{TRUE}}
+  clang_analyzer_eval(obj->fvcl(1) == 9); // expected-warning{{FALSE}} expected-warning{{TRUE}}
+}
+
+int main() {
+  clang_analyzer_eval(f(3) == 2); // expected-warning{{TRUE}}
+  clang_analyzer_eval(f(4) == 3); // expected-warning{{TRUE}}
+  clang_analyzer_eval(f(5) == 3); // expected-warning{{FALSE}}
+  clang_analyzer_eval(g(4) == 6); // expected-warning{{TRUE}}
+  clang_analyzer_eval(h(2) == 8); // expected-warning{{TRUE}}
+
+  clang_analyzer_eval(myns::fns(2) == 9);                   // expected-warning{{TRUE}}
+  clang_analyzer_eval(myns::embed_ns::fens(2) == -1);       // expected-warning{{TRUE}}
+  clang_analyzer_eval(mycls().fcl(1) == 6);                 // expected-warning{{TRUE}}
+  clang_analyzer_eval(mycls::fscl(1) == 7);                 // expected-warning{{TRUE}}
+  clang_analyzer_eval(myns::embed_cls().fecl(1) == -6);     // expected-warning{{TRUE}}
+  clang_analyzer_eval(mycls::embed_cls2().fecl2(0) == -11); // expected-warning{{TRUE}}
+
+  clang_analyzer_eval(chns::chf1(4) == 12);           // expected-warning{{TRUE}}
+  clang_analyzer_eval(fun_using_anon_struct(8) == 8); // expected-warning{{TRUE}}
+
+  clang_analyzer_eval(other_macro_diag(1) == 1); // expected-warning{{TRUE}}
+  // expected-warning@ctudir/ctu-other.cpp:93{{REACHABLE}}
+  MACRODIAG(); // expected-warning{{REACHABLE}}
+}

clang/test/Analysis/ctu-unknown-parts-in-triples.cpp

 // We do not expect any error when one part of the triple is unknown, but other
 // known parts are equal.
 
 // RUN: rm -rf %t && mkdir %t
 // RUN: mkdir -p %t/ctudir
 // RUN: %clang_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
 // RUN:   -emit-pch -o %t/ctudir/ctu-other.cpp.ast %S/Inputs/ctu-other.cpp
-// RUN: cp %S/Inputs/ctu-other.cpp.externalDefMap.txt %t/ctudir/externalDefMap.txt
+// RUN: cp %S/Inputs/ctu-other.cpp.externalDefMap.ast-dump.txt %t/ctudir/externalDefMap.txt
 // RUN: %clang_analyze_cc1 -std=c++14 -triple x86_64-unknown-linux-gnu \
 // RUN:   -analyzer-checker=core,debug.ExprInspection \
 // RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
 // RUN:   -analyzer-config ctu-dir=%t/ctudir \
 // RUN:   -Werror=ctu \
 // RUN:   -verify %s
 
 // expected-no-diagnostics
 
 int f(int);
 
 int main() {
   return f(5);
 }

clang/unittests/CrossTU/CrossTranslationUnitTest.cpp

 //===- unittest/Tooling/CrossTranslationUnitTest.cpp - Tooling unit tests -===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "clang/CrossTU/CrossTranslationUnit.h"
-#include "clang/Frontend/CompilerInstance.h"
 #include "clang/AST/ASTConsumer.h"
+#include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/FrontendAction.h"
 #include "clang/Tooling/Tooling.h"
+#include "llvm/ADT/Optional.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/ToolOutputFile.h"
 #include "gtest/gtest.h"
 #include <cassert>
 
 namespace clang {
 namespace cross_tu {
TEST(CrossTranslationUnit, IndexFormatCanBeParsed) {
   llvm::SmallString<256> IndexFileName;
   ASSERT_FALSE(llvm::sys::fs::createTemporaryFile("index", "txt", IndexFD,
                                                   IndexFileName));
   llvm::ToolOutputFile IndexFile(IndexFileName, IndexFD);
   IndexFile.os() << IndexText;
   IndexFile.os().flush();
   EXPECT_TRUE(llvm::sys::fs::exists(IndexFileName));
   llvm::Expected<llvm::StringMap<std::string>> IndexOrErr =
-      parseCrossTUIndex(IndexFileName, "");
+      parseCrossTUIndex(IndexFileName);
   EXPECT_TRUE((bool)IndexOrErr);
   llvm::StringMap<std::string> ParsedIndex = IndexOrErr.get();
   for (const auto &E : Index) {
     EXPECT_TRUE(ParsedIndex.count(E.getKey()));
     EXPECT_EQ(ParsedIndex[E.getKey()], E.getValue());
   }
   for (const auto &E : ParsedIndex)
     EXPECT_TRUE(Index.count(E.getKey()));
 }
 
-TEST(CrossTranslationUnit, CTUDirIsHandledCorrectly) {
-  llvm::StringMap<std::string> Index;
-  Index["a"] = "/b/c/d";
-  std::string IndexText = createCrossTUIndexString(Index);
-
-  int IndexFD;
-  llvm::SmallString<256> IndexFileName;
-  ASSERT_FALSE(llvm::sys::fs::createTemporaryFile("index", "txt", IndexFD,
-                                                  IndexFileName));
-  llvm::ToolOutputFile IndexFile(IndexFileName, IndexFD);
-  IndexFile.os() << IndexText;
-  IndexFile.os().flush();
-  EXPECT_TRUE(llvm::sys::fs::exists(IndexFileName));
-  llvm::Expected<llvm::StringMap<std::string>> IndexOrErr =
-      parseCrossTUIndex(IndexFileName, "/ctudir");
-  EXPECT_TRUE((bool)IndexOrErr);
-  llvm::StringMap<std::string> ParsedIndex = IndexOrErr.get();
-  EXPECT_EQ(ParsedIndex["a"], "/ctudir/b/c/d");
-}
-
 } // end namespace cross_tu
 } // end namespace clang