This patch adds an alternative way of loading ASTs to provide the CTU definitions needed during analysis.
The additional approach is to use Tooling with a user-provided compile_commands.json,
and look up the needed file from the output of the ext-def-mapping tool (which is now used for both approaches).

Great job, this seems to be progressing nicely! please see my comments inline.

• Remove link reference from commit message
• Remove superfluous semicolons left in the code

• Add capture by reference test
• Add another class to ensure reachability inside function call is independent from each other in the first 2 test cases

Please feel free to add more reviewers.

Nice improvements! It was a good catch that a substantial part of the checker could be cut out. Accurate modelling of CString and related API-s is really important, and this is a good step in that direction.
I would test this on a larger codebase as well before committing, but LGTM otherwise.

@chrish_ericsson_atx Thanks for fixing this! Your help is much appreciated :)

• Remove unused member Limit
• Rebase to current master

Lovely documentation with practical use-cases!
I left a few inline remarks.
Also wouldn't it be nice to have a section which introduces the -ast-merge command-line option. It would be helpful to see an example where a PCH is dumped and merged into another TU. You could mention, that this can be used to debug ASTImporter functionality.
Cheers!

Specify the exact meaning of successful storage

Updated the revision.
I find this solution definitely more compact with responsibilities more separated. One more thing that comes to mind is that maybe the whole explicit passing of CTUDir and IndexName arguments is a bit verbose. What are your thoughts about these being injected earlier than the query operations (eg.: in getASTUnitForFunction and getFileForFunction)?

• Merge RAII class
• Update comments

Thanks for pointing out these issues. Most of them are agreed. Merging the RAII counter with the threshold checker class, however, does not seem like a good decision for me. What would be the benefits of merging the two?

Too much autoformat fixed

Refactor functionality into local classes

32f220c5fbe5 is the more sophisticated solution to the problem.

There are multiple new additions to stream formatters, and this patch is way too old. I may consider creating a new checker like this with an extended feature set later.

• Add implementation rationale, fix forgotten build-step

Incremental change

Incremental change

Incremental change

Incremental change

Incremental change

I have applied the fixes, which seemed fitting. Also see my reasoning behind the command-prefix implementation inline.

• Remove redundant parameter handling
• Pass env keyword argument to every ShellCommand step
• Use RemoveDirectory where applicable

@gkistanova Your review is very much appreciated!
I have answered the inline comments, and I am currently applying the suggested fixes. After short testing, I will update this revision.

Rebase again

Missed arcanist diff. Disregard this revision.

Rebase onto current master

Friendly ping @gkistanova. What do you think about this modification? Also, should we add our buildslave to the silent (if I remember correctly it is called staging), or the reporting buildmaster? Thanks!

Revert unnecessary clang-formating of AnalysisConsumer.cpp

I could greatly simplify the API of getCrossTUDefinition by injecting the threshold value in the constructor. A minor drawback with this approach is that testing code is a bit more complicated, as I had to patch the CompilerInstance to override the AnalyzerOption value for CTUImportThreshold. Thanks for the suggestion!

Apply review suggestions by Xazax

In D59934#1449221, @Szelethus wrote:

Hmm. Is your clang recent enough to contain @bruntib's patch D57892? Is it possible that this patch solves the same issue? With this patch applied, are you able to get a macro expansions from a different TU in the plist output, or does this patch only resolve the regression?

In D59934#1449109, @Szelethus wrote:

I would still like to learn more about this issue. I am somewhat afraid that the our macro expansion is faulty. Can you provide a stacktrace maybe?

In D59934#1446059, @lebedev.ri wrote:

Test?

Just have a bit more context, I have the following information from a debug session at the execution point of the unreachable:

In D59934#1446736, @Szelethus wrote:

In D59934#1446051, @gamesh411 wrote:

Hi!

This issue came up during the generation BugReports of BugPaths containing macro-expansions, where the spelling location and expansion locations were in different files.
With this change, we make such SourceLocations comparable by their FileIDs.

Do you mean regular macro expansions or the one we're generating in the plist output via -analyzer-config expand-macros=true?

This issue came up during the generation BugReports of BugPaths containing macro-expansions, where the spelling location and expansion locations were in different files.
With this change, we make such SourceLocations comparable by their FileIDs.

ping @a_sidorin @shafik

ping @a_sidorin

Updated option handling location to use AnalyzerOptions instead of CC1

During CTU analysis, not only StructuralEquivalence but the main implementation of ASTImporter can also emit ODR-related diagnostics. After some consideration, I have chosen not to make these dependent on a switch, as ASTImporter is not nearly as widely used as StructuralEquivalence (thru Sema), and it would also complicate the code unnecessarily.
What do you think?

• Revert member order to original

This revision is the alternative of the abandoned D55646.
Now Sema uses the old behavior of emitting ODR errors while merging and importing is done in a more lenient way.

I am creating a new revision that keeps the old handling of ODR violations the same when used by parts of the Sema, but emit warnings when used by the ASTImporter.

I am going to abandon this modification, as setting ODR violations as warnings, seems like a change that could have unforeseen consequences.

Remove unnecessary ToTU variable from test case.

Ping.

Hey Aleksei!
Thank you for the input! ODR violations being warnings would be beneficial from the code maintenance point of view, as we would not have to resort to duplicate some (if not most) of them as errors. There is also a flexibility advantage in the diagnostics, as warnings can be propagated to error level or suppressed, whereas errors cannot be "demoted" (AFAIK). So while I am not opposed to providing an error AND a warning for these kinds of violations, if the SEMA or other modules do not need them to be explicitly defined as errors (for I don't know... tooling support or other reasons), then it would be cleaner to only have warnings for these.

In order to be able to handle ODR-related diagnostics with command line options, these diagnostics were moved from Error category to Warning. What are your thoughts?

In my opinion, after migrating relevant test cases from D33826, this is ready.

Hi!
Thanks for your reviews, although I haven't been active for some time now.
I personally do not have commit rights, so could someone else take care of it?

I have implemented the std::transform. The previous version used std::for_each because the iterator for enum declarations was not a random access iterator, but it turned out that I can solve this problem via std::distance. Thanks for sticking to your opinion on this one, because of it I could learn something new.

Ping. @NoQ would you please have a look? Thanks!

As for the the loss of precision problem, in the special case of char the size of char is known. However does the analysis have the necessary information in this stage to know the size of an int for example? I found bit-width specifying information in the llvm::Type class which is used in the code generation phase. It could be done by checking on a per type basis, but then again, it could possibly lead to false positives. Correct me if I am wrong.

Applied most of the suggested changes, thanks for all the insights!