Add preliminary Cross Translation Unit support library

[xazax.hun]

Summary

This patch introduces a class that can help to build tools that require cross translation unit facilities.
This class allows function definitions to be loaded from external AST files based on an index.
In order to use this functionality an index is required. USRs are used as names to look up the functions.
This class also does caching to avoid redundant loading of AST files.

Right now only function defnitions can be loaded using this API, because this is what the Static Analyzer requires.
In to future this could be extended to classes, types etc.

More tests will come together with the first user once it is accepted: https://reviews.llvm.org/D30691

[xazax.hun]

• Add a tool to dump USRs for function definitions. It can be used to create index files.

[klimek]

General direction looks good.

I think in addition to integration tests through the static analyzer, we should still have unit tests.

include/clang/Tooling/CrossTranslationUnit.h

47 ↗

(On Diff #103571)

I'd just spell out Cross & s/CTUDIR/IndexDir/ perhaps? (if CTUDir is to contain the indices)
I think as this is the major API entry point, a comment that explains the meaning of the arguments and how to use them would be nice.

[xazax.hun]

• Added test for the USR dumping tool.
• Added unit test for the CrossTranslationUnit class
• Added documentation for the API entry point
• Renamed some functions and variables

[xazax.hun]

Unfortunately, in order to make the Unit Test pass, I also had to modify the AST Importer. Are you ok with having that change as part of this patch (Aleksei might be a suitable person to review that part) or should I make a separate differential for that?

[klimek]

Adding folks interested in the indexing discussion.

include/clang/Tooling/CrossTranslationUnit.h

53–58 ↗

(On Diff #103707)

In the future we'll want to create an index interface around this (which will probably serve also what the refactoring integration would be based on), instead of piping files and directories into all classes.

Perhaps we can start this by already pulling out a class ProjectIndex or somesuch, with methods like loadASTDefining(...)?

[xazax.hun]

include/clang/Tooling/CrossTranslationUnit.h

53–58 ↗

(On Diff #103707)

While I do agree to have an interface for that would be really good, but maybe it would be better to first review and accept this patch and after that design the interface in a follow-up patch (so https://reviews.llvm.org/D30691 is not blocked). What do you think?

[xazax.hun]

• Removed an unrelated change
• Made the unit test more strict

[klimek]

include/clang/Tooling/CrossTranslationUnit.h

53–58 ↗

(On Diff #103707)

I'm generally fine with that, and mainly looped in other folks to see whether anybody else has concerns.

[xazax.hun]

• Updated to compile with trunk
• Minor style fixes
• Proper diagnostic when the index format is wrong

[xazax.hun]

It looks like Richard approved libTooling as a dependency for clang on the mailing list (http://lists.llvm.org/pipermail/cfe-dev/2017-July/054536.html).
If it is ok to have this code in libTooling (for now), I think we could start/continue the review of this patch.

[klimek]

In D34512#800490, @xazax.hun wrote:

It looks like Richard approved libTooling as a dependency for clang on the mailing list (http://lists.llvm.org/pipermail/cfe-dev/2017-July/054536.html).
If it is ok to have this code in libTooling (for now), I think we could start/continue the review of this patch.

I read that somewhat differently? It seems like Richard basically proposes adding a new library for things that control how we run clang in a multi-TU scenario. I'd call it libIndex, but that already exists :)

[xazax.hun]

In D34512#800499, @klimek wrote:

In D34512#800490, @xazax.hun wrote:

It looks like Richard approved libTooling as a dependency for clang on the mailing list (http://lists.llvm.org/pipermail/cfe-dev/2017-July/054536.html).
If it is ok to have this code in libTooling (for now), I think we could start/continue the review of this patch.

I read that somewhat differently? It seems like Richard basically proposes adding a new library for things that control how we run clang in a multi-TU scenario. I'd call it libIndex, but that already exists :)

No, but since Richard said he did not see any justifications to include this in libTooling, I had the impression this is not his final word, and in case you see it justified, this remains the suggested direction.
But in this case I will rewrite this patch to create a new library. Are you still interested in reviewing it? Do you have any other name in mind? What about libCrossTU?

[xazax.hun]

• Move CrossTU functionality into its own library

[klimek]

+Richard as top-level code owner for new libs.

For bikeshedding the name: I'd have liked libIndex, but that's already taken. CrossTU doesn't seem too bad to me, too, though.

[arphaman]

unittests/CrossTU/CMakeLists.txt

11

CrossTUTests?

[xazax.hun]

In D34512#800618, @klimek wrote:

+Richard as top-level code owner for new libs.

For bikeshedding the name: I'd have liked libIndex, but that's already taken. CrossTU doesn't seem too bad to me, too, though.

Some brainstorming for the bikeshedding: libProjet, libProjectIndex, libImport
The reason I am not sure about the Index in the name because this code does not contain (yet) an interface to create/handle indexes.
The import might be a good once, since this library is basically wrapping the ASTImporter to import code from external sources, but it might be misleading, since ASTImporter is in the AST module.

[whisperity]

In D34512#800502, @xazax.hun wrote:

In D34512#800499, @klimek wrote:

In D34512#800490, @xazax.hun wrote:

It looks like Richard approved libTooling as a dependency for clang on the mailing list (http://lists.llvm.org/pipermail/cfe-dev/2017-July/054536.html).
If it is ok to have this code in libTooling (for now), I think we could start/continue the review of this patch.

I read that somewhat differently? It seems like Richard basically proposes adding a new library for things that control how we run clang in a multi-TU scenario. I'd call it libIndex, but that already exists :)

No, but since Richard said he did not see any justifications to include this in libTooling, I had the impression this is not his final word, and in case you see it justified, this remains the suggested direction.
But in this case I will rewrite this patch to create a new library. Are you still interested in reviewing it? Do you have any other name in mind? What about libCrossTU?

In D34512#800625, @xazax.hun wrote:

In D34512#800618, @klimek wrote:

+Richard as top-level code owner for new libs.

For bikeshedding the name: I'd have liked libIndex, but that's already taken. CrossTU doesn't seem too bad to me, too, though.

Some brainstorming for the bikeshedding: libProjet, libProjectIndex, libImport
The reason I am not sure about the Index in the name because this code does not contain (yet) an interface to create/handle indexes.
The import might be a good once, since this library is basically wrapping the ASTImporter to import code from external sources, but it might be misleading, since ASTImporter is in the AST module.

Cross-TU is now used (in the lifeline of these features introduced by the team) in multiple contexts: CTU analysis, and now CTU lookup.
Maybe the name of this lib should reflect on the fact that the lib is used to support cross-TU definition lookup.

Hmm... libCrossTULocator?

[xazax.hun]

• Fix a copy and paste error and removed an unintended change.

[klimek]

In D34512#800626, @whisperity wrote:

In D34512#800502, @xazax.hun wrote:

In D34512#800499, @klimek wrote:

In D34512#800490, @xazax.hun wrote:

It looks like Richard approved libTooling as a dependency for clang on the mailing list (http://lists.llvm.org/pipermail/cfe-dev/2017-July/054536.html).
If it is ok to have this code in libTooling (for now), I think we could start/continue the review of this patch.

I read that somewhat differently? It seems like Richard basically proposes adding a new library for things that control how we run clang in a multi-TU scenario. I'd call it libIndex, but that already exists :)

No, but since Richard said he did not see any justifications to include this in libTooling, I had the impression this is not his final word, and in case you see it justified, this remains the suggested direction.
But in this case I will rewrite this patch to create a new library. Are you still interested in reviewing it? Do you have any other name in mind? What about libCrossTU?

In D34512#800625, @xazax.hun wrote:

In D34512#800618, @klimek wrote:

+Richard as top-level code owner for new libs.

For bikeshedding the name: I'd have liked libIndex, but that's already taken. CrossTU doesn't seem too bad to me, too, though.

Some brainstorming for the bikeshedding: libProjet, libProjectIndex, libImport
The reason I am not sure about the Index in the name because this code does not contain (yet) an interface to create/handle indexes.
The import might be a good once, since this library is basically wrapping the ASTImporter to import code from external sources, but it might be misleading, since ASTImporter is in the AST module.

Cross-TU is now used (in the lifeline of these features introduced by the team) in multiple contexts: CTU analysis, and now CTU lookup.
Maybe the name of this lib should reflect on the fact that the lib is used to support cross-TU definition lookup.

Hmm... libCrossTULocator?

Well, a better name for locator is index :)

[xazax.hun]

gentle ping

[klimek]

Specifically, ping Richard for new top-level lib in clang.

[xazax.hun]

In D34512#803724, @klimek wrote:

Specifically, ping Richard for new top-level lib in clang.

Richard proposed pulling this out into a separate library in the first place. Do we need his approval for the name? Or we want him to consider if this patch justifies the existence of a separate library?

[klimek]

Yep, I want Richard's approval for the name. I think he already expressed a pro-pulling-out stance.

[xazax.hun]

In D34512#806505, @klimek wrote:

Yep, I want Richard's approval for the name. I think he already expressed a pro-pulling-out stance.

Great! Ping for the name approval.

[rsmith]

Organizationally, this seems fine. Carry on :)

include/clang/Basic/DiagnosticFrontendKinds.td

229–231

Please add a new DiagnosticCrossTUKinds.td file for the new subdirectory.

include/clang/CrossTU/CrossTranslationUnit.h

13–14

This should be updated to match the new directory.

30

This doesn't follow either of our normal conventions for namespace names within Clang and LLVM (CamelCase and snake_case, with the latter being more common). Maybe cross_tu?

[xazax.hun]

• Addressed review comments.

[xazax.hun]

In D34512#829712, @rsmith wrote:

Organizationally, this seems fine. Carry on :)

Great news! Thank you!

[krasimir]

include/clang/CrossTU/CrossTUDiagnostic.h

16 ↗

(On Diff #109552)

LLVM Style uses no indent for namespaces. Reformat with clang-format.

include/clang/CrossTU/CrossTranslationUnit.h

71

Maybe rename to FunctionASTUnitMap for consistency with the previous line?

[xazax.hun]

• Address further review comments.

[whisperity]

Apart from those in the in-line comments, I have a question: how safe is this library to Release builds? I know this is only a submodule dependency for the "real deal" in D30691, but I have seen some asserts that "imported function should already have a body" and such.

Will the static analyzer handle these errors gracefully and fall back to "function is unknown, let's throw my presumptions out of the window" or will it bail away? I'm explicitly thinking of the assert-lacking Release build.

include/clang/Basic/AllDiagnostics.h

20 ↗

(On Diff #109559)

Import file order?

lib/CrossTU/CrossTranslationUnit.cpp

46

"visits (...) and looks up" or "visit (...) and look up"

tools/CMakeLists.txt

22

The other "blocks" in this file look somewhat in alphabetic order, perhaps this should be added a few lines above?

[whisperity]

include/clang/CrossTU/CrossTranslationUnit.h

43

Does the name of this class make sense? If I say

CrossTranslationUnit *ctuptr = new CrossTranslationUnit(...);

did I construct a new CrossTranslationUnit? What even is a CrossTranslationUnit? Other class names, such as ASTImporter, DiagOpts, and FrontendAction, etc. somehow feel better at transmitting reflecting upon their meaning in their name.

lib/CrossTU/CrossTranslationUnit.cpp

47

If we are using USR for lookup, then why does this look up "based on mangled name"?

[xazax.hun]

In D34512#836831, @whisperity wrote:

Apart from those in the in-line comments, I have a question: how safe is this library to Release builds? I know this is only a submodule dependency for the "real deal" in D30691, but I have seen some asserts that "imported function should already have a body" and such.

Will the static analyzer handle these errors gracefully and fall back to "function is unknown, let's throw my presumptions out of the window" or will it bail away? I'm explicitly thinking of the assert-lacking Release build.

The basic idea is that, if a function already have a body in the current translation unit and you still want to import it from somewhere else, it might be a programmer error, so we do not handle this gracefully.

include/clang/CrossTU/CrossTranslationUnit.h

43

What would you think about CrossTranslationUnitContext?

The functionality of this class is have all the relevant data required for doing Cross TU lookups and also provide the interface for that.

[whisperity]

include/clang/CrossTU/CrossTranslationUnit.h

43

WFM. I'm not sure whose authority is the class names, but I like it a lot better.

[xazax.hun]

• Address review comments

[dkrupp]

The creation of this library (libCrossTU) is approved for importing function definitions. @zaks.anna, @NoQ , @klimek could you please help us reviewing the code itself?

Then, when this is approved, we could progress with the review of the dependent "Support for naive cross translational unit analysis" (https://reviews.llvm.org/D30691) feature. The two patch is now in sync.
Thanks a lot in advance.

[dcoughlin]

Except for some drive-by nits, this is a high-level review.

I have some high-level questions about the design of the library:

1. How do you intend to handle the case when there are multiple function definitions with the same USR? Whose responsibility it is to deal with this: the client (for example, the static analyzer) or the index/database (libCrossTU)? This seems to me to be a fundamental part of the design for this feature that is missing. If you expect the client to handle this scenario (for example, to pick a definition) I would expect the library API to return more than a single potential result for a query. If you expect the the library to pick a definition then at a minimum you should document how this will be done. If the library will treat this as an error then you should communicate this to the client so it can attempt to recover from it.

2. Whose responsibility is it to handle errors that the library runs into? Right now several errors appear to reported to the end user as diagnostics. It seems to me that the decision of how (or whether) to report these errors to the end user should be up to a particular client. Are these errors even actionable on the part of end users? My sense it that with the envisioned workflow users would not know/care about the format of the database file.

3. Where should the code that understands the database file format live? Right now the logic for the parsing of the index format is in libCrossTU and the emission of the index is in the command-line tool. I think it would be easier to maintain if there were a single place that understand the file format. This way consumers and emitters of the index could be easily updated when the format/representation changes. Additionally, I think it is important to document this format.

include/clang/Basic/DiagnosticCrossTUKinds.td

13 ↗

(On Diff #110559)

Is this a user-facing diagnostic? Do we expect users to know what 'CrossTU' means? Or what 'USR' means?

include/clang/CrossTU/CrossTranslationUnit.h

61

Can this document what happens when the function declaration cannot be found? And what happens when multiple function declarations are found?

lib/CrossTU/CrossTranslationUnit.cpp

49

Should this assert that DC is not null? What is the reason to accept null here?

82

What is the rationale behind emitting this as a diagnostic? In general for a library I would expect that errors would be communicated to the client, which then would have responsibility for handling them, reporting them to the user, or aborting.

93

It looks like this is parsing. Is the file format documented anywhere? Also, it would be good to keep the parsing and generation code in the same place so that it is easy to figure out what to update when the file format changes.

tools/clang-func-mapping/ClangFnMapGen.cpp

87

It seems like the functionality that writes an entry and the functionality that reads an entry should ideally be in the same place. This way when the format is updated it is obvious what other places need to be updated. Can the code that writes a mapping be moved to libCrossTU and can we have this tool link against it?

[xazax.hun]

In D34512#854729, @dcoughlin wrote:

Except for some drive-by nits, this is a high-level review.

I have some high-level questions about the design of the library:

1. How do you intend to handle the case when there are multiple function definitions with the same USR? Whose responsibility it is to deal with this: the client (for example, the static analyzer) or the index/database (libCrossTU)? This seems to me to be a fundamental part of the design for this feature that is missing. If you expect the client to handle this scenario (for example, to pick a definition) I would expect the library API to return more than a single potential result for a query. If you expect the the library to pick a definition then at a minimum you should document how this will be done. If the library will treat this as an error then you should communicate this to the client so it can attempt to recover from it.

Right now we rely on the script that is using the analyzer such as scan-build-py or codechecker to remove the duplicate USRs from the index before starting the analysis. Having an ErrorOr return type so the user can handle the case when multiple definitions or none of them is found could be useful. It is not feasible to return multiple definitions, however. The importing alters the ASTContext, so the only way to choose between the definitions would be via a callback that is triggered before the import is done. What do you think?

2. Whose responsibility is it to handle errors that the library runs into? Right now several errors appear to reported to the end user as diagnostics. It seems to me that the decision of how (or whether) to report these errors to the end user should be up to a particular client. Are these errors even actionable on the part of end users? My sense it that with the envisioned workflow users would not know/care about the format of the database file.

The intention was that to report to the user that the index format is invalid, since it is a configuration issue that she might be able to handle. But we could also return an error code and leave the reporting to the client. The reason we introduced the diagnostic is that we assumed that the client can not recover from a configuration error and will end up reporting the problem to the user.

3. Where should the code that understands the database file format live? Right now the logic for the parsing of the index format is in libCrossTU and the emission of the index is in the command-line tool. I think it would be easier to maintain if there were a single place that understand the file format. This way consumers and emitters of the index could be easily updated when the format/representation changes. Additionally, I think it is important to document this format.

This is a very good point! I will update the library.

include/clang/Basic/DiagnosticCrossTUKinds.td

13 ↗

(On Diff #110559)

It is. Do you have any alternative suggestion? What about error parsing index file without mentioning any of the terms you mentioned?

include/clang/CrossTU/CrossTranslationUnit.h

61

Sure. Good point.

lib/CrossTU/CrossTranslationUnit.cpp

49

Indeed, an assert should be better.

82

Since this is likely to be a configuration error we were thinking that only the user can handle it. But I could transform this to an error code if you would prefer that solution.

93

Good point!

tools/clang-func-mapping/ClangFnMapGen.cpp

87

Sure, good point!

[xazax.hun]

• Updates according to review comments.

[dcoughlin]

The importing alters the ASTContext, so the only way to choose between the definitions would be via a callback that is triggered before the import is done. What do you think?

I think that could work. Another possibility would be to have a two step process. The first step would return a representation of possible definitions to import; then the client would chose which one to important and call another API to perform the actual import.

In either case, the important scenario I think we should support is choosing at a call site to a C function the most likely definition of the called function, based on number and type of parameters, from multiple possible definitions in other translation units. If the API is rich enough to support this then I think that is a good indication it will be useful for other scenarios as well.

The reason we introduced the diagnostic is that we assumed that the client can not recover from a configuration error and will end up reporting the problem to the user.

For the static analyzer client, at least, one possible recovery is performing the existing invalidation that we do when calling a function defined in another translation unit. (I'm not really sure this a good idea; I just wanted to point out that the decision probably belongs with the client). I think it is reasonable to report an error as a diagnostic -- but I think this should be up to the client and I don't think it should show up in the index file itself. In an ideal world the user wouldn't even know that file existed. Further, for large projects/incremental rebuilds a text file is unlikely to be a suitable representation for the index. In the long term I doubt the file will be end-user editable/readable.

[xazax.hun]

In D34512#856184, @dcoughlin wrote:

In either case, the important scenario I think we should support is choosing at a call site to a C function the most likely definition of the called function, based on number and type of parameters, from multiple possible definitions in other translation units. If the API is rich enough to support this then I think that is a good indication it will be useful for other scenarios as well.

Note that the lookup is already based on USR which is similar to mangled names in a sense that it contains information about the function parameters. So the only way to get multiple candidates from the lookup is having multiple function definitions with the same signature. It is only possible to disambiguate with knowledge about the linking graph. I think that information is better to be stored in the index in the future and do the disambiguation within the library instead of making the client do that.

The reason we introduced the diagnostic is that we assumed that the client can not recover from a configuration error and will end up reporting the problem to the user.

For the static analyzer client, at least, one possible recovery is performing the existing invalidation that we do when calling a function defined in another translation unit. (I'm not really sure this a good idea; I just wanted to point out that the decision probably belongs with the client). I think it is reasonable to report an error as a diagnostic -- but I think this should be up to the client and I don't think it should show up in the index file itself. In an ideal world the user wouldn't even know that file existed. Further, for large projects/incremental rebuilds a text file is unlikely to be a suitable representation for the index. In the long term I doubt the file will be end-user editable/readable.

I agree that we do not consider this format to be final for the index. In the future, it would be great to include linking information as well and also have utilities to merge partial indexes. This is a first version to make the machinery work for the most common cases which are already pretty usable for the open source projects we tested.

[xazax.hun]

• Added unit test to ensure the produced index format can be parsed.
• Added further diagnostics.

[benlangmuir]

In D34512#856301, @xazax.hun wrote:

In D34512#856184, @dcoughlin wrote:

In either case, the important scenario I think we should support is choosing at a call site to a C function the most likely definition of the called function, based on number and type of parameters, from multiple possible definitions in other translation units. If the API is rich enough to support this then I think that is a good indication it will be useful for other scenarios as well.

Note that the lookup is already based on USR which is similar to mangled names in a sense that it contains information about the function parameters. So the only way to get multiple candidates from the lookup is having multiple function definitions with the same signature.

I just want to clarify that C function USRs do not contain type information, although C++ USRs do.

[xazax.hun]

In D34512#856821, @benlangmuir wrote:

In D34512#856301, @xazax.hun wrote:

In D34512#856184, @dcoughlin wrote:

In either case, the important scenario I think we should support is choosing at a call site to a C function the most likely definition of the called function, based on number and type of parameters, from multiple possible definitions in other translation units. If the API is rich enough to support this then I think that is a good indication it will be useful for other scenarios as well.

Note that the lookup is already based on USR which is similar to mangled names in a sense that it contains information about the function parameters. So the only way to get multiple candidates from the lookup is having multiple function definitions with the same signature.

I just want to clarify that C function USRs do not contain type information, although C++ USRs do.

Just double checked and indeed: https://github.com/llvm-mirror/clang/blob/master/lib/Index/USRGeneration.cpp#L236
Thank you for the clarification, it looks like I did not notice this detail. In this case, it might make sense to let the client disambiguate.

[danielmarjamaki]

small nits

include/clang/CrossTU/CrossTranslationUnit.h

59

I suggest that "can" is removed.

63

there is no \return or \returns here.

63

maybe: each line consists of an USR and a filepath separated by a space

69

Maybe /can be/is/ unless you envision that tools that require cross translation unit capability might use some other classes.

93

you should split out some of this information to a \return or \returns

lib/CrossTU/CrossTranslationUnit.cpp

61

redundant assignment

80

I suggest that LineNo is 1 on the first line.

82

Pos can be const

83

LineRef can be const

85

FunctionName and FileName can be moved here and it is possible to make these variables const.

86

I would like to see some FunctionName validation. For instance "123" should not be a valid function name.

90

Stupid question .. how will this work if the path is longer than 256 bytes?

103

how about std::ostringstream , imho that is cleaner

122

/funtion/function/

126

LookupFnName could be const right?

149

I believe LookupFnName can be const

190

It is possible to make ASTFileName const

224

sorry I am probably missing something here.. you first assert !FD->hasBody() and here you assert FD->hasBody(). Is it changed in this function somewhere?

tools/clang-func-mapping/ClangFnMapGen.cpp

79

I believe that realpath() is a posix function

unittests/CrossTU/CrossTranslationUnitTest.cpp

41

should this be assert(FD && FD->getName() == "f")

[xazax.hun]

• Make the API capable of using custom lookup strategies for CTU
• Fix review comments

lib/CrossTU/CrossTranslationUnit.cpp

83

StringRef is an immutable reference to a string, I think it is not idiomatic in LLVM codebase to make it const.

86

This is not a real function name but an USR. I updated the name of the variable to reflect that this name is only used for lookup.

90

If the path is shorter than 256 bytes it will be stored on stack, otherwise SmallString will allocate on the heap.

126

In LLVM we usually do not mark StringRefs as consts, they represent a const reference to a string.

224

Yes, after the importer imports the new declaration with a body we should be able to find the body through the original declaration. The importer modifies the AST and the supporting data structures.

[dcoughlin]

Thanks Gabor! Some additional comments in line.

include/clang/CrossTU/CrossTranslationUnit.h

119

Is this comment correct? Does it return a map?

129

I found this comment confusing. Is 'Unit' the separate ASTUnit or it the current one?

135

If this is public API, it deserves a comment. If not, can we make it private?

lib/CrossTU/CrossTranslationUnit.cpp

36

Is this FIXME still relevant? What will need to be transitioned to llvm::Error for it to be removed? Can you make the comment a bit more specific so that future maintainers will know when/how to address the FIXME?

195

What is this cast for? Is it to suppress an analyzer dead store false positive? I thought we got all those!

200

If I understand correctly, there is no way to call loadExternalAST() without the potential for a diagnostic showing up to the user. Can this diagnostic emission be moved to the client?

250

Does this method need to take the unit? Can it just get the ASTContext from 'FD'? If not, can we add an assertion that FD has the required relationship to Unit? (And document it in the header doc.)

tools/clang-func-mapping/ClangFnMapGen.cpp

57

Is getLookupName() here used?

73

Should this instead reuse the logic for generating a lookup name from CrossTranslationUnitContext::getLookupName()? That way if we change how the lookup name is generated we only have to do it in one place.

[xazax.hun]

• Address review comments.

lib/CrossTU/CrossTranslationUnit.cpp

36

Unfortunately, it is. IndexError needs to implement convertToErrorCode because it is pure virtual in the base class, and that requires the existence of this class. So this can only be removed, once convertToErrorCode is pruned globally from the codebase. I think all of such classes are marked with similar FIXMEs.

195

The point was to avoid an assertion fail where an error was not checked. The reason, there is no really good way to propagate errors right now. But with this diagnostic emission moved to the client this is no longer required.

[xazax.hun]

Any further reviews?
What are the criteria to accept this patch? Who or how many people should accept this?

[dcoughlin]

I'm OK with this going into the repo a is (although it is light on tests!), as long as we have an agreement that you'll be OK with iteration on both the interface and the implementation to handle real-world projects.

More specifically, for this to work well on large/complicated projects we'll need to:

• Move conflict resolution from index generation where it is now to query time so that clients can pick the best implementation of a function when multiple are found. This is important for projects that have multiple functions with the same name or that build the same file in multiple ways.
• Change function lookup from needing to traverse over the entire AST.
• Move away from a linear, flat-file index to something that can handle larger projects more efficiently.

For this last point, I suspect it will be good to adopt whatever Clangd ends up using to support indexing. (There has been some discussion of this relatively recently on the cfe-dev list.)

[xazax.hun]

In D34512#877032, @dcoughlin wrote:

I'm OK with this going into the repo a is (although it is light on tests!), as long as we have an agreement that you'll be OK with iteration on both the interface and the implementation to handle real-world projects.

I agree that it could have more tests, but while the main features are already tested, more tests are also coming with the first usage of the library (the Static Analyzer). We are also willing to add new tests as the interface/library evolves.

More specifically, for this to work well on large/complicated projects we'll need to:

• Move conflict resolution from index generation where it is now to query time so that clients can pick the best implementation of a function when multiple are found. This is important for projects that have multiple functions with the same name or that build the same file in multiple ways.

I agree. Right now the way we use this internally, the collisions are handled after index generation by an external tool (CodeChecker).

• Change function lookup from needing to traverse over the entire AST.
• Move away from a linear, flat-file index to something that can handle larger projects more efficiently.

For this last point, I suspect it will be good to adopt whatever Clangd ends up using to support indexing. (There has been some discussion of this relatively recently on the cfe-dev list.)

The index right now is a minimal implementation to make this functionality work. This part was never the bottleneck according to our measurements so far. It would be great to reuse the index format from ClangD in case it supports to generate lightweight indexes that only contain the data we need for analysis. (We only need an index of function definitions for the analyzer and nothing else. I suspect other clients also would not need other information.)

I think it would be a waste of effort to change the lookup strategy or move the collision handling mechanisms to index generation until we do not know how the index will work. But I also think it wouldn't be good to block this until ClanD indexing reaching a mature state.

All in all, we are willing to reuse as much of ClangD indexing as we can in the future, but I think it is also more aligned with the LLVM Developer Policy to have something working committed and do the development in the repo rather than working separately on a fork.

[xazax.hun]

• Unittests now creates temporary files at the correct location
• Temporary files are also cleaned up when the process is terminated
• Absolute paths are handled correctly by the library

[dcoughlin]

But I also think it wouldn't be good to block this until ClanD indexing reaching a mature state.

I agree 100%.

All in all, we are willing to reuse as much of ClangD indexing as we can in the future, but I think it is also more aligned with the LLVM Developer Policy to have something working committed and do the development in the repo rather than working separately on a fork.

This sounds good to me. I just wanted to make sure that you were onboard with changing this index to use the common indexing infrastructure in the future. (Assuming it makes sense to do so, of course).

[r.stahl]

In D34512#877643, @xazax.hun wrote:

• Unittests now creates temporary files at the correct location
• Temporary files are also cleaned up when the process is terminated
• Absolute paths are handled correctly by the library

I think there is an issue with this change.

First, the function map generation writes absolute paths to the map file. Now when the parseCrossTUIndex function parses it, it will no longer prepend the paths with the CTUDir and therefore expect the precompiled AST files at the exact path of the source file. This seems like a bad requirement, because the user would have to overwrite his source files.

If I understand your intention correctly, a fix would be to replace the is_absolute check with a check for CTUDir == "" in the parseCrossTUIndex function.

[xazax.hun]

In D34512#890537, @r.stahl wrote:

In D34512#877643, @xazax.hun wrote:

• Unittests now creates temporary files at the correct location
• Temporary files are also cleaned up when the process is terminated
• Absolute paths are handled correctly by the library

I think there is an issue with this change.

First, the function map generation writes absolute paths to the map file. Now when the parseCrossTUIndex function parses it, it will no longer prepend the paths with the CTUDir and therefore expect the precompiled AST files at the exact path of the source file. This seems like a bad requirement, because the user would have to overwrite his source files.

If I understand your intention correctly, a fix would be to replace the is_absolute check with a check for CTUDir == "" in the parseCrossTUIndex function.

Good catch, that is right!

include/clang/Basic/DiagnosticFrontendKinds.td

 
 def err_missing_vfs_overlay_file : Error<
   "virtual filesystem overlay file '%0' not found">, DefaultFatal;
 def err_invalid_vfs_overlay : Error<
   "invalid virtual filesystem overlay file '%0'">, DefaultFatal;
 
 def warn_option_invalid_ocl_version : Warning<
   "OpenCL version %0 does not support the option '%1'">, InGroup<Deprecated>;
+
+def err_fnmap_parsing : Error<
+  "error parsing CrossTU index file: '%0' line: %1 'USR filename' format "
+  "expected">;
 }

include/clang/CrossTU/CrossTranslationUnit.h

+//===--- CrossTranslationUnit.h - -------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+//  This file implements the CrossTranslationUnit interface.
+//
+//===----------------------------------------------------------------------===//
+#ifndef LLVM_CLANG_TOOLING_CROSSTRANSLATIONUNIT_H
+#define LLVM_CLANG_TOOLING_CROSSTRANSLATIONUNIT_H
+
+#include "clang/Basic/LLVM.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/StringMap.h"
+
+namespace clang {
+class CompilerInstance;
+class ASTContext;
+class ASTImporter;
+class ASTUnit;
+class DeclContext;
+class FunctionDecl;
+class NamedDecl;
+class TranslationUnitDecl;
+
+namespace crossTU {
+
+/// \brief This class can be used for tools that requires cross translation
+///        unit capability.
+///
+/// This class can load function definitions from external AST files.
+/// The loaded definition will be merged back to the original AST using the
+/// AST Importer.
+/// In order to use this class, an index file is required that describes
+/// the locations of the AST files for each function definition.
+///
+/// Note that this class also implements caching.
+class CrossTranslationUnit {
+public:
+  CrossTranslationUnit(CompilerInstance &CI);
+  ~CrossTranslationUnit();
+
+  /// \brief This function can load a function definition from an external AST
+  ///        file and merge it into the original AST.
+  ///
+  /// This method should only be used on functions that have no definitions in
+  /// the current translation unit. A function definition with the same
+  /// declaration will be looked up in the index file which should be in the
+  /// \p CrossTUDir directory, called \p IndexName. In case the declaration is
+  /// found in the index the corresponding AST file will be loaded and the
+  /// definition of the function will be merged into the original AST using
+  /// the AST Importer. The declaration with the definition will be returned.
+  ///
+  /// Note that the AST files should also be in the \p CrossTUDir.
+  const FunctionDecl *getCrossTUDefinition(const FunctionDecl *FD,
+                                           StringRef CrossTUDir,
+                                           StringRef IndexName);
+
+private:
+  ASTImporter &getOrCreateASTImporter(ASTContext &From);
+  std::string getLookupName(const NamedDecl *ND);
+  const FunctionDecl *findFunctionInDeclContext(const DeclContext *DC,
+                                                StringRef LookupFnName);
+
+  llvm::StringMap<std::unique_ptr<clang::ASTUnit>> FileASTUnitMap;
+  llvm::StringMap<clang::ASTUnit *> FunctionAstUnitMap;
+  llvm::StringMap<std::string> FunctionFileMap;
+  llvm::DenseMap<TranslationUnitDecl *, std::unique_ptr<ASTImporter>>
+      ASTUnitImporterMap;
+  CompilerInstance &CI;
+  ASTContext &Context;
+};
+
+} // namespace tooling
+} // namespace clang
+
+#endif // LLVM_CLANG_TOOLING_CROSSTRANSLATIONUNIT_H

lib/AST/ASTImporter.cpp

Decl *ASTNodeImporter::VisitFunctionDecl(FunctionDecl *D) {
   DeclarationName Name;
   SourceLocation Loc;
   NamedDecl *ToD;
   if (ImportDeclParts(D, DC, LexicalDC, Name, ToD, Loc))
     return nullptr;
   if (ToD)
     return ToD;
 
+  const FunctionDecl *FoundWithoutBody = nullptr;
+
   // Try to find a function in our own ("to") context with the same name, same
   // type, and in the same context as the function we're importing.
   if (!LexicalDC->isFunctionOrMethod()) {
     SmallVector<NamedDecl *, 4> ConflictingDecls;
     unsigned IDNS = Decl::IDNS_Ordinary;
     SmallVector<NamedDecl *, 2> FoundDecls;
     DC->getRedeclContext()->localUncachedLookup(Name, FoundDecls);
     for (unsigned I = 0, N = FoundDecls.size(); I != N; ++I) {
       if (!FoundDecls[I]->isInIdentifierNamespace(IDNS))
         continue;
 
       if (FunctionDecl *FoundFunction = dyn_cast<FunctionDecl>(FoundDecls[I])) {
         if (FoundFunction->hasExternalFormalLinkage() &&
             D->hasExternalFormalLinkage()) {
           if (Importer.IsStructurallyEquivalent(D->getType(), 
                                                 FoundFunction->getType())) {
             // FIXME: Actually try to merge the body and other attributes.
+            const FunctionDecl *FromBodyDecl = nullptr;
+            D->hasBody(FromBodyDecl);
+            if (D == FromBodyDecl && !FoundFunction->hasBody()) {
+              // This function is needed to merge completely.
+              FoundWithoutBody = FoundFunction;
+              break;
+            }
             return Importer.Imported(D, FoundFunction);
           }
 
           // FIXME: Check for overloading more carefully, e.g., by boosting
           // Sema::IsOverload out to the AST library.
 
           // Function overloading is okay in C++.
           if (Importer.getToContext().getLangOpts().CPlusPlus)
Decl *ASTNodeImporter::VisitFunctionDecl(FunctionDecl *D) {
 
   // Set the parameters.
   for (unsigned I = 0, N = Parameters.size(); I != N; ++I) {
     Parameters[I]->setOwningFunction(ToFunction);
     ToFunction->addDeclInternal(Parameters[I]);
   }
   ToFunction->setParams(Parameters);
 
+  if (FoundWithoutBody) {
+    auto *Recent = const_cast<FunctionDecl *>(
+          FoundWithoutBody->getMostRecentDecl());
+    ToFunction->setPreviousDecl(Recent);
+  }
+
   if (usedDifferentExceptionSpec) {
     // Update FunctionProtoType::ExtProtoInfo.
     QualType T = Importer.Import(D->getType());
     if (T.isNull())
       return nullptr;
     ToFunction->setType(T);
   }
 

lib/CrossTU/CMakeLists.txt

+set(LLVM_LINK_COMPONENTS
+  Support
+  )
+
+add_clang_library(clangCrossTU
+  CrossTranslationUnit.cpp
+
+  LINK_LIBS
+  clangAST
+  clangBasic
+  clangFrontend
+  clangIndex
+  )

lib/CrossTU/CrossTranslationUnit.cpp

+//===--- CrossTranslationUnit.cpp - -----------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+//  This file provides an interface to load binary AST dumps on demand. This
+//  feature can be utilized for tools that require cross translation unit
+//  support.
+//
+//===----------------------------------------------------------------------===//
+#include "clang/CrossTU/CrossTranslationUnit.h"
+#include "clang/AST/ASTImporter.h"
+#include "clang/AST/Decl.h"
+#include "clang/Basic/TargetInfo.h"
+#include "clang/Frontend/ASTUnit.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Frontend/FrontendDiagnostic.h"
+#include "clang/Frontend/TextDiagnosticPrinter.h"
+#include "clang/Index/USRGeneration.h"
+#include "llvm/ADT/Triple.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/raw_ostream.h"
+#include <fstream>
+
+namespace clang {
+namespace crossTU {
+
+CrossTranslationUnit::CrossTranslationUnit(CompilerInstance &CI)
+    : CI(CI), Context(CI.getASTContext()) {}
+
+CrossTranslationUnit::~CrossTranslationUnit() {}
+
+std::string CrossTranslationUnit::getLookupName(const NamedDecl *ND) {
+  SmallString<128> DeclUSR;
+  bool Ret = index::generateUSRForDecl(ND, DeclUSR);
+  assert(!Ret);
+  return DeclUSR.str();
+}
+
+/// Recursively visit the funtion decls of a DeclContext, and looks up a
+/// function based on mangled name.
+const FunctionDecl *
+CrossTranslationUnit::findFunctionInDeclContext(const DeclContext *DC,
+                                                StringRef LookupFnName) {
+  if (!DC)
+    return nullptr;
+  for (const Decl *D : DC->decls()) {
+    const auto *SubDC = dyn_cast<DeclContext>(D);
+    if (const auto *FD = findFunctionInDeclContext(SubDC, LookupFnName))
+      return FD;
+
+    const auto *ND = dyn_cast<FunctionDecl>(D);
+    const FunctionDecl *ResultDecl;
+    if (!ND || !ND->hasBody(ResultDecl))
+      continue;
+    if (getLookupName(ResultDecl) != LookupFnName)
+      continue;
+    return ResultDecl;
+  }
+  return nullptr;
+}
+
+const FunctionDecl *CrossTranslationUnit::getCrossTUDefinition(
+    const FunctionDecl *FD, StringRef CrossTUDir, StringRef IndexName) {
+  assert(!FD->hasBody() && "FD has a definition in current translation unit!");
+
+  std::string LookupFnName = getLookupName(FD);
+  if (LookupFnName.empty())
+    return nullptr;
+  ASTUnit *Unit = nullptr;
+  auto FnUnitCacheEntry = FunctionAstUnitMap.find(LookupFnName);
+  if (FnUnitCacheEntry == FunctionAstUnitMap.end()) {
+    if (FunctionFileMap.empty()) {
+      SmallString<256> ExternalFunctionMap = CrossTUDir;
+      llvm::sys::path::append(ExternalFunctionMap, IndexName);
+      std::ifstream ExternalFnMapFile(ExternalFunctionMap.c_str());
+      if (!ExternalFnMapFile) {
+        Context.getDiagnostics().Report(diag::err_fe_error_opening)
+            << ExternalFunctionMap << "required by the CrossTU functionality";
+        return nullptr;
+      }
+
+      StringRef FunctionName, FileName;
+      std::string Line;
+      unsigned LineNo = 0;
+      while (std::getline(ExternalFnMapFile, Line)) {
+        size_t Pos = Line.find(" ");
+        StringRef LineRef{Line};
+        if (Pos > 0 && Pos != std::string::npos) {
+          FunctionName = LineRef.substr(0, Pos);
+          FileName = LineRef.substr(Pos + 1);
+          SmallString<256> FilePath = CrossTUDir;
+          llvm::sys::path::append(FilePath, FileName);
+          FunctionFileMap[FunctionName] = FilePath.str().str();
+        } else {
+          Context.getDiagnostics().Report(diag::err_fnmap_parsing)
+              << ExternalFunctionMap << (LineNo + 1);
+        }
+        LineNo++;
+      }
+    }
+
+    StringRef ASTFileName;
+    auto It = FunctionFileMap.find(LookupFnName);
+    if (It == FunctionFileMap.end())
+      return nullptr; // No definition found even in some other build unit.
+    ASTFileName = It->second;
+    auto ASTCacheEntry = FileASTUnitMap.find(ASTFileName);
+    if (ASTCacheEntry == FileASTUnitMap.end()) {
+      IntrusiveRefCntPtr<DiagnosticOptions> DiagOpts = new DiagnosticOptions();
+      TextDiagnosticPrinter *DiagClient =
+          new TextDiagnosticPrinter(llvm::errs(), &*DiagOpts);
+      IntrusiveRefCntPtr<DiagnosticIDs> DiagID(new DiagnosticIDs());
+      IntrusiveRefCntPtr<DiagnosticsEngine> Diags(
+          new DiagnosticsEngine(DiagID, &*DiagOpts, DiagClient));
+
+      std::unique_ptr<ASTUnit> LoadedUnit(ASTUnit::LoadFromASTFile(
+          ASTFileName, CI.getPCHContainerOperations()->getRawReader(),
+          ASTUnit::LoadEverything, Diags, CI.getFileSystemOpts()));
+      Unit = LoadedUnit.get();
+      FileASTUnitMap[ASTFileName] = std::move(LoadedUnit);
+    } else {
+      Unit = ASTCacheEntry->second.get();
+    }
+    FunctionAstUnitMap[LookupFnName] = Unit;
+  } else {
+    Unit = FnUnitCacheEntry->second;
+  }
+
+  if (!Unit)
+    return nullptr;
+  assert(&Unit->getFileManager() ==
+         &Unit->getASTContext().getSourceManager().getFileManager());
+  ASTImporter &Importer = getOrCreateASTImporter(Unit->getASTContext());
+  TranslationUnitDecl *TU = Unit->getASTContext().getTranslationUnitDecl();
+  if (const FunctionDecl *ResultDecl =
+          findFunctionInDeclContext(TU, LookupFnName)) {
+    auto *ToDecl = cast<FunctionDecl>(
+        Importer.Import(const_cast<FunctionDecl *>(ResultDecl)));
+    assert(ToDecl->hasBody());
+    assert(FD->hasBody() && "Functions already imported should have body.");
+    return ToDecl;
+  }
+  return nullptr;
+}
+
+ASTImporter &CrossTranslationUnit::getOrCreateASTImporter(ASTContext &From) {
+  auto I = ASTUnitImporterMap.find(From.getTranslationUnitDecl());
+  if (I != ASTUnitImporterMap.end())
+    return *I->second;
+  ASTImporter *NewImporter =
+      new ASTImporter(Context, Context.getSourceManager().getFileManager(),
+                      From, From.getSourceManager().getFileManager(), false);
+  ASTUnitImporterMap[From.getTranslationUnitDecl()].reset(NewImporter);
+  return *NewImporter;
+}
+
+} // namespace tooling
+} // namespace clang

test/Analysis/func-mapping-test.cpp

+// RUN: %clang_func_map %s -- | FileCheck %s
+
+int f(int) {
+  return 0;
+}
+
+// CHECK: c:@F@f#I#

test/CMakeLists.txt

 
 list(APPEND CLANG_TEST_DEPS
   clang clang-headers
   clang-format
   c-index-test diagtool
   clang-tblgen
   clang-offload-bundler
   clang-import-test
+  clang-func-mapping
   )
   
 if(CLANG_ENABLE_STATIC_ANALYZER)
   list(APPEND CLANG_TEST_DEPS
     clang-check
     )
 endif()
 

test/lit.cfg

config.substitutions.append( ('%clang_cc1',
                               % (config.clang,
                                  getClangBuiltinIncludeDir(config.clang))) )
 config.substitutions.append( ('%clang_cpp', ' ' + config.clang +
                               ' --driver-mode=cpp '))
 config.substitutions.append( ('%clang_cl', ' ' + config.clang +
                               ' --driver-mode=cl '))
 config.substitutions.append( ('%clangxx', ' ' + config.clang +
                               ' --driver-mode=g++ '))
+config.substitutions.append( ('%clang_func_map', ' ' + lit.util.which('clang-func-mapping', config.environment['PATH']) + ' ') )
 config.substitutions.append( ('%clang', ' ' + config.clang + ' ') )
 config.substitutions.append( ('%test_debuginfo', ' ' + config.llvm_src_root + '/utils/test_debuginfo.pl ') )
 config.substitutions.append( ('%itanium_abi_triple', makeItaniumABITriple(config.target_triple)) )
 config.substitutions.append( ('%ms_abi_triple', makeMSABITriple(config.target_triple)) )
 config.substitutions.append( ('%resource_dir', getClangBuiltinIncludeDir(config.clang)) )
 
 # The host triple might not be set, at least if we're compiling clang from
 # an already installed llvm.

tools/CMakeLists.txt

if(CLANG_ENABLE_ARCMT)
   add_clang_subdirectory(arcmt-test)
   add_clang_subdirectory(c-arcmt-test)
 endif()
 
 if(CLANG_ENABLE_STATIC_ANALYZER)
   add_clang_subdirectory(clang-check)
   add_clang_subdirectory(scan-build)
   add_clang_subdirectory(scan-view)
+  add_clang_subdirectory(clang-func-mapping)
 endif()
 
 # We support checking out the clang-tools-extra repository into the 'extra'
 # subdirectory. It contains tools developed as part of the Clang/LLVM project
 # on top of the Clang tooling platform. We keep them in a separate repository
 # to keep the primary Clang repository small and focused.
 # It also may be included by LLVM_EXTERNAL_CLANG_TOOLS_EXTRA_SOURCE_DIR.
 add_llvm_external_project(clang-tools-extra extra)
 
 # libclang may require clang-tidy in clang-tools-extra.
 add_clang_subdirectory(libclang)

tools/clang-func-mapping/CMakeLists.txt

+set(LLVM_LINK_COMPONENTS
+  ${LLVM_TARGETS_TO_BUILD}
+  asmparser
+  support
+  mc
+  )
+
+add_clang_executable(clang-func-mapping
+  ClangFnMapGen.cpp
+  )
+
+target_link_libraries(clang-func-mapping
+  clangAST
+  clangBasic
+  clangFrontend
+  clangIndex
+  clangTooling
+  )
+
+install(TARGETS clang-func-mapping
+  RUNTIME DESTINATION bin)

tools/clang-func-mapping/ClangFnMapGen.cpp

+//===- ClangFnMapGen.cpp -----------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===--------------------------------------------------------------------===//
+//
+// Clang tool which creates a list of defined functions and the files in which
+// they are defined.
+//
+//===--------------------------------------------------------------------===//
+
+#include "clang/AST/ASTConsumer.h"
+#include "clang/AST/ASTContext.h"
+#include "clang/AST/GlobalDecl.h"
+#include "clang/AST/Mangle.h"
+#include "clang/AST/StmtVisitor.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Basic/TargetInfo.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Frontend/FrontendActions.h"
+#include "clang/Index/USRGeneration.h"
+#include "clang/Tooling/CommonOptionsParser.h"
+#include "clang/Tooling/Tooling.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/Signals.h"
+#include <sstream>
+#include <string>
+#include <vector>
+
+using namespace llvm;
+using namespace clang;
+using namespace clang::tooling;
+
+static cl::OptionCategory ClangFnMapGenCategory("clang-fnmapgen options");
+
+class MapFunctionNamesConsumer : public ASTConsumer {
+public:
+  MapFunctionNamesConsumer(ASTContext &Context) : Ctx(Context) {}
+
+  ~MapFunctionNamesConsumer() {
+    // Flush results to standard output.
+    llvm::outs() << DefinedFuncsStr.str();
+  }
+
+  virtual void HandleTranslationUnit(ASTContext &Ctx) {
+    handleDecl(Ctx.getTranslationUnitDecl());
+  }
+
+private:
+  std::string getLookupName(const FunctionDecl *FD);
+  void handleDecl(const Decl *D);
+
+  ASTContext &Ctx;
+  std::stringstream DefinedFuncsStr;
+  std::string CurrentFileName;
+};
+
+void MapFunctionNamesConsumer::handleDecl(const Decl *D) {
+  if (!D)
+    return;
+
+  if (const auto *FD = dyn_cast<FunctionDecl>(D)) {
+    if (FD->isThisDeclarationADefinition()) {
+      if (const Stmt *Body = FD->getBody()) {
+        SmallString<128> LookupName;
+        bool Res = index::generateUSRForDecl(D, LookupName);
+        assert(!Res);
+        const SourceManager &SM = Ctx.getSourceManager();
+        if (CurrentFileName.empty()) {
+          StringRef SMgrName =
+              SM.getFileEntryForID(SM.getMainFileID())->getName();
+          char *Path = realpath(SMgrName.str().c_str(), nullptr);
+          CurrentFileName = Path;
+          free(Path);
+        }
+
+        switch (FD->getLinkageInternal()) {
+        case ExternalLinkage:
+        case VisibleNoLinkage:
+        case UniqueExternalLinkage:
+          if (SM.isInMainFile(Body->getLocStart()))
+            DefinedFuncsStr << LookupName.str().str() << " " << CurrentFileName
+                            << "\n";
+        default:
+          break;
+        }
+      }
+    }
+  }
+
+  if (const auto *DC = dyn_cast<DeclContext>(D))
+    for (const Decl *D : DC->decls())
+      handleDecl(D);
+}
+
+class MapFunctionNamesAction : public ASTFrontendAction {
+protected:
+  std::unique_ptr<ASTConsumer> CreateASTConsumer(CompilerInstance &CI,
+                                                 llvm::StringRef) {
+    std::unique_ptr<ASTConsumer> PFC(
+        new MapFunctionNamesConsumer(CI.getASTContext()));
+    return PFC;
+  }
+};
+
+static cl::extrahelp CommonHelp(CommonOptionsParser::HelpMessage);
+
+int main(int argc, const char **argv) {
+  // Print a stack trace if we signal out.
+  sys::PrintStackTraceOnErrorSignal(argv[0], false);
+  PrettyStackTraceProgram X(argc, argv);
+
+  const char *Overview = "\nThis tool collects the USR name and location "
+                         "of all functions definitions in the source files "
+                         "(excluding headers).\n";
+  CommonOptionsParser OptionsParser(argc, argv, ClangFnMapGenCategory,
+                                    cl::ZeroOrMore, Overview);
+
+  ClangTool Tool(OptionsParser.getCompilations(),
+                 OptionsParser.getSourcePathList());
+  Tool.run(newFrontendActionFactory<MapFunctionNamesAction>().get());
+  return 0;
+}

unittests/CrossTU/CMakeLists.txt

+set(LLVM_LINK_COMPONENTS
+  ${LLVM_TARGETS_TO_BUILD}
+  Support
+  )
+
+add_clang_unittest(CrossTUTests
+  CrossTranslationUnitTest.cpp
+  )
+
+target_link_libraries(CrossTUTests
+  clangAST
+  clangBasic
+  clangCrossTU
+  clangFrontend
+  clangTooling
+  )

unittests/CrossTU/CrossTranslationUnitTest.cpp

+//===- unittest/Tooling/CrossTranslationUnitTest.cpp - Tooling unit tests -===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/AST/ASTConsumer.h"
+#include "clang/CrossTU/CrossTranslationUnit.h"
+#include "clang/Frontend/FrontendAction.h"
+#include "clang/Tooling/Tooling.h"
+#include "llvm/Config/llvm-config.h"
+#include "llvm/Support/Path.h"
+#include "gtest/gtest.h"
+#include <cassert>
+
+namespace clang {
+namespace crossTU {
+
+namespace {
+StringRef IndexFileName = "index.txt";
+StringRef ASTFileName = "f.ast";
+StringRef DefinitionFileName = "input.cc";
+
+class CTUASTConsumer : public clang::ASTConsumer {
+public:
+  explicit CTUASTConsumer(clang::CompilerInstance &CI, bool *Success)
+      : CTU(CI), Success(Success) {}
+
+  void HandleTranslationUnit(ASTContext &Ctx) {
+    const TranslationUnitDecl *TU = Ctx.getTranslationUnitDecl();
+    const FunctionDecl *FD = nullptr;
+    for (const Decl *D : TU->decls()) {
+      FD = dyn_cast<FunctionDecl>(D);
+      if (FD && FD->getName() == "f")
+        break;
+    }
+    assert(FD);
+    bool OrigFDHasBody = FD->hasBody();
+
+    // Prepare the index file and the AST file.
+    std::error_code EC;
+    llvm::raw_fd_ostream OS(IndexFileName, EC, llvm::sys::fs::F_Text);
+    OS << "c:@F@f#I# " << ASTFileName << "\n";
+    OS.flush();
+    StringRef SourceText = "int f(int) { return 0; }\n";
+    // This file must exist since the saved ASTFile will reference it.
+    llvm::raw_fd_ostream OS2(DefinitionFileName, EC, llvm::sys::fs::F_Text);
+    OS2 << SourceText;
+    OS2.flush();
+    std::unique_ptr<ASTUnit> ASTWithDefinition =
+        tooling::buildASTFromCode(SourceText);
+    ASTWithDefinition->Save(ASTFileName);
+
+    // Load the definition from the AST file.
+    const FunctionDecl *NewFD =
+        CTU.getCrossTUDefinition(FD, ".", IndexFileName);
+
+    *Success = NewFD && NewFD->hasBody() && !OrigFDHasBody;
+  }
+
+private:
+  CrossTranslationUnit CTU;
+  bool *Success;
+};
+
+class CTUAction : public clang::ASTFrontendAction {
+public:
+  CTUAction(bool *Success) : Success(Success) {}
+
+protected:
+  std::unique_ptr<clang::ASTConsumer>
+  CreateASTConsumer(clang::CompilerInstance &CI, StringRef) override {
+    return llvm::make_unique<CTUASTConsumer>(CI, Success);
+  }
+
+private:
+  bool *Success;
+};
+
+} // end namespace
+
+TEST(CrossTranslationUnit, CanLoadFunctionDefinition) {
+  bool Success = false;
+  EXPECT_TRUE(tooling::runToolOnCode(new CTUAction(&Success), "int f(int);"));
+  EXPECT_TRUE(Success);
+  EXPECT_TRUE(llvm::sys::fs::exists(IndexFileName));
+  EXPECT_FALSE((bool)llvm::sys::fs::remove(IndexFileName));
+  EXPECT_TRUE(llvm::sys::fs::exists(ASTFileName));
+  EXPECT_FALSE((bool)llvm::sys::fs::remove(ASTFileName));
+  EXPECT_TRUE(llvm::sys::fs::exists(DefinitionFileName));
+  EXPECT_FALSE((bool)llvm::sys::fs::remove(DefinitionFileName));
+}
+
+} // end namespace tooling
+} // end namespace clang