Add index-while-building support to Clang

[jkorous]

Summary

Adds a new -index-store-path option that causes Clang to additionally collect and output source code indexing information to the supplied path. This is done by wrapping the FrontendAction otherwise setup by the invocation with a WrappingIndexRecordAction. This action simply delegates to the wrapped action, but additionally multiplexes in its own IndexASTConsumer to collect symbol information from the AST and tracks the source file and module dependencies of the translation unit (via the IndexDependencyProvider class).

When the action completes, it then writes this information out to the supplied index store path in the form of a unit file, which stores the dependency information of the translation unit, and record files, that store the symbol and symbol occurrences seen in each source file. These are written out in the LLVM Bitstream format.

For a better (and more detailed) description of these changes, see the design document at: https://docs.google.com/document/d/1cH2sTpgSnJZCkZtJl1aY-rzy4uGPcrI-6RrUpdATO2Q/edit?usp=sharing

and the mailing list discussion 'RFC: Adding index-while-building support to Clang'

[arphaman]

lib/Index/IndexRecordHasher.cpp

104

Should you hash the return type as well?

205

You can use Qualifiers::Const here or make your own enum instead of raw constants.

[nathawes]

Thanks @arphaman! I'll work through your comments and update.

include/clang/Index/IndexDataStoreSymbolUtils.h

14

They're used by IndexRecordWriter below to convert from libIndex's representation of things to the index store's.

[nathawes]

lib/Index/IndexRecordHasher.cpp

104

The return type doesn't affect the function's USR, so there's no need to consider it when hashing the function decl. The hashing here is happening per decl occurrence (source offset + role set + Decl) collected by the index AST walker, so changing the return type will still change the record hash when any decl occurrences it contains are hashed in.

[nathawes]

Based on @arphaman's feedback:

• Pulled the index store related diagnostics out into their own category/diagnostic group
• Removed the CLANG_PROJECT_INDEX_PATH env var check.
• Swapped "/" used in a few places as a separator/root with the equivalent llvm::sys::path call.
• Fixed the typo/convention/documentation issues and simplifications pointed out so far

[malaperle]

lib/Index/IndexUnitWriter.cpp

213

extra semi-colon (noticed this warning while compiling)

[malaperle]

lib/Index/IndexingAction.cpp

559

As a first attempt, I tried to use index::createIndexDataRecordingAction in combination with ASTUnit::LoadFromCompilerInvocationAction but one problem is that right before it calls EndSourceFileAction in LoadFromCompilerInvocationAction, it calls transferASTDataFromCompilerInstance which means that the SourceManager in CompilerInstance is nulled out as it gets "transfered" to the AST. So this line crashes in this case. To be fair, at this point I don't need the ASTUnit so I can look at executing the action differently, but I thought I'd point it out!

[malaperle]

lib/Index/IndexRecordWriter.cpp

156

I'm getting quite a bit of those while indexing Clangd, it looks like it comes from some LLVM/Support headers:
Index: Duplicate USR! c:@N@std@ST>2#NI#Nb@__try_lock_impl
Index: Duplicate USR! c:@N@llvm@ST>1#T@DenseMapInfo
Index: Duplicate USR! c:@N@llvm@ST>1#T@isPodLike
Index: Duplicate USR! c:@N@llvm@N@detail@ST>1#T@unit
Index: Duplicate USR! c:@N@llvm@ST>2#T#T@format_provider
Index: Duplicate USR! c:@N@llvm@ST>2#T#T@format_provider
Index: Duplicate USR! c:@N@llvm@ST>1#T@PointerLikeTypeTraits
Index: Duplicate USR! c:@N@llvm@ST>1#T@simplify_type
Index: Duplicate USR! c:@N@std@ST>1#T@atomic
Index: Duplicate USR! c:@N@llvm@ST>1#T@isPodLike
Index: Duplicate USR! c:@N@llvm@ST>1#T@DenseMapInfo

I think it would be good to have the file name at least in the log. I also assume those duplication are issues that would have to be fixed in USRGenerator (i.e. in separate patches) ?

[malaperle]

include/indexstore/IndexStoreCXX.h

75

I know this is an old revision but I thought I should ask for the future patch... Would it be possible to not use "blocks"? This will affect portability of the code. I'm not familiar with blocks but I would think it would be possible to replace with C++11 lambdas, or something else that's standard. I was just playing around with this code and since I used GCC, it did not work.

[ioeric]

In D39050#900830, @arphaman wrote:

I think this patch should be split into a number of smaller patches to help the review process.

Things like tools/IndexStore, DirectoryWatcher and other components that are not directly needed right now should definitely be in their own patches.
It would be nice to find some way to split the implementation into multiple patches as well.

+1.

This is a lot of work (but great work!) for one patch. Smaller/incremental patches help reviewers understand and (hopefully) capture potential improvement of the design. I would really appreciate it if you could further split the patch.

Some comments/ideas:

• The lack of tests is a bit concerning.
• I think the implementation of the index output logic (e.g. IndexUnitWriter and bit format file) can be abstracted away (and split into separate patches) so that you can unit-test the action with a custom/mock unit writer; this would also make the action reusable with (potentially) other storage formats.
• I would suggest that you start with a patch that implement the index action and just enough components so that you could test the action.

Thanks!

[nathawes]

In D39050#920451, @ioeric wrote:

In D39050#900830, @arphaman wrote:

I think this patch should be split into a number of smaller patches to help the review process.

Things like tools/IndexStore, DirectoryWatcher and other components that are not directly needed right now should definitely be in their own patches.
It would be nice to find some way to split the implementation into multiple patches as well.

+1.

This is a lot of work (but great work!) for one patch. Smaller/incremental patches help reviewers understand and (hopefully) capture potential improvement of the design. I would really appreciate it if you could further split the patch.

Thanks for taking a look @ioeric! I'll have a go at splitting it further.

Some comments/ideas:

• The lack of tests is a bit concerning.

I moved all the code for reading the index store data into a separate patch (to come after this one) in order to slim this one down for review, and most of the tests went with it because they're based around reading and dumping the stored data for FileCheck. The original version of this patch has them all (https://reviews.llvm.org/D39050?id=118854). The ones that remain here are just those checking that the unit/record files are written out and that the hashing mechanism is producing distinct record files when the symbolic content of the source file changes.

• I think the implementation of the index output logic (e.g. IndexUnitWriter and bit format file) can be abstracted away (and split into separate patches) so that you can unit-test the action with a custom/mock unit writer; this would also make the action reusable with (potentially) other storage formats.

The added IndexRecordAction and existing IndexAction use the same functionality from libIndex to collect the indexing data, so I'm not sure mocking the unit writer to unit test IndexRecordAction would add very much value – writing the index data out is the new behavior. The existing tests for IndexAction (under test/Index/Core) are already covering the correctness of the majority of the collected indexing info and the tests coming in the follow-up patch (seen in the original version of this patch) test it's still correct after the write/read round trip.

• I would suggest that you start with a patch that implement the index action and just enough components so that you could test the action.

Thanks!

[ioeric]

• I think the implementation of the index output logic (e.g. IndexUnitWriter and bit format file) can be abstracted away (and split into separate patches) so that you can unit-test the action with a custom/mock unit writer; this would also make the action reusable with (potentially) other storage formats.

The added IndexRecordAction and existing IndexAction use the same functionality from libIndex to collect the indexing data, so I'm not sure mocking the unit writer to unit test IndexRecordAction would add very much value – writing the index data out is the new behavior. The existing tests for IndexAction (under test/Index/Core) are already covering the correctness of the majority of the collected indexing info and the tests coming in the follow-up patch (seen in the original version of this patch) test it's still correct after the write/read round trip.

Thanks for the clarification! I still think it's useful to decouple the IndexAction from the bit format file so that it could be reusable elsewhere. For example, I can see the index action be useful to clangd for building in-memory index.

I also tried applying your original patch locally but couldn't get it to work mostly due to portability issues (e.g. blocks and if (APPLE) in make files). AFAIK, many folks compile clang with GCC and/or without APPLE, so it's important that you get the portability right from the very beginning. Thanks!

Index-while-build is awesome! I'm looking forward to your patches!

[akyrtzi]

Hey Eric,

In D39050#921748, @ioeric wrote:

• I think the implementation of the index output logic (e.g. IndexUnitWriter and bit format file) can be abstracted away (and split into separate patches) so that you can unit-test the action with a custom/mock unit writer; this would also make the action reusable with (potentially) other storage formats.

The added IndexRecordAction and existing IndexAction use the same functionality from libIndex to collect the indexing data, so I'm not sure mocking the unit writer to unit test IndexRecordAction would add very much value – writing the index data out is the new behavior. The existing tests for IndexAction (under test/Index/Core) are already covering the correctness of the majority of the collected indexing info and the tests coming in the follow-up patch (seen in the original version of this patch) test it's still correct after the write/read round trip.

Thanks for the clarification! I still think it's useful to decouple the IndexAction from the bit format file so that it could be reusable elsewhere. For example, I can see the index action be useful to clangd for building in-memory index.

As Nathan mentioned, we believe the indexing action, as it exists in the trunk, is decoupled enough to be useful, for example Marc was already able to use it and write out the indexing data in a completely different format for his fork of clangd. Of course, we are definitely interested in any additional refactorings that would structure things better and we are eager to see and discuss follow-up patches from anyone that is interested in improving the code, but could we treat this as potential follow-up improvements ?

We are eager to provide the functionality so others can start experimenting with it; I'd propose that we discuss ideas for refactoring of the code as follow-up, what do you think ? Getting the initial functionality in and iterating on it, while getting more experience with applying it on various use-cases, is a common operating mindset of the llvm/clang projects.

I also tried applying your original patch locally but couldn't get it to work mostly due to portability issues (e.g. blocks and if (APPLE) in make files). AFAIK, many folks compile clang with GCC and/or without APPLE, so it's important that you get the portability right from the very beginning. Thanks!

Nathan will look into making using blocks optional, providing additional function pointer+context APIs where appropriate and having the common implementation using lambdas.
For the APPLE specific parts it, the only specific darwin-specific part is the part using FSEvents, the other 'if (APPLE)' checks can likely be removed. We would generally need help from people with linux expertise to provide the 'FSEvents' equivalent functionality but this is a small part of the overall feature, it's not important for getting the index-while-building data.

But these things are not part of the current patch, we can discuss again with the follow-up patches that will contain those things.

Index-while-build is awesome! I'm looking forward to your patches!

[ioeric]

In D39050#922597, @akyrtzi wrote:

Hey Eric,

In D39050#921748, @ioeric wrote:

• I think the implementation of the index output logic (e.g. IndexUnitWriter and bit format file) can be abstracted away (and split into separate patches) so that you can unit-test the action with a custom/mock unit writer; this would also make the action reusable with (potentially) other storage formats.

The added IndexRecordAction and existing IndexAction use the same functionality from libIndex to collect the indexing data, so I'm not sure mocking the unit writer to unit test IndexRecordAction would add very much value – writing the index data out is the new behavior. The existing tests for IndexAction (under test/Index/Core) are already covering the correctness of the majority of the collected indexing info and the tests coming in the follow-up patch (seen in the original version of this patch) test it's still correct after the write/read round trip.

Thanks for the clarification! I still think it's useful to decouple the IndexAction from the bit format file so that it could be reusable elsewhere. For example, I can see the index action be useful to clangd for building in-memory index.

As Nathan mentioned, we believe the indexing action, as it exists in the trunk, is decoupled enough to be useful, for example Marc was already able to use it and write out the indexing data in a completely different format for his fork of clangd. Of course, we are definitely interested in any additional refactorings that would structure things better and we are eager to see and discuss follow-up patches from anyone that is interested in improving the code, but could we treat this as potential follow-up improvements ?

We are eager to provide the functionality so others can start experimenting with it; I'd propose that we discuss ideas for refactoring of the code as follow-up, what do you think ? Getting the initial functionality in and iterating on it, while getting more experience with applying it on various use-cases, is a common operating mindset of the llvm/clang projects.

To be honest, I want this functionality to get in as much as you do, and I'm more than happy to prioritize the code review for it :) But the current patch size makes the reviewing really hard (e.g. I would never have caught the BLOCK issues hadn't I tried running the original patch myself). I'm not sure if it's really a common practice to check in a big chunk of code without careful code review and leave potential improvements as followups. I'm sure @klimek would have thoughts about this.

If the index action is already flexible enough, would you mind splitting the code for the index action out so that we can start reviewing it? Given that the current patch has very few tests, I guess it wouldn't be too much worse to split out the action without proper test.

[akyrtzi]

To be honest, I want this functionality to get in as much as you do, and I'm more than happy to prioritize the code review for it :) But the current patch size makes the reviewing really hard (e.g. I would never have caught the BLOCK issues hadn't I tried running the original patch myself). I'm not sure if it's really a common practice to check in a big chunk of code without careful code review and leave potential improvements as followups. I'm sure @klimek would have thoughts about this.

To be clear, I didn't mean to imply we don't want careful code review, we are really happy for people to point out issues. For example the building problems on linux are serious issues that we will fix and we are grateful for your feedback!

If the index action is already flexible enough, would you mind splitting the code for the index action out so that we can start reviewing it? Given that the current patch has very few tests, I guess it wouldn't be too much worse to split out the action without proper test.

To clarify, the index action Nathan and I are referring to, is the indexing action that exists currently in trunk and is the source of the index symbols, feeding index symbols to an abstract IndexDataConsumer. See here: https://llvm.org/svn/llvm-project/cfe/trunk/include/clang/Index/IndexingAction.h
This is what Marc used to get the index symbols and store them in his own format. Tests for this functionality are in: https://llvm.org/svn/llvm-project/cfe/trunk/test/Index/Core/

[ioeric]

If the index action is already flexible enough, would you mind splitting the code for the index action out so that we can start reviewing it? Given that the current patch has very few tests, I guess it wouldn't be too much worse to split out the action without proper test.

To clarify, the index action Nathan and I are referring to, is the indexing action that exists currently in trunk and is the source of the index symbols, feeding index symbols to an abstract IndexDataConsumer. See here: https://llvm.org/svn/llvm-project/cfe/trunk/include/clang/Index/IndexingAction.h
This is what Marc used to get the index symbols and store them in his own format. Tests for this functionality are in: https://llvm.org/svn/llvm-project/cfe/trunk/test/Index/Core/

Ah, sorry, I was referring to IndexRecordAction and its friends (record readers/writers). I didn't notice the newly added index action added and really didn't mean to ask you to refactor the existing code. Apologies for the miscommunication!

What I wanted to proposed is that we could decouple reading/writing of record/unit from the bit file format, so that the record output is not tied to a single output format (e.g. bit format, directory-based) and thus make the compiler more flexible. This might already be the case, but it's not really easy to tell from the current patch...

[malaperle]

Hi! I got a bit further in my experiment in integrating this in Clangd. I put some comments (in the first more complete revision). But since the scope of this patch changed, if you feel like we should take the discussions elsewhere, please let me know! Thanks!

include/indexstore/IndexStoreCXX.h

84

From what I understand, this returns the beginning of the occurrence. It would be useful to also have the end of the occurrence. From what I tested in Xcode, when you do "Find Selected Symbol in Workspace", it highlights the symbol name in yellow in the list of matches, so it mush use that LineCol then highlight the matching name. This is works in many situations but others occurrences won't have the name of the symbol. For example:
"MyClass o1, o2;"
If I use "Find Selected Symbol in Workspace" on MyClass constructor, if won't be able to highlight o1 and o2
Do you think it would be possible to add that (EndLineCol)? If not, how would one go about extending libindexstore in order to add additional information per occurrence? It is not obvious to me so far.
We also need other things, for example in the case of definitions, we need the full range of the definition so that users can "peek" at definitions in a pop-up. Without storing the end of the definition in the index, we would need to reparse the file.

374

As part of this dependency tracking mechanism, I haven't found that it could provide information about about the files including a specific header. So given a header (or source file in odd cases), I do not see a way to get all the files that would need to be reindexed if that header changed. Is that something you achieve outside the index? Or perhaps this is something I missed in the code.

377

Could there be a bit of explanation about what's a File dependency versus record and unit? All units and records are file dependencies, right? Are there any files that are neither records or units?

[nathawes]

Thanks for the feedback @malaperle!

include/indexstore/IndexStoreCXX.h

84

Our approach to related locations (e.g. name, signature, body, and doc comment start/end locs) has been to not include them in the index and derive them from the start location later. There's less data to collect and write out during the build that way, and deriving the other locations isn't that costly usually, as in most cases you 1) don't need to type check or even preprocess to get the related locations, as with finding the end of o1 and o2 in your example, and 2) usually only need to derive locations for a single or small number of occurrences, like when 'peeking' at a definition.

Are there cases where you think this approach won't work/perform well enough for the indexer-backed queries clangd needs to support?

374

The unit files store the path of the header/source files they depend on as 'File' dependencies. So any unit file with 'File' dependency on header/source file that was modified may need to be re-indexed.

To support finding which specific files include or are included by a given header (rather than which units somehow transitively include it) we also store the file-to-file inclusions in the unit file (retrieved via IndexUnitReader's foreachInclude method below).

377

I'll rename this to SourceFile and add some comments to explain. Unit file dependencies separate the source dependencies into 'File' dependencies and 'Record' dependencies. The 'File' dependencies track the paths of the header/source files seen in the translation unit, while the 'Record' dependencies track which record files have the symbolic content seen in those source files – the header/source file path doesn't appear anywhere in the record file. This separation lets us depend on a single record file corresponding to multiple source files (e.g. when two source files have the same symbolic content), and on a single source file corresponding multiple record files (e.g. when a single header is included multiple times with different preprocessor contexts changing its symbolic content).

[ioeric]

First round of comments. Mostly around indexing actions and file records; I haven't started reviewing the data writing and storing code. I think it might make sense to split the index writing and storing logics into a separate patch, which should be possible if writeUnitData is abstracted into an interface.

include/clang/Frontend/CompilerInstance.h

188

It might make sense to define an alias for std::function<std::unique_ptr<FrontendAction>(const FrontendOptions &opts, std::unique_ptr<FrontendAction> action)>, which is used multiple times.

include/clang/Frontend/FrontendOptions.h

262

It might make sense to also have documentations for these options here.

include/indexstore/IndexStoreCXX.h

84

I agree that we should try to keep the serialized symbol size minimal, but I think it's worthwhile to also store end locations for occurrences because 1) it's cheap, 2) it's not necessary easy to compute without parsing for occurrences like a::b::c or a::b<X>, 3) it would be useful for many LSP use cases.

lib/Frontend/CompilerInstance.cpp

1175

nit: no braces around one liners.

lib/FrontendTool/ExecuteCompilerInvocation.cpp

170

Could you comment on what this does? The Act above is already wrapped. Why do we need setGenModuleActionWrapper to createIndexDataRecordingAction again? Also, createIndexDataRecordingAction doesn't seem related to GenModule.

lib/Index/FileIndexRecord.cpp

24

Why?

38

Please comment when this would happen.

40

Why do we need Decls to be sorted by offset? If we want this for printing, it might make sense to just do a sort there.

lib/Index/FileIndexRecord.h

25

Please add documentation.

42

Is this clang-formatted? You might want to run git-clang-format on the whole patch.

lib/Index/IndexingAction.cpp

281

Again, you don't need the full IndexingContext and RecordOptions here.

291

Note that getDecomposedExpansionLoc can also return invalid decomposed loc.

305

Do we want better error handling here?

324

Please provide documentation.

501

Can we get this state from the base class instead of maintaining a another state, which seems to be identical?

521

Just StringRef BuildNumber = RepositoryPath;

698

Please provide a brief documentation for this class.

700

Again, it doesn't seem necessary for this class to have information about all record options. It seems that you only need RecordSystemDependencies here.

710

readability nit: avoid using auto if the return type is short to spell but hard to infer from the value expression. Same else where.

759

Could you add a comment explaining why we are not allowing searching.

766

It's a bit worrying that IndexDataRecorder and IndexContext reference each other. If you only need some information from the IndexingContext, simply pass it into Recorder. In this case, I think you only need the SourceManager from the ASTContext in the recorder to calculate whether a file is a system header. I see you also cache result of IndexingContext::isSystemFile in the indexing context, but I think it would be more sensible for the callers to handle caching for this call.

768

nit: no braces around one liners.

776

nit: redundant empty line

834

Just auto pair = getIndexOptionsFromFrontendOptions(FEOpts); and then use pair.first and pair.second?

Same below.

lib/Index/IndexingContext.h

39

Please define the scope of this class to avoid throwing random states into it, which usually happens to a "context" class.

[malaperle]

include/indexstore/IndexStoreCXX.h

84

There's a few reason I think it's better to store the end loc.

• When doing "find references", computing the end loc of each occurrence will be costly. Imagine having thousands of occurrences and for each of them having to run logic to find the end of the occurrence.
• The AST and preprocessor are the best tools I know to figure out the proper end loc. Not using them means having to write a mini-preprocessor with some knowledge about the language semantics to cover some cases.

MyClass |o1, o2;

Here, I have to stop at the comma. So it's basically take any alpha-numerical character, right?

bool operator<(const Foo&, const Foo&)
Ret operator()(Params ...params)

No, in those cases, we have to take < and the first ().

• In the case of body start/end locations, similarly, it can be non-trivial.

void foo() {
  if (0) {
  }
}

We have to count the balanced { } until we finish the body.

#define FUNC_BODY {\
\
}
void foo() FUNC_BODY

Oops, where's the body? We need another special logic for this, etc.

I think overall, it puts a lot of burden on the client of libIndexStore, burden that would be much more work and more inaccurate than using the AST/Preprocessor while indexing.

374

Thanks! I'll play around with this a bit more with this new information.

377

It's more clear now, thanks!

[akyrtzi]

@malaperle, to clarify we are not suggesting that you write your own parser, the suggestion is to use clang in 'fast-scan' mode to get the structure of the declarations of a single file, see CXTranslationUnit_SingleFileParse (along with enabling skipping of bodies). We have found clang is super fast when you only try to get the structure of a file like this. We can make convenient APIs to provide the syntactic structure of declarations based on their location.

But let's say we added the end-loc, is it enough ? If you want to implement the 'peek the definition' like Eclipse, then it is not enough, you also need to figure out if there are documentation comments associated with the declaration and also show those. Also what if you want to highlight the type signature of a function, then just storing the location of the closing brace of its body is not enough. There can be any arbitrary things you may want to get from the structure of the declaration (e.g. the parameter ranges), but we could provide an API to gather any syntactic structure info you may want.

I would encourage you to try CXTranslationUnit_SingleFileParse|CXTranslationUnit_SkipFunctionBodies, you will be pleasantly surprised with how fast this mode is. The c-index-test option is -single-file-parse.

[nathawes]

lib/FrontendTool/ExecuteCompilerInvocation.cpp

170

It's to wrap any GenerateModuleActions that get created as needed when/if Act ends up loading any modules, so that we output index data for them too. I'll add a comment.

lib/Index/FileIndexRecord.cpp

40

It's mostly for when we hash them, so that ordering doesn't change the hash, but it's also for printing. The IndexASTConsumer doesn't always report symbol occurrences in source order, due to the preprocessor and a few other cases.
We can sort them when the IndexRecordDataConsumer's finish() is called rather than as they're added to avoid the copying from repeated insert calls if that's the concern.

lib/Index/IndexingAction.cpp

501

I don't see this state in either base class (WrapperFrontendAction and IndexRecordActionBase). WrappingIndexAction and WrappingIndexRecordAction both have this, though. Were you thinking a new intermediate common base class between them and WrapperFrontendAction?

766

Good point. The IndexingContext was actually already calling IsSystemFile before it calls IndexDataRecorder's handleDeclOccurrence and handleModuleOccurrence anyway, so I'll change it to pass that through as an extra param and remove IndexDataRecorder's dependency on the IndexingContext.

[nathawes]

Worked through the comments from @ioeric and split the code for writing out the collected indexing data into a separate patch.

[malaperle]

In D39050#948500, @akyrtzi wrote:

@malaperle, to clarify we are not suggesting that you write your own parser, the suggestion is to use clang in 'fast-scan' mode to get the structure of the declarations of a single file, see CXTranslationUnit_SingleFileParse (along with enabling skipping of bodies). We have found clang is super fast when you only try to get the structure of a file like this.

Thank you, that sounds very useful. I will try that and get some measurements.

We can make convenient APIs to provide the syntactic structure of declarations based on their location.

Perhaps just for the end-loc since it's pretty much guaranteed to be needed by everyone. But if it's very straightforward, perhaps that's not needed. I'll try and see.

But let's say we added the end-loc, is it enough ? If you want to implement the 'peek the definition' like Eclipse, then it is not enough, you also need to figure out if there are documentation comments associated with the declaration and also show those. Also what if you want to highlight the type signature of a function, then just storing the location of the closing brace of its body is not enough. There can be any arbitrary things you may want to get from the structure of the declaration (e.g. the parameter ranges), but we could provide an API to gather any syntactic structure info you may want.

That's a very good point. I guess in the back of my mind, I have the worry that one cannot extend what is stored, either for a different performance trade-off or for additional things. The fact that both clang and clangd have to agree on the format so that index-while-building can be used seems to make it inherently not possible to extend. But perhaps it's better to not overthink this for now.

[ioeric]

Thanks a lot for the changes! Some more comments inlined.

Please mark addressed comments as done so that reviewers could know what to look :) Thanks!

include/clang/Frontend/CompilerInstance.h

187

nit: LLVM variable names start with upper-case letters.

include/clang/Index/IndexingAction.h

34

This should be removed?

Some forward declarations above are not used as well.

lib/Driver/Job.cpp

293

Could you share this code with line 278 above, which already has a nice comment?

lib/Index/FileIndexRecord.cpp

40

I would leave the sorting to the point where records are hashed to avoid making the record stateful. Consider changing getDeclOccurrences to getOccurrencesSortedByOffset; this should make the behavior more explicit.

lib/Index/FileIndexRecord.h

52

s/isSystem/IsSystem/

Also, I wonder if we can filter out system decls proactively and avoid creating file index record for them. We could also avoid propogating IsSystem here.

lib/Index/IndexingAction.cpp

362

IsSystemFileCache &SysrootPath? What is this parameter?

451

Please document this class. This can be easily confused with IndexActionBase which has a similar name. Same for IndexAction/IndexRecordAction and WrappingIndexRecordAction/WrappingIndexRecordAction. I think these pairs share (especially the wrapping actions) some common logics and could probably be merged.

477

This does a lot of stuff... please document the behavior!

501

I thought this could be a state in the WrapperFrontendAction since both derived classes maintain this state, but after a closer look, this seems to depend on both base classes. I'm not a big fun of maintaining states in multi-stage classes (e.g. FrontendAction), which could be confusing and hard to follow; I think IndexRecordActionBase::finish(...) should be able to handle the case where no index consumer is created (i.e. no record/dependency/... is collected).

Also, IndexRecordActionBase (and the existing IndexActionBase ) should really be a component instead of a base class since none of its methods is virtual.

569

nit: no need for braces. Same below.

581

In the previous patch, writeUnitData does several things including handling modules, dependencies, includes and index records, as well as writing data. It might make sense to add an abstract class (UnitDataCollector?) that defines interfaces which make these behavior more explicit. We can then have users pass in an implementation via createIndexDataRecordingAction which would also decouple the data collection from data storage in the library.

616

I'm a bit nervous about propagating the entire FrontendOptions into the index library. I would simply expose getIndexOptionsFromFrontendOptions and have callers parse FrontendOptions and pass in only index-related options.

lib/Index/IndexingContext.h

41

This name is really confusing... Is* is usually used for booleans. Simply call this SystemFileCache.

42

I think it would be more straightforward to have context own the cache. If setSysrootPath is the problem, it might make sense to propagate it via the context or, if necessary, create a new cache when a new SysrootPath is set.

53

How does this affect the existing cached results? Do you need to invalidate them?

[nathawes]

Thanks for taking another look @ioeric – I'll work through your comments and update.

[nathawes]

lib/Index/FileIndexRecord.h

52

If the -index-ignore-system-symbols flag is set system decls are filtered out in IndexingContext::handleDeclOccurrence and aren't reported to the IndexDataConsumer, so FileIndexRecords won't be created. The IsSystem here is for clients that want index data for system files, but want to be able to distinguish them from regular files.

[nathawes]

I've refactored the indexing/dependency data collection out from the writing with the new IndexUnitDataConsumer class, and made other smaller changes to address the feedback from @ioeric.

[nathawes]

Fix out of date header comment in FileIndexData.h

[ioeric]

Thanks a lot for further cleaning up the patch! It is now much easier to review. I really appreciate it!

Some more comments on the public APIs and the layering of classes. There are a lot of helper classes in the implementation, so I think it's important to get a clear layering so that they could be easily understood by future contributors :)

Also, with the IndexUnitDataConsumer abstraction, it seems to be feasible now to add some unit tests for createUnitIndexingAction. With all the recent major changes, I think it's important that we have some degree of testing to make sure components actually work together.

include/clang/Frontend/CompilerInstance.h

187

opts and action are still lower-case.

include/clang/Index/DeclOccurrence.h

38 ↗

(On Diff #127412)

Nit: indentation.

Tip: git-clang-format against the diff base can format all changed lines in your patch.

include/clang/Index/IndexUnitDataConsumer.h

1 ↗

(On Diff #127568)

IIUC, this is the index data for a translation unit, as opposed to an AST. If so, consider calling this UnitIndexDataConsumer to match (AST)IndexDataConsumer which is parallel to this. We might want to rename them to be either index::UnitDataConsumer vs index::ASTDataConsumer or index::UnitIndexDataConsumer vs index::ASTIndexDataConsumer . I am inclined to the first pair as index is already implied in the namespace.

67 ↗

(On Diff #127568)

Comment? Why do we actually need this?

include/clang/Index/IndexingAction.h

48

We are now mixing functionalities for Unit indexing and AST indexing actions in the same file. We might want to separate these into two headers e..g UnitIndexingAction.h and ASTIndexingAction.h. This would make it easier for users to find the right functions :)

65

Please add documentation for each field. It's not trivial what each field is for, especially some fields seem to be optional and some seem to be mutually exclusive.

66

These pointers suggest the life time of this struct is tied to some other struct, which makes the struct look a bit dangerous to use. Should we also carry a reference or a smart pointer to the underlying object that keeps these pointers valid? Would it be a CompilerInstance (guessing from IndexUnitDataConsumerFactory )?

95

What is the intended user of this function? It's unclear how users could obtain a ConsumerFactory (i.e. UnitDetails) without the functionalities in UnitDataConsumerActionImpl .

(Also see comment in the implementation of createIndexDataRecordingAction.)

109

This is likely only useful for compiler invocation. I would put it in the compiler invocation code.

lib/Driver/Job.cpp

211

nit: Comment should start with an overview of what the function does.

Returns a directory path that is ...

Also, consider calling this getDirAdjacentToModCache. buildDir can be ambiguous.

220

Please clang-format the code. Without indentation, this looks like an no-op statement.

lib/Index/IndexingAction.cpp

90

Use class for interfaces.

98

Does CI here have to be the same instance as the one in createIndexASTConsumer ? Might worth documenting.

136

nit: Move this after Impl->createIndexASTConsumer(CI).

Do we need to reset this flag? Calling CreateASTConsumer multiple times on the same instance seems to be allowed?

221

This seems to be related to files. Maybe FileIndexDataCollector?

228

override

232

Simply begin, if the class is called FileIndexDataCollector .

Similar below to match iterator naming convention.

243

I think this should be public as this is still implementing IndexDataConsumer.

301

I'd simply do:

if FileIncludeFilter  == UnitIndexingOptions::FileIncludeFilterKind::UserOnly)
  if (isSystem...)
    return;

315

Same here. This should be public

332

The naming convention for the callback interfaces is forEach* e.g. forEachFileDependency.

s/visitor/Callback/
(same below).

336

forEachInclude

339

forEachModuleImport

347

This is two classes in one, which is difficult to understand. Could you split it into FileIndexDependencyCollector and FileIndexDependencyProvider and have FileIndexDependencyCollector returns a provider on finish (e.g. Provider consume();; you might want to copy/move the collected data into the provider). It would be easier to justify the behavior (e.g. what happens when you access the provider while collector is still working?)

351

What does Entries contain? What files are added?

479

Instead of passing ParentUnitConsumer, consider checking the Mod before calling the function.

482

Non-factory static method is often a code smell. Any reason not to make these static methods private members? With that, you wouldn't need to pass along so many parameters. You could make them const if you don't want members to be modified.

489

Why is this overload public while others are private? Aren't they all used only in this class?

509

Any reason to close the anonymous namespace here? Shouldn't outlined definitions of UnitDataConsumerActionImpl's methods also in the anonymous namespace?

736

I think the inheritance of IndexUnitDataConsumer and the creation of factory should be in user code (e.g. implementation for on-disk persist-index-data should come from the compiler invocation code ExecuteCompilerInvocation.cpp or at least a separate file in the library that compiler invocation can use), and the user should only use createUnitIndexingAction by providing a factory. Currently, createUnitIndexingAction and createIndexDataRecordingAction are mostly identical except for the code that implements IndexUnitDataConsumer and creates the factory.

The current createIndexDataRecordingAction would probably only used by the compiler invocation, and we can keep the generalized createUnitIndexingAction in the public APIs.

743

The UnitInfo is ignored? What do we actually need it for?

747

Base doesn't seem to be a very meaningful name here.

[ioeric]

include/clang/Index/IndexUnitDataConsumer.h

1 ↗

(On Diff #127568)

Sorry, asking you to also rename IndexDataConsumer is probably too much and out of the scope of this patch. I'm fine with UnitIndexDataConsumer or UnitDataConsumer or something similar for now without touching IndexDataConsumer :)

[ioeric]

(I think I forgot to update the patch status :)

[nathawes]

@ioeric I should have an updated patch up shortly with your inline comments addressed + new tests. Thanks again for reviewing!

include/clang/Index/IndexUnitDataConsumer.h

67 ↗

(On Diff #127568)

From here, my understanding is that it's an optimization to avoid the vtable being included in multiple translation units. I'm not sure if that's actually a problem, I was just following IndexDataConsumer's lead. Added a comment.

include/clang/Index/IndexingAction.h

95

Sorry, I'm not sure what you mean here. Users shouldn't need to know anything about UnitDataConsumerActionImpl, they just need to provide a lambda/function reference that takes a CompilerInstance& and a UnitDetails and returns an IndexUnitDataConsumer (or UnitIndexDataConsumer once I rename it). This gets called once per translation unit to get a distinct data consumer for each unit, i.e. for the main translation unit as well as for each of its dependent modules that the main unit's data consumer says should be indexed via shouldIndexModuleDependency(...).

109

There's another public index:: API for writing out index data for individual clang module files in the follow up patch that takes a RecordingOptions and is used externally, from Swift. This function's useful on the Swift side to get the RecordingOptions from FrontendOptions it has already set up.

lib/Index/IndexingAction.cpp

136

Oops. Yes, we do :-)

482

Sorry, there's missing context – they're used from another public API that's in the follow-up patch. I'll bring that over and make these top-level static functions, since they don't belong exclusively to IndexDataConsumerActionImpl.

489

Same as above – this is called from a public index:: API in the follow-up patch.

736

IndexUnitDataRecorder here is just a stub I added when I split the patch up – the follow-up revision has it in a separate file. I'll move the separate files to this patch and stub out the method bodies with TODOs instead.

I've made createIndexDataRecordingAction call createUnitIndexingAction to remove the duplication, and pulled it, RecordingOptions and getRecordingOptionsFromFrontendOptions to a new header (RecordingAction.h) that ExecuteComilerInvocation.cpp uses. Does that sound ok?

743

It should be passed to IndexUnitDataRecorder to write out info about the unit itself. This was just me splitting the patch badly.

[nathawes]

• Applied the various refactorings suggested by @ioeric
• Extended c-index-test with a new option to print out the collected unit indexing data, and
• Added tests for the unit indexing functionality using the new option
• Fixed formatting

[ioeric]

Nice! Thanks for making the refactoring and adding tests! I think this is good to go now.

I'm not very familiar with code outside of the index library (Driver, Basic etc), but the changes seem reasonable to me. Feel free to get another pair of eyes for them ;)

include/clang/Index/RecordingAction.h

42 ↗

(On Diff #130496)

Add a FIXME that this is not implemented yet.

lib/Index/IndexingAction.cpp

736

Sounds good. Thanks for the explanation!

[ioeric]

I'm wondering if there is any further plan for this? ;)

[malaperle]

In D39050#1004937, @ioeric wrote:

I'm wondering if there is any further plan for this? ;)

I'd like to comment on the amount of data that will be stored but that can be done outside this review. I still have a few things to figure out before reaching a conclusion.

[nathawes]

@ioeric I'm working on a few other priorities over the next few weeks, sorry, but should get back to this relatively soon after that.
I would just land it, but I expect some downstream breakage I want to make sure I have time to fix.

@malaperle Sounds good – I'll keep an eye out for it!

[malaperle]

In D39050#949185, @malaperle wrote:

In D39050#948500, @akyrtzi wrote:

@malaperle, to clarify we are not suggesting that you write your own parser, the suggestion is to use clang in 'fast-scan' mode to get the structure of the declarations of a single file, see CXTranslationUnit_SingleFileParse (along with enabling skipping of bodies). We have found clang is super fast when you only try to get the structure of a file like this.

Thank you, that sounds very useful. I will try that and get some measurements.

We can make convenient APIs to provide the syntactic structure of declarations based on their location.

Perhaps just for the end-loc since it's pretty much guaranteed to be needed by everyone. But if it's very straightforward, perhaps that's not needed. I'll try and see.

But let's say we added the end-loc, is it enough ? If you want to implement the 'peek the definition' like Eclipse, then it is not enough, you also need to figure out if there are documentation comments associated with the declaration and also show those. Also what if you want to highlight the type signature of a function, then just storing the location of the closing brace of its body is not enough. There can be any arbitrary things you may want to get from the structure of the declaration (e.g. the parameter ranges), but we could provide an API to gather any syntactic structure info you may want.

That's a very good point. I guess in the back of my mind, I have the worry that one cannot extend what is stored, either for a different performance trade-off or for additional things. The fact that both clang and clangd have to agree on the format so that index-while-building can be used seems to make it inherently not possible to extend. But perhaps it's better to not overthink this for now.

I did a bit more of experimenting. For the end-loc, I changed my prototype so that the end-loc is not stored in the index but rather computed "on the fly" using SourceManager and Lexer only. For my little benchmark, I used the LLVM/Clang/Clangd code base which I queried for all references of "std" (the namespace) which is around 46K references in the index.

With end-loc in index: 3.45s on average (20 samples)
With end-loc computed on the fly: 11.33s on average (20 samples)
I also tried with Xcode but without too much success: it took about 30 secs to reach 45K results and then carried on for a long time and hung (although I didn't try to leave it for hours to see if it finished).

From my perspective, it seems that the extra time is quite substantial and it doesn't seem worth to save an integer per occurrence in this case.

For computing the start/end-loc of function bodies, I tried the SingleFileParseMode and SkipFunctionBodies separately ( as a start). The source I use this on looks like this:

#include "MyClass.h"

MyClass::MyClass() {
}

void MyClass::doOperation() {
}

With SingleFileParseMode, I get several errors:

MyClass.cpp:5:1: error: use of undeclared identifier 'MyClass'
MyClass.cpp:8:6: error: use of undeclared identifier 'MyClass'

Then I cannot obtain any Decl* at the position of doOperation. With SingleFileParseMode, I'm also a bit weary that not processing headers will result in many inaccuracies. From our perspective, we are more wiling to sacrifice disk space in order to have more accuracy and speed. For comparison, the index I worked with containing all end-loc for occurrences and also function start/end is 201M for LLVM/Clang/Clangd which is small to us.

With SkipFunctionBodies alone, I can get the Decl* but FunctionDecl::getSourceRange() doesn't include the body, rather, it stops after the arguments.
It would be very nice if we could do this cheaply but it doesn't seem possible with those two flags alone. What did you have in mind for implementing an "API to gather any syntactic structure info" ?

[malaperle]

In D39050#1021204, @malaperle wrote:

For computing the start/end-loc of function bodies, I tried the SingleFileParseMode and SkipFunctionBodies separately ( as a start). The source I use this on looks like this:

Given the discussion in https://reviews.llvm.org/D44247, I think we can do without the start/end-loc of function bodies and try some heuristics client-side. We can always revisit this later if necessary.

However, for the end-loc of occurrences, would you be OK with this being added? I think it would be a good compromise in terms of performance, simplicity and index size.

[nathawes]

In D39050#1036249, @malaperle wrote:

In D39050#1021204, @malaperle wrote:

For computing the start/end-loc of function bodies, I tried the SingleFileParseMode and SkipFunctionBodies separately ( as a start). The source I use this on looks like this:

Given the discussion in https://reviews.llvm.org/D44247, I think we can do without the start/end-loc of function bodies and try some heuristics client-side. We can always revisit this later if necessary.

However, for the end-loc of occurrences, would you be OK with this being added? I think it would be a good compromise in terms of performance, simplicity and index size.

@malaperle Just to clarify, what's the particular end-loc we're talking about here? e.g. for a function call, would this be the end of the function's name, or the closing paren?
For the end of the name, couldn't this be derived from the start loc + symbol name length (barring token pastes and escaped new lines in the middle of identifiers, which hopefully aren't too common)?
I can see the value for the closing paren though.

@akyrtzi Are the numbers from Marc-Andre's experiment what you'd expect to see and is there anything else to try? I'm not familiar with those modes at all to comment, sorry. I assume any API to gather syntactic structure info would be based on those modes, right?

[malaperle]

In D39050#1036394, @nathawes wrote:

@malaperle Just to clarify, what's the particular end-loc we're talking about here? e.g. for a function call, would this be the end of the function's name, or the closing paren?
For the end of the name, couldn't this be derived from the start loc + symbol name length (barring token pastes and escaped new lines in the middle of identifiers, which hopefully aren't too common)?
I can see the value for the closing paren though.

I mean the end of the name referencing the symbol, so that it can be highlighted properly when using the "find references in workspace" feature. There are cases where the name of the symbol itself is not present, for example "MyClass o1, o2;" (o1 and o2 reference the constructor), references to overloaded operators, etc.

[nathawes]

In D39050#1037008, @malaperle wrote:

In D39050#1036394, @nathawes wrote:

@malaperle Just to clarify, what's the particular end-loc we're talking about here? e.g. for a function call, would this be the end of the function's name, or the closing paren?
For the end of the name, couldn't this be derived from the start loc + symbol name length (barring token pastes and escaped new lines in the middle of identifiers, which hopefully aren't too common)?
I can see the value for the closing paren though.

I mean the end of the name referencing the symbol, so that it can be highlighted properly when using the "find references in workspace" feature. There are cases where the name of the symbol itself is not present, for example "MyClass o1, o2;" (o1 and o2 reference the constructor), references to overloaded operators, etc.

Ah, I see – thanks! I was thinking all occurrences whose symbol name didn't actually appear at their location were marked with SymbolRole::Implicit, but that only seems to be true for the ObjC index data.

[akyrtzi]

Hey Marc,

The fact that both clang and clangd have to agree on the format so that index-while-building can be used seems to make it inherently not possible to extend

I don't think "not possible to extend" is quite correct, we can make it so that the format allows optional data to be recorded.

On the topic of recording the end-loc, I agree it's not much data overall, but it will be useful to examine the uses closely and to figure out whether it's really required and whether it is at the same time inadequate for other uses.

I changed my prototype so that the end-loc is not stored in the index but rather computed "on the fly" using SourceManager and Lexer only.

I assume you used SingleFileParseMode+SkipFunctionBodies for this, right ?

For my little benchmark, I used the LLVM/Clang/Clangd code base which I queried for all references of "std" (the namespace) which is around 46K references in the index.

This is an interesting use case, and I can say we have some experience because Xcode has this functionality without requiring the end-loc for every reference.
So what it does is that it has a 'name' to look for (say 'foo' for the variable foo) and if it finds the name in the location then it highlights, otherwise if it doesn't find it (e.g. because it is an implicit reference) then it points to the location but doesn't highlight something. The same thing happens for operator overloads (the operators get highlighted at the reference location).
For implicit references it's most likely there's nothing to highlight so the end-loc will most likely be empty anyway (or same as start-loc ?) to indicate an empty range.

With SingleFileParseMode, I get several errors:

Good point, the parser definitely needs recovery improvements in C++.

With SkipFunctionBodies alone, I can get the Decl* but FunctionDecl::getSourceRange() doesn't include the body

This seems strange, there's an EndRangeLoc field that should have been filled in, not exactly sure if it is a bug or omission.

Going back to the topic of what use cases end-loc covers, note that it still seems inadequate for peek-definition functionality. You can't set it to body-end loc (otherwise occurrences highlighting would highlight the whole body which I think is undesirable) and you still need to include doc-comments if they exist.

[malaperle]

In D39050#1037796, @akyrtzi wrote:

Hey Marc,

The fact that both clang and clangd have to agree on the format so that index-while-building can be used seems to make it inherently not possible to extend

I don't think "not possible to extend" is quite correct, we can make it so that the format allows optional data to be recorded.

That would be good. How would one go about asking Clang to generate this extra information? Would a Clang Plugin be suitable for this? I don't know much about those but perhaps that could be one way to extent the basic behavior of "-index_store_path" in this way?

I changed my prototype so that the end-loc is not stored in the index but rather computed "on the fly" using SourceManager and Lexer only.

I assume you used SingleFileParseMode+SkipFunctionBodies for this, right ?

No, sorry the end-locs I meant there is for occurrences. Only the lexer was needed to get the end of the token. So for "MyClass o1, o2;" o1 and o2 get highlighted as references to the MyClass constructor.

For my little benchmark, I used the LLVM/Clang/Clangd code base which I queried for all references of "std" (the namespace) which is around 46K references in the index.

This is an interesting use case, and I can say we have some experience because Xcode has this functionality without requiring the end-loc for every reference.
So what it does is that it has a 'name' to look for (say 'foo' for the variable foo) and if it finds the name in the location then it highlights, otherwise if it doesn't find it (e.g. because it is an implicit reference) then it points to the location but doesn't highlight something.

I think it's useful to highlight something even when the name is not there. For example in "MyClass o1, o2;" it feels natural that o1 and o2 would get highlighted.

The same thing happens for operator overloads (the operators get highlighted at the reference location).

It does? I can only seem to do a textual search. For example, if I look at "FileId::operator<", if I right-click in the middle of "operator<" and do "Find selected symbol in workspace", it seems to start a text based search because there are many results that are semantically unrelated.

For implicit references it's most likely there's nothing to highlight so the end-loc will most likely be empty anyway (or same as start-loc ?) to indicate an empty range.

I think for those cases the end of the token is probably suitable. Can you give examples which implicit references you have in mind? Maybe another one (other than the constructor mentioned above) could be a function call like "passMeAStdString(MyStringRef)", here the "operator std::string" would be called and MyStringRef could be highlighted, I think it would make sense to the user that is gets called by passing this parameter by seeing the highlight.

Going back to the topic of what use cases end-loc covers, note that it still seems inadequate for peek-definition functionality. You can't set it to body-end loc (otherwise occurrences highlighting would highlight the whole body which I think is undesirable) and you still need to include doc-comments if they exist.

I think maybe I wasn't clear, I was thinking about two end-locs: end-locs of occurrences and end-locs of bodies. The end-loc of occurrences would be used for highlight when searching for all occurrences and the end-loc for bodies would be used for the peek definition. I think we can disregard end-locs of bodies for now.

[akyrtzi]

That would be good. How would one go about asking Clang to generate this extra information? Would a Clang Plugin be suitable for this?

That's an interesting idea that we could explore, but I don't have much experience with that mechanism to comment on.

Only the lexer was needed to get the end of the token

Ok, that's interesting, not sure why Xcode is so fast to highlight, did you reuse same SourceManager/Lexer/buffers for occurrences from same file ? We'd definitely add the end-loc if we cannot come up with a mechanism to highlight fast enough without it.

I think it's useful to highlight something even when the name is not there. For example in "MyClass o1, o2;" it feels natural that o1 and o2 would get highlighted.

To clarify, even with implicit references the start loc points to something. In this case the implicit references can have start locs for the o1 and o2 identifiers and the end result for the UI will be the same (o1 and o2 get highlighted) even without having end-locs for all references.

It does? I can only seem to do a textual search.

The example I tried is the following. If you could file a bug report for the test case that did not work as you expected it would be much appreciated!

class Something1 {
public:
    Something1() {}
    ~Something1() {}
    operator int() {
        return 0;
    }

    friend int operator <<(Something1 &p, Something1 &p2) {
        return 0;
    }
};

void foo1(Something1 p1, Something1 p2) {
    p1 << p2;
    p1 << p2;
}

here the "operator std::string" would be called and MyStringRef could be highlighted

Even without end-loc, the start loc could point to MyStringRef and you could highlight it.

[malaperle]

In D39050#1040501, @akyrtzi wrote:

That would be good. How would one go about asking Clang to generate this extra information? Would a Clang Plugin be suitable for this?

That's an interesting idea that we could explore, but I don't have much experience with that mechanism to comment on.

Only the lexer was needed to get the end of the token

Ok, that's interesting, not sure why Xcode is so fast to highlight, did you reuse same SourceManager/Lexer/buffers for occurrences from same file ? We'd definitely add the end-loc if we cannot come up with a mechanism to highlight fast enough without it.

I don't think Xcode is quite fast, it's about 10 times slower (although I'm not sure it really finished) than when I use my branch that has the end-loc. I would try end-locs in Xcode if I could, to compare :) So I don't really know where the bottleneck is in Xcode. Comparing oranges to oranges, it's 4 times slower without end-locs compared to with end-locs on my branch. I does use the same SourceManager for the 46K references and I verified that it uses the same buffers, etc.
I'll put the numbers here again for readability.

For my little benchmark, I used the LLVM/Clang/Clangd code base which I queried for all references of "std" (the namespace) which is around 46K references in the index.

With end-loc in index: 3.45s on average (20 samples)
With end-loc computed on the fly: 11.33s on average (20 samples)
I also tried with Xcode but without too much success: it took about 30 secs to reach 45K results and then carried on for a long time and hung (although I didn't try to leave it for hours to see if it finished).

I think it's useful to highlight something even when the name is not there. For example in "MyClass o1, o2;" it feels natural that o1 and o2 would get highlighted.

To clarify, even with implicit references the start loc points to something. In this case the implicit references can have start locs for the o1 and o2 identifiers and the end result for the UI will be the same (o1 and o2 get highlighted) even without having end-locs for all references.

It's the same but slower. IMO, the trade off is not great. It's entirely subjective but I think 4-10 times slower in order to save an integer per occurrence is not worth it from my point of view.

Even without end-loc, the start loc could point to MyStringRef and you could highlight it.

(Same here, it's doable but faster if already in the index.)

It does? I can only seem to do a textual search.

The example I tried is the following. If you could file a bug report for the test case that did not work as you expected it would be much appreciated!

Sure, I'll give that a try and isolate it as much as I can. BTW, does it work for you on the LLVM code base?

[nathawes]

Updated to apply on top-of-tree.

[gribozavr]

include/clang/Frontend/FrontendOptions.h

262

Please end comments with a period.

265

Would it make more sense to flip this boolean to positive? "IndexIncludeSystemSymbols"?

lib/Index/IndexingAction.cpp

107

Please don't duplicate the information from the signature in comments. No need to say that this function returns an IndexASTConsumer (twice, in the first sentence and in the \returns clause), the code already says that.

Also, "The compiler instance used to process the input" does not mean much to me either.

142

No semicolon.

151

No semicolon.

155

No semicolon.

242

Please don't duplicate type information from the signature in the comment.

250

I don't understand... this is not really the user-specified output file.

270

Please don't duplicate type information from the signature in the comment.

lib/Index/IndexingContext.h

40

Please add a period at the end of the comment.

44

DirEntries => IsSystemDirEntry?

46

Triple slashes for doc comments.

46

Unclear how a boolean can keep track of the last check.

Did you mean "Whether the file is a system file or not. This value is a cache." If so, please rename the variable to something like IsSystemFileCache.

test/Index/Core/index-source.mm

2 ↗

(On Diff #153190)

No need to specify check-prefixes=CHECK.

test/Index/Core/index-unit.mm

1 ↗

(On Diff #153190)

This test is very difficult to read... it is just a dump of random internal data structures... what do you think about converting it to a unit test?

[mgrang]

lib/Index/FileIndexData.cpp

31 ↗

(On Diff #153190)

Please use range-based llvm::sort instead of std::sort:

llvm::sort(Sorted);

See https://llvm.org/docs/CodingStandards.html#beware-of-non-deterministic-sorting-order-of-equal-elements

[jkorous]

It's time to officially abandon these patches in favor of new push for upstreaming index-while-building.

Current reviews in progress
https://reviews.llvm.org/D58749
https://reviews.llvm.org/D58418

RFC
http://lists.llvm.org/pipermail/cfe-dev/2019-February/061432.html

[jkorous]

I'll address comments for this patch in the new set of patches.

@gribozavr I haven't put up this part of code for the new round of review yet. I will keep this on mind.

@mgrang This already landed in edbbe470f66 as clang/lib/Index/FileIndexRecord.cpp but luckily the implementation isn't using sort() at all. Thanks for pointing this out anyway!

[akyrtzi]

include/clang/Frontend/FrontendOptions.h

265

@jkorous I noticed this name can be misleading, it may seem as if what this does is "avoid indexing system symbol occurrences" but what it actually does is "avoid indexing symbol occurrences from system files". We should rename it to "IndexIgnoreSystemHeaders" or "IndexIncludeSystemHeaders" per Dmitri's suggestion.

include/clang/Basic/DiagnosticFrontendKinds.td

 def remark_module_build : Remark<"building module '%0' as '%1'">,
   InGroup<ModuleBuild>;
 def remark_module_build_done : Remark<"finished building module '%0'">,
   InGroup<ModuleBuild>;
 def err_modules_embed_file_not_found :
   Error<"file '%0' specified by '-fmodules-embed-file=' not found">,
   DefaultFatal;
 
+def remark_index_producing_module_file_data : Remark<"producing index data for "
+  "module file '%0'">,
+  InGroup<IndexStore>;
+
 def err_test_module_file_extension_version : Error<
   "test module file extension '%0' has different version (%1.%2) than expected "
   "(%3.%4)">;
 
 def err_missing_vfs_overlay_file : Error<
   "virtual filesystem overlay file '%0' not found">, DefaultFatal;
 def err_invalid_vfs_overlay : Error<
   "invalid virtual filesystem overlay file '%0'">, DefaultFatal;
 
 def warn_option_invalid_ocl_version : Warning<
   "OpenCL version %0 does not support the option '%1'">, InGroup<Deprecated>;
 }

include/clang/Basic/DiagnosticGroups.td

 def MethodSignatures : DiagGroup<"method-signatures">;
 def MismatchedParameterTypes : DiagGroup<"mismatched-parameter-types">;
 def MismatchedReturnTypes : DiagGroup<"mismatched-return-types">;
 def MismatchedTags : DiagGroup<"mismatched-tags">;
 def MissingFieldInitializers : DiagGroup<"missing-field-initializers">;
 def ModuleBuild : DiagGroup<"module-build">;
 def ModuleConflict : DiagGroup<"module-conflict">;
 def ModuleFileExtension : DiagGroup<"module-file-extension">;
+def IndexStore : DiagGroup<"index-store">;
 def NewlineEOF : DiagGroup<"newline-eof">;
 def Nullability : DiagGroup<"nullability">;
 def NullabilityDeclSpec : DiagGroup<"nullability-declspec">;
 def NullabilityInferredOnNestedType : DiagGroup<"nullability-inferred-on-nested-type">;
 def NullableToNonNullConversion : DiagGroup<"nullable-to-nonnull-conversion">;
 def NullabilityCompletenessOnArrays : DiagGroup<"nullability-completeness-on-arrays">;
 def NullabilityCompleteness : DiagGroup<"nullability-completeness",
                                         [NullabilityCompletenessOnArrays]>;

include/clang/DirectoryWatcher/DirectoryWatcher.h

+//===- DirectoryWatcher.h - Listens for directory file changes --*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+/// \file
+/// \brief Utility class for listening for file system changes in a directory.
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_DIRECTORYWATCHER_DIRECTORYWATCHER_H
+#define LLVM_CLANG_DIRECTORYWATCHER_DIRECTORYWATCHER_H
+
+#include "clang/Basic/LLVM.h"
+#include "clang/Index/IndexDataStore.h"
+#include <functional>
+#include <memory>
+#include <string>
+
+namespace clang {
+
+/// Provides notifications for file system changes in a directory.
+///
+/// Guarantees that the first time the directory is processed, the receiver will
+/// be invoked even if the directory is empty.
+class DirectoryWatcher : public index::AbstractDirectoryWatcher {
+  struct Implementation;
+  Implementation &Impl;
+
+  DirectoryWatcher();
+
+  DirectoryWatcher(const DirectoryWatcher&) = delete;
+  DirectoryWatcher &operator =(const DirectoryWatcher&) = delete;
+
+public:
+  ~DirectoryWatcher();
+
+  static std::unique_ptr<DirectoryWatcher>
+    create(StringRef Path, EventReceiver Receiver, bool waitInitialSync,
+           std::string &Error);
+};
+
+} // namespace clang
+
+#endif

include/clang/Driver/Job.h

 class InputInfo;
 
 // Re-export this as clang::driver::ArgStringList.
 using llvm::opt::ArgStringList;
 
 struct CrashReportInfo {
   StringRef Filename;
   StringRef VFSPath;
+  StringRef IndexStorePath;
 
-  CrashReportInfo(StringRef Filename, StringRef VFSPath)
-      : Filename(Filename), VFSPath(VFSPath) {}
+  CrashReportInfo(StringRef Filename, StringRef VFSPath,
+                  StringRef IndexStorePath)
+      : Filename(Filename), VFSPath(VFSPath), IndexStorePath(IndexStorePath) {}
 };
 
 /// Command - An executable path/name and argument vector to
 /// execute.
 class Command {
   /// Source - The action which caused the creation of this job.
   const Action &Source;
 

include/clang/Driver/Options.td

 def objcmt_migrate_designated_init : Flag<["-"], "objcmt-migrate-designated-init">, Flags<[CC1Option]>,
   HelpText<"Enable migration to infer NS_DESIGNATED_INITIALIZER for initializer methods">;
 def objcmt_whitelist_dir_path: Joined<["-"], "objcmt-whitelist-dir-path=">, Flags<[CC1Option]>,
   HelpText<"Only modify files with a filename contained in the provided directory path">;
 // The misspelt "white-list" [sic] alias is due for removal.
 def : Joined<["-"], "objcmt-white-list-dir-path=">, Flags<[CC1Option]>,
     Alias<objcmt_whitelist_dir_path>;
 
+def index_store_path : Separate<["-"], "index-store-path">, Flags<[CC1Option]>,
+  HelpText<"Enable indexing with the specified data store path">;
+def index_ignore_system_symbols : Flag<["-"], "index-ignore-system-symbols">, Flags<[CC1Option]>,
+  HelpText<"Ignore symbols from system headers">;
+def index_record_codegen_name : Flag<["-"], "index-record-codegen-name">, Flags<[CC1Option]>,
+  HelpText<"Record the codegen name for symbols">;
+
 // Make sure all other -ccc- options are rejected.
 def ccc_ : Joined<["-"], "ccc-">, Group<internal_Group>, Flags<[Unsupported]>;
 
 // Standard Options
 
 def _HASH_HASH_HASH : Flag<["-"], "###">, Flags<[DriverOption, CoreOption]>,
     HelpText<"Print (but do not run) the commands to run for this compilation">;
 def _DASH_DASH : Option<["--"], "", KIND_REMAINING_ARGS>,

include/clang/Frontend/CompilerInstance.h

class CompilerInstance : public ModuleLoader {
   /// If the output doesn't support seeking (terminal, pipe). we switch
   /// the stream to a buffer_ostream. These are the buffer and the original
   /// stream.
   std::unique_ptr<llvm::raw_fd_ostream> NonSeekStream;
 
   /// The list of active output files.
   std::list<OutputFile> OutputFiles;
 
+  /// \brief An optional callback function used to wrap all FrontendActions
+  /// produced to generate imported modules before they are executed.
+  std::function<std::unique_ptr<FrontendAction>
+    (const FrontendOptions &opts, std::unique_ptr<FrontendAction> action)>
+    GenModuleActionWrapper;
+
   CompilerInstance(const CompilerInstance &) = delete;
   void operator=(const CompilerInstance &) = delete;
 public:
   explicit CompilerInstance(
       std::shared_ptr<PCHContainerOperations> PCHContainerOps =
           std::make_shared<PCHContainerOperations>(),
       MemoryBufferCache *SharedPCMCache = nullptr);
   ~CompilerInstance() override;
public:
   bool hadModuleLoaderFatalFailure() const {
     return ModuleLoader::HadFatalFailure;
   }
 
   GlobalModuleIndex *loadGlobalModuleIndex(SourceLocation TriggerLoc) override;
 
   bool lookupMissingImports(StringRef Name, SourceLocation TriggerLoc) override;
 
+  void setGenModuleActionWrapper(std::function<std::unique_ptr<FrontendAction>
+    (const FrontendOptions &Opts, std::unique_ptr<FrontendAction> Action)> Wrapper) {
+    GenModuleActionWrapper = Wrapper;
+  };
+
+  std::function<std::unique_ptr<FrontendAction>
+    (const FrontendOptions &Opts, std::unique_ptr<FrontendAction> Action)>
+  getGenModuleActionWrapper() const { return GenModuleActionWrapper; }
+
   void addDependencyCollector(std::shared_ptr<DependencyCollector> Listener) {
     DependencyCollectors.push_back(std::move(Listener));
   }
 
   void setExternalSemaSource(IntrusiveRefCntPtr<ExternalSemaSource> ESS);
 
   MemoryBufferCache &getPCMCache() const { return *PCMCache; }
 };
 
 } // end namespace clang
 
 #endif

include/clang/Frontend/FrontendOptions.h

ObjCMT_MigrateAll = (ObjCMT_Literals | ObjCMT_Subscripting |
                          ObjCMT_MigrateDecls | ObjCMT_PropertyDotSyntax)
   };
   unsigned ObjCMTAction;
   std::string ObjCMTWhiteListPath;
 
   std::string MTMigrateDir;
   std::string ARCMTMigrateReportOut;
 
+  std::string IndexStorePath;
+  unsigned IndexIgnoreSystemSymbols : 1;
+  unsigned IndexRecordCodegenName : 1;
+
   /// The input files and their types.
   std::vector<FrontendInputFile> Inputs;
 
   /// When the input is a module map, the original module map file from which
   /// that map was inferred, if any (for umbrella modules).
   std::string OriginalModuleMap;
 
   /// The output file, if any.
public:
   FrontendOptions() :
     DisableFree(false), RelocatablePCH(false), ShowHelp(false),
     ShowStats(false), ShowTimers(false), ShowVersion(false),
     FixWhatYouCan(false), FixOnlyWarnings(false), FixAndRecompile(false),
     FixToTemporaries(false), ARCMTMigrateEmitARCErrors(false),
     SkipFunctionBodies(false), UseGlobalModuleIndex(true),
     GenerateGlobalModuleIndex(true), ASTDumpDecls(false), ASTDumpLookups(false),
     BuildingImplicitModule(false), ModulesEmbedAllFiles(false),
-    IncludeTimestamps(true), ARCMTAction(ARCMT_None),
-    ObjCMTAction(ObjCMT_None), ProgramAction(frontend::ParseSyntaxOnly)
+    IncludeTimestamps(true), ARCMTAction(ARCMT_None), ObjCMTAction(ObjCMT_None),
+    IndexIgnoreSystemSymbols(false), IndexRecordCodegenName(false),
+    ProgramAction(frontend::ParseSyntaxOnly)
   {}
 
   /// getInputKindForExtension - Return the appropriate input kind for a file
   /// extension. For example, "c" would return InputKind::C.
   ///
   /// \return The input kind for the extension, or InputKind::Unknown if the
   /// extension is not recognized.
   static InputKind getInputKindForExtension(StringRef Extension);
 };
 
 }  // end namespace clang
 
 #endif

include/clang/Index/IndexDataStore.h

+//===--- IndexDataStore.h - Index data store info -------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEX_INDEXDATASTORE_H
+#define LLVM_CLANG_INDEX_INDEXDATASTORE_H
+
+#include "clang/Basic/LLVM.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/ADT/STLExtras.h"
+#include <functional>
+#include <memory>
+#include <string>
+#include <vector>
+
+namespace clang {
+namespace index {
+
+class AbstractDirectoryWatcher {
+public:
+  enum class EventKind {
+    /// A file was added.
+    Added,
+    /// A file was removed.
+    Removed,
+    /// A file was modified.
+    Modified,
+    /// The watched directory got deleted. No more events will follow.
+    DirectoryDeleted,
+  };
+
+  struct Event {
+    EventKind Kind;
+    std::string Filename;
+    timespec ModTime;
+  };
+
+  typedef std::function<void(ArrayRef<Event> Events, bool isInitial)> EventReceiver;
+  typedef std::unique_ptr<AbstractDirectoryWatcher>(CreateFnTy)
+    (StringRef Path, EventReceiver Receiver, bool waitInitialSync, std::string &Error);
+
+  virtual ~AbstractDirectoryWatcher() {}
+};
+
+class IndexDataStore {
+public:
+  ~IndexDataStore();
+
+  static std::unique_ptr<IndexDataStore>
+    create(StringRef IndexStorePath, std::string &Error);
+
+  StringRef getFilePath() const;
+  bool foreachUnitName(bool sorted,
+                       llvm::function_ref<bool(StringRef unitName)> receiver);
+
+  static unsigned getFormatVersion();
+
+  enum class UnitEventKind {
+    Added,
+    Removed,
+    Modified,
+    /// The directory got deleted. No more events will follow.
+    DirectoryDeleted,
+  };
+  struct UnitEvent {
+    UnitEventKind Kind;
+    StringRef UnitName;
+    timespec ModTime;
+  };
+  struct UnitEventNotification {
+    bool IsInitial;
+    ArrayRef<UnitEvent> Events;
+  };
+  typedef std::function<void(UnitEventNotification)> UnitEventHandler;
+
+  void setUnitEventHandler(UnitEventHandler Handler);
+  /// \returns true if an error occurred.
+  bool startEventListening(llvm::function_ref<AbstractDirectoryWatcher::CreateFnTy> createFn,
+                           bool waitInitialSync, std::string &Error);
+  void stopEventListening();
+
+  void discardUnit(StringRef UnitName);
+  void discardRecord(StringRef RecordName);
+
+  void purgeStaleData();
+
+private:
+  IndexDataStore(void *Impl) : Impl(Impl) {}
+
+  void *Impl; // An IndexDataStoreImpl.
+};
+
+} // namespace index
+} // namespace clang
+
+#endif

include/clang/Index/IndexDataStoreSymbolUtils.h

+//===--- IndexDataStoreSymbolUtils.h - Utilities for indexstore symbols ---===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEX_INDEXDATASTORESYMBOLUTILS_H
+#define LLVM_CLANG_INDEX_INDEXDATASTORESYMBOLUTILS_H
+
+#include "indexstore/indexstore.h"
+#include "clang/Index/IndexSymbol.h"
+
+namespace clang {
+namespace index {
+
+/// Map an indexstore_symbol_kind_t to a SymbolKind, handling unknown values.
+SymbolKind getSymbolKind(indexstore_symbol_kind_t K);
+
+SymbolSubKind getSymbolSubKind(indexstore_symbol_subkind_t K);
+
+/// Map an indexstore_symbol_language_t to a SymbolLanguage, handling unknown
+/// values.
+SymbolLanguage getSymbolLanguage(indexstore_symbol_language_t L);
+
+/// Map an indexstore representation to a SymbolPropertySet, handling
+/// unknown values.
+SymbolPropertySet getSymbolProperties(uint64_t Props);
+
+/// Map an indexstore representation to a SymbolRoleSet, handling unknown
+/// values.
+SymbolRoleSet getSymbolRoles(uint64_t Roles);
+
+/// Map a SymbolLanguage to a indexstore_symbol_language_t.
+indexstore_symbol_kind_t getIndexStoreKind(SymbolKind K);
+
+indexstore_symbol_subkind_t getIndexStoreSubKind(SymbolSubKind K);
+
+/// Map a SymbolLanguage to a indexstore_symbol_language_t.
+indexstore_symbol_language_t getIndexStoreLang(SymbolLanguage L);
+
+/// Map a SymbolPropertySet to its indexstore representation.
+uint64_t getIndexStoreProperties(SymbolPropertySet Props);
+
+/// Map a SymbolRoleSet to its indexstore representation.
+uint64_t getIndexStoreRoles(SymbolRoleSet Roles);
+
+} // end namespace index
+} // end namespace clang
+
+#endif // LLVM_CLANG_INDEX_INDEXDATASTORESYMBOLUTILS_H

include/clang/Index/IndexRecordReader.h

+//===--- IndexRecordReader.h - Index record deserialization ---------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEX_INDEXRECORDREADER_H
+#define LLVM_CLANG_INDEX_INDEXRECORDREADER_H
+
+#include "clang/Index/IndexSymbol.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringRef.h"
+#include <memory>
+
+namespace llvm {
+  class MemoryBuffer;
+}
+
+namespace clang {
+namespace index {
+
+struct IndexRecordDecl {
+  unsigned DeclID;
+  SymbolInfo SymInfo;
+  SymbolRoleSet Roles;
+  SymbolRoleSet RelatedRoles;
+  StringRef Name;
+  StringRef USR;
+  StringRef CodeGenName;
+};
+
+struct IndexRecordRelation {
+  SymbolRoleSet Roles;
+  const IndexRecordDecl *Dcl = nullptr;
+
+  IndexRecordRelation() = default;
+  IndexRecordRelation(SymbolRoleSet Roles, const IndexRecordDecl *Dcl)
+    : Roles(Roles), Dcl(Dcl) {}
+};
+
+struct IndexRecordOccurrence {
+  const IndexRecordDecl *Dcl;
+  SmallVector<IndexRecordRelation, 4> Relations;
+  SymbolRoleSet Roles;
+  unsigned Line;
+  unsigned Column;
+};
+
+class IndexRecordReader {
+  IndexRecordReader();
+
+public:
+  static std::unique_ptr<IndexRecordReader>
+    createWithRecordFilename(StringRef RecordFilename, StringRef StorePath,
+                             std::string &Error);
+  static std::unique_ptr<IndexRecordReader>
+    createWithFilePath(StringRef FilePath, std::string &Error);
+  static std::unique_ptr<IndexRecordReader>
+    createWithBuffer(std::unique_ptr<llvm::MemoryBuffer> Buffer,
+                     std::string &Error);
+
+  ~IndexRecordReader();
+
+  struct DeclSearchReturn {
+    bool AcceptDecl;
+    bool ContinueSearch;
+  };
+  typedef DeclSearchReturn(DeclSearchCheck)(const IndexRecordDecl &);
+
+  /// Goes through and passes record decls, after filtering using a \c Checker
+  /// function.
+  ///
+  /// Resulting decls can be used as filter for \c foreachOccurrence. This
+  /// allows allocating memory only for the record decls that the caller is
+  /// interested in.
+  bool searchDecls(llvm::function_ref<DeclSearchCheck> Checker,
+                   llvm::function_ref<void(const IndexRecordDecl *)> Receiver);
+
+  /// \param NoCache if true, avoids allocating memory for the decls.
+  /// Useful when the caller does not intend to keep \c IndexRecordReader
+  /// for more queries.
+  bool foreachDecl(bool NoCache,
+                   llvm::function_ref<bool(const IndexRecordDecl *)> Receiver);
+
+  /// \param DeclsFilter if non-empty indicates the list of decls that we want
+  /// to get occurrences for. An empty array indicates that we want occurrences
+  /// for all decls.
+  /// \param RelatedDeclsFilter Same as \c DeclsFilter but for related decls.
+  bool foreachOccurrence(ArrayRef<const IndexRecordDecl *> DeclsFilter,
+                         ArrayRef<const IndexRecordDecl *> RelatedDeclsFilter,
+              llvm::function_ref<bool(const IndexRecordOccurrence &)> Receiver);
+  bool foreachOccurrence(
+              llvm::function_ref<bool(const IndexRecordOccurrence &)> Receiver);
+
+  bool foreachOccurrenceInLineRange(unsigned lineStart, unsigned lineCount,
+              llvm::function_ref<bool(const IndexRecordOccurrence &)> Receiver);
+
+  struct Implementation;
+private:
+  Implementation &Impl;
+};
+
+} // namespace index
+} // namespace clang
+
+#endif

include/clang/Index/IndexRecordWriter.h

+//===--- IndexRecordWriter.h - Index record serialization -----------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEX_INDEXRECORDWRITER_H
+#define LLVM_CLANG_INDEX_INDEXRECORDWRITER_H
+
+#include "clang/Index/IndexSymbol.h"
+#include "llvm/ADT/SmallString.h"
+
+namespace clang {
+namespace index {
+
+namespace writer {
+/// An opaque pointer to a declaration or other symbol used by the
+/// IndexRecordWriter to identify when two occurrences refer to the same symbol,
+/// and as a token for getting information about a symbol from the caller.
+typedef const void *OpaqueDecl;
+
+/// An indexer symbol suitable for serialization.
+///
+/// This includes all the information about the symbol that will be serialized
+/// except for roles, which are synthesized by looking at all the occurrences.
+///
+/// \seealso IndexRecordDecl
+/// \note this struct is generally accompanied by a buffer that owns the string
+/// storage.  It should not be stored permanently.
+struct Symbol {
+  SymbolInfo SymInfo;
+  StringRef Name;
+  StringRef USR;
+  StringRef CodeGenName;
+};
+
+/// An relation to an opaque symbol.
+/// \seealso IndexRecordRelation
+struct SymbolRelation {
+  OpaqueDecl RelatedSymbol;
+  SymbolRoleSet Roles;
+};
+
+typedef llvm::function_ref<Symbol(OpaqueDecl, SmallVectorImpl<char> &Scratch)>
+    SymbolWriterCallback;
+} // end namespace writer
+
+/// A language-independent utility for serializing index record files.
+///
+/// Internally, this class is a small state machine.  Users should first call
+/// beginRecord, and if the file does not already exist, then proceed to add
+/// all symbol occurrences (addOccurrence) and finally finish with endRecord.
+class IndexRecordWriter {
+  SmallString<64> RecordsPath; ///< The records directory path.
+  void *Record = nullptr;      ///< The state of the current record.
+public:
+  IndexRecordWriter(StringRef IndexPath);
+
+  enum class Result {
+    Success,
+    Failure,
+    AlreadyExists,
+  };
+
+  /// Begin writing a record for the file \p Filename with contents uniquely
+  /// identified by \p RecordHash.
+  ///
+  /// \param Filename the name of the file this is a record for.
+  /// \param RecordHash the unique hash of the record contents.
+  /// \param Error on failure, set to the error message.
+  /// \param RecordFile if non-null, this is set to the name of the record file.
+  ///
+  /// \returns Success if we should continue writing this record, AlreadyExists
+  /// if the record file has already been written, or Failure if there was an
+  /// error, in which case \p Error will be set.
+  Result beginRecord(StringRef Filename, llvm::hash_code RecordHash,
+                     std::string &Error, std::string *RecordFile = nullptr);
+
+  /// Finish writing the record file.
+  ///
+  /// \param Error on failure, set to the error message.
+  /// \param GetSymbolForDecl a callback mapping an writer::OpaqueDecl to its
+  /// writer::Symbol. This is how the language-specific symbol information is
+  /// provided to the IndexRecordWriter. The scratch parameter can be used for
+  /// any necessary storage.
+  ///
+  /// \return Success, or Failure and sets \p Error.
+  Result endRecord(std::string &Error,
+                   writer::SymbolWriterCallback GetSymbolForDecl);
+
+  /// Add an occurrence of the symbol \p D with the given \p Roles and location.
+  void addOccurrence(writer::OpaqueDecl D, SymbolRoleSet Roles, unsigned Line,
+                     unsigned Column, ArrayRef<writer::SymbolRelation> Related);
+};
+
+} // end namespace index
+} // end namespace clang
+
+#endif // LLVM_CLANG_INDEX_INDEXRECORDWRITER_H

include/clang/Index/IndexUnitReader.h

+//===--- IndexUnitReader.h - Index unit deserialization -------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEX_INDEXUNITREADER_H
+#define LLVM_CLANG_INDEX_INDEXUNITREADER_H
+
+#include "clang/Basic/LLVM.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/Support/Chrono.h"
+
+namespace clang {
+namespace index {
+
+class IndexUnitReader {
+public:
+  enum class DependencyKind {
+    Unit,
+    Record,
+    File,
+  };
+
+  ~IndexUnitReader();
+
+  static std::unique_ptr<IndexUnitReader>
+    createWithUnitFilename(StringRef UnitFilename, StringRef StorePath,
+                           std::string &Error);
+  static std::unique_ptr<IndexUnitReader>
+    createWithFilePath(StringRef FilePath, std::string &Error);
+
+  static Optional<llvm::sys::TimePoint<>>
+    getModificationTimeForUnit(StringRef UnitFilename, StringRef StorePath,
+                               std::string &Error);
+
+  StringRef getProviderIdentifier() const;
+  StringRef getProviderVersion() const;
+
+  llvm::sys::TimePoint<> getModificationTime() const;
+  StringRef getWorkingDirectory() const;
+  StringRef getOutputFile() const;
+  StringRef getSysrootPath() const;
+  StringRef getMainFilePath() const;
+  StringRef getModuleName() const;
+  StringRef getTarget() const;
+  bool hasMainFile() const;
+  bool isSystemUnit() const;
+  bool isModuleUnit() const;
+  bool isDebugCompilation() const;
+
+  struct DependencyInfo {
+    DependencyKind Kind;
+    bool IsSystem;
+    StringRef UnitOrRecordName;
+    StringRef FilePath;
+    StringRef ModuleName;
+    size_t FileSize;
+    time_t ModTime;
+  };
+  struct IncludeInfo {
+    StringRef SourcePath;
+    unsigned SourceLine;
+    StringRef TargetPath;
+  };
+  /// Unit dependencies are provided ahead of record ones, record ones
+  /// ahead of the file ones.
+  bool foreachDependency(llvm::function_ref<bool(const DependencyInfo &Info)> Receiver);
+
+  bool foreachInclude(llvm::function_ref<bool(const IncludeInfo &Info)> Receiver);
+
+private:
+  IndexUnitReader(void *Impl) : Impl(Impl) {}
+
+  void *Impl; // An IndexUnitReaderImpl.
+};
+
+} // namespace index
+} // namespace clang
+
+#endif

include/clang/Index/IndexUnitWriter.h

+//===--- IndexUnitWriter.h - Index unit serialization ---------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEX_INDEXUNITWRITER_H
+#define LLVM_CLANG_INDEX_INDEXUNITWRITER_H
+
+#include "clang/Basic/LLVM.h"
+#include "llvm/ADT/DenseSet.h"
+#include "llvm/ADT/SmallString.h"
+#include <string>
+#include <vector>
+
+namespace llvm {
+  class BitstreamWriter;
+}
+
+namespace clang {
+  class FileEntry;
+  class FileManager;
+
+namespace index {
+
+namespace writer {
+/// An opaque pointer to a module used by the IndexUnitWriter to associate
+/// record and file dependencies with a module, and as a token for getting
+/// information about the module from the caller.
+typedef const void *OpaqueModule;
+
+/// Module info suitable for serialization.
+///
+/// This is used for top-level modules and sub-modules.
+struct ModuleInfo {
+  /// Full, dot-separate, module name.
+  StringRef Name;
+};
+
+typedef llvm::function_ref<ModuleInfo(OpaqueModule, SmallVectorImpl<char> &Scratch)>
+    ModuleInfoWriterCallback;
+} // end namespace writer
+
+class IndexUnitWriter {
+  FileManager &FileMgr;
+  SmallString<64> UnitsPath;
+  std::string ProviderIdentifier;
+  std::string ProviderVersion;
+  std::string OutputFile;
+  std::string ModuleName;
+  const FileEntry *MainFile;
+  bool IsSystemUnit;
+  bool IsModuleUnit;
+  bool IsDebugCompilation;
+  std::string TargetTriple;
+  std::string WorkDir;
+  std::string SysrootPath;
+  std::function<writer::ModuleInfo(writer::OpaqueModule,
+                            SmallVectorImpl<char> &Scratch)> GetInfoForModuleFn;
+  struct FileInclude {
+    int Index;
+    unsigned Line;
+  };
+  struct FileEntryData {
+    const FileEntry *File;
+    bool IsSystem;
+    int ModuleIndex;
+    std::vector<FileInclude> Includes;
+  };
+  std::vector<FileEntryData> Files;
+  std::vector<writer::OpaqueModule> Modules;
+  llvm::DenseMap<const FileEntry *, int> IndexByFile;
+  llvm::DenseMap<writer::OpaqueModule, int> IndexByModule;
+  llvm::DenseSet<const FileEntry *> SeenASTFiles;
+  struct RecordOrUnitData {
+    std::string Name;
+    int FileIndex;
+    int ModuleIndex;
+    bool IsSystem;
+  };
+  std::vector<RecordOrUnitData> Records;
+  std::vector<RecordOrUnitData> ASTFileUnits;
+
+public:
+  /// \param MainFile the main file for a compiled source file. This should be
+  /// null for PCH and module units.
+  /// \param IsSystem true for system module units, false otherwise.
+  IndexUnitWriter(FileManager &FileMgr,
+                  StringRef StorePath,
+                  StringRef ProviderIdentifier, StringRef ProviderVersion,
+                  StringRef OutputFile,
+                  StringRef ModuleName,
+                  const FileEntry *MainFile,
+                  bool IsSystem,
+                  bool IsModuleUnit,
+                  bool IsDebugCompilation,
+                  StringRef TargetTriple,
+                  StringRef SysrootPath,
+                  writer::ModuleInfoWriterCallback GetInfoForModule);
+  ~IndexUnitWriter();
+
+  int addFileDependency(const FileEntry *File, bool IsSystem,
+                        writer::OpaqueModule Mod);
+  void addRecordFile(StringRef RecordFile, const FileEntry *File, bool IsSystem,
+                     writer::OpaqueModule Mod);
+  void addASTFileDependency(const FileEntry *File, bool IsSystem,
+                            writer::OpaqueModule Mod, bool withoutUnitName = false);
+  void addUnitDependency(StringRef UnitFile, const FileEntry *File, bool IsSystem,
+                         writer::OpaqueModule Mod);
+  bool addInclude(const FileEntry *Source, unsigned Line, const FileEntry *Target);
+
+  bool write(std::string &Error);
+
+  void getUnitNameForOutputFile(StringRef FilePath, SmallVectorImpl<char> &Str);
+  void getUnitPathForOutputFile(StringRef FilePath, SmallVectorImpl<char> &Str);
+  /// If the unit file exists and \p timeCompareFilePath is provided, it will
+  /// return true if \p timeCompareFilePath is older than the unit file.
+  Optional<bool> isUnitUpToDateForOutputFile(StringRef FilePath,
+                                             Optional<StringRef> TimeCompareFilePath,
+                                             std::string &Error);
+  static void getUnitNameForAbsoluteOutputFile(StringRef FilePath, SmallVectorImpl<char> &Str);
+  static bool initIndexDirectory(StringRef StorePath, std::string &Error);
+
+private:
+  class PathStorage;
+  int addModule(writer::OpaqueModule Mod);
+  void writeUnitInfo(llvm::BitstreamWriter &Stream, PathStorage &PathStore);
+  void writeDependencies(llvm::BitstreamWriter &Stream, PathStorage &PathStore);
+  void writeIncludes(llvm::BitstreamWriter &Stream, PathStorage &PathStore);
+  void writePaths(llvm::BitstreamWriter &Stream, PathStorage &PathStore);
+  void writeModules(llvm::BitstreamWriter &Stream);
+};
+
+} // end namespace index
+} // end namespace clang
+
+#endif

include/clang/Index/IndexingAction.h

 //===--- IndexingAction.h - Frontend index action -------------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_INDEX_INDEXINGACTION_H
 #define LLVM_CLANG_INDEX_INDEXINGACTION_H
 
 #include "clang/Basic/LLVM.h"
 #include "llvm/ADT/ArrayRef.h"
 #include <memory>
+#include <string>
 
 namespace clang {
   class ASTContext;
   class ASTReader;
   class ASTUnit;
+  class CompilerInstance;
   class Decl;
   class FrontendAction;
+  class FrontendOptions;
+  class Module;
 
 namespace serialization {
   class ModuleFile;
 }
 
 namespace index {
   class IndexDataConsumer;
+  class IndexUnitWriter;
 
 struct IndexingOptions {
   enum class SystemSymbolFilterKind {
     None,
     DeclarationsOnly,
     All,
   };
 
   SystemSymbolFilterKind SystemSymbolFilter
     = SystemSymbolFilterKind::DeclarationsOnly;
   bool IndexFunctionLocals = false;
 };
 
+struct RecordingOptions {
+  enum class IncludesRecordingKind {
+    None,
+    UserOnly, // only record includes inside non-system files.
+    All,
+  };
+
+  std::string DataDirPath;
+  bool RecordSymbolCodeGenName = false;
+  bool RecordSystemDependencies = true;
+  IncludesRecordingKind RecordIncludes = IncludesRecordingKind::UserOnly;
+};
+
 /// \param WrappedAction another frontend action to wrap over or null.
 std::unique_ptr<FrontendAction>
 createIndexingAction(std::shared_ptr<IndexDataConsumer> DataConsumer,
                      IndexingOptions Opts,
                      std::unique_ptr<FrontendAction> WrappedAction);
 
 void indexASTUnit(ASTUnit &Unit,
                   std::shared_ptr<IndexDataConsumer> DataConsumer,
                   IndexingOptions Opts);
 
 void indexTopLevelDecls(ASTContext &Ctx, ArrayRef<const Decl *> Decls,
                         std::shared_ptr<IndexDataConsumer> DataConsumer,
                         IndexingOptions Opts);
 
 void indexModuleFile(serialization::ModuleFile &Mod, ASTReader &Reader,
                      std::shared_ptr<IndexDataConsumer> DataConsumer,
                      IndexingOptions Opts);
 
+/// \param WrappedAction another frontend action to wrap over or null.
+std::unique_ptr<FrontendAction>
+createIndexDataRecordingAction(const FrontendOptions &FEOpts,
+                               std::unique_ptr<FrontendAction> WrappedAction);
+
+/// Checks if the unit file exists for the module file, if it doesn't it
+/// generates index data for it.
+///
+/// \returns true if the index data were generated, false otherwise.
+bool emitIndexDataForModuleFile(const Module *Mod, const CompilerInstance &CI,
+                                IndexUnitWriter &ParentUnitWriter);
+
 } // namespace index
 } // namespace clang
 
 #endif

include/indexstore/IndexStoreCXX.h

+//===--- IndexStoreCXX.h - C++ wrapper for the Index Store C API. ---------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// Header-only C++ wrapper for the Index Store C API.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEXSTORE_INDEXSTORECXX_H
+#define LLVM_CLANG_INDEXSTORE_INDEXSTORECXX_H
+
+#include "indexstore/indexstore.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/Optional.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/SmallString.h"
+#include <ctime>
+
+namespace indexstore {
+  using llvm::ArrayRef;
+  using llvm::Optional;
+  using llvm::StringRef;
+
+static inline StringRef stringFromIndexStoreStringRef(indexstore_string_ref_t str) {
+  return StringRef(str.data, str.length);
+}
+
+class IndexRecordSymbol {
+  indexstore_symbol_t obj;
+  friend class IndexRecordReader;
+
+public:
+  IndexRecordSymbol(indexstore_symbol_t obj) : obj(obj) {}
+
+  indexstore_symbol_language_t getLanguage() {
+    return indexstore_symbol_get_language(obj);
+  }
+  indexstore_symbol_kind_t getKind() { return indexstore_symbol_get_kind(obj); }
+  indexstore_symbol_subkind_t getSubKind() { return indexstore_symbol_get_subkind(obj); }
+  uint64_t getProperties() {
+    return indexstore_symbol_get_properties(obj);
+  }
+  uint64_t getRoles() { return indexstore_symbol_get_roles(obj); }
+  uint64_t getRelatedRoles() { return indexstore_symbol_get_related_roles(obj); }
+  StringRef getName() { return stringFromIndexStoreStringRef(indexstore_symbol_get_name(obj)); }
+  StringRef getUSR() { return stringFromIndexStoreStringRef(indexstore_symbol_get_usr(obj)); }
+  StringRef getCodegenName() { return stringFromIndexStoreStringRef(indexstore_symbol_get_codegen_name(obj)); }
+};
+
+class IndexSymbolRelation {
+  indexstore_symbol_relation_t obj;
+
+public:
+  IndexSymbolRelation(indexstore_symbol_relation_t obj) : obj(obj) {}
+
+  uint64_t getRoles() { return indexstore_symbol_relation_get_roles(obj); }
+  IndexRecordSymbol getSymbol() { return indexstore_symbol_relation_get_symbol(obj); }
+};
+
+class IndexRecordOccurrence {
+  indexstore_occurrence_t obj;
+
+public:
+  IndexRecordOccurrence(indexstore_occurrence_t obj) : obj(obj) {}
+
+  IndexRecordSymbol getSymbol() { return indexstore_occurrence_get_symbol(obj); }
+  uint64_t getRoles() { return indexstore_occurrence_get_roles(obj); }
+
+  bool foreachRelation(llvm::function_ref<bool(IndexSymbolRelation)> receiver) {
+#if INDEXSTORE_HAS_BLOCKS
+    return indexstore_occurrence_relations_apply(obj, ^bool(indexstore_symbol_relation_t sym_rel) {
+      return receiver(sym_rel);
+    });
+#else
+    return false;
+#endif
+  }
+
+  std::pair<unsigned, unsigned> getLineCol() {
+    unsigned line, col;
+    indexstore_occurrence_get_line_col(obj, &line, &col);
+    return std::make_pair(line, col);
+  }
+};
+
+class IndexStore;
+typedef std::shared_ptr<IndexStore> IndexStoreRef;
+
+class IndexStore {
+  indexstore_t obj;
+  friend class IndexRecordReader;
+  friend class IndexUnitReader;
+
+public:
+  IndexStore(StringRef path, std::string &error) {
+    llvm::SmallString<64> buf = path;
+    indexstore_error_t c_err = nullptr;
+    obj = indexstore_store_create(buf.c_str(), &c_err);
+    if (c_err) {
+      error = indexstore_error_get_description(c_err);
+      indexstore_error_dispose(c_err);
+    }
+  }
+
+  IndexStore(IndexStore &&other) : obj(other.obj) {
+    other.obj = nullptr;
+  }
+
+  ~IndexStore() {
+    indexstore_store_dispose(obj);
+  }
+
+  static IndexStoreRef create(StringRef path, std::string &error) {
+    auto storeRef = std::make_shared<IndexStore>(path, error);
+    if (storeRef->isInvalid())
+      return nullptr;
+    return storeRef;
+  }
+
+  static unsigned formatVersion() {
+    return indexstore_format_version();
+  }
+
+  bool isValid() const { return obj; }
+  bool isInvalid() const { return !isValid(); }
+  explicit operator bool() const { return isValid(); }
+
+  bool foreachUnit(bool sorted, llvm::function_ref<bool(StringRef unitName)> receiver) {
+#if INDEXSTORE_HAS_BLOCKS
+    return indexstore_store_units_apply(obj, sorted, ^bool(indexstore_string_ref_t unit_name) {
+      return receiver(stringFromIndexStoreStringRef(unit_name));
+    });
+#else
+    return false;
+#endif
+  }
+
+  class UnitEvent {
+    indexstore_unit_event_t obj;
+  public:
+    UnitEvent(indexstore_unit_event_t obj) : obj(obj) {}
+
+    enum class Kind {
+      Added,
+      Removed,
+      Modified,
+      DirectoryDeleted,
+    };
+    Kind getKind() const {
+      indexstore_unit_event_kind_t c_k = indexstore_unit_event_get_kind(obj);
+      Kind K;
+      switch (c_k) {
+      case INDEXSTORE_UNIT_EVENT_ADDED: K = Kind::Added; break;
+      case INDEXSTORE_UNIT_EVENT_REMOVED: K = Kind::Removed; break;
+      case INDEXSTORE_UNIT_EVENT_MODIFIED: K = Kind::Modified; break;
+      case INDEXSTORE_UNIT_EVENT_DIRECTORY_DELETED: K = Kind::DirectoryDeleted; break;
+      }
+      return K;
+    }
+
+    StringRef getUnitName() const {
+      return stringFromIndexStoreStringRef(indexstore_unit_event_get_unit_name(obj));
+    }
+
+    timespec getModificationTime() const { return indexstore_unit_event_get_modification_time(obj); }
+  };
+
+  class UnitEventNotification {
+    indexstore_unit_event_notification_t obj;
+  public:
+    UnitEventNotification(indexstore_unit_event_notification_t obj) : obj(obj) {}
+
+    bool isInitial() const { return indexstore_unit_event_notification_is_initial(obj); }
+    size_t getEventsCount() const { return indexstore_unit_event_notification_get_events_count(obj); }
+    UnitEvent getEvent(size_t index) const { return indexstore_unit_event_notification_get_event(obj, index); }
+  };
+
+  typedef std::function<void(UnitEventNotification)> UnitEventHandler;
+
+  void setUnitEventHandler(UnitEventHandler handler) {
+#if INDEXSTORE_HAS_BLOCKS
+    if (!handler) {
+      indexstore_store_set_unit_event_handler(obj, nullptr);
+      return;
+    }
+
+    indexstore_store_set_unit_event_handler(obj, ^(indexstore_unit_event_notification_t evt_note) {
+      handler(UnitEventNotification(evt_note));
+    });
+#endif
+  }
+
+  bool startEventListening(bool waitInitialSync, std::string &error) {
+    indexstore_unit_event_listen_options_t opts;
+    opts.wait_initial_sync = waitInitialSync;
+    indexstore_error_t c_err = nullptr;
+    bool ret = indexstore_store_start_unit_event_listening(obj, &opts, sizeof(opts), &c_err);
+    if (c_err) {
+      error = indexstore_error_get_description(c_err);
+      indexstore_error_dispose(c_err);
+    }
+    return ret;
+  }
+
+  void stopEventListening() {
+    return indexstore_store_stop_unit_event_listening(obj);
+  }
+
+  void discardUnit(StringRef UnitName) {
+    llvm::SmallString<64> buf = UnitName;
+    indexstore_store_discard_unit(obj, buf.c_str());
+  }
+
+  void discardRecord(StringRef RecordName) {
+    llvm::SmallString<64> buf = RecordName;
+    indexstore_store_discard_record(obj, buf.c_str());
+  }
+
+  void getUnitNameFromOutputPath(StringRef outputPath, llvm::SmallVectorImpl<char> &nameBuf) {
+    llvm::SmallString<256> buf = outputPath;
+    size_t nameLen = indexstore_store_get_unit_name_from_output_path(obj, buf.c_str(), nameBuf.data(), nameBuf.size());
+    if (nameLen+1 > nameBuf.size()) {
+      nameBuf.resize(nameLen+1);
+      indexstore_store_get_unit_name_from_output_path(obj, buf.c_str(), nameBuf.data(), nameBuf.size());
+    }
+  }
+
+  llvm::Optional<timespec>
+  getUnitModificationTime(StringRef unitName, std::string &error) {
+    llvm::SmallString<64> buf = unitName;
+    int64_t seconds, nanoseconds;
+    indexstore_error_t c_err = nullptr;
+    bool err = indexstore_store_get_unit_modification_time(obj, buf.c_str(),
+      &seconds, &nanoseconds, &c_err);
+    if (err && c_err) {
+      error = indexstore_error_get_description(c_err);
+      indexstore_error_dispose(c_err);
+      return llvm::None;
+    }
+    timespec ts;
+    ts.tv_sec = seconds;
+    ts.tv_nsec = nanoseconds;
+    return ts;
+  }
+
+  void purgeStaleData() {
+    indexstore_store_purge_stale_data(obj);
+  }
+};
+
+class IndexRecordReader {
+  indexstore_record_reader_t obj;
+
+public:
+  IndexRecordReader(IndexStore &store, StringRef recordName, std::string &error) {
+    llvm::SmallString<64> buf = recordName;
+    indexstore_error_t c_err = nullptr;
+    obj = indexstore_record_reader_create(store.obj, buf.c_str(), &c_err);
+    if (c_err) {
+      error = indexstore_error_get_description(c_err);
+      indexstore_error_dispose(c_err);
+    }
+  }
+
+  IndexRecordReader(IndexRecordReader &&other) : obj(other.obj) {
+    other.obj = nullptr;
+  }
+
+  ~IndexRecordReader() {
+    indexstore_record_reader_dispose(obj);
+  }
+
+  bool isValid() const { return obj; }
+  bool isInvalid() const { return !isValid(); }
+  explicit operator bool() const { return isValid(); }
+
+  /// Goes through and passes record decls, after filtering using a \c Checker
+  /// function.
+  ///
+  /// Resulting decls can be used as filter for \c foreachOccurrence. This
+  /// allows allocating memory only for the record decls that the caller is
+  /// interested in.
+  bool searchSymbols(llvm::function_ref<bool(IndexRecordSymbol, bool &stop)> filter,
+                     llvm::function_ref<void(IndexRecordSymbol)> receiver) {
+#if INDEXSTORE_HAS_BLOCKS
+    return indexstore_record_reader_search_symbols(obj, ^bool(indexstore_symbol_t symbol, bool *stop) {
+      return filter(symbol, *stop);
+    }, ^(indexstore_symbol_t symbol) {
+      receiver(symbol);
+    });
+#else
+    return false;
+#endif
+  }
+
+  bool foreachSymbol(bool noCache, llvm::function_ref<bool(IndexRecordSymbol)> receiver) {
+#if INDEXSTORE_HAS_BLOCKS
+    return indexstore_record_reader_symbols_apply(obj, noCache, ^bool(indexstore_symbol_t sym) {
+      return receiver(sym);
+    });
+#else
+    return false;
+#endif
+  }
+
+  /// \param DeclsFilter if non-empty indicates the list of decls that we want
+  /// to get occurrences for. An empty array indicates that we want occurrences
+  /// for all decls.
+  /// \param RelatedDeclsFilter Same as \c DeclsFilter but for related decls.
+  bool foreachOccurrence(ArrayRef<IndexRecordSymbol> symbolsFilter,
+                         ArrayRef<IndexRecordSymbol> relatedSymbolsFilter,
+              llvm::function_ref<bool(IndexRecordOccurrence)> receiver) {
+#if INDEXSTORE_HAS_BLOCKS
+    llvm::SmallVector<indexstore_symbol_t, 16> c_symbolsFilter;
+    c_symbolsFilter.reserve(symbolsFilter.size());
+    for (IndexRecordSymbol sym : symbolsFilter) {
+      c_symbolsFilter.push_back(sym.obj);
+    }
+    llvm::SmallVector<indexstore_symbol_t, 16> c_relatedSymbolsFilter;
+    c_relatedSymbolsFilter.reserve(relatedSymbolsFilter.size());
+    for (IndexRecordSymbol sym : relatedSymbolsFilter) {
+      c_relatedSymbolsFilter.push_back(sym.obj);
+    }
+    return indexstore_record_reader_occurrences_of_symbols_apply(obj,
+                                c_symbolsFilter.data(), c_symbolsFilter.size(),
+                                c_relatedSymbolsFilter.data(),
+                                c_relatedSymbolsFilter.size(),
+                                ^bool(indexstore_occurrence_t occur) {
+                                  return receiver(occur);
+                                });
+#else
+    return false;
+#endif
+  }
+
+  bool foreachOccurrence(
+              llvm::function_ref<bool(IndexRecordOccurrence)> receiver) {
+#if INDEXSTORE_HAS_BLOCKS
+    return indexstore_record_reader_occurrences_apply(obj, ^bool(indexstore_occurrence_t occur) {
+      return receiver(occur);
+    });
+#else
+    return false;
+#endif
+  }
+
+  bool foreachOccurrenceInLineRange(unsigned lineStart, unsigned lineEnd,
+              llvm::function_ref<bool(IndexRecordOccurrence)> receiver) {
+#if INDEXSTORE_HAS_BLOCKS
+    return indexstore_record_reader_occurrences_in_line_range_apply(obj,
+                                                                    lineStart,
+                                                                    lineEnd,
+                                          ^bool(indexstore_occurrence_t occur) {
+      return receiver(occur);
+    });
+#else
+    return false;
+#endif
+  }
+};
+
+class IndexUnitDependency {
+  indexstore_unit_dependency_t obj;
+  friend class IndexUnitReader;
+
+public:
+  IndexUnitDependency(indexstore_unit_dependency_t obj) : obj(obj) {}
+
+  enum class DependencyKind {
+    Unit,
+    Record,
+    File,
+  };
+  DependencyKind getKind() {
+    switch (indexstore_unit_dependency_get_kind(obj)) {
+    case INDEXSTORE_UNIT_DEPENDENCY_UNIT: return DependencyKind::Unit;
+    case INDEXSTORE_UNIT_DEPENDENCY_RECORD: return DependencyKind::Record;
+    case INDEXSTORE_UNIT_DEPENDENCY_FILE: return DependencyKind::File;
+    }
+  }
+  bool isSystem() { return indexstore_unit_dependency_is_system(obj); }
+  StringRef getName() { return stringFromIndexStoreStringRef(indexstore_unit_dependency_get_name(obj)); }
+  StringRef getFilePath() { return stringFromIndexStoreStringRef(indexstore_unit_dependency_get_filepath(obj)); }
+  StringRef getModuleName() { return stringFromIndexStoreStringRef(indexstore_unit_dependency_get_modulename(obj)); }
+  time_t getModificationTime() { return indexstore_unit_dependency_get_modification_time(obj); }
+  size_t getFileSize() { return indexstore_unit_dependency_get_file_size(obj); }
+
+};
+
+class IndexUnitInclude {
+  indexstore_unit_include_t obj;
+  friend class IndexUnitReader;
+
+public:
+  IndexUnitInclude(indexstore_unit_include_t obj) : obj(obj) {}
+
+  StringRef getSourcePath() {
+    return stringFromIndexStoreStringRef(indexstore_unit_include_get_source_path(obj));
+  }
+  StringRef getTargetPath() {
+    return stringFromIndexStoreStringRef(indexstore_unit_include_get_target_path(obj));
+  }
+  unsigned getSourceLine() {
+    return indexstore_unit_include_get_source_line(obj);
+  }
+};
+
+class IndexUnitReader {
+  indexstore_unit_reader_t obj;
+
+public:
+  IndexUnitReader(IndexStore &store, StringRef unitName, std::string &error) {
+    llvm::SmallString<64> buf = unitName;
+    indexstore_error_t c_err = nullptr;
+    obj = indexstore_unit_reader_create(store.obj, buf.c_str(), &c_err);
+    if (c_err) {
+      error = indexstore_error_get_description(c_err);
+      indexstore_error_dispose(c_err);
+    }
+  }
+
+  IndexUnitReader(IndexUnitReader &&other) : obj(other.obj) {
+    other.obj = nullptr;
+  }
+
+  ~IndexUnitReader() {
+    indexstore_unit_reader_dispose(obj);
+  }
+
+  bool isValid() const { return obj; }
+  bool isInvalid() const { return !isValid(); }
+  explicit operator bool() const { return isValid(); }
+
+  StringRef getProviderIdentifier() {
+    return stringFromIndexStoreStringRef(indexstore_unit_reader_get_provider_identifier(obj));
+  }
+  StringRef getProviderVersion() {
+    return stringFromIndexStoreStringRef(indexstore_unit_reader_get_provider_version(obj));
+  }
+
+  timespec getModificationTime() {
+    int64_t seconds, nanoseconds;
+    indexstore_unit_reader_get_modification_time(obj, &seconds, &nanoseconds);
+    timespec ts;
+    ts.tv_sec = seconds;
+    ts.tv_nsec = nanoseconds;
+    return ts;
+  }
+
+  bool isSystemUnit() { return indexstore_unit_reader_is_system_unit(obj); }
+  bool isModuleUnit() { return indexstore_unit_reader_is_module_unit(obj); }
+  bool isDebugCompilation() { return indexstore_unit_reader_is_debug_compilation(obj); }
+  bool hasMainFile() { return indexstore_unit_reader_has_main_file(obj); }
+
+  StringRef getMainFilePath() {
+    return stringFromIndexStoreStringRef(indexstore_unit_reader_get_main_file(obj));
+  }
+  StringRef getModuleName() {
+    return stringFromIndexStoreStringRef(indexstore_unit_reader_get_module_name(obj));
+  }
+  StringRef getWorkingDirectory() {
+    return stringFromIndexStoreStringRef(indexstore_unit_reader_get_working_dir(obj));
+  }
+  StringRef getOutputFile() {
+    return stringFromIndexStoreStringRef(indexstore_unit_reader_get_output_file(obj));
+  }
+  StringRef getSysrootPath() {
+    return stringFromIndexStoreStringRef(indexstore_unit_reader_get_sysroot_path(obj));
+  }
+  StringRef getTarget() {
+    return stringFromIndexStoreStringRef(indexstore_unit_reader_get_target(obj));
+  }
+
+  bool foreachDependency(llvm::function_ref<bool(IndexUnitDependency)> receiver) {
+#if INDEXSTORE_HAS_BLOCKS
+    return indexstore_unit_reader_dependencies_apply(obj, ^bool(indexstore_unit_dependency_t dep) {
+      return receiver(dep);
+    });
+#else
+    return false;
+#endif
+  }
+
+  bool foreachInclude(llvm::function_ref<bool(IndexUnitInclude)> receiver) {
+#if INDEXSTORE_HAS_BLOCKS
+    return indexstore_unit_reader_includes_apply(obj, ^bool(indexstore_unit_include_t inc) {
+      return receiver(inc);
+    });
+#else
+    return false;
+#endif
+  }
+};
+
+} // namespace indexstore
+
+#endif

include/indexstore/indexstore.h

+/*===-- indexstore/indexstore.h - Index Store C API ----------------- C -*-===*\
+|*                                                                            *|
+|*                     The LLVM Compiler Infrastructure                       *|
+|*                                                                            *|
+|* This file is distributed under the University of Illinois Open Source      *|
+|* License. See LICENSE.TXT for details.                                      *|
+|*                                                                            *|
+|*===----------------------------------------------------------------------===*|
+|*                                                                            *|
+|* This header provides a C API for the index store.                          *|
+|*                                                                            *|
+\*===----------------------------------------------------------------------===*/
+
+#ifndef LLVM_CLANG_C_INDEXSTORE_INDEXSTORE_H
+#define LLVM_CLANG_C_INDEXSTORE_INDEXSTORE_H
+
+#include <stdint.h>
+#include <stddef.h>
+#include <ctime>
+
+/**
+ * \brief The version constants for the Index Store C API.
+ * INDEXSTORE_VERSION_MINOR should increase when there are API additions.
+ * INDEXSTORE_VERSION_MAJOR is intended for "major" source/ABI breaking changes.
+ */
+#define INDEXSTORE_VERSION_MAJOR 0
+#define INDEXSTORE_VERSION_MINOR 9
+
+#define INDEXSTORE_VERSION_ENCODE(major, minor) ( \
+      ((major) * 10000)                           \
+    + ((minor) *     1))
+
+#define INDEXSTORE_VERSION INDEXSTORE_VERSION_ENCODE( \
+    INDEXSTORE_VERSION_MAJOR,                         \
+    INDEXSTORE_VERSION_MINOR )
+
+#define INDEXSTORE_VERSION_STRINGIZE_(major, minor)   \
+    #major"."#minor
+#define INDEXSTORE_VERSION_STRINGIZE(major, minor)    \
+    INDEXSTORE_VERSION_STRINGIZE_(major, minor)
+
+#define INDEXSTORE_VERSION_STRING INDEXSTORE_VERSION_STRINGIZE( \
+    INDEXSTORE_VERSION_MAJOR,                                   \
+    INDEXSTORE_VERSION_MINOR)
+
+#ifdef  __cplusplus
+# define INDEXSTORE_BEGIN_DECLS  extern "C" {
+# define INDEXSTORE_END_DECLS    }
+#else
+# define INDEXSTORE_BEGIN_DECLS
+# define INDEXSTORE_END_DECLS
+#endif
+
+#ifndef INDEXSTORE_PUBLIC
+# if defined (_MSC_VER)
+#  define INDEXSTORE_PUBLIC __declspec(dllimport)
+# else
+#  define INDEXSTORE_PUBLIC
+# endif
+#endif
+
+#ifndef __has_feature
+# define __has_feature(x) 0
+#endif
+
+#if __has_feature(blocks)
+# define INDEXSTORE_HAS_BLOCKS 1
+#else
+# define INDEXSTORE_HAS_BLOCKS 0
+#endif
+
+INDEXSTORE_BEGIN_DECLS
+
+typedef void *indexstore_error_t;
+
+INDEXSTORE_PUBLIC const char *
+indexstore_error_get_description(indexstore_error_t);
+
+INDEXSTORE_PUBLIC void
+indexstore_error_dispose(indexstore_error_t);
+
+typedef struct {
+  const char *data;
+  size_t length;
+} indexstore_string_ref_t;
+
+INDEXSTORE_PUBLIC unsigned
+indexstore_format_version(void);
+
+typedef void *indexstore_t;
+
+INDEXSTORE_PUBLIC indexstore_t
+indexstore_store_create(const char *store_path, indexstore_error_t *error);
+
+INDEXSTORE_PUBLIC void
+indexstore_store_dispose(indexstore_t);
+
+#if INDEXSTORE_HAS_BLOCKS
+INDEXSTORE_PUBLIC bool
+indexstore_store_units_apply(indexstore_t, unsigned sorted,
+                             bool(^applier)(indexstore_string_ref_t unit_name));
+#endif
+
+typedef void *indexstore_unit_event_notification_t;
+typedef void *indexstore_unit_event_t;
+
+INDEXSTORE_PUBLIC size_t
+indexstore_unit_event_notification_get_events_count(indexstore_unit_event_notification_t);
+
+INDEXSTORE_PUBLIC indexstore_unit_event_t
+indexstore_unit_event_notification_get_event(indexstore_unit_event_notification_t, size_t index);
+
+INDEXSTORE_PUBLIC bool
+indexstore_unit_event_notification_is_initial(indexstore_unit_event_notification_t);
+
+typedef enum {
+  INDEXSTORE_UNIT_EVENT_ADDED = 1,
+  INDEXSTORE_UNIT_EVENT_REMOVED = 2,
+  INDEXSTORE_UNIT_EVENT_MODIFIED = 3,
+  INDEXSTORE_UNIT_EVENT_DIRECTORY_DELETED = 4,
+} indexstore_unit_event_kind_t;
+
+INDEXSTORE_PUBLIC indexstore_unit_event_kind_t
+indexstore_unit_event_get_kind(indexstore_unit_event_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_unit_event_get_unit_name(indexstore_unit_event_t);
+
+INDEXSTORE_PUBLIC timespec
+indexstore_unit_event_get_modification_time(indexstore_unit_event_t);
+
+#if INDEXSTORE_HAS_BLOCKS
+typedef void (^indexstore_unit_event_handler_t)(indexstore_unit_event_notification_t);
+
+INDEXSTORE_PUBLIC void
+indexstore_store_set_unit_event_handler(indexstore_t,
+                                        indexstore_unit_event_handler_t handler);
+#endif
+
+typedef struct {
+  /// If true, \c indexstore_store_start_unit_event_listening will block until
+  /// the initial set of units is passed to the unit event handler, otherwise
+  /// the function will return and the initial set will be passed asynchronously.
+  bool wait_initial_sync;
+} indexstore_unit_event_listen_options_t;
+
+INDEXSTORE_PUBLIC bool
+indexstore_store_start_unit_event_listening(indexstore_t,
+                                            indexstore_unit_event_listen_options_t *,
+                                            size_t listen_options_struct_size,
+                                            indexstore_error_t *error);
+
+INDEXSTORE_PUBLIC void
+indexstore_store_stop_unit_event_listening(indexstore_t);
+
+INDEXSTORE_PUBLIC void
+indexstore_store_discard_unit(indexstore_t, const char *unit_name);
+
+INDEXSTORE_PUBLIC void
+indexstore_store_discard_record(indexstore_t, const char *record_name);
+
+INDEXSTORE_PUBLIC void
+indexstore_store_purge_stale_data(indexstore_t);
+
+/// Determines the unit name from the \c output_path and writes it out in the
+/// \c name_buf buffer. It doesn't write more than \c buf_size.
+/// \returns the length of the name. If this is larger than \c buf_size, the
+/// caller should call the function again with a buffer of the appropriate size.
+INDEXSTORE_PUBLIC size_t
+indexstore_store_get_unit_name_from_output_path(indexstore_t store,
+                                                const char *output_path,
+                                                char *name_buf,
+                                                size_t buf_size);
+
+/// \returns true if an error occurred, false otherwise.
+INDEXSTORE_PUBLIC bool
+indexstore_store_get_unit_modification_time(indexstore_t store,
+                                            const char *unit_name,
+                                            int64_t *seconds,
+                                            int64_t *nanoseconds,
+                                            indexstore_error_t *error);
+
+typedef void *indexstore_symbol_t;
+
+typedef enum {
+  INDEXSTORE_SYMBOL_KIND_UNKNOWN = 0,
+  INDEXSTORE_SYMBOL_KIND_MODULE = 1,
+  INDEXSTORE_SYMBOL_KIND_NAMESPACE = 2,
+  INDEXSTORE_SYMBOL_KIND_NAMESPACEALIAS = 3,
+  INDEXSTORE_SYMBOL_KIND_MACRO = 4,
+  INDEXSTORE_SYMBOL_KIND_ENUM = 5,
+  INDEXSTORE_SYMBOL_KIND_STRUCT = 6,
+  INDEXSTORE_SYMBOL_KIND_CLASS = 7,
+  INDEXSTORE_SYMBOL_KIND_PROTOCOL = 8,
+  INDEXSTORE_SYMBOL_KIND_EXTENSION = 9,
+  INDEXSTORE_SYMBOL_KIND_UNION = 10,
+  INDEXSTORE_SYMBOL_KIND_TYPEALIAS = 11,
+  INDEXSTORE_SYMBOL_KIND_FUNCTION = 12,
+  INDEXSTORE_SYMBOL_KIND_VARIABLE = 13,
+  INDEXSTORE_SYMBOL_KIND_FIELD = 14,
+  INDEXSTORE_SYMBOL_KIND_ENUMCONSTANT = 15,
+  INDEXSTORE_SYMBOL_KIND_INSTANCEMETHOD = 16,
+  INDEXSTORE_SYMBOL_KIND_CLASSMETHOD = 17,
+  INDEXSTORE_SYMBOL_KIND_STATICMETHOD = 18,
+  INDEXSTORE_SYMBOL_KIND_INSTANCEPROPERTY = 19,
+  INDEXSTORE_SYMBOL_KIND_CLASSPROPERTY = 20,
+  INDEXSTORE_SYMBOL_KIND_STATICPROPERTY = 21,
+  INDEXSTORE_SYMBOL_KIND_CONSTRUCTOR = 22,
+  INDEXSTORE_SYMBOL_KIND_DESTRUCTOR = 23,
+  INDEXSTORE_SYMBOL_KIND_CONVERSIONFUNCTION = 24,
+  INDEXSTORE_SYMBOL_KIND_PARAMETER = 25,
+  INDEXSTORE_SYMBOL_KIND_USING = 26,
+
+  INDEXSTORE_SYMBOL_KIND_COMMENTTAG = 1000,
+} indexstore_symbol_kind_t;
+
+typedef enum {
+  INDEXSTORE_SYMBOL_SUBKIND_NONE = 0,
+  INDEXSTORE_SYMBOL_SUBKIND_CXXCOPYCONSTRUCTOR = 1,
+  INDEXSTORE_SYMBOL_SUBKIND_CXXMOVECONSTRUCTOR = 2,
+  INDEXSTORE_SYMBOL_SUBKIND_ACCESSORGETTER = 3,
+  INDEXSTORE_SYMBOL_SUBKIND_ACCESSORSETTER = 4,
+  INDEXSTORE_SYMBOL_SUBKIND_USINGTYPENAME = 5,
+  INDEXSTORE_SYMBOL_SUBKIND_USINGVALUE = 6,
+} indexstore_symbol_subkind_t;
+
+typedef enum {
+  INDEXSTORE_SYMBOL_PROPERTY_GENERIC                          = 1 << 0,
+  INDEXSTORE_SYMBOL_PROPERTY_TEMPLATE_PARTIAL_SPECIALIZATION  = 1 << 1,
+  INDEXSTORE_SYMBOL_PROPERTY_TEMPLATE_SPECIALIZATION          = 1 << 2,
+  INDEXSTORE_SYMBOL_PROPERTY_UNITTEST                         = 1 << 3,
+  INDEXSTORE_SYMBOL_PROPERTY_IBANNOTATED                      = 1 << 4,
+  INDEXSTORE_SYMBOL_PROPERTY_IBOUTLETCOLLECTION               = 1 << 5,
+  INDEXSTORE_SYMBOL_PROPERTY_GKINSPECTABLE                    = 1 << 6,
+  INDEXSTORE_SYMBOL_PROPERTY_LOCAL                            = 1 << 7,
+} indexstore_symbol_property_t;
+
+typedef enum {
+  INDEXSTORE_SYMBOL_LANG_C = 0,
+  INDEXSTORE_SYMBOL_LANG_OBJC = 1,
+  INDEXSTORE_SYMBOL_LANG_CXX = 2,
+
+  INDEXSTORE_SYMBOL_LANG_SWIFT = 100,
+} indexstore_symbol_language_t;
+
+typedef enum {
+  INDEXSTORE_SYMBOL_ROLE_DECLARATION = 1 << 0,
+  INDEXSTORE_SYMBOL_ROLE_DEFINITION  = 1 << 1,
+  INDEXSTORE_SYMBOL_ROLE_REFERENCE   = 1 << 2,
+  INDEXSTORE_SYMBOL_ROLE_READ        = 1 << 3,
+  INDEXSTORE_SYMBOL_ROLE_WRITE       = 1 << 4,
+  INDEXSTORE_SYMBOL_ROLE_CALL        = 1 << 5,
+  INDEXSTORE_SYMBOL_ROLE_DYNAMIC     = 1 << 6,
+  INDEXSTORE_SYMBOL_ROLE_ADDRESSOF   = 1 << 7,
+  INDEXSTORE_SYMBOL_ROLE_IMPLICIT    = 1 << 8,
+
+  // Relation roles.
+  INDEXSTORE_SYMBOL_ROLE_REL_CHILDOF     = 1 << 9,
+  INDEXSTORE_SYMBOL_ROLE_REL_BASEOF      = 1 << 10,
+  INDEXSTORE_SYMBOL_ROLE_REL_OVERRIDEOF  = 1 << 11,
+  INDEXSTORE_SYMBOL_ROLE_REL_RECEIVEDBY  = 1 << 12,
+  INDEXSTORE_SYMBOL_ROLE_REL_CALLEDBY    = 1 << 13,
+  INDEXSTORE_SYMBOL_ROLE_REL_EXTENDEDBY  = 1 << 14,
+  INDEXSTORE_SYMBOL_ROLE_REL_ACCESSOROF  = 1 << 15,
+  INDEXSTORE_SYMBOL_ROLE_REL_CONTAINEDBY = 1 << 16,
+  INDEXSTORE_SYMBOL_ROLE_REL_IBTYPEOF    = 1 << 17,
+  INDEXSTORE_SYMBOL_ROLE_REL_SPECIALIZATIONOF = 1 << 18,
+} indexstore_symbol_role_t;
+
+INDEXSTORE_PUBLIC indexstore_symbol_language_t
+indexstore_symbol_get_language(indexstore_symbol_t);
+
+INDEXSTORE_PUBLIC indexstore_symbol_kind_t
+indexstore_symbol_get_kind(indexstore_symbol_t);
+
+INDEXSTORE_PUBLIC indexstore_symbol_subkind_t
+indexstore_symbol_get_subkind(indexstore_symbol_t);
+
+INDEXSTORE_PUBLIC uint64_t
+indexstore_symbol_get_properties(indexstore_symbol_t);
+
+INDEXSTORE_PUBLIC uint64_t
+indexstore_symbol_get_roles(indexstore_symbol_t);
+
+INDEXSTORE_PUBLIC uint64_t
+indexstore_symbol_get_related_roles(indexstore_symbol_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_symbol_get_name(indexstore_symbol_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_symbol_get_usr(indexstore_symbol_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_symbol_get_codegen_name(indexstore_symbol_t);
+
+typedef void *indexstore_symbol_relation_t;
+
+INDEXSTORE_PUBLIC uint64_t
+indexstore_symbol_relation_get_roles(indexstore_symbol_relation_t);
+
+INDEXSTORE_PUBLIC indexstore_symbol_t
+indexstore_symbol_relation_get_symbol(indexstore_symbol_relation_t);
+
+typedef void *indexstore_occurrence_t;
+
+INDEXSTORE_PUBLIC indexstore_symbol_t
+indexstore_occurrence_get_symbol(indexstore_occurrence_t);
+
+#if INDEXSTORE_HAS_BLOCKS
+INDEXSTORE_PUBLIC bool
+indexstore_occurrence_relations_apply(indexstore_occurrence_t,
+                      bool(^applier)(indexstore_symbol_relation_t symbol_rel));
+#endif
+
+INDEXSTORE_PUBLIC uint64_t
+indexstore_occurrence_get_roles(indexstore_occurrence_t);
+
+INDEXSTORE_PUBLIC void
+indexstore_occurrence_get_line_col(indexstore_occurrence_t,
+                              unsigned *line, unsigned *column);
+
+typedef void *indexstore_record_reader_t;
+
+INDEXSTORE_PUBLIC indexstore_record_reader_t
+indexstore_record_reader_create(indexstore_t store, const char *record_name,
+                                indexstore_error_t *error);
+
+INDEXSTORE_PUBLIC void
+indexstore_record_reader_dispose(indexstore_record_reader_t);
+
+#if INDEXSTORE_HAS_BLOCKS
+/// Goes through the symbol data and passes symbols to \c receiver, for the
+/// symbol data that \c filter returns true on.
+///
+/// This allows allocating memory only for the record symbols that the caller is
+/// interested in.
+INDEXSTORE_PUBLIC bool
+indexstore_record_reader_search_symbols(indexstore_record_reader_t,
+    bool(^filter)(indexstore_symbol_t symbol, bool *stop),
+    void(^receiver)(indexstore_symbol_t symbol));
+
+/// \param nocache if true, avoids allocating memory for the symbols.
+/// Useful when the caller does not intend to keep \c indexstore_record_reader_t
+/// for more queries.
+INDEXSTORE_PUBLIC bool
+indexstore_record_reader_symbols_apply(indexstore_record_reader_t,
+                                       bool nocache,
+                                    bool(^applier)(indexstore_symbol_t symbol));
+
+INDEXSTORE_PUBLIC bool
+indexstore_record_reader_occurrences_apply(indexstore_record_reader_t,
+                                 bool(^applier)(indexstore_occurrence_t occur));
+
+INDEXSTORE_PUBLIC bool
+indexstore_record_reader_occurrences_in_line_range_apply(indexstore_record_reader_t,
+                                                         unsigned line_start,
+                                                         unsigned line_count,
+                                 bool(^applier)(indexstore_occurrence_t occur));
+
+/// \param symbols if non-zero \c symbols_count, indicates the list of symbols
+/// that we want to get occurrences for. An empty array indicates that we want
+/// occurrences for all symbols.
+/// \param related_symbols Same as \c symbols but for related symbols.
+INDEXSTORE_PUBLIC bool
+indexstore_record_reader_occurrences_of_symbols_apply(indexstore_record_reader_t,
+        indexstore_symbol_t *symbols, size_t symbols_count,
+        indexstore_symbol_t *related_symbols, size_t related_symbols_count,
+        bool(^applier)(indexstore_occurrence_t occur));
+#endif
+
+
+typedef void *indexstore_unit_reader_t;
+
+INDEXSTORE_PUBLIC indexstore_unit_reader_t
+indexstore_unit_reader_create(indexstore_t store, const char *unit_name,
+                              indexstore_error_t *error);
+
+INDEXSTORE_PUBLIC void
+indexstore_unit_reader_dispose(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_unit_reader_get_provider_identifier(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_unit_reader_get_provider_version(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC void
+indexstore_unit_reader_get_modification_time(indexstore_unit_reader_t,
+                                             int64_t *seconds,
+                                             int64_t *nanoseconds);
+
+INDEXSTORE_PUBLIC bool
+indexstore_unit_reader_is_system_unit(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC bool
+indexstore_unit_reader_is_module_unit(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC bool
+indexstore_unit_reader_is_debug_compilation(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC bool
+indexstore_unit_reader_has_main_file(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_unit_reader_get_main_file(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_unit_reader_get_module_name(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_unit_reader_get_working_dir(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_unit_reader_get_output_file(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_unit_reader_get_sysroot_path(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_unit_reader_get_target(indexstore_unit_reader_t);
+
+typedef void *indexstore_unit_dependency_t;
+typedef void *indexstore_unit_include_t;
+
+typedef enum {
+  INDEXSTORE_UNIT_DEPENDENCY_UNIT = 1,
+  INDEXSTORE_UNIT_DEPENDENCY_RECORD = 2,
+  INDEXSTORE_UNIT_DEPENDENCY_FILE = 3,
+} indexstore_unit_dependency_kind_t;
+
+INDEXSTORE_PUBLIC indexstore_unit_dependency_kind_t
+indexstore_unit_dependency_get_kind(indexstore_unit_dependency_t);
+
+INDEXSTORE_PUBLIC bool
+indexstore_unit_dependency_is_system(indexstore_unit_dependency_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_unit_dependency_get_filepath(indexstore_unit_dependency_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_unit_dependency_get_modulename(indexstore_unit_dependency_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_unit_dependency_get_name(indexstore_unit_dependency_t);
+
+INDEXSTORE_PUBLIC time_t
+indexstore_unit_dependency_get_modification_time(indexstore_unit_dependency_t);
+
+INDEXSTORE_PUBLIC size_t
+indexstore_unit_dependency_get_file_size(indexstore_unit_dependency_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_unit_include_get_source_path(indexstore_unit_include_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_unit_include_get_target_path(indexstore_unit_include_t);
+
+INDEXSTORE_PUBLIC unsigned
+indexstore_unit_include_get_source_line(indexstore_unit_include_t);
+
+#if INDEXSTORE_HAS_BLOCKS
+INDEXSTORE_PUBLIC bool
+indexstore_unit_reader_dependencies_apply(indexstore_unit_reader_t,
+                             bool(^applier)(indexstore_unit_dependency_t));
+
+INDEXSTORE_PUBLIC bool
+indexstore_unit_reader_includes_apply(indexstore_unit_reader_t,
+                             bool(^applier)(indexstore_unit_include_t));
+
+#endif
+
+INDEXSTORE_END_DECLS
+
+#endif

lib/CMakeLists.txt

if(CLANG_ENABLE_ARCMT)
   add_subdirectory(ARCMigrate)
 endif()
 add_subdirectory(Driver)
 add_subdirectory(Serialization)
 add_subdirectory(Frontend)
 add_subdirectory(FrontendTool)
 add_subdirectory(Tooling)
 add_subdirectory(Index)
+add_subdirectory(DirectoryWatcher)
 if(CLANG_ENABLE_STATIC_ANALYZER)
   add_subdirectory(StaticAnalyzer)
 endif()
 add_subdirectory(Format)

lib/DirectoryWatcher/CMakeLists.txt

+set(LLVM_LINK_COMPONENTS support)
+
+add_clang_library(clangDirectoryWatcher
+  DirectoryWatcher.cpp
+
+  LINK_LIBS
+  clangBasic
+  )

lib/DirectoryWatcher/DirectoryWatcher.cpp

+//===- DirectoryWatcher.cpp - Listens for directory file changes ----------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+/// \file
+/// \brief Utility class for listening for file system changes in a directory.
+//===----------------------------------------------------------------------===//
+
+#include "clang/DirectoryWatcher/DirectoryWatcher.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/StringMap.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/raw_ostream.h"
+
+#define HAVE_CORESERVICES 0
+
+#if defined(__has_include)
+#if __has_include(<CoreServices/CoreServices.h>)
+
+#include <CoreServices/CoreServices.h>
+#undef HAVE_CORESERVICES
+#define HAVE_CORESERVICES 1
+
+#endif
+#endif
+
+using namespace clang;
+using namespace llvm;
+
+static timespec toTimeSpec(sys::TimePoint<> tp) {
+  std::chrono::seconds sec = std::chrono::time_point_cast<std::chrono::seconds>(
+                 tp).time_since_epoch();
+  std::chrono::nanoseconds nsec =
+    std::chrono::time_point_cast<std::chrono::nanoseconds>(tp - sec)
+      .time_since_epoch();
+  timespec ts;
+  ts.tv_sec = sec.count();
+  ts.tv_nsec = nsec.count();
+  return ts;
+}
+
+static Optional<timespec> getModTime(StringRef path) {
+  sys::fs::file_status Status;
+  std::error_code EC = status(path, Status);
+  if (EC)
+    return None;
+  return toTimeSpec(Status.getLastModificationTime());
+}
+
+struct DirectoryWatcher::Implementation {
+#if HAVE_CORESERVICES
+  FSEventStreamRef EventStream = nullptr;
+
+  bool setupFSEventStream(StringRef path, EventReceiver receiver,
+                          dispatch_queue_t queue);
+  void stopFSEventStream();
+
+  ~Implementation() {
+    stopFSEventStream();
+  };
+#endif
+};
+
+#if HAVE_CORESERVICES
+namespace {
+struct EventStreamContextData {
+  std::string WatchedPath;
+  DirectoryWatcher::EventReceiver Receiver;
+
+  EventStreamContextData(std::string watchedPath, DirectoryWatcher::EventReceiver receiver)
+  : WatchedPath(std::move(watchedPath)), Receiver(std::move(receiver)) {
+  }
+
+  static void dispose(const void *ctx) {
+    delete static_cast<const EventStreamContextData*>(ctx);
+  }
+};
+}
+
+static void eventStreamCallback(
+                       ConstFSEventStreamRef stream,
+                       void *clientCallBackInfo,
+                       size_t numEvents,
+                       void *eventPaths,
+                       const FSEventStreamEventFlags eventFlags[],
+                       const FSEventStreamEventId eventIds[]) {
+  auto *ctx = static_cast<EventStreamContextData*>(clientCallBackInfo);
+
+  std::vector<DirectoryWatcher::Event> Events;
+  for (size_t i = 0; i < numEvents; ++i) {
+    StringRef path = ((const char **)eventPaths)[i];
+    const FSEventStreamEventFlags flags = eventFlags[i];
+    if (!(flags & kFSEventStreamEventFlagItemIsFile)) {
+      if ((flags & kFSEventStreamEventFlagItemRemoved) && path == ctx->WatchedPath) {
+        DirectoryWatcher::Event Evt{DirectoryWatcher::EventKind::DirectoryDeleted, path, timespec{}};
+        Events.push_back(Evt);
+        break;
+      }
+      continue;
+    }
+    DirectoryWatcher::EventKind K = DirectoryWatcher::EventKind::Modified;
+    if ((flags & kFSEventStreamEventFlagItemCreated) ||
+        (flags & kFSEventStreamEventFlagItemRenamed))
+      K = DirectoryWatcher::EventKind::Added;
+    if (flags & kFSEventStreamEventFlagItemRemoved)
+      K = DirectoryWatcher::EventKind::Removed;
+    timespec modTime{};
+    if (K != DirectoryWatcher::EventKind::Removed) {
+      auto modTimeOpt = getModTime(path);
+      if (!modTimeOpt.hasValue())
+        continue;
+      modTime = modTimeOpt.getValue();
+    }
+    DirectoryWatcher::Event Evt{K, path, modTime};
+    Events.push_back(Evt);
+  }
+
+  ctx->Receiver(Events, /*isInitial=*/false);
+}
+
+bool DirectoryWatcher::Implementation::setupFSEventStream(StringRef path,
+                                                          EventReceiver receiver,
+                                                          dispatch_queue_t queue) {
+  if (path.empty())
+    return true;
+
+  CFMutableArrayRef pathsToWatch = CFArrayCreateMutable(nullptr, 0, &kCFTypeArrayCallBacks);
+  CFStringRef cfPathStr = CFStringCreateWithBytes(nullptr, (const UInt8 *)path.data(), path.size(), kCFStringEncodingUTF8, false);
+  CFArrayAppendValue(pathsToWatch, cfPathStr);
+  CFRelease(cfPathStr);
+  CFAbsoluteTime latency = 0.2; // Latency in seconds.
+
+  std::string realPath;
+  {
+    SmallString<128> Storage;
+    StringRef P = llvm::Twine(path).toNullTerminatedStringRef(Storage);
+    char Buffer[PATH_MAX];
+    // Use ::realpath to get the real path name
+    if (::realpath(P.begin(), Buffer) != nullptr)
+      realPath = Buffer;
+    else
+      realPath = path;
+  }
+
+  EventStreamContextData *ctxData = new EventStreamContextData(std::move(realPath), std::move(receiver));
+  FSEventStreamContext context;
+  context.version = 0;
+  context.info = ctxData;
+  context.retain = nullptr;
+  context.release = EventStreamContextData::dispose;
+  context.copyDescription = nullptr;
+
+  EventStream = FSEventStreamCreate(nullptr,
+                                    eventStreamCallback,
+                                    &context,
+                                    pathsToWatch,
+                                    kFSEventStreamEventIdSinceNow,
+                                    latency,
+                                    kFSEventStreamCreateFlagFileEvents |
+                                    kFSEventStreamCreateFlagNoDefer);
+  CFRelease(pathsToWatch);
+  if (!EventStream) {
+    return true;
+  }
+  FSEventStreamSetDispatchQueue(EventStream, queue);
+  FSEventStreamStart(EventStream);
+  return false;
+}
+
+void DirectoryWatcher::Implementation::stopFSEventStream() {
+  if (!EventStream)
+    return;
+  FSEventStreamStop(EventStream);
+  FSEventStreamInvalidate(EventStream);
+  FSEventStreamRelease(EventStream);
+  EventStream = nullptr;
+}
+#endif
+
+DirectoryWatcher::DirectoryWatcher()
+  : Impl(*new Implementation()) {}
+
+DirectoryWatcher::~DirectoryWatcher() {
+  delete &Impl;
+}
+
+#if HAVE_CORESERVICES
+static std::vector<DirectoryWatcher::Event> scanDirectory(StringRef Path) {
+  using namespace llvm::sys;
+
+  std::vector<DirectoryWatcher::Event> Events;
+  std::error_code EC;
+  for (auto It = fs::directory_iterator(Path, EC), End = fs::directory_iterator();
+         !EC && It != End; It.increment(EC)) {
+    auto modTime = getModTime(It->path());
+    if (!modTime.hasValue())
+      continue;
+    DirectoryWatcher::Event Event{DirectoryWatcher::EventKind::Added, It->path(), modTime.getValue()};
+    Events.push_back(std::move(Event));
+  }
+  return Events;
+}
+#endif
+
+std::unique_ptr<DirectoryWatcher> DirectoryWatcher::create(StringRef Path,
+        EventReceiver Receiver, bool waitInitialSync, std::string &Error) {
+#if HAVE_CORESERVICES
+
+  using namespace llvm::sys;
+
+  if (!fs::exists(Path)) {
+    std::error_code EC = fs::create_directories(Path);
+    if (EC) {
+      Error = EC.message();
+      return nullptr;
+    }
+  }
+
+  bool IsDir;
+  std::error_code EC = fs::is_directory(Path, IsDir);
+  if (EC) {
+    Error = EC.message();
+    return nullptr;
+  }
+  if (!IsDir) {
+    Error = "path is not a directory: ";
+    Error += Path;
+    return nullptr;
+  }
+
+  std::unique_ptr<DirectoryWatcher> DirWatch;
+  DirWatch.reset(new DirectoryWatcher());
+  auto &Impl = DirWatch->Impl;
+
+  dispatch_queue_t queue = dispatch_queue_create("DirectoryWatcher", DISPATCH_QUEUE_SERIAL);
+  dispatch_semaphore_t initScanSema = dispatch_semaphore_create(0);
+  dispatch_semaphore_t setupFSEventsSema = dispatch_semaphore_create(0);
+
+  std::string copiedPath = Path;
+  dispatch_retain(initScanSema);
+  dispatch_retain(setupFSEventsSema);
+  dispatch_async(queue, ^{
+    // Wait for the event stream to be setup before doing the initial scan,
+    // to make sure we won't miss any events.
+    dispatch_semaphore_wait(setupFSEventsSema, DISPATCH_TIME_FOREVER);
+    auto events = scanDirectory(copiedPath);
+    Receiver(events, /*isInitial=*/true);
+    dispatch_semaphore_signal(initScanSema);
+    dispatch_release(setupFSEventsSema);
+    dispatch_release(initScanSema);
+  });
+  bool fsErr = Impl.setupFSEventStream(Path, Receiver, queue);
+  dispatch_semaphore_signal(setupFSEventsSema);
+
+  if (waitInitialSync) {
+    dispatch_semaphore_wait(initScanSema, DISPATCH_TIME_FOREVER);
+  }
+  dispatch_release(setupFSEventsSema);
+  dispatch_release(initScanSema);
+  dispatch_release(queue);
+
+  if (fsErr) {
+    raw_string_ostream(Error) << "failed to setup FSEvents stream for path: " << Path;
+    return nullptr;
+  }
+
+  return DirWatch;
+#else
+  return nullptr;
+#endif
+}

lib/Driver/Driver.cpp

if (StringRef(TempFile).endswith(".cache")) {
       // In some cases (modules) we'll dump extra data to help with reproducing
       // the crash into a directory next to the output.
       VFS = llvm::sys::path::filename(TempFile);
       llvm::sys::path::append(VFS, "vfs", "vfs.yaml");
     }
   }
 
   // Assume associated files are based off of the first temporary file.
-  CrashReportInfo CrashInfo(TempFiles[0], VFS);
+  CrashReportInfo CrashInfo(
+      TempFiles[0], VFS,
+      C.getArgs().getLastArgValue(options::OPT_index_store_path));
 
   std::string Script = CrashInfo.Filename.rsplit('.').first.str() + ".sh";
   std::error_code EC;
   llvm::raw_fd_ostream ScriptOS(Script, EC, llvm::sys::fs::F_Excl);
   if (EC) {
     Diag(clang::diag::note_drv_command_failed_diag_msg)
         << "Error generating run script: " + Script + " " + EC.message();
   } else {

lib/Driver/Job.cpp

IsInclude = llvm::StringSwitch<bool>(Flag)
     .Cases("-idirafter", "-internal-isystem", "-iwithprefix", true)
     .Cases("-internal-externc-isystem", "-iprefix", true)
     .Cases("-iwithprefixbefore", "-isystem", "-iquote", true)
     .Cases("-isysroot", "-I", "-F", "-resource-dir", true)
     .Cases("-iframework", "-include-pch", true)
     .Default(false);
   if (IsInclude)
     return HaveCrashVFS ? false : true;
+  if (StringRef(Flag).startswith("-index-store-path"))
+    return true;
 
   // The remaining flags are treated as a single argument.
 
   // These flags are all of the form -Flag and have no second argument.
   ShouldSkip = llvm::StringSwitch<bool>(Flag)
     .Cases("-M", "-MM", "-MG", "-MP", "-MD", true)
     .Case("-MMD", true)
     .Default(false);
rewriteIncludes(const llvm::ArrayRef<const char *> &Args, size_t Idx,
   assert(NumArgs == 2 && "Not expecting more than two arguments");
   StringRef Inc(Args[Idx + NumArgs - 1]);
   if (!getAbsPath(Inc, NewInc))
     return;
   IncFlags.push_back(SmallString<128>(Args[Idx]));
   IncFlags.push_back(std::move(NewInc));
 }
 
 void Command::Print(raw_ostream &OS, const char *Terminator, bool Quote,
                     CrashReportInfo *CrashInfo) const {
   // Always quote the exe.
   OS << ' ';
   printArg(OS, Executable, /*Quote=*/true);
 
   llvm::ArrayRef<const char *> Args = Arguments;
   llvm::SmallVector<const char *, 128> ArgsRespFile;
   if (ResponseFile != nullptr) {
     buildArgvForResponseFile(ArgsRespFile);
     Args = ArrayRef<const char *>(ArgsRespFile).slice(1); // no executable name
   }
 
   bool HaveCrashVFS = CrashInfo && !CrashInfo->VFSPath.empty();
+  bool HaveIndexStorePath = CrashInfo && !CrashInfo->IndexStorePath.empty();
   for (size_t i = 0, e = Args.size(); i < e; ++i) {
     const char *const Arg = Args[i];
 
     if (CrashInfo) {
       int NumArgs = 0;
       bool IsInclude = false;
       if (skipArgs(Arg, HaveCrashVFS, NumArgs, IsInclude)) {
         i += NumArgs - 1;
if (CrashInfo && HaveCrashVFS) {
 
     std::string ModCachePath = "-fmodules-cache-path=";
     ModCachePath.append(RelModCacheDir.c_str());
 
     OS << ' ';
     printArg(OS, ModCachePath, Quote);
   }
 
+  if (CrashInfo && HaveIndexStorePath) {
+    SmallString<128> IndexStoreDir;
+
+    if (HaveCrashVFS) {
+      IndexStoreDir = llvm::sys::path::parent_path(
+          llvm::sys::path::parent_path(CrashInfo->VFSPath));
+      llvm::sys::path::append(IndexStoreDir, "index-store");
+    } else {
+      IndexStoreDir = "index-store";
+    }
+
+    OS << ' ';
+    printArg(OS, "-index-store-path", Quote);
+    OS << ' ';
+    printArg(OS, IndexStoreDir.c_str(), Quote);
+  }
+
+
   if (ResponseFile != nullptr) {
     OS << "\n Arguments passed via response file:\n";
     writeResponseFile(OS);
     // Avoiding duplicated newline terminator, since FileLists are
     // newline-separated.
     if (Creator.getResponseFilesSupport() != Tool::RF_FileList)
       OS << "\n";
     OS << " (end of response file)";

lib/Driver/ToolChains/Clang.cpp

#endif
   // Pass the path to compiler resource files.
   CmdArgs.push_back("-resource-dir");
   CmdArgs.push_back(D.ResourceDir.c_str());
 
   Args.AddLastArg(CmdArgs, options::OPT_working_directory);
 
   RenderARCMigrateToolOptions(D, Args, CmdArgs);
 
+  if (Args.hasArg(options::OPT_index_store_path)) {
+    Args.AddLastArg(CmdArgs, options::OPT_index_store_path);
+    Args.AddLastArg(CmdArgs, options::OPT_index_ignore_system_symbols);
+    Args.AddLastArg(CmdArgs, options::OPT_index_record_codegen_name);
+
+    // If '-o' is passed along with '-fsyntax-only' pass it along the cc1
+    // invocation so that the index action knows what the out file is.
+    if (isa<CompileJobAction>(JA) && JA.getType() == types::TY_Nothing) {
+      Args.AddLastArg(CmdArgs, options::OPT_o);
+    }
+  }
+
+  if (const char *IdxStorePath = ::getenv("CLANG_PROJECT_INDEX_PATH")) {
+    CmdArgs.push_back("-index-store-path");
+    CmdArgs.push_back(IdxStorePath);
+    CmdArgs.push_back("-index-ignore-system-symbols");
+    CmdArgs.push_back("-index-record-codegen-name");
+  }
+
+
   // Add preprocessing options like -I, -D, etc. if we are using the
   // preprocessor.
   //
   // FIXME: Support -fpreprocessed
   if (types::getPreprocessedType(InputType) != types::TY_INVALID)
     AddPreprocessingOptions(C, JA, D, Args, CmdArgs, Output, Inputs);
 
   // Don't warn about "clang -c -DPIC -fPIC test.i" because libtool.m4 assumes

lib/Driver/ToolChains/Darwin.cpp

void darwin::Linker::ConstructJob(Compilation &C, const JobAction &JA,
   // -filelist linker option.
   llvm::opt::ArgStringList InputFileList;
 
   // The logic here is derived from gcc's behavior; most of which
   // comes from specs (starting with link_command). Consult gcc for
   // more information.
   ArgStringList CmdArgs;
 
+  Args.ClaimAllArgs(options::OPT_index_store_path);
+  Args.ClaimAllArgs(options::OPT_index_ignore_system_symbols);
+  Args.ClaimAllArgs(options::OPT_index_record_codegen_name);
+
   /// Hack(tm) to ignore linking errors when we are doing ARC migration.
   if (Args.hasArg(options::OPT_ccc_arcmt_check,
                   options::OPT_ccc_arcmt_migrate)) {
     for (const auto &Arg : Args)
       Arg->claim();
     const char *Exec =
         Args.MakeArgString(getToolChain().GetProgramPath("touch"));
     CmdArgs.push_back(Output.getFilename());

lib/Frontend/CompilerInstance.cpp

 #include "clang/Frontend/FrontendAction.h"
 #include "clang/Frontend/FrontendActions.h"
 #include "clang/Frontend/FrontendDiagnostic.h"
 #include "clang/Frontend/LogDiagnosticPrinter.h"
 #include "clang/Frontend/SerializedDiagnosticPrinter.h"
 #include "clang/Frontend/TextDiagnosticPrinter.h"
 #include "clang/Frontend/Utils.h"
 #include "clang/Frontend/VerifyDiagnosticConsumer.h"
+#include "clang/Index/IndexingAction.h"
 #include "clang/Lex/HeaderSearch.h"
 #include "clang/Lex/PTHManager.h"
 #include "clang/Lex/Preprocessor.h"
 #include "clang/Lex/PreprocessorOptions.h"
 #include "clang/Sema/CodeCompleteConsumer.h"
 #include "clang/Sema/Sema.h"
 #include "clang/Serialization/ASTReader.h"
 #include "clang/Serialization/GlobalModuleIndex.h"
compileModuleImpl(CompilerInstance &ImportingInstance, SourceLocation ImportLoc,
   Instance.setFileManager(&ImportingInstance.getFileManager());
   Instance.createSourceManager(Instance.getFileManager());
   SourceManager &SourceMgr = Instance.getSourceManager();
   SourceMgr.setModuleBuildStack(
     ImportingInstance.getSourceManager().getModuleBuildStack());
   SourceMgr.pushModuleBuildStack(ModuleName,
     FullSourceLoc(ImportLoc, ImportingInstance.getSourceManager()));
 
+  // Pass along the GenModuleActionWrapper callback
+  auto wrapGenModuleAction = ImportingInstance.getGenModuleActionWrapper();
+  Instance.setGenModuleActionWrapper(wrapGenModuleAction);
+
   // If we're collecting module dependencies, we need to share a collector
   // between all of the module CompilerInstances. Other than that, we don't
   // want to produce any dependency output from the module build.
   Instance.setModuleDepCollector(ImportingInstance.getModuleDepCollector());
   Inv.getDependencyOutputOpts() = DependencyOutputOptions();
 
   ImportingInstance.getDiagnostics().Report(ImportLoc,
                                             diag::remark_module_build)
     << ModuleName << ModuleFileName;
 
   PreBuildStep(Instance);
 
   // Execute the action to actually build the module in-place. Use a separate
   // thread so that we get a stack large enough.
   const unsigned ThreadStackSize = 8 << 20;
   llvm::CrashRecoveryContext CRC;
   CRC.RunSafelyOnThread(
       [&]() {
-        GenerateModuleFromModuleMapAction Action;
-        Instance.ExecuteAction(Action);
+        std::unique_ptr<FrontendAction> Action(
+            new GenerateModuleFromModuleMapAction);
+        if (wrapGenModuleAction) {
+          Action = wrapGenModuleAction(FrontendOpts, std::move(Action));
+        }
+        Instance.ExecuteAction(*Action);
       },
       ThreadStackSize);
 
   PostBuildStep(Instance);
 
   ImportingInstance.getDiagnostics().Report(ImportLoc,
                                             diag::remark_module_build_done)
     << ModuleName;

lib/Frontend/CompilerInvocation.cpp

static InputKind ParseFrontendArgs(FrontendOptions &Opts, ArgList &Args,
   Opts.ObjCMTWhiteListPath = Args.getLastArgValue(OPT_objcmt_whitelist_dir_path);
 
   if (Opts.ARCMTAction != FrontendOptions::ARCMT_None &&
       Opts.ObjCMTAction != FrontendOptions::ObjCMT_None) {
     Diags.Report(diag::err_drv_argument_not_allowed_with)
       << "ARC migration" << "ObjC migration";
   }
 
+  Opts.IndexStorePath = Args.getLastArgValue(OPT_index_store_path);
+  Opts.IndexIgnoreSystemSymbols = Args.hasArg(OPT_index_ignore_system_symbols);
+  Opts.IndexRecordCodegenName = Args.hasArg(OPT_index_record_codegen_name);
+
   InputKind DashX(InputKind::Unknown);
   if (const Arg *A = Args.getLastArg(OPT_x)) {
     StringRef XValue = A->getValue();
 
     // Parse suffixes: '<lang>(-header|[-module-map][-cpp-output])'.
     // FIXME: Supporting '<lang>-header-cpp-output' would be useful.
     bool Preprocessed = XValue.consume_back("-cpp-output");
     bool ModuleMap = XValue.consume_back("-module-map");

lib/FrontendTool/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   Option
   Support
   )
 
 set(link_libs
   clangBasic
   clangCodeGen
   clangDriver
   clangFrontend
+  clangIndex
   clangRewriteFrontend
   )
 
 if(CLANG_ENABLE_ARCMT)
   list(APPEND link_libs
     clangARCMigrate
     )
 endif()

lib/FrontendTool/ExecuteCompilerInvocation.cpp

 #include "clang/Config/config.h"
 #include "clang/Driver/Options.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/CompilerInvocation.h"
 #include "clang/Frontend/FrontendActions.h"
 #include "clang/Frontend/FrontendDiagnostic.h"
 #include "clang/Frontend/FrontendPluginRegistry.h"
 #include "clang/Frontend/Utils.h"
+#include "clang/Index/IndexingAction.h"
 #include "clang/Rewrite/Frontend/FrontendActions.h"
 #include "clang/StaticAnalyzer/Frontend/FrontendActions.h"
 #include "llvm/Option/OptTable.h"
 #include "llvm/Option/Option.h"
 #include "llvm/Support/DynamicLibrary.h"
 #include "llvm/Support/ErrorHandling.h"
 using namespace clang;
 using namespace llvm::opt;
if (CI.getFrontendOpts().ProgramAction != frontend::MigrateSource &&
     if (FEOpts.ObjCMTAction != FrontendOptions::ObjCMT_None) {
       Act = llvm::make_unique<arcmt::ObjCMigrateAction>(std::move(Act),
                                                         FEOpts.MTMigrateDir,
                                                         FEOpts.ObjCMTAction);
     }
   }
 #endif
 
+  if (!FEOpts.IndexStorePath.empty()) {
+    Act = index::createIndexDataRecordingAction(FEOpts, std::move(Act));
+    CI.setGenModuleActionWrapper(&index::createIndexDataRecordingAction);
+  }
+
   // If there are any AST files to merge, create a frontend action
   // adaptor to perform the merge.
   if (!FEOpts.ASTMergeFiles.empty())
     Act = llvm::make_unique<ASTMergeAction>(std::move(Act),
                                             FEOpts.ASTMergeFiles);
 
   return Act;
 }

lib/Index/BitstreamVisitor.h

+//===--- BitstreamVisitor.h - Helper for reading a bitstream --------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_LIB_INDEX_BITSTREAMVISITOR_H
+#define LLVM_CLANG_LIB_INDEX_BITSTREAMVISITOR_H
+
+#include "llvm/Bitcode/BitstreamReader.h"
+#include "clang/Basic/LLVM.h"
+#include "clang/Serialization/ASTReader.h"
+#include <string>
+
+namespace clang {
+namespace index {
+namespace store {
+
+/// Helper class that saves the current stream position and
+/// then restores it when destroyed.
+struct SavedStreamPosition {
+  explicit SavedStreamPosition(llvm::BitstreamCursor &Cursor)
+    : Cursor(Cursor), Offset(Cursor.GetCurrentBitNo()) { }
+
+  ~SavedStreamPosition() {
+    Cursor.JumpToBit(Offset);
+  }
+
+private:
+  llvm::BitstreamCursor &Cursor;
+  uint64_t Offset;
+};
+
+enum class StreamVisit {
+  Continue,
+  Skip,
+  Abort
+};
+
+template <typename ImplClass>
+class BitstreamVisitor {
+  SmallVector<unsigned, 4> BlockStack;
+
+protected:
+  llvm::BitstreamCursor &Stream;
+  Optional<llvm::BitstreamBlockInfo> BlockInfo;
+  std::string *Error;
+
+public:
+  BitstreamVisitor(llvm::BitstreamCursor &Stream)
+    : Stream(Stream) {}
+
+  StreamVisit visitBlock(unsigned ID) {
+    return StreamVisit::Continue;
+  }
+
+  bool visit(std::string &Error) {
+    this->Error = &Error;
+
+    ASTReader::RecordData Record;
+    while (1) {
+      llvm::BitstreamEntry Entry = Stream.advance(llvm::BitstreamCursor::AF_DontPopBlockAtEnd);
+
+      switch (Entry.Kind) {
+      case llvm::BitstreamEntry::Error:
+        Error = "malformed serialization";
+        return false;
+
+      case llvm::BitstreamEntry::EndBlock:
+        if (BlockStack.empty())
+          return true;
+        BlockStack.pop_back();
+        if (Stream.ReadBlockEnd()) {
+          Error = "malformed serialization";
+          return false;
+        }
+        if (Stream.AtEndOfStream())
+          return true;
+        break;
+
+      case llvm::BitstreamEntry::SubBlock: {
+        if (Entry.ID == llvm::bitc::BLOCKINFO_BLOCK_ID) {
+          BlockInfo = Stream.ReadBlockInfoBlock();
+          if (!BlockInfo) {
+            Error = "malformed BlockInfoBlock";
+            return false;
+          }
+          Stream.setBlockInfo(&*BlockInfo);
+          break;
+        }
+
+        StreamVisit Ret = static_cast<ImplClass*>(this)->visitBlock(Entry.ID);
+        switch (Ret) {
+        case StreamVisit::Continue:
+          if (Stream.EnterSubBlock(Entry.ID)) {
+            Error = "malformed block record";
+            return false;
+          }
+          readBlockAbbrevs(Stream);
+          BlockStack.push_back(Entry.ID);
+          break;
+
+        case StreamVisit::Skip: 
+          if (Stream.SkipBlock()) {
+            Error = "malformed serialization";
+            return false;
+          }
+          if (Stream.AtEndOfStream())
+            return true;
+          break;
+
+        case StreamVisit::Abort:
+          return false;
+        }
+        break;
+      }
+
+      case llvm::BitstreamEntry::Record: {
+        Record.clear();
+        StringRef Blob;
+        unsigned RecID = Stream.readRecord(Entry.ID, Record, &Blob);
+        unsigned BlockID = BlockStack.empty() ? 0 : BlockStack.back();
+        StreamVisit Ret = static_cast<ImplClass*>(this)->visitRecord(BlockID, RecID, Record, Blob);
+        switch (Ret) {
+        case StreamVisit::Continue:
+          break;
+
+        case StreamVisit::Skip: 
+          Stream.skipRecord(Entry.ID);
+          break;
+
+        case StreamVisit::Abort:
+          return false;
+        }
+        break;
+      }
+      }
+    }
+  }
+
+  static void readBlockAbbrevs(llvm::BitstreamCursor &Cursor) {
+    while (true) {
+      uint64_t Offset = Cursor.GetCurrentBitNo();
+      unsigned Code = Cursor.ReadCode();
+
+      // We expect all abbrevs to be at the start of the block.
+      if (Code != llvm::bitc::DEFINE_ABBREV) {
+        Cursor.JumpToBit(Offset);
+        return;
+      }
+      Cursor.ReadAbbrevRecord();
+    }
+  }
+};
+
+} // end namespace store
+} // end namespace index
+} // end namespace clang
+
+#endif

lib/Index/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
+  BitReader
   Core
   Support
   )
 
 add_clang_library(clangIndex
+  ClangIndexRecordWriter.cpp
   CodegenNameGenerator.cpp
   CommentToXML.cpp
+  FileIndexRecord.cpp
   IndexBody.cpp
+  IndexDataStore.cpp
+  IndexDataStoreUtils.cpp
   IndexDecl.cpp
   IndexingAction.cpp
   IndexingContext.cpp
+  IndexRecordHasher.cpp
+  IndexRecordReader.cpp
+  IndexRecordWriter.cpp
   IndexSymbol.cpp
   IndexTypeSourceInfo.cpp
+  IndexUnitReader.cpp
+  IndexUnitWriter.cpp
   USRGeneration.cpp
 
   ADDITIONAL_HEADERS
   IndexingContext.h
   SimpleFormatContext.h
 
   LINK_LIBS
   clangAST
   clangBasic
   clangFormat
   clangFrontend
+  clangLex
   clangRewrite
   clangSerialization
   clangToolingCore
   )

lib/Index/ClangIndexRecordWriter.h

+//===--- ClangIndexRecordWriter.h - Index record serialization ------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_LIB_INDEX_CLANGINDEXRECORDWRITER_H
+#define LLVM_CLANG_LIB_INDEX_CLANGINDEXRECORDWRITER_H
+
+#include "IndexRecordHasher.h"
+#include "clang/Index/IndexRecordWriter.h"
+#include "clang/Index/IndexingAction.h"
+#include "clang/Index/CodegenNameGenerator.h"
+#include "llvm/ADT/SmallString.h"
+
+namespace clang {
+  class ASTContext;
+  class Decl;
+
+namespace index {
+  class FileIndexRecord;
+
+class ClangIndexRecordWriter {
+  IndexRecordWriter Impl;
+
+  ASTContext &Ctx;
+  RecordingOptions RecordOpts;
+
+  std::unique_ptr<CodegenNameGenerator> CGNameGen;
+  llvm::BumpPtrAllocator Allocator;
+  llvm::DenseMap<const Decl *, StringRef> USRByDecl;
+  IndexRecordHasher Hasher;
+
+public:
+  ClangIndexRecordWriter(ASTContext &Ctx, RecordingOptions Opts);
+  ~ClangIndexRecordWriter();
+
+  ASTContext &getASTContext() { return Ctx; }
+  CodegenNameGenerator *getCGNameGen() { return CGNameGen.get(); }
+
+  bool writeRecord(StringRef Filename, const FileIndexRecord &Record,
+                   std::string &Error, std::string *RecordFile = nullptr);
+  StringRef getUSR(const Decl *D);
+
+private:
+  StringRef getUSRNonCached(const Decl *D);
+};
+
+} // end namespace index
+} // end namespace clang
+
+#endif

lib/Index/ClangIndexRecordWriter.cpp

+//===--- ClangIndexRecordWriter.cpp - Index record serialization ----------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "ClangIndexRecordWriter.h"
+#include "FileIndexRecord.h"
+#include "clang/Index/IndexSymbol.h"
+#include "clang/Index/IndexRecordReader.h"
+#include "clang/Index/USRGeneration.h"
+#include "clang/AST/ASTContext.h"
+#include "clang/AST/DeclCXX.h"
+#include "clang/AST/DeclObjC.h"
+
+using namespace clang;
+using namespace clang::index;
+
+StringRef ClangIndexRecordWriter::getUSR(const Decl *D) {
+  assert(D->isCanonicalDecl());
+  auto Insert = USRByDecl.insert(std::make_pair(D, StringRef()));
+  if (Insert.second) {
+    Insert.first->second = getUSRNonCached(D);
+  }
+  return Insert.first->second;
+}
+
+StringRef ClangIndexRecordWriter::getUSRNonCached(const Decl *D) {
+  SmallString<256> Buf;
+  bool Ignore = generateUSRForDecl(D, Buf);
+  if (Ignore)
+    return StringRef();
+  StringRef USR = Buf.str();
+  char *Ptr = Allocator.Allocate<char>(USR.size());
+  std::copy(USR.begin(), USR.end(), Ptr);
+  return StringRef(Ptr, USR.size());
+}
+
+ClangIndexRecordWriter::ClangIndexRecordWriter(ASTContext &Ctx,
+                                               RecordingOptions Opts)
+    : Impl(Opts.DataDirPath), Ctx(Ctx), RecordOpts(std::move(Opts)),
+      Hasher(Ctx) {
+  if (Opts.RecordSymbolCodeGenName)
+    CGNameGen.reset(new CodegenNameGenerator(Ctx));
+}
+
+ClangIndexRecordWriter::~ClangIndexRecordWriter() {}
+
+bool ClangIndexRecordWriter::writeRecord(StringRef Filename,
+                                         const FileIndexRecord &IdxRecord,
+                                         std::string &Error,
+                                         std::string *OutRecordFile) {
+
+  auto RecordHash = Hasher.hashRecord(IdxRecord);
+
+  switch (Impl.beginRecord(Filename, RecordHash, Error, OutRecordFile)) {
+  case IndexRecordWriter::Result::Success:
+    break; // Continue writing.
+  case IndexRecordWriter::Result::Failure:
+    return true;
+  case IndexRecordWriter::Result::AlreadyExists:
+    return false;
+  }
+
+  ASTContext &Ctx = getASTContext();
+  SourceManager &SM = Ctx.getSourceManager();
+  FileID FID = IdxRecord.getFileID();
+  auto getLineCol = [&](unsigned Offset) -> std::pair<unsigned, unsigned> {
+    unsigned LineNo = SM.getLineNumber(FID, Offset);
+    unsigned ColNo = SM.getColumnNumber(FID, Offset);
+    return std::make_pair(LineNo, ColNo);
+  };
+
+  for (auto &Occur : IdxRecord.getDeclOccurrences()) {
+    unsigned Line, Col;
+    std::tie(Line, Col) = getLineCol(Occur.Offset);
+    SmallVector<writer::SymbolRelation, 3> Related;
+    Related.reserve(Occur.Relations.size());
+    for (auto &Rel : Occur.Relations)
+      Related.push_back(writer::SymbolRelation{Rel.RelatedSymbol, Rel.Roles});
+
+    Impl.addOccurrence(Occur.Dcl, Occur.Roles, Line, Col, Related);
+  }
+
+  PrintingPolicy Policy(Ctx.getLangOpts());
+  Policy.SuppressTemplateArgsInCXXConstructors = true;
+
+  auto Result = Impl.endRecord(Error,
+      [&](writer::OpaqueDecl OD, SmallVectorImpl<char> &Scratch) {
+    const Decl *D = static_cast<const Decl *>(OD);
+    auto Info = getSymbolInfo(D);
+
+    writer::Symbol Sym;
+    Sym.SymInfo = Info;
+
+    auto *ND = dyn_cast<NamedDecl>(D);
+    if (ND) {
+      llvm::raw_svector_ostream OS(Scratch);
+      DeclarationName DeclName = ND->getDeclName();
+      if (!DeclName.isEmpty())
+        DeclName.print(OS, Policy);
+    }
+    unsigned NameLen = Scratch.size();
+    Sym.Name = StringRef(Scratch.data(), NameLen);
+
+    Sym.USR = getUSR(D);
+    assert(!Sym.USR.empty() && "Recorded decl without USR!");
+
+    if (CGNameGen && ND) {
+      llvm::raw_svector_ostream OS(Scratch);
+      CGNameGen->writeName(ND, OS);
+    }
+    unsigned CGNameLen = Scratch.size() - NameLen;
+    Sym.CodeGenName = StringRef(Scratch.data() + NameLen, CGNameLen);
+    return Sym;
+  });
+
+  switch (Result) {
+  case IndexRecordWriter::Result::Success:
+  case IndexRecordWriter::Result::AlreadyExists:
+    return false;
+  case IndexRecordWriter::Result::Failure:
+    return true;
+  }
+}

lib/Index/FileIndexRecord.h

+//===--- FileIndexRecord.h - Index data per file --------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_LIB_INDEX_FILEINDEXRECORD_H
+#define LLVM_CLANG_LIB_INDEX_FILEINDEXRECORD_H
+
+#include "clang/Index/IndexSymbol.h"
+#include "clang/Basic/SourceLocation.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/SmallVector.h"
+#include <vector>
+
+namespace clang {
+  class IdentifierInfo;
+
+namespace index {
+
+class FileIndexRecord {
+public:
+  struct DeclOccurrence {
+    SymbolRoleSet Roles;
+    unsigned Offset;
+    const Decl *Dcl;
+    SmallVector<SymbolRelation, 3> Relations;
+
+    DeclOccurrence(SymbolRoleSet R,
+                   unsigned Offset,
+                   const Decl *D,
+                   ArrayRef<SymbolRelation> Relations)
+      : Roles(R),
+        Offset(Offset),
+        Dcl(D),
+        Relations(Relations.begin(), Relations.end()) {}
+
+    friend bool operator <(const DeclOccurrence &LHS, const DeclOccurrence &RHS) {
+      return LHS.Offset < RHS.Offset;
+    }
+  };
+
+private:
+  FileID FID;
+  bool IsSystem;
+  std::vector<DeclOccurrence> Decls;
+
+public:
+  FileIndexRecord(FileID FID, bool isSystem) : FID(FID), IsSystem(isSystem) {}
+
+  ArrayRef<DeclOccurrence> getDeclOccurrences() const { return Decls; }
+
+  FileID getFileID() const { return FID; }
+  bool isSystem() const { return IsSystem; }
+
+  void addDeclOccurence(SymbolRoleSet Roles,
+                        unsigned Offset,
+                        const Decl *D,
+                        ArrayRef<SymbolRelation> Relations);
+  void print(llvm::raw_ostream &OS);
+};
+
+} // end namespace index
+} // end namespace clang
+
+#endif

lib/Index/FileIndexRecord.cpp

+//===--- FileIndexRecord.cpp - Index data per file ------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "FileIndexRecord.h"
+#include "clang/AST/ASTContext.h"
+#include "clang/AST/DeclTemplate.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/Support/Path.h"
+
+using namespace clang;
+using namespace clang::index;
+
+void FileIndexRecord::addDeclOccurence(SymbolRoleSet Roles,
+                                       unsigned Offset,
+                                       const Decl *D,
+                                       ArrayRef<SymbolRelation> Relations) {
+  assert(D->isCanonicalDecl());
+
+  auto IsNextOccurence = [&]()->bool {
+    if (Decls.empty())
+      return true;
+    auto &Last = Decls.back();
+    return Last.Offset < Offset;
+  };
+
+  if (IsNextOccurence()) {
+    Decls.emplace_back(Roles, Offset, D, Relations);
+    return;
+  }
+
+  DeclOccurrence NewInfo(Roles, Offset, D, Relations);
+  auto It = std::upper_bound(Decls.begin(), Decls.end(), NewInfo);
+  Decls.insert(It, std::move(NewInfo));
+}
+
+void FileIndexRecord::print(llvm::raw_ostream &OS) {
+  OS << "DECLS BEGIN ---\n";
+  for (auto &DclInfo : Decls) {
+    auto D = DclInfo.Dcl;
+    SourceManager &SM = D->getASTContext().getSourceManager();
+    SourceLocation Loc = SM.getFileLoc(D->getLocation());
+    PresumedLoc PLoc = SM.getPresumedLoc(Loc);
+    OS << llvm::sys::path::filename(PLoc.getFilename()) << ':' << PLoc.getLine()
+       << ':' << PLoc.getColumn();
+
+    if (auto ND = dyn_cast<NamedDecl>(D)) {
+      OS << ' ' << ND->getNameAsString();
+    }
+
+    OS << '\n';
+  }
+  OS << "DECLS END ---\n";
+}

lib/Index/IndexDataStore.cpp

+//===--- IndexDataStore.cpp - Index data store info -----------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Index/IndexDataStore.h"
+#include "IndexDataStoreUtils.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/Mutex.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/raw_ostream.h"
+
+using namespace clang;
+using namespace clang::index;
+using namespace clang::index::store;
+using namespace llvm;
+
+static void appendSubDir(StringRef subdir, SmallVectorImpl<char> &StorePathBuf) {
+  SmallString<10> VersionPath;
+  raw_svector_ostream(VersionPath) << 'v' << STORE_FORMAT_VERSION;
+
+  sys::path::append(StorePathBuf, VersionPath);
+  sys::path::append(StorePathBuf, subdir);
+}
+
+void store::appendInteriorUnitPath(StringRef UnitName,
+                                   SmallVectorImpl<char> &PathBuf) {
+  sys::path::append(PathBuf, UnitName);
+}
+
+void store::appendUnitSubDir(SmallVectorImpl<char> &StorePathBuf) {
+  return appendSubDir("units", StorePathBuf);
+}
+
+void store::appendRecordSubDir(SmallVectorImpl<char> &StorePathBuf) {
+  return appendSubDir("records", StorePathBuf);
+}
+
+void store::appendInteriorRecordPath(StringRef RecordName,
+                                     SmallVectorImpl<char> &PathBuf) {
+  // To avoid putting a huge number of files into the records directory, create
+  // subdirectories based on the last 2 characters from the hash.
+  StringRef hash2chars = RecordName.substr(RecordName.size()-2);
+  sys::path::append(PathBuf, hash2chars);
+  sys::path::append(PathBuf, RecordName);
+}
+
+//===----------------------------------------------------------------------===//
+// IndexDataStore
+//===----------------------------------------------------------------------===//
+
+namespace {
+
+class UnitEventHandlerData {
+  mutable sys::Mutex Mtx;
+  IndexDataStore::UnitEventHandler Handler;
+
+public:
+  void setHandler(IndexDataStore::UnitEventHandler handler) {
+    sys::ScopedLock L(Mtx);
+    Handler = std::move(handler);
+  }
+  IndexDataStore::UnitEventHandler getHandler() const {
+    sys::ScopedLock L(Mtx);
+    return Handler;
+  }
+};
+
+class IndexDataStoreImpl {
+  std::string FilePath;
+  std::shared_ptr<UnitEventHandlerData> TheUnitEventHandlerData;
+  std::unique_ptr<AbstractDirectoryWatcher> DirWatcher;
+
+public:
+  explicit IndexDataStoreImpl(StringRef indexStorePath)
+    : FilePath(indexStorePath) {
+    TheUnitEventHandlerData = std::make_shared<UnitEventHandlerData>();
+  }
+
+  StringRef getFilePath() const { return FilePath; }
+  bool foreachUnitName(bool sorted,
+                       llvm::function_ref<bool(StringRef unitName)> receiver);
+  void setUnitEventHandler(IndexDataStore::UnitEventHandler Handler);
+  bool startEventListening(llvm::function_ref<AbstractDirectoryWatcher::CreateFnTy> createFn,
+                           bool waitInitialSync, std::string &Error);
+  void stopEventListening();
+  void discardUnit(StringRef UnitName);
+  void discardRecord(StringRef RecordName);
+  void purgeStaleData();
+};
+
+} // anonymous namespace
+
+bool IndexDataStoreImpl::foreachUnitName(bool sorted,
+                        llvm::function_ref<bool(StringRef unitName)> receiver) {
+  SmallString<128> UnitPath;
+  UnitPath = FilePath;
+  appendUnitSubDir(UnitPath);
+
+  std::vector<std::string> filenames;
+
+  std::error_code EC;
+  for (auto It = sys::fs::directory_iterator(UnitPath, EC),
+           End = sys::fs::directory_iterator();
+       !EC && It != End; It.increment(EC)) {
+    StringRef unitName = sys::path::filename(It->path());
+    if (!sorted) {
+      if (!receiver(unitName))
+        return false;
+    } else {
+      filenames.push_back(unitName);
+    }
+  }
+
+  if (sorted) {
+    llvm::array_pod_sort(filenames.begin(), filenames.end());
+    for (auto &fname : filenames)
+      if (!receiver(fname))
+        return false;
+  }
+  return true;
+}
+
+void IndexDataStoreImpl::setUnitEventHandler(IndexDataStore::UnitEventHandler handler) {
+  TheUnitEventHandlerData->setHandler(std::move(handler));
+}
+
+bool IndexDataStoreImpl::startEventListening(llvm::function_ref<AbstractDirectoryWatcher::CreateFnTy> createFn,
+                                             bool waitInitialSync, std::string &Error) {
+  if (DirWatcher) {
+    Error = "event listener already active";
+    return true;
+  }
+
+  SmallString<128> UnitPath;
+  UnitPath = FilePath;
+  appendUnitSubDir(UnitPath);
+
+  auto localUnitEventHandlerData = TheUnitEventHandlerData;
+  auto OnUnitsChange = [localUnitEventHandlerData](ArrayRef<AbstractDirectoryWatcher::Event> Events, bool isInitial) {
+    SmallVector<IndexDataStore::UnitEvent, 16> UnitEvents;
+    UnitEvents.reserve(Events.size());
+    for (const AbstractDirectoryWatcher::Event &evt : Events) {
+      IndexDataStore::UnitEventKind K;
+      StringRef UnitName = sys::path::filename(evt.Filename);
+      switch (evt.Kind) {
+      case AbstractDirectoryWatcher::EventKind::Added:
+        K = IndexDataStore::UnitEventKind::Added; break;
+      case AbstractDirectoryWatcher::EventKind::Removed:
+        K = IndexDataStore::UnitEventKind::Removed; break;
+      case AbstractDirectoryWatcher::EventKind::Modified:
+        K = IndexDataStore::UnitEventKind::Modified; break;
+      case AbstractDirectoryWatcher::EventKind::DirectoryDeleted:
+        K = IndexDataStore::UnitEventKind::DirectoryDeleted;
+        UnitName = StringRef();
+        break;
+      }
+      UnitEvents.push_back(IndexDataStore::UnitEvent{K, UnitName, evt.ModTime});
+    }
+
+    if (auto handler = localUnitEventHandlerData->getHandler()) {
+      IndexDataStore::UnitEventNotification EventNote{isInitial, UnitEvents};
+      handler(EventNote);
+    }
+  };
+
+  DirWatcher = createFn(UnitPath.str(), OnUnitsChange, waitInitialSync, Error);
+  if (!DirWatcher)
+    return true;
+
+  return false;
+}
+
+void IndexDataStoreImpl::stopEventListening() {
+  DirWatcher.reset();
+}
+
+void IndexDataStoreImpl::discardUnit(StringRef UnitName) {
+  SmallString<128> UnitPath;
+  UnitPath = FilePath;
+  appendUnitSubDir(UnitPath);
+  appendInteriorUnitPath(UnitName, UnitPath);
+  sys::fs::remove(UnitPath);
+}
+
+void IndexDataStoreImpl::discardRecord(StringRef RecordName) {
+  SmallString<128> RecordPath;
+  RecordPath = FilePath;
+  appendRecordSubDir(RecordPath);
+  appendInteriorRecordPath(RecordName, RecordPath);
+  sys::fs::remove(RecordPath);
+}
+
+void IndexDataStoreImpl::purgeStaleData() {
+  // FIXME: Implement.
+}
+
+
+std::unique_ptr<IndexDataStore>
+IndexDataStore::create(StringRef IndexStorePath, std::string &Error) {
+  if (!sys::fs::exists(IndexStorePath)) {
+    raw_string_ostream OS(Error);
+    OS << "index store path does not exist: " << IndexStorePath;
+    return nullptr;
+  }
+
+  return std::unique_ptr<IndexDataStore>(
+    new IndexDataStore(new IndexDataStoreImpl(IndexStorePath)));
+}
+
+#define IMPL static_cast<IndexDataStoreImpl*>(Impl)
+
+IndexDataStore::~IndexDataStore() {
+  delete IMPL;
+}
+
+StringRef IndexDataStore::getFilePath() const {
+  return IMPL->getFilePath();
+}
+
+bool IndexDataStore::foreachUnitName(bool sorted,
+                     llvm::function_ref<bool(StringRef unitName)> receiver) {
+  return IMPL->foreachUnitName(sorted, std::move(receiver));
+}
+
+unsigned IndexDataStore::getFormatVersion() {
+  return STORE_FORMAT_VERSION;
+}
+
+void IndexDataStore::setUnitEventHandler(UnitEventHandler Handler) {
+  return IMPL->setUnitEventHandler(std::move(Handler));
+}
+
+bool IndexDataStore::startEventListening(llvm::function_ref<AbstractDirectoryWatcher::CreateFnTy> createFn,
+                                         bool waitInitialSync, std::string &Error) {
+  return IMPL->startEventListening(std::move(createFn), waitInitialSync, Error);
+}
+
+void IndexDataStore::stopEventListening() {
+  return IMPL->stopEventListening();
+}
+
+void IndexDataStore::discardUnit(StringRef UnitName) {
+  IMPL->discardUnit(UnitName);
+}
+
+void IndexDataStore::discardRecord(StringRef RecordName) {
+  IMPL->discardRecord(RecordName);
+}
+
+void IndexDataStore::purgeStaleData() {
+  IMPL->purgeStaleData();
+}

lib/Index/IndexDataStoreUtils.h

+//===--- IndexDataStoreUtils.h - Functions/constants for the data store ---===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_LIB_INDEX_INDEXDATASTOREUTILS_H
+#define LLVM_CLANG_LIB_INDEX_INDEXDATASTOREUTILS_H
+
+#include "llvm/Bitcode/BitCodes.h"
+#include "clang/Basic/LLVM.h"
+
+namespace llvm {
+  class BitstreamWriter;
+}
+
+namespace clang {
+namespace index {
+namespace store {
+
+static const unsigned STORE_FORMAT_VERSION = 5;
+
+void appendUnitSubDir(SmallVectorImpl<char> &StorePathBuf);
+void appendInteriorUnitPath(StringRef UnitName,
+                            SmallVectorImpl<char> &PathBuf);
+void appendRecordSubDir(SmallVectorImpl<char> &StorePathBuf);
+void appendInteriorRecordPath(StringRef RecordName,
+                              SmallVectorImpl<char> &PathBuf);
+
+enum RecordBitRecord {
+  REC_VERSION         = 0,
+  REC_DECLINFO        = 1,
+  REC_DECLOFFSETS     = 2,
+  REC_DECLOCCURRENCE  = 3,
+};
+
+enum RecordBitBlock {
+  REC_VERSION_BLOCK_ID = llvm::bitc::FIRST_APPLICATION_BLOCKID,
+  REC_DECLS_BLOCK_ID,
+  REC_DECLOFFSETS_BLOCK_ID,
+  REC_DECLOCCURRENCES_BLOCK_ID,
+};
+
+enum UnitBitRecord {
+  UNIT_VERSION        = 0,
+  UNIT_INFO           = 1,
+  UNIT_DEPENDENCY     = 2,
+  UNIT_INCLUDE        = 3,
+  UNIT_PATH           = 4,
+  UNIT_PATH_BUFFER    = 5,
+  UNIT_MODULE         = 6,
+  UNIT_MODULE_BUFFER  = 7,
+};
+
+enum UnitBitBlock {
+  UNIT_VERSION_BLOCK_ID = llvm::bitc::FIRST_APPLICATION_BLOCKID,
+  UNIT_INFO_BLOCK_ID,
+  UNIT_DEPENDENCIES_BLOCK_ID,
+  UNIT_INCLUDES_BLOCK_ID,
+  UNIT_PATHS_BLOCK_ID,
+  UNIT_MODULES_BLOCK_ID,
+};
+
+enum UnitDependencyKind {
+  UNIT_DEPEND_KIND_FILE = 0,
+  UNIT_DEPEND_KIND_RECORD = 1,
+  UNIT_DEPEND_KIND_UNIT = 2,
+};
+static const unsigned UnitDependencyKindBitNum = 2;
+
+enum UnitFilePathPrefixKind {
+  UNIT_PATH_PREFIX_NONE = 0,
+  UNIT_PATH_PREFIX_WORKDIR = 1,
+  UNIT_PATH_PREFIX_SYSROOT = 2,
+};
+static const unsigned UnitFilePathPrefixKindBitNum = 2;
+
+typedef SmallVector<uint64_t, 64> RecordData;
+typedef SmallVectorImpl<uint64_t> RecordDataImpl;
+
+struct BitPathComponent {
+  size_t Offset = 0;
+  size_t Size = 0;
+  BitPathComponent(size_t Offset, size_t Size) : Offset(Offset), Size(Size) {}
+  BitPathComponent() = default;
+};
+
+struct DirBitPath {
+  UnitFilePathPrefixKind PrefixKind = UNIT_PATH_PREFIX_NONE;
+  BitPathComponent Dir;
+  DirBitPath(UnitFilePathPrefixKind Kind,
+             BitPathComponent Dir) : PrefixKind(Kind), Dir(Dir) {}
+  DirBitPath() = default;
+};
+
+struct FileBitPath : DirBitPath {
+  BitPathComponent Filename;
+  FileBitPath(UnitFilePathPrefixKind Kind, BitPathComponent Dir,
+              BitPathComponent Filename) : DirBitPath(Kind, Dir), Filename(Filename) {}
+  FileBitPath() = default;
+};
+
+void emitBlockID(unsigned ID, const char *Name,
+                 llvm::BitstreamWriter &Stream, RecordDataImpl &Record);
+
+void emitRecordID(unsigned ID, const char *Name,
+                  llvm::BitstreamWriter &Stream, RecordDataImpl &Record);
+
+} // end namespace store
+} // end namespace index
+} // end namespace clang
+
+#endif

lib/Index/IndexDataStoreUtils.cpp

+//===--- IndexDataStoreUtils.cpp - Functions/constants for the data store -===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Index/IndexDataStoreSymbolUtils.h"
+#include "IndexDataStoreUtils.h"
+#include "llvm/Bitcode/BitstreamWriter.h"
+
+using namespace clang;
+using namespace clang::index;
+using namespace clang::index::store;
+using namespace llvm;
+
+void store::emitBlockID(unsigned ID, const char *Name,
+                        BitstreamWriter &Stream, RecordDataImpl &Record) {
+  Record.clear();
+  Record.push_back(ID);
+  Stream.EmitRecord(bitc::BLOCKINFO_CODE_SETBID, Record);
+
+  // Emit the block name if present.
+  if (!Name || Name[0] == 0)
+    return;
+  Record.clear();
+  while (*Name)
+    Record.push_back(*Name++);
+  Stream.EmitRecord(bitc::BLOCKINFO_CODE_BLOCKNAME, Record);
+}
+
+void store::emitRecordID(unsigned ID, const char *Name,
+                         BitstreamWriter &Stream,
+                         RecordDataImpl &Record) {
+  Record.clear();
+  Record.push_back(ID);
+  while (*Name)
+    Record.push_back(*Name++);
+  Stream.EmitRecord(bitc::BLOCKINFO_CODE_SETRECORDNAME, Record);
+}
+
+/// Map an indexstore_symbol_kind_t to a SymbolKind, handling unknown values.
+SymbolKind index::getSymbolKind(indexstore_symbol_kind_t K) {
+  switch ((uint64_t)K) {
+  default:
+  case INDEXSTORE_SYMBOL_KIND_UNKNOWN:
+    return SymbolKind::Unknown;
+  case INDEXSTORE_SYMBOL_KIND_MODULE:
+    return SymbolKind::Module;
+  case INDEXSTORE_SYMBOL_KIND_NAMESPACE:
+    return SymbolKind::Namespace;
+  case INDEXSTORE_SYMBOL_KIND_NAMESPACEALIAS:
+    return SymbolKind::NamespaceAlias;
+  case INDEXSTORE_SYMBOL_KIND_MACRO:
+    return SymbolKind::Macro;
+  case INDEXSTORE_SYMBOL_KIND_ENUM:
+    return SymbolKind::Enum;
+  case INDEXSTORE_SYMBOL_KIND_STRUCT:
+    return SymbolKind::Struct;
+  case INDEXSTORE_SYMBOL_KIND_CLASS:
+    return SymbolKind::Class;
+  case INDEXSTORE_SYMBOL_KIND_PROTOCOL:
+    return SymbolKind::Protocol;
+  case INDEXSTORE_SYMBOL_KIND_EXTENSION:
+    return SymbolKind::Extension;
+  case INDEXSTORE_SYMBOL_KIND_UNION:
+    return SymbolKind::Union;
+  case INDEXSTORE_SYMBOL_KIND_TYPEALIAS:
+    return SymbolKind::TypeAlias;
+  case INDEXSTORE_SYMBOL_KIND_FUNCTION:
+    return SymbolKind::Function;
+  case INDEXSTORE_SYMBOL_KIND_VARIABLE:
+    return SymbolKind::Variable;
+  case INDEXSTORE_SYMBOL_KIND_FIELD:
+    return SymbolKind::Field;
+  case INDEXSTORE_SYMBOL_KIND_ENUMCONSTANT:
+    return SymbolKind::EnumConstant;
+  case INDEXSTORE_SYMBOL_KIND_INSTANCEMETHOD:
+    return SymbolKind::InstanceMethod;
+  case INDEXSTORE_SYMBOL_KIND_CLASSMETHOD:
+    return SymbolKind::ClassMethod;
+  case INDEXSTORE_SYMBOL_KIND_STATICMETHOD:
+    return SymbolKind::StaticMethod;
+  case INDEXSTORE_SYMBOL_KIND_INSTANCEPROPERTY:
+    return SymbolKind::InstanceProperty;
+  case INDEXSTORE_SYMBOL_KIND_CLASSPROPERTY:
+    return SymbolKind::ClassProperty;
+  case INDEXSTORE_SYMBOL_KIND_STATICPROPERTY:
+    return SymbolKind::StaticProperty;
+  case INDEXSTORE_SYMBOL_KIND_CONSTRUCTOR:
+    return SymbolKind::Constructor;
+  case INDEXSTORE_SYMBOL_KIND_DESTRUCTOR:
+    return SymbolKind::Destructor;
+  case INDEXSTORE_SYMBOL_KIND_CONVERSIONFUNCTION:
+    return SymbolKind::ConversionFunction;
+  case INDEXSTORE_SYMBOL_KIND_PARAMETER:
+    return SymbolKind::Parameter;
+  case INDEXSTORE_SYMBOL_KIND_USING:
+    return SymbolKind::Using;
+  }
+}
+
+SymbolSubKind index::getSymbolSubKind(indexstore_symbol_subkind_t K) {
+  switch ((uint64_t)K) {
+  default:
+  case INDEXSTORE_SYMBOL_SUBKIND_NONE:
+    return SymbolSubKind::None;
+  case INDEXSTORE_SYMBOL_SUBKIND_CXXCOPYCONSTRUCTOR:
+    return SymbolSubKind::CXXCopyConstructor;
+  case INDEXSTORE_SYMBOL_SUBKIND_CXXMOVECONSTRUCTOR:
+    return SymbolSubKind::CXXMoveConstructor;
+  case INDEXSTORE_SYMBOL_SUBKIND_ACCESSORGETTER:
+      return SymbolSubKind::AccessorGetter;
+  case INDEXSTORE_SYMBOL_SUBKIND_ACCESSORSETTER:
+      return SymbolSubKind::AccessorSetter;
+  case INDEXSTORE_SYMBOL_SUBKIND_USINGTYPENAME:
+      return SymbolSubKind::UsingTypename;
+  case INDEXSTORE_SYMBOL_SUBKIND_USINGVALUE:
+      return SymbolSubKind::UsingValue;
+  }
+}
+
+/// Map an indexstore_symbol_language_t to a SymbolLanguage, handling unknown
+/// values.
+SymbolLanguage index::getSymbolLanguage(indexstore_symbol_language_t L) {
+  switch ((uint64_t)L) {
+  default: // FIXME: add an unknown language?
+  case INDEXSTORE_SYMBOL_LANG_C:
+    return SymbolLanguage::C;
+  case INDEXSTORE_SYMBOL_LANG_OBJC:
+    return SymbolLanguage::ObjC;
+  case INDEXSTORE_SYMBOL_LANG_CXX:
+    return SymbolLanguage::CXX;
+  case INDEXSTORE_SYMBOL_LANG_SWIFT:
+    return SymbolLanguage::Swift;
+  }
+}
+
+/// Map an indexstore representation to a SymbolPropertySet, handling
+/// unknown values.
+SymbolPropertySet index::getSymbolProperties(uint64_t Props) {
+  // FIXME: currently these enums must be kept in sync.
+  return (uint64_t)Props;
+}
+
+/// Map an indexstore representation to a SymbolRoleSet, handling unknown
+/// values.
+SymbolRoleSet index::getSymbolRoles(uint64_t Roles) {
+  // FIXME: currently these enums must be kept in sync.
+  return (uint64_t)Roles;
+}
+
+/// Map a SymbolLanguage to a indexstore_symbol_language_t.
+indexstore_symbol_kind_t index::getIndexStoreKind(SymbolKind K) {
+  switch (K) {
+  case SymbolKind::Unknown:
+    return INDEXSTORE_SYMBOL_KIND_UNKNOWN;
+  case SymbolKind::Module:
+    return INDEXSTORE_SYMBOL_KIND_MODULE;
+  case SymbolKind::Namespace:
+    return INDEXSTORE_SYMBOL_KIND_NAMESPACE;
+  case SymbolKind::NamespaceAlias:
+    return INDEXSTORE_SYMBOL_KIND_NAMESPACEALIAS;
+  case SymbolKind::Macro:
+    return INDEXSTORE_SYMBOL_KIND_MACRO;
+  case SymbolKind::Enum:
+    return INDEXSTORE_SYMBOL_KIND_ENUM;
+  case SymbolKind::Struct:
+    return INDEXSTORE_SYMBOL_KIND_STRUCT;
+  case SymbolKind::Class:
+    return INDEXSTORE_SYMBOL_KIND_CLASS;
+  case SymbolKind::Protocol:
+    return INDEXSTORE_SYMBOL_KIND_PROTOCOL;
+  case SymbolKind::Extension:
+    return INDEXSTORE_SYMBOL_KIND_EXTENSION;
+  case SymbolKind::Union:
+    return INDEXSTORE_SYMBOL_KIND_UNION;
+  case SymbolKind::TypeAlias:
+    return INDEXSTORE_SYMBOL_KIND_TYPEALIAS;
+  case SymbolKind::Function:
+    return INDEXSTORE_SYMBOL_KIND_FUNCTION;
+  case SymbolKind::Variable:
+    return INDEXSTORE_SYMBOL_KIND_VARIABLE;
+  case SymbolKind::Field:
+    return INDEXSTORE_SYMBOL_KIND_FIELD;
+  case SymbolKind::EnumConstant:
+    return INDEXSTORE_SYMBOL_KIND_ENUMCONSTANT;
+  case SymbolKind::InstanceMethod:
+    return INDEXSTORE_SYMBOL_KIND_INSTANCEMETHOD;
+  case SymbolKind::ClassMethod:
+    return INDEXSTORE_SYMBOL_KIND_CLASSMETHOD;
+  case SymbolKind::StaticMethod:
+    return INDEXSTORE_SYMBOL_KIND_STATICMETHOD;
+  case SymbolKind::InstanceProperty:
+    return INDEXSTORE_SYMBOL_KIND_INSTANCEPROPERTY;
+  case SymbolKind::ClassProperty:
+    return INDEXSTORE_SYMBOL_KIND_CLASSPROPERTY;
+  case SymbolKind::StaticProperty:
+    return INDEXSTORE_SYMBOL_KIND_STATICPROPERTY;
+  case SymbolKind::Constructor:
+    return INDEXSTORE_SYMBOL_KIND_CONSTRUCTOR;
+  case SymbolKind::Destructor:
+    return INDEXSTORE_SYMBOL_KIND_DESTRUCTOR;
+  case SymbolKind::ConversionFunction:
+    return INDEXSTORE_SYMBOL_KIND_CONVERSIONFUNCTION;
+  case SymbolKind::Parameter:
+    return INDEXSTORE_SYMBOL_KIND_PARAMETER;
+  case SymbolKind::Using:
+    return INDEXSTORE_SYMBOL_KIND_USING;
+  }
+  llvm_unreachable("unexpected symbol kind");
+}
+
+indexstore_symbol_subkind_t index::getIndexStoreSubKind(SymbolSubKind K) {
+  switch (K) {
+  case SymbolSubKind::None:
+    return INDEXSTORE_SYMBOL_SUBKIND_NONE;
+  case SymbolSubKind::CXXCopyConstructor:
+    return INDEXSTORE_SYMBOL_SUBKIND_CXXCOPYCONSTRUCTOR;
+  case SymbolSubKind::CXXMoveConstructor:
+    return INDEXSTORE_SYMBOL_SUBKIND_CXXMOVECONSTRUCTOR;
+  case SymbolSubKind::AccessorGetter:
+    return INDEXSTORE_SYMBOL_SUBKIND_ACCESSORGETTER;
+  case SymbolSubKind::AccessorSetter:
+    return INDEXSTORE_SYMBOL_SUBKIND_ACCESSORSETTER;
+  case SymbolSubKind::UsingTypename:
+    return INDEXSTORE_SYMBOL_SUBKIND_USINGTYPENAME;
+  case SymbolSubKind::UsingValue:
+    return INDEXSTORE_SYMBOL_SUBKIND_USINGVALUE;
+  }
+  llvm_unreachable("unexpected symbol subkind");
+}
+
+/// Map a SymbolLanguage to a indexstore_symbol_language_t.
+indexstore_symbol_language_t index::getIndexStoreLang(SymbolLanguage L) {
+  switch (L) {
+  case SymbolLanguage::C:
+    return INDEXSTORE_SYMBOL_LANG_C;
+  case SymbolLanguage::ObjC:
+    return INDEXSTORE_SYMBOL_LANG_OBJC;
+  case SymbolLanguage::CXX:
+    return INDEXSTORE_SYMBOL_LANG_CXX;
+  case SymbolLanguage::Swift:
+    return INDEXSTORE_SYMBOL_LANG_SWIFT;
+  }
+  llvm_unreachable("unexpected symbol language");
+}
+
+/// Map a SymbolPropertySet to its indexstore representation.
+uint64_t index::getIndexStoreProperties(SymbolPropertySet Props) {
+  uint64_t storeProp = 0;
+  applyForEachSymbolProperty(Props, [&](SymbolProperty prop) {
+    switch (prop) {
+    case SymbolProperty::Generic:
+      storeProp |= INDEXSTORE_SYMBOL_PROPERTY_GENERIC;
+      break;
+    case SymbolProperty::TemplatePartialSpecialization:
+      storeProp |= INDEXSTORE_SYMBOL_PROPERTY_TEMPLATE_PARTIAL_SPECIALIZATION;
+      break;
+    case SymbolProperty::TemplateSpecialization:
+      storeProp |= INDEXSTORE_SYMBOL_PROPERTY_TEMPLATE_SPECIALIZATION;
+      break;
+    case SymbolProperty::UnitTest:
+      storeProp |= INDEXSTORE_SYMBOL_PROPERTY_UNITTEST;
+      break;
+    case SymbolProperty::IBAnnotated:
+      storeProp |= INDEXSTORE_SYMBOL_PROPERTY_IBANNOTATED;
+      break;
+    case SymbolProperty::IBOutletCollection:
+      storeProp |= INDEXSTORE_SYMBOL_PROPERTY_IBOUTLETCOLLECTION;
+      break;
+    case SymbolProperty::GKInspectable:
+      storeProp |= INDEXSTORE_SYMBOL_PROPERTY_GKINSPECTABLE;
+      break;
+    case SymbolProperty::Local:
+      storeProp |= INDEXSTORE_SYMBOL_PROPERTY_LOCAL;
+      break;
+    }
+  });
+  return storeProp;
+}
+
+/// Map a SymbolRoleSet to its indexstore representation.
+uint64_t index::getIndexStoreRoles(SymbolRoleSet Roles) {
+  uint64_t storeRoles = 0;
+  applyForEachSymbolRole(Roles, [&](SymbolRole role) {
+    switch (role) {
+    case SymbolRole::Declaration:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_DECLARATION;
+      break;
+    case SymbolRole::Definition:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_DEFINITION;
+      break;
+    case SymbolRole::Reference:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REFERENCE;
+      break;
+    case SymbolRole::Read:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_READ;
+      break;
+    case SymbolRole::Write:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_WRITE;
+      break;
+    case SymbolRole::Call:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_CALL;
+      break;
+    case SymbolRole::Dynamic:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_DYNAMIC;
+      break;
+    case SymbolRole::AddressOf:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_ADDRESSOF;
+      break;
+    case SymbolRole::Implicit:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_IMPLICIT;
+      break;
+    case SymbolRole::RelationChildOf:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REL_CHILDOF;
+      break;
+    case SymbolRole::RelationBaseOf:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REL_BASEOF;
+      break;
+    case SymbolRole::RelationOverrideOf:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REL_OVERRIDEOF;
+      break;
+    case SymbolRole::RelationReceivedBy:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REL_RECEIVEDBY;
+      break;
+    case SymbolRole::RelationCalledBy:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REL_CALLEDBY;
+      break;
+    case SymbolRole::RelationExtendedBy:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REL_EXTENDEDBY;
+      break;
+    case SymbolRole::RelationAccessorOf:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REL_ACCESSOROF;
+      break;
+    case SymbolRole::RelationContainedBy:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REL_CONTAINEDBY;
+      break;
+    case SymbolRole::RelationIBTypeOf:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REL_IBTYPEOF;
+      break;
+    case SymbolRole::RelationSpecializationOf:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REL_SPECIALIZATIONOF;
+      break;
+    }
+  });
+  return storeRoles;
+}

lib/Index/IndexRecordHasher.h

+//===--- IndexRecordHasher.h - Index record hashing -----------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_LIB_INDEX_INDEXRECORDHASHER_H
+#define LLVM_CLANG_LIB_INDEX_INDEXRECORDHASHER_H
+
+#include "clang/Basic/LLVM.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/Hashing.h"
+
+namespace clang {
+  class ASTContext;
+  class Decl;
+  class DeclarationName;
+  class NestedNameSpecifier;
+  class QualType;
+  class Type;
+  template <typename> class CanQual;
+  typedef CanQual<Type> CanQualType;
+
+namespace index {
+  class FileIndexRecord;
+
+class IndexRecordHasher {
+  ASTContext &Ctx;
+  llvm::DenseMap<const void *, llvm::hash_code> HashByPtr;
+
+public:
+  explicit IndexRecordHasher(ASTContext &Ctx) : Ctx(Ctx) {}
+  ASTContext &getASTContext() { return Ctx; }
+
+  llvm::hash_code hashRecord(const FileIndexRecord &Record);
+  llvm::hash_code hash(const Decl *D);
+  llvm::hash_code hash(QualType Ty);
+  llvm::hash_code hash(CanQualType Ty);
+  llvm::hash_code hash(DeclarationName Name);
+  llvm::hash_code hash(const NestedNameSpecifier *NNS);
+
+private:
+  template <typename T>
+  llvm::hash_code tryCache(const void *Ptr, T Obj);
+
+  llvm::hash_code hashImpl(const Decl *D);
+  llvm::hash_code hashImpl(CanQualType Ty);
+  llvm::hash_code hashImpl(DeclarationName Name);
+  llvm::hash_code hashImpl(const NestedNameSpecifier *NNS);
+};
+
+} // end namespace index
+} // end namespace clang
+
+#endif

lib/Index/IndexRecordHasher.cpp

+//===--- IndexRecordHasher.cpp - Index record hashing ---------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "IndexRecordHasher.h"
+#include "FileIndexRecord.h"
+#include "clang/AST/ASTContext.h"
+#include "clang/AST/Decl.h"
+#include "clang/AST/DeclVisitor.h"
+#include "llvm/Support/Path.h"
+
+#define INITIAL_HASH 5381
+#define COMBINE_HASH(...) (Hash = hash_combine(Hash, __VA_ARGS__))
+
+using namespace clang;
+using namespace clang::index;
+using namespace llvm;
+
+static hash_code computeHash(const TemplateArgument &Arg,
+                             IndexRecordHasher &Hasher);
+
+namespace {
+class DeclHashVisitor : public ConstDeclVisitor<DeclHashVisitor, hash_code> {
+  IndexRecordHasher &Hasher;
+
+public:
+  DeclHashVisitor(IndexRecordHasher &Hasher) : Hasher(Hasher) {}
+
+  hash_code VisitDecl(const Decl *D) {
+    return VisitDeclContext(D->getDeclContext());
+  }
+
+  hash_code VisitNamedDecl(const NamedDecl *D) {
+    hash_code Hash = VisitDecl(D);
+    if (auto *attr = D->getExternalSourceSymbolAttr()) {
+      COMBINE_HASH(hash_value(attr->getDefinedIn()));
+    }
+    return COMBINE_HASH(Hasher.hash(D->getDeclName()));
+  }
+
+  hash_code VisitTagDecl(const TagDecl *D) {
+    if (D->getDeclName().isEmpty()) {
+      if (const TypedefNameDecl *TD = D->getTypedefNameForAnonDecl())
+        return Visit(TD);
+
+      hash_code Hash = VisitDeclContext(D->getDeclContext());
+      if (D->isEmbeddedInDeclarator() && !D->isFreeStanding()) {
+        COMBINE_HASH(hashLoc(D->getLocation(), /*IncludeOffset=*/true));
+      } else
+        COMBINE_HASH('a');
+      return Hash;
+    }
+
+    hash_code Hash = VisitTypeDecl(D);
+    return COMBINE_HASH('T');
+  }
+
+  hash_code VisitClassTemplateSpecializationDecl(const ClassTemplateSpecializationDecl *D) {
+    hash_code Hash = VisitCXXRecordDecl(D);
+    const TemplateArgumentList &Args = D->getTemplateArgs();
+    COMBINE_HASH('>');
+    for (unsigned I = 0, N = Args.size(); I != N; ++I) {
+      COMBINE_HASH(computeHash(Args.get(I), Hasher));
+    }
+    return Hash;
+  }
+
+  hash_code VisitObjCContainerDecl(const ObjCContainerDecl *D) {
+    hash_code Hash = VisitNamedDecl(D);
+    return COMBINE_HASH('I');
+  }
+
+  hash_code VisitObjCImplDecl(const ObjCImplDecl *D) {
+    if (auto *ID = D->getClassInterface())
+      return VisitObjCInterfaceDecl(ID);
+    else
+      return 0;
+  }
+
+  hash_code VisitObjCCategoryDecl(const ObjCCategoryDecl *D) {
+    // FIXME: Differentiate between category and the interface ?
+    if (auto *ID = D->getClassInterface())
+      return VisitObjCInterfaceDecl(ID);
+    else
+      return 0;
+  }
+
+  hash_code VisitFunctionDecl(const FunctionDecl *D) {
+    hash_code Hash = VisitNamedDecl(D);
+    ASTContext &Ctx = Hasher.getASTContext();
+    if ((!Ctx.getLangOpts().CPlusPlus && !D->hasAttr<OverloadableAttr>())
+        || D->isExternC())
+      return Hash;
+
+    for (auto param : D->parameters()) {
+      COMBINE_HASH(Hasher.hash(param->getType()));
+    }
+    return Hash;
+  }
+
+  hash_code VisitUnresolvedUsingTypenameDecl(const UnresolvedUsingTypenameDecl *D) {
+    hash_code Hash = VisitNamedDecl(D);
+    COMBINE_HASH(Hasher.hash(D->getQualifier()));
+    return Hash;
+  }
+
+  hash_code VisitUnresolvedUsingValueDecl(const UnresolvedUsingValueDecl *D) {
+    hash_code Hash = VisitNamedDecl(D);
+    COMBINE_HASH(Hasher.hash(D->getQualifier()));
+    return Hash;
+  }
+
+  hash_code VisitDeclContext(const DeclContext *DC) {
+    // FIXME: Add location if this is anonymous namespace ?
+    DC = DC->getRedeclContext();
+    const Decl *D = cast<Decl>(DC)->getCanonicalDecl();
+    if (auto *ND = dyn_cast<NamedDecl>(D))
+      return Hasher.hash(ND);
+    else
+      return 0;
+  }
+
+  hash_code hashLoc(SourceLocation Loc, bool IncludeOffset) {
+    if (Loc.isInvalid()) {
+      return 0;
+    }
+    hash_code Hash = INITIAL_HASH;
+    const SourceManager &SM = Hasher.getASTContext().getSourceManager();
+    Loc = SM.getFileLoc(Loc);
+    const std::pair<FileID, unsigned> &Decomposed = SM.getDecomposedLoc(Loc);
+    const FileEntry *FE = SM.getFileEntryForID(Decomposed.first);
+    if (FE) {
+      COMBINE_HASH(llvm::sys::path::filename(FE->getName()));
+    } else {
+      // This case really isn't interesting.
+      return 0;
+    }
+    if (IncludeOffset) {
+      // Use the offest into the FileID to represent the location.  Using
+      // a line/column can cause us to look back at the original source file,
+      // which is expensive.
+      COMBINE_HASH(Decomposed.second);
+    }
+    return Hash;
+  }
+};
+}
+
+hash_code IndexRecordHasher::hashRecord(const FileIndexRecord &Record) {
+  hash_code Hash = INITIAL_HASH;
+  for (auto &Info : Record.getDeclOccurrences()) {
+    COMBINE_HASH(Info.Roles, Info.Offset, hash(Info.Dcl));
+    for (auto &Rel : Info.Relations) {
+      COMBINE_HASH(hash(Rel.RelatedSymbol));
+    }
+  }
+  return Hash;
+}
+
+hash_code IndexRecordHasher::hash(const Decl *D) {
+  assert(D->isCanonicalDecl());
+
+  if (isa<TagDecl>(D) || isa<ObjCContainerDecl>(D)) {
+    return tryCache(D, D);
+  } else  if (auto *NS = dyn_cast<NamespaceDecl>(D)) {
+    if (NS->isAnonymousNamespace())
+      return hash_value(StringRef("@aN"));
+    return tryCache(D, D);
+  } else {
+    // There's a balance between caching results and not growing the cache too
+    // much. Measurements showed that avoiding caching all decls is beneficial
+    // particularly when including all of Cocoa.
+    return hashImpl(D);
+  }
+}
+
+hash_code IndexRecordHasher::hash(QualType NonCanTy) {
+  CanQualType CanTy = Ctx.getCanonicalType(NonCanTy);
+  return hash(CanTy);
+}
+
+hash_code IndexRecordHasher::hash(CanQualType CT) {
+  // Do some hashing without going to the cache, for example we can avoid
+  // storing the hash for both the type and its const-qualified version.
+  hash_code Hash = INITIAL_HASH;
+
+  auto asCanon = [](QualType Ty) -> CanQualType {
+    return CanQualType::CreateUnsafe(Ty);
+  };
+
+  while (true) {
+    Qualifiers Q = CT.getQualifiers();
+    CT = CT.getUnqualifiedType();
+    const Type *T = CT.getTypePtr();
+    unsigned qVal = 0;
+    if (Q.hasConst())
+      qVal |= 0x1;
+    if (Q.hasVolatile())
+      qVal |= 0x2;
+    if (Q.hasRestrict())
+      qVal |= 0x4;
+    if(qVal)
+      COMBINE_HASH(qVal);
+
+    // Hash in ObjC GC qualifiers?
+
+    if (const BuiltinType *BT = dyn_cast<BuiltinType>(T)) {
+      return COMBINE_HASH(BT->getKind());
+    }
+    if (const PointerType *PT = dyn_cast<PointerType>(T)) {
+      COMBINE_HASH('*');
+      CT = asCanon(PT->getPointeeType());
+      continue;
+    }
+    if (const ReferenceType *RT = dyn_cast<ReferenceType>(T)) {
+      COMBINE_HASH('&');
+      CT = asCanon(RT->getPointeeType());
+      continue;
+    }
+    if (const BlockPointerType *BT = dyn_cast<BlockPointerType>(T)) {
+      COMBINE_HASH('B');
+      CT = asCanon(BT->getPointeeType());
+      continue;
+    }
+    if (const ObjCObjectPointerType *OPT = dyn_cast<ObjCObjectPointerType>(T)) {
+      COMBINE_HASH('*');
+      CT = asCanon(OPT->getPointeeType());
+      continue;
+    }
+    if (const TagType *TT = dyn_cast<TagType>(T)) {
+      return COMBINE_HASH('$', hash(TT->getDecl()->getCanonicalDecl()));
+    }
+    if (const ObjCInterfaceType *OIT = dyn_cast<ObjCInterfaceType>(T)) {
+      return COMBINE_HASH('$', hash(OIT->getDecl()->getCanonicalDecl()));
+    }
+    if (const ObjCObjectType *OIT = dyn_cast<ObjCObjectType>(T)) {
+      for (auto *Prot : OIT->getProtocols())
+        COMBINE_HASH(hash(Prot));
+      CT = asCanon(OIT->getBaseType());
+      continue;
+    }
+    if (const TemplateTypeParmType *TTP = dyn_cast<TemplateTypeParmType>(T)) {
+      return COMBINE_HASH('t', TTP->getDepth(), TTP->getIndex());
+    }
+    if (const InjectedClassNameType *InjT = dyn_cast<InjectedClassNameType>(T)) {
+      CT = asCanon(InjT->getInjectedSpecializationType().getCanonicalType());
+      continue;
+    }
+
+    break;
+  }
+
+  return COMBINE_HASH(tryCache(CT.getAsOpaquePtr(), CT));
+}
+
+hash_code IndexRecordHasher::hash(DeclarationName Name) {
+  assert(!Name.isEmpty());
+  // Measurements for using cache or not here, showed significant slowdown when
+  // using the cache for all DeclarationNames when parsing Cocoa, and minor
+  // improvement or no difference for a couple of C++ single translation unit
+  // files. So we avoid caching DeclarationNames.
+  return hashImpl(Name);
+}
+
+hash_code IndexRecordHasher::hash(const NestedNameSpecifier *NNS) {
+  assert(NNS);
+  // Measurements for the C++ single translation unit files did not show much
+  // difference here; choosing to cache them currently.
+  return tryCache(NNS, NNS);
+}
+
+template <typename T>
+hash_code IndexRecordHasher::tryCache(const void *Ptr, T Obj) {
+  auto It = HashByPtr.find(Ptr);
+  if (It != HashByPtr.end())
+    return It->second;
+
+  hash_code Hash = hashImpl(Obj);
+  // hashImpl() may call into tryCache recursively and mutate
+  // HashByPtr, so we use find() earlier and insert the hash with another
+  // lookup here instead of calling insert() earlier and utilizing the iterator
+  // that insert() returns.
+  HashByPtr[Ptr] = Hash;
+  return Hash;
+}
+
+hash_code IndexRecordHasher::hashImpl(const Decl *D) {
+  return DeclHashVisitor(*this).Visit(D);
+}
+
+static hash_code computeHash(const IdentifierInfo *II) {
+  return hash_value(II->getName());
+}
+
+static hash_code computeHash(Selector Sel) {
+  unsigned N = Sel.getNumArgs();
+  if (N == 0)
+    ++N;
+  hash_code Hash = INITIAL_HASH;
+  for (unsigned I = 0; I != N; ++I)
+    if (IdentifierInfo *II = Sel.getIdentifierInfoForSlot(I))
+      COMBINE_HASH(computeHash(II));
+  return Hash;
+}
+
+static hash_code computeHash(TemplateName Name, IndexRecordHasher &Hasher) {
+  hash_code Hash = INITIAL_HASH;
+  if (TemplateDecl *Template = Name.getAsTemplateDecl()) {
+    if (TemplateTemplateParmDecl *TTP
+        = dyn_cast<TemplateTemplateParmDecl>(Template)) {
+      return COMBINE_HASH('t', TTP->getDepth(), TTP->getIndex());
+    }
+
+    return COMBINE_HASH(Hasher.hash(Template->getCanonicalDecl()));
+  }
+
+  // FIXME: Hash dependent template names.
+  return Hash;
+}
+
+static hash_code computeHash(const TemplateArgument &Arg,
+                             IndexRecordHasher &Hasher) {
+  hash_code Hash = INITIAL_HASH;
+
+  switch (Arg.getKind()) {
+  case TemplateArgument::Null:
+    break;
+
+  case TemplateArgument::Declaration:
+    COMBINE_HASH(Hasher.hash(Arg.getAsDecl()));
+    break;
+
+  case TemplateArgument::NullPtr:
+    break;
+
+  case TemplateArgument::TemplateExpansion:
+    COMBINE_HASH('P'); // pack expansion of...
+    // Fall through
+  case TemplateArgument::Template:
+    COMBINE_HASH(computeHash(Arg.getAsTemplateOrTemplatePattern(), Hasher));
+    break;
+      
+  case TemplateArgument::Expression:
+    // FIXME: Hash expressions.
+    break;
+      
+  case TemplateArgument::Pack:
+    COMBINE_HASH('p');
+    for (const auto &P : Arg.pack_elements())
+      COMBINE_HASH(computeHash(P, Hasher));
+    break;
+      
+  case TemplateArgument::Type:
+    COMBINE_HASH(Hasher.hash(Arg.getAsType()));
+    break;
+      
+  case TemplateArgument::Integral:
+    COMBINE_HASH('V', Hasher.hash(Arg.getIntegralType()), Arg.getAsIntegral());
+    break;
+  }
+
+  return Hash;
+}
+
+hash_code IndexRecordHasher::hashImpl(CanQualType CQT) {
+  hash_code Hash = INITIAL_HASH;
+
+  auto asCanon = [](QualType Ty) -> CanQualType {
+    return CanQualType::CreateUnsafe(Ty);
+  };
+
+  const Type *T = CQT.getTypePtr();
+
+  if (const PackExpansionType *Expansion = dyn_cast<PackExpansionType>(T)) {
+    return COMBINE_HASH('P', hash(asCanon(Expansion->getPattern())));
+  }
+  if (const RValueReferenceType *RT = dyn_cast<RValueReferenceType>(T)) {
+    return COMBINE_HASH('%', hash(asCanon(RT->getPointeeType())));
+  }
+  if (const FunctionProtoType *FT = dyn_cast<FunctionProtoType>(T)) {
+    COMBINE_HASH('F', hash(asCanon(FT->getReturnType())));
+    for (const auto &I : FT->param_types())
+      COMBINE_HASH(hash(asCanon(I)));
+    return COMBINE_HASH(FT->isVariadic());
+  }
+  if (const ComplexType *CT = dyn_cast<ComplexType>(T)) {
+    return COMBINE_HASH('<', hash(asCanon(CT->getElementType())));
+  }
+  if (const TemplateSpecializationType *Spec
+      = dyn_cast<TemplateSpecializationType>(T)) {
+    COMBINE_HASH('>', computeHash(Spec->getTemplateName(), *this));
+    for (unsigned I = 0, N = Spec->getNumArgs(); I != N; ++I)
+      COMBINE_HASH(computeHash(Spec->getArg(I), *this));
+    return Hash;
+  }
+  if (const DependentNameType *DNT = dyn_cast<DependentNameType>(T)) {
+    COMBINE_HASH('^');
+    if (const NestedNameSpecifier *NNS = DNT->getQualifier())
+      COMBINE_HASH(hash(NNS));
+    return COMBINE_HASH(computeHash(DNT->getIdentifier()));
+  }
+
+  // Unhandled type.
+  return Hash;
+}
+
+hash_code IndexRecordHasher::hashImpl(DeclarationName Name) {
+  hash_code Hash = INITIAL_HASH;
+  COMBINE_HASH(Name.getNameKind());
+
+  switch (Name.getNameKind()) {
+    case DeclarationName::Identifier:
+      COMBINE_HASH(computeHash(Name.getAsIdentifierInfo()));
+      break;
+    case DeclarationName::ObjCZeroArgSelector:
+    case DeclarationName::ObjCOneArgSelector:
+    case DeclarationName::ObjCMultiArgSelector:
+      COMBINE_HASH(computeHash(Name.getObjCSelector()));
+      break;
+    case DeclarationName::CXXConstructorName:
+    case DeclarationName::CXXDestructorName:
+    case DeclarationName::CXXConversionFunctionName:
+      break;
+    case DeclarationName::CXXOperatorName:
+      COMBINE_HASH(Name.getCXXOverloadedOperator());
+      break;
+    case DeclarationName::CXXLiteralOperatorName:
+      COMBINE_HASH(computeHash(Name.getCXXLiteralIdentifier()));
+    case DeclarationName::CXXUsingDirective:
+    case DeclarationName::CXXDeductionGuideName:
+      break;
+  }
+
+  return Hash;
+}
+
+hash_code IndexRecordHasher::hashImpl(const NestedNameSpecifier *NNS) {
+  hash_code Hash = INITIAL_HASH;
+  if (auto *Pre = NNS->getPrefix())
+    COMBINE_HASH(hash(Pre));
+
+  COMBINE_HASH(NNS->getKind());
+
+  switch (NNS->getKind()) {
+  case NestedNameSpecifier::Identifier:
+    COMBINE_HASH(computeHash(NNS->getAsIdentifier()));
+    break;
+
+  case NestedNameSpecifier::Namespace:
+    COMBINE_HASH(hash(NNS->getAsNamespace()->getCanonicalDecl()));
+    break;
+
+  case NestedNameSpecifier::NamespaceAlias:
+    COMBINE_HASH(hash(NNS->getAsNamespaceAlias()->getCanonicalDecl()));
+    break;
+
+  case NestedNameSpecifier::Global:
+    break;
+
+  case NestedNameSpecifier::Super:
+    break;
+
+  case NestedNameSpecifier::TypeSpecWithTemplate:
+    // Fall through to hash the type.
+
+  case NestedNameSpecifier::TypeSpec:
+    COMBINE_HASH(hash(QualType(NNS->getAsType(), 0)));
+    break;
+  }
+
+  return Hash;
+}

lib/Index/IndexRecordReader.cpp

+//===--- IndexRecordReader.cpp - Index record deserialization -------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Index/IndexRecordReader.h"
+#include "IndexDataStoreUtils.h"
+#include "BitstreamVisitor.h"
+#include "clang/Index/IndexDataStoreSymbolUtils.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/Bitcode/BitstreamReader.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/raw_ostream.h"
+
+using namespace clang;
+using namespace clang::index;
+using namespace clang::index::store;
+using namespace llvm;
+
+struct IndexRecordReader::Implementation {
+  BumpPtrAllocator Allocator;
+  std::unique_ptr<MemoryBuffer> Buffer;
+  llvm::BitstreamCursor DeclCursor;
+  llvm::BitstreamCursor OccurCursor;
+  ArrayRef<uint32_t> DeclOffsets;
+  const IndexRecordDecl **Decls;
+
+  void setDeclOffsets(ArrayRef<uint32_t> Offs) {
+    DeclOffsets = Offs;
+    Decls = Allocator.Allocate<const IndexRecordDecl*>(Offs.size());
+    memset(Decls, 0, sizeof(IndexRecordDecl*)*Offs.size());
+  }
+
+  unsigned getNumDecls() const { return DeclOffsets.size(); }
+
+  const IndexRecordDecl *getDeclByID(unsigned DeclID) {
+    if (DeclID == 0)
+      return nullptr;
+    return getDecl(DeclID-1);
+  }
+
+  const IndexRecordDecl *getDecl(unsigned Index) {
+    assert(Index < getNumDecls());
+    if (const IndexRecordDecl *D = Decls[Index])
+      return D;
+
+    IndexRecordDecl *D = Allocator.Allocate<IndexRecordDecl>();
+    readDecl(Index, *D);
+    Decls[Index] = D;
+    return D;
+  }
+
+  /// Goes through the decls and populates a vector of record decls, based on
+  /// what the given function returns.
+  ///
+  /// The advantage of this function is to allocate memory only for the record
+  /// decls that the caller is interested in.
+  bool searchDecls(llvm::function_ref<DeclSearchCheck> Checker,
+                   llvm::function_ref<void(const IndexRecordDecl *)> Receiver) {
+    for (unsigned I = 0, E = getNumDecls(); I != E; ++I) {
+      if (const IndexRecordDecl *D = Decls[I]) {
+        DeclSearchReturn Ret = Checker(*D);
+        if (Ret.AcceptDecl)
+          Receiver(D);
+        if (!Ret.ContinueSearch)
+          return false;
+        continue;
+      }
+
+      IndexRecordDecl LocalD;
+      readDecl(I, LocalD);
+      DeclSearchReturn Ret = Checker(LocalD);
+      if (Ret.AcceptDecl) {
+        IndexRecordDecl *D = Allocator.Allocate<IndexRecordDecl>();
+        *D = LocalD;
+        Decls[I] = D;
+        Receiver(D);
+      }
+      if (!Ret.ContinueSearch)
+        return false;
+    }
+    return true;
+  }
+
+  void readDecl(unsigned Index, IndexRecordDecl &RecD) {
+    RecordData Record;
+    StringRef Blob;
+    DeclCursor.JumpToBit(DeclOffsets[Index]);
+    unsigned Code = DeclCursor.ReadCode();
+    unsigned RecID = DeclCursor.readRecord(Code, Record, &Blob);
+    assert(RecID == REC_DECLINFO);
+    (void)RecID;
+
+    unsigned I = 0;
+    RecD.DeclID = Index+1;
+    RecD.SymInfo.Kind = getSymbolKind((indexstore_symbol_kind_t)read(Record, I));
+    RecD.SymInfo.SubKind = getSymbolSubKind((indexstore_symbol_subkind_t)read(Record, I));
+    RecD.SymInfo.Lang =
+        getSymbolLanguage((indexstore_symbol_language_t)read(Record, I));
+    RecD.SymInfo.Properties = getSymbolProperties(read(Record, I));
+    RecD.Roles = getSymbolRoles(read(Record, I));
+    RecD.RelatedRoles = getSymbolRoles(read(Record, I));
+    size_t NameLen = read(Record, I);
+    size_t USRLen = read(Record, I);
+    RecD.Name = Blob.substr(0, NameLen);
+    RecD.USR = Blob.substr(NameLen, USRLen);
+    RecD.CodeGenName = Blob.substr(NameLen+USRLen);
+  }
+
+  /// Reads occurrence data.
+  /// \param DeclsFilter if non-empty indicates the list of decls that we want
+  /// to get occurrences for. If empty then indicates that we want occurrences
+  /// for all decls.
+  /// \param RelatedDeclsFilter Same as \c DeclsFilter but for related decls.
+  /// \returns true if the occurrence info was filled out, false if occurrence
+  /// was ignored.
+  bool readOccurrence(RecordDataImpl &Record, StringRef Blob,
+                      ArrayRef<const IndexRecordDecl *> DeclsFilter,
+                      ArrayRef<const IndexRecordDecl *> RelatedDeclsFilter,
+                      IndexRecordOccurrence &RecOccur) {
+
+    auto isDeclIDContained = [](unsigned DeclID,
+                                ArrayRef<const IndexRecordDecl *> Ds) -> bool {
+      if (Ds.empty())
+        return true; // empty means accept all.
+      auto pred = [DeclID](const IndexRecordDecl *D) { return D->DeclID == DeclID; };
+      return std::find_if(Ds.begin(), Ds.end(), pred) != Ds.end();
+    };
+
+    unsigned I = 0;
+    unsigned DeclID = read(Record, I);
+    if (!isDeclIDContained(DeclID, DeclsFilter))
+      return false;
+
+    if (!RelatedDeclsFilter.empty()) {
+      unsigned RelI = I+3;
+      unsigned NumRelated = read(Record, RelI);
+      bool FoundRelated = false;
+      while (NumRelated--) {
+        ++RelI; // roles;
+        unsigned RelDID = read(Record, RelI);
+        if (isDeclIDContained(RelDID, RelatedDeclsFilter)) {
+          FoundRelated = true;
+          break;
+        }
+      }
+      if (!FoundRelated)
+        return false;
+    }
+
+    RecOccur.Dcl = getDeclByID(DeclID);
+    RecOccur.Roles = getSymbolRoles(read(Record, I));
+    RecOccur.Line = read(Record, I);
+    RecOccur.Column = read(Record, I);
+
+    unsigned NumRelated = read(Record, I);
+    while (NumRelated--) {
+      SymbolRoleSet RelRoles = getSymbolRoles(read(Record, I));
+      const IndexRecordDecl *RelD = getDeclByID(read(Record, I));
+      RecOccur.Relations.emplace_back(RelRoles, RelD);
+    }
+
+    return true;
+  }
+
+  bool foreachDecl(bool NoCache,
+                   function_ref<bool(const IndexRecordDecl *)> Receiver) {
+    for (unsigned I = 0, E = getNumDecls(); I != E; ++I) {
+      if (const IndexRecordDecl *D = Decls[I]) {
+        if (!Receiver(D))
+          return false;
+        continue;
+      }
+
+      if (NoCache) {
+        IndexRecordDecl LocalD;
+        readDecl(I, LocalD);
+        if (!Receiver(&LocalD))
+          return false;
+      } else {
+        if (!Receiver(getDecl(I)))
+          return false;
+      }
+    }
+    return true;
+  }
+
+  bool foreachOccurrence(ArrayRef<const IndexRecordDecl *> DeclsFilter,
+                         ArrayRef<const IndexRecordDecl *> RelatedDeclsFilter,
+                         function_ref<bool(const IndexRecordOccurrence &)> Receiver) {
+    class OccurBitVisitor : public BitstreamVisitor<OccurBitVisitor> {
+      IndexRecordReader::Implementation &Reader;
+      ArrayRef<const IndexRecordDecl *> DeclsFilter;
+      ArrayRef<const IndexRecordDecl *> RelatedDeclsFilter;
+      function_ref<bool(const IndexRecordOccurrence &)> Receiver;
+
+    public:
+      OccurBitVisitor(llvm::BitstreamCursor &Stream,
+                      IndexRecordReader::Implementation &Reader,
+                      ArrayRef<const IndexRecordDecl *> DeclsFilter,
+                      ArrayRef<const IndexRecordDecl *> RelatedDeclsFilter,
+                      function_ref<bool(const IndexRecordOccurrence &)> Receiver)
+        : BitstreamVisitor(Stream),
+          Reader(Reader),
+          DeclsFilter(DeclsFilter),
+          RelatedDeclsFilter(RelatedDeclsFilter),
+          Receiver(std::move(Receiver)) {}
+
+      StreamVisit visitRecord(unsigned BlockID, unsigned RecID,
+                              RecordDataImpl &Record, StringRef Blob) {
+        assert(RecID == REC_DECLOCCURRENCE);
+        IndexRecordOccurrence RecOccur;
+        if (Reader.readOccurrence(Record, Blob, DeclsFilter, RelatedDeclsFilter,
+                                   RecOccur))
+          if (!Receiver(RecOccur))
+            return StreamVisit::Abort;
+        return StreamVisit::Continue;
+      }
+    };
+
+    SavedStreamPosition SavedPosition(OccurCursor);
+    OccurBitVisitor Visitor(OccurCursor, *this, DeclsFilter, RelatedDeclsFilter,
+                            Receiver);
+    std::string Error;
+    return Visitor.visit(Error);
+  }
+
+  bool foreachOccurrenceInLineRange(unsigned lineStart, unsigned lineCount,
+            llvm::function_ref<bool(const IndexRecordOccurrence &)> receiver) {
+    // FIXME: Use binary search and make this more efficient.
+    unsigned lineEnd = lineStart+lineCount;
+    return foreachOccurrence(None, None, [&](const IndexRecordOccurrence &occur) -> bool {
+      if (occur.Line > lineEnd)
+        return false; // we're done.
+      if (occur.Line >= lineStart) {
+        if (!receiver(occur))
+          return false;
+      }
+      return true;
+    });
+  }
+
+  static uint64_t read(RecordDataImpl &Record, unsigned &I) {
+    return Record[I++];
+  }
+};
+
+namespace {
+
+class IndexBitstreamVisitor : public BitstreamVisitor<IndexBitstreamVisitor> {
+  IndexRecordReader::Implementation &Reader;
+
+public:
+  IndexBitstreamVisitor(llvm::BitstreamCursor &Stream,
+                        IndexRecordReader::Implementation &Reader)
+    : BitstreamVisitor(Stream), Reader(Reader) {}
+
+  StreamVisit visitBlock(unsigned ID) {
+    switch ((RecordBitBlock)ID) {
+    case REC_VERSION_BLOCK_ID:
+    case REC_DECLOFFSETS_BLOCK_ID:
+      return StreamVisit::Continue;
+
+    case REC_DECLS_BLOCK_ID:
+      Reader.DeclCursor = Stream;
+      if (Reader.DeclCursor.EnterSubBlock(ID)) {
+        *Error = "malformed block record";
+        return StreamVisit::Abort;
+      }
+      readBlockAbbrevs(Reader.DeclCursor);
+      return StreamVisit::Skip;
+
+    case REC_DECLOCCURRENCES_BLOCK_ID:
+      Reader.OccurCursor = Stream;
+      if (Reader.OccurCursor.EnterSubBlock(ID)) {
+        *Error = "malformed block record";
+        return StreamVisit::Abort;
+      }
+      readBlockAbbrevs(Reader.OccurCursor);
+      return StreamVisit::Skip;
+    }
+
+    // Some newly introduced block in a minor version update that we cannot
+    // handle.
+    return StreamVisit::Skip;
+  }
+
+  StreamVisit visitRecord(unsigned BlockID, unsigned RecID,
+                          RecordDataImpl &Record, StringRef Blob) {
+    switch (BlockID) {
+    case REC_VERSION_BLOCK_ID: {
+      unsigned StoreFormatVersion = Record[0];
+      if (StoreFormatVersion != STORE_FORMAT_VERSION) {
+        llvm::raw_string_ostream OS(*Error);
+        OS << "Store format version mismatch: " << StoreFormatVersion;
+        OS << " , expected: " << STORE_FORMAT_VERSION;
+        return StreamVisit::Abort;
+      }
+      break;
+    }
+    case REC_DECLOFFSETS_BLOCK_ID:
+      assert(RecID == REC_DECLOFFSETS);
+      Reader.setDeclOffsets(makeArrayRef((uint32_t*)Blob.data(), Record[0]));
+      break;
+
+    case REC_DECLS_BLOCK_ID:
+    case REC_DECLOCCURRENCES_BLOCK_ID:
+      llvm_unreachable("shouldn't visit this block'");
+    }
+    return StreamVisit::Continue;
+  }
+};
+
+} // anonymous namespace
+
+std::unique_ptr<IndexRecordReader>
+IndexRecordReader::createWithRecordFilename(StringRef RecordFilename,
+                                            StringRef StorePath,
+                                            std::string &Error) {
+  SmallString<128> PathBuf = StorePath;
+  appendRecordSubDir(PathBuf);
+  appendInteriorRecordPath(RecordFilename, PathBuf);
+  return createWithFilePath(PathBuf.str(), Error);
+}
+
+std::unique_ptr<IndexRecordReader>
+IndexRecordReader::createWithFilePath(StringRef FilePath, std::string &Error) {
+  auto ErrOrBuf = MemoryBuffer::getFile(FilePath, /*FileSize=*/-1,
+                                        /*RequiresNullTerminator=*/false);
+  if (!ErrOrBuf) {
+    raw_string_ostream(Error) << "failed opening index record '"
+      << FilePath << "': " << ErrOrBuf.getError().message();
+    return nullptr;
+  }
+  return createWithBuffer(std::move(*ErrOrBuf), Error);
+}
+
+std::unique_ptr<IndexRecordReader>
+IndexRecordReader::createWithBuffer(std::unique_ptr<llvm::MemoryBuffer> Buffer,
+                                    std::string &Error) {
+
+  std::unique_ptr<IndexRecordReader> Reader;
+  Reader.reset(new IndexRecordReader());
+  auto &Impl = Reader->Impl;
+  Impl.Buffer = std::move(Buffer);
+  llvm::BitstreamCursor Stream(*Impl.Buffer);
+
+  // Sniff for the signature.
+  if (Stream.Read(8) != 'I' ||
+      Stream.Read(8) != 'D' ||
+      Stream.Read(8) != 'X' ||
+      Stream.Read(8) != 'R') {
+    Error = "not a serialized index record file";
+    return nullptr;
+  }
+
+  IndexBitstreamVisitor BitVisitor(Stream, Impl);
+  if (!BitVisitor.visit(Error))
+    return nullptr;
+
+  return Reader;
+}
+
+IndexRecordReader::IndexRecordReader()
+  : Impl(*new Implementation()) {
+
+}
+
+IndexRecordReader::~IndexRecordReader() {
+  delete &Impl;
+}
+
+bool IndexRecordReader::searchDecls(
+                        llvm::function_ref<DeclSearchCheck> Checker,
+                  llvm::function_ref<void(const IndexRecordDecl *)> Receiver) {
+  return Impl.searchDecls(std::move(Checker), std::move(Receiver));
+}
+
+bool IndexRecordReader::foreachDecl(bool NoCache,
+                                    function_ref<bool(const IndexRecordDecl *)> Receiver) {
+  return Impl.foreachDecl(NoCache, std::move(Receiver));
+}
+
+bool IndexRecordReader::foreachOccurrence(
+                  ArrayRef<const IndexRecordDecl *> DeclsFilter,
+                  ArrayRef<const IndexRecordDecl *> RelatedDeclsFilter,
+                  function_ref<bool(const IndexRecordOccurrence &)> Receiver) {
+  return Impl.foreachOccurrence(DeclsFilter, RelatedDeclsFilter,
+                                std::move(Receiver));
+}
+
+bool IndexRecordReader::foreachOccurrence(
+            llvm::function_ref<bool(const IndexRecordOccurrence &)> Receiver) {
+  return foreachOccurrence(None, None, std::move(Receiver));
+}
+
+bool IndexRecordReader::foreachOccurrenceInLineRange(unsigned lineStart,
+                                                     unsigned lineCount,
+             llvm::function_ref<bool(const IndexRecordOccurrence &)> Receiver) {
+  return Impl.foreachOccurrenceInLineRange(lineStart, lineCount, Receiver);
+}

lib/Index/IndexRecordWriter.cpp

+//===--- IndexRecordWriter.cpp - Index record serialization ---------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Index/IndexRecordWriter.h"
+#include "IndexDataStoreUtils.h"
+#include "indexstore/indexstore.h"
+#include "clang/Index/IndexDataStoreSymbolUtils.h"
+#include "llvm/ADT/APInt.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/StringSet.h"
+#include "llvm/Bitcode/BitstreamWriter.h"
+#include "llvm/Support/Errc.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/raw_ostream.h"
+
+using namespace clang;
+using namespace clang::index;
+using namespace clang::index::store;
+using namespace llvm;
+
+using writer::OpaqueDecl;
+
+namespace {
+struct DeclInfo {
+  OpaqueDecl D;
+  SymbolRoleSet Roles;
+  SymbolRoleSet RelatedRoles;
+};
+
+struct OccurrenceInfo {
+  unsigned DeclID;
+  OpaqueDecl D;
+  SymbolRoleSet Roles;
+  unsigned Line;
+  unsigned Column;
+  SmallVector<std::pair<writer::SymbolRelation, unsigned>, 4> Related;
+};
+
+struct RecordState {
+  std::string RecordPath;
+  SmallString<512> Buffer;
+  BitstreamWriter Stream;
+
+  DenseMap<OpaqueDecl, unsigned> IndexForDecl;
+  std::vector<DeclInfo> Decls;
+  std::vector<OccurrenceInfo> Occurrences;
+
+  RecordState(std::string &&RecordPath)
+      : RecordPath(std::move(RecordPath)), Stream(Buffer) {}
+};
+} // end anonymous namespace
+
+static void writeBlockInfo(BitstreamWriter &Stream) {
+  RecordData Record;
+
+  Stream.EnterBlockInfoBlock();
+#define BLOCK(X) emitBlockID(X ## _ID, #X, Stream, Record)
+#define RECORD(X) emitRecordID(X, #X, Stream, Record)
+
+  BLOCK(REC_VERSION_BLOCK);
+  RECORD(REC_VERSION);
+
+  BLOCK(REC_DECLS_BLOCK);
+  RECORD(REC_DECLINFO);
+
+  BLOCK(REC_DECLOFFSETS_BLOCK);
+  RECORD(REC_DECLOFFSETS);
+
+  BLOCK(REC_DECLOCCURRENCES_BLOCK);
+  RECORD(REC_DECLOCCURRENCE);
+
+#undef RECORD
+#undef BLOCK
+  Stream.ExitBlock();
+}
+
+static void writeVersionInfo(BitstreamWriter &Stream) {
+  using namespace llvm::sys;
+
+  Stream.EnterSubblock(REC_VERSION_BLOCK_ID, 3);
+
+  auto Abbrev = std::make_shared<BitCodeAbbrev>();
+  Abbrev->Add(BitCodeAbbrevOp(REC_VERSION));
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 6)); // Store format version
+  unsigned AbbrevCode = Stream.EmitAbbrev(std::move(Abbrev));
+
+  RecordData Record;
+  Record.push_back(REC_VERSION);
+  Record.push_back(STORE_FORMAT_VERSION);
+  Stream.EmitRecordWithAbbrev(AbbrevCode, Record);
+
+  Stream.ExitBlock();
+}
+
+template <typename T, typename Allocator>
+static StringRef data(const std::vector<T, Allocator> &v) {
+  if (v.empty())
+    return StringRef();
+  return StringRef(reinterpret_cast<const char *>(&v[0]), sizeof(T) * v.size());
+}
+
+template <typename T> static StringRef data(const SmallVectorImpl<T> &v) {
+  return StringRef(reinterpret_cast<const char *>(v.data()),
+                   sizeof(T) * v.size());
+}
+
+static void writeDecls(BitstreamWriter &Stream, ArrayRef<DeclInfo> Decls,
+                       ArrayRef<OccurrenceInfo> Occurrences,
+                       writer::SymbolWriterCallback GetSymbolForDecl) {
+  SmallVector<uint32_t, 32> DeclOffsets;
+  DeclOffsets.reserve(Decls.size());
+
+  //===--------------------------------------------------------------------===//
+  // DECLS_BLOCK_ID
+  //===--------------------------------------------------------------------===//
+
+  Stream.EnterSubblock(REC_DECLS_BLOCK_ID, 3);
+
+  auto Abbrev = std::make_shared<BitCodeAbbrev>();
+  Abbrev->Add(BitCodeAbbrevOp(REC_DECLINFO));
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 5)); // Kind
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 5)); // SubKind
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 5)); // Language
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, SymbolPropertyBitNum)); // Properties
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, SymbolRoleBitNum)); // Roles
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, SymbolRoleBitNum)); // Related Roles
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 6)); // Length of name in block
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 6)); // Length of USR in block
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Blob)); // Name + USR + CodeGen symbol name
+  unsigned AbbrevCode = Stream.EmitAbbrev(std::move(Abbrev));
+
+#ifndef NDEBUG
+  StringSet<> USRSet;
+#endif
+
+  RecordData Record;
+  llvm::SmallString<256> Blob;
+  llvm::SmallString<256> Scratch;
+  for (auto &Info : Decls) {
+    DeclOffsets.push_back(Stream.GetCurrentBitNo());
+    Blob.clear();
+    Scratch.clear();
+
+    writer::Symbol SymInfo = GetSymbolForDecl(Info.D, Scratch);
+    assert(SymInfo.SymInfo.Kind != SymbolKind::Unknown);
+    assert(!SymInfo.USR.empty() && "Recorded decl without USR!");
+
+    Blob += SymInfo.Name;
+    Blob += SymInfo.USR;
+    Blob += SymInfo.CodeGenName;
+
+#ifndef NDEBUG
+    bool IsNew = USRSet.insert(SymInfo.USR).second;
+    if (!IsNew) {
+      llvm::errs() << "Index: Duplicate USR! " << SymInfo.USR << "\n";
+      // FIXME: print more information so it's easier to find the declaration.
+    }
+#endif
+
+    Record.clear();
+    Record.push_back(REC_DECLINFO);
+    Record.push_back(getIndexStoreKind(SymInfo.SymInfo.Kind));
+    Record.push_back(getIndexStoreSubKind(SymInfo.SymInfo.SubKind));
+    Record.push_back(getIndexStoreLang(SymInfo.SymInfo.Lang));
+    Record.push_back(getIndexStoreProperties(SymInfo.SymInfo.Properties));
+    Record.push_back(getIndexStoreRoles(Info.Roles));
+    Record.push_back(getIndexStoreRoles(Info.RelatedRoles));
+    Record.push_back(SymInfo.Name.size());
+    Record.push_back(SymInfo.USR.size());
+    Stream.EmitRecordWithBlob(AbbrevCode, Record, Blob);
+  }
+
+  Stream.ExitBlock();
+
+  //===--------------------------------------------------------------------===//
+  // DECLOFFSETS_BLOCK_ID
+  //===--------------------------------------------------------------------===//
+
+  Stream.EnterSubblock(REC_DECLOFFSETS_BLOCK_ID, 3);
+
+  Abbrev = std::make_shared<BitCodeAbbrev>();
+  Abbrev->Add(BitCodeAbbrevOp(REC_DECLOFFSETS));
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 32)); // Number of Decls
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Blob)); // Offsets array
+  AbbrevCode = Stream.EmitAbbrev(std::move(Abbrev));
+
+  Record.clear();
+  Record.push_back(REC_DECLOFFSETS);
+  Record.push_back(DeclOffsets.size());
+  Stream.EmitRecordWithBlob(AbbrevCode, Record, data(DeclOffsets));
+
+  Stream.ExitBlock();
+
+  //===--------------------------------------------------------------------===//
+  // DECLOCCURRENCES_BLOCK_ID
+  //===--------------------------------------------------------------------===//
+
+  Stream.EnterSubblock(REC_DECLOCCURRENCES_BLOCK_ID, 3);
+
+  Abbrev = std::make_shared<BitCodeAbbrev>();
+  Abbrev->Add(BitCodeAbbrevOp(REC_DECLOCCURRENCE));
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 8)); // Decl ID
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, SymbolRoleBitNum)); // Roles
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 12)); // Line
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 8)); // Column
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 4)); // Num related
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Array)); // Related Roles/IDs
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 16)); // Roles or ID
+  AbbrevCode = Stream.EmitAbbrev(std::move(Abbrev));
+
+  for (auto &Occur : Occurrences) {
+    Record.clear();
+    Record.push_back(REC_DECLOCCURRENCE);
+    Record.push_back(Occur.DeclID);
+    Record.push_back(getIndexStoreRoles(Occur.Roles));
+    Record.push_back(Occur.Line);
+    Record.push_back(Occur.Column);
+    Record.push_back(Occur.Related.size());
+    for (auto &Rel : Occur.Related) {
+      Record.push_back(getIndexStoreRoles(Rel.first.Roles));
+      Record.push_back(Rel.second);
+    }
+    Stream.EmitRecordWithAbbrev(AbbrevCode, Record);
+  }
+  Stream.ExitBlock();
+}
+
+IndexRecordWriter::IndexRecordWriter(StringRef IndexPath)
+    : RecordsPath(IndexPath) {
+  store::appendRecordSubDir(RecordsPath);
+}
+
+IndexRecordWriter::Result
+IndexRecordWriter::beginRecord(StringRef Filename, hash_code RecordHash,
+                               std::string &Error, std::string *OutRecordFile) {
+  using namespace llvm::sys;
+  assert(!Record && "called beginRecord before calling endRecord on previous");
+
+  std::string RecordName;
+  {
+    llvm::raw_string_ostream RN(RecordName);
+    RN << path::filename(Filename);
+    RN << "-" << APInt(64, RecordHash).toString(36, /*Signed=*/false);
+  }
+  SmallString<256> RecordPath = RecordsPath.str();
+  appendInteriorRecordPath(RecordName, RecordPath);
+
+  if (OutRecordFile)
+    *OutRecordFile = RecordName;
+
+  if (std::error_code EC =
+          fs::access(RecordPath.c_str(), fs::AccessMode::Exist)) {
+    if (EC != errc::no_such_file_or_directory) {
+      llvm::raw_string_ostream Err(Error);
+      Err << "could not access record '" << RecordPath
+          << "': " << EC.message();
+      return Result::Failure;
+    }
+  } else {
+    return Result::AlreadyExists;
+  }
+
+  // Write the record header.
+  auto *State = new RecordState(RecordPath.str());
+  Record = State;
+  llvm::BitstreamWriter &Stream = State->Stream;
+  Stream.Emit('I', 8);
+  Stream.Emit('D', 8);
+  Stream.Emit('X', 8);
+  Stream.Emit('R', 8);
+
+  writeBlockInfo(Stream);
+  writeVersionInfo(Stream);
+
+  return Result::Success;
+}
+
+IndexRecordWriter::Result
+IndexRecordWriter::endRecord(std::string &Error,
+                             writer::SymbolWriterCallback GetSymbolForDecl) {
+  assert(Record && "called endRecord without calling beginRecord");
+  auto &State = *static_cast<RecordState *>(Record);
+  Record = nullptr;
+  struct ScopedDelete {
+    RecordState *S;
+    ScopedDelete(RecordState *S) : S(S) {}
+    ~ScopedDelete() { delete S; }
+  } Deleter(&State);
+
+  if (!State.Decls.empty()) {
+    writeDecls(State.Stream, State.Decls, State.Occurrences, GetSymbolForDecl);
+  }
+
+  if (std::error_code EC = sys::fs::create_directory(sys::path::parent_path(State.RecordPath))) {
+    llvm::raw_string_ostream Err(Error);
+    Err << "failed to create directory '" << sys::path::parent_path(State.RecordPath) << "': " << EC.message();
+    return Result::Failure;
+  }
+
+  // Create a unique file to write to so that we can move the result into place
+  // atomically. If this process crashes we don't want to interfere with any
+  // other concurrent processes.
+  SmallString<128> TempPath(State.RecordPath);
+  TempPath += "-temp-%%%%%%%%";
+  int TempFD;
+  if (sys::fs::createUniqueFile(TempPath.str(), TempFD, TempPath)) {
+    llvm::raw_string_ostream Err(Error);
+    Err << "failed to create temporary file: " << TempPath;
+    return Result::Failure;
+  }
+
+  raw_fd_ostream OS(TempFD, /*shouldClose=*/true);
+  OS.write(State.Buffer.data(), State.Buffer.size());
+  OS.close();
+
+  // Atomically move the unique file into place.
+  if (std::error_code EC =
+          sys::fs::rename(TempPath.c_str(), State.RecordPath.c_str())) {
+    llvm::raw_string_ostream Err(Error);
+    Err << "failed to rename '" << TempPath << "' to '" << State.RecordPath << "': " << EC.message();
+    return Result::Failure;
+  }
+
+  return Result::Success;
+}
+
+void IndexRecordWriter::addOccurrence(
+    OpaqueDecl D, SymbolRoleSet Roles, unsigned Line, unsigned Column,
+    ArrayRef<writer::SymbolRelation> Related) {
+  assert(Record && "called addOccurrence without calling beginRecord");
+  auto &State = *static_cast<RecordState *>(Record);
+
+  auto insertDecl = [&](OpaqueDecl D, SymbolRoleSet Roles,
+                        SymbolRoleSet RelatedRoles) -> unsigned {
+    auto Insert =
+        State.IndexForDecl.insert(std::make_pair(D, State.Decls.size()));
+    unsigned Index = Insert.first->second;
+
+    if (Insert.second) {
+      State.Decls.push_back(DeclInfo{D, Roles, RelatedRoles});
+    } else {
+      State.Decls[Index].Roles |= Roles;
+      State.Decls[Index].RelatedRoles |= RelatedRoles;
+    }
+    return Index + 1;
+  };
+
+  unsigned DeclID = insertDecl(D, Roles, SymbolRoleSet());
+
+  decltype(OccurrenceInfo::Related) RelatedDecls;
+  RelatedDecls.reserve(Related.size());
+  for (auto &Rel : Related) {
+    unsigned ID = insertDecl(Rel.RelatedSymbol, SymbolRoleSet(), Rel.Roles);
+    RelatedDecls.emplace_back(Rel, ID);
+  }
+
+  State.Occurrences.push_back(
+      OccurrenceInfo{DeclID, D, Roles, Line, Column, std::move(RelatedDecls)});
+}

lib/Index/IndexUnitReader.cpp

+//===--- IndexUnitReader.cpp - Index unit deserialization -----------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Index/IndexUnitReader.h"
+#include "IndexDataStoreUtils.h"
+#include "BitstreamVisitor.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/Optional.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Bitcode/BitstreamReader.h"
+#include "llvm/Support/Chrono.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/raw_ostream.h"
+
+#include <unistd.h>
+
+using namespace clang;
+using namespace clang::index;
+using namespace clang::index::store;
+using namespace llvm;
+
+namespace {
+
+typedef function_ref<bool(const IndexUnitReader::DependencyInfo &)> DependencyReceiver;
+typedef function_ref<bool(const IndexUnitReader::IncludeInfo &)> IncludeReceiver;
+
+class IndexUnitReaderImpl {
+  sys::TimePoint<> ModTime;
+  std::unique_ptr<MemoryBuffer> MemBuf;
+
+public:
+  StringRef ProviderIdentifier;
+  StringRef ProviderVersion;
+  llvm::BitstreamCursor DependCursor;
+  llvm::BitstreamCursor IncludeCursor;
+  bool IsSystemUnit;
+  bool IsModuleUnit;
+  bool IsDebugCompilation;
+  StringRef WorkingDir;
+  StringRef OutputFile;
+  StringRef SysrootPath;
+  StringRef ModuleName;
+  SmallString<128> MainFilePath;
+  StringRef Target;
+  std::vector<FileBitPath> Paths;
+  StringRef PathsBuffer;
+
+  struct ModuleInfo {
+    unsigned NameOffset;
+    unsigned NameSize;
+  };
+  std::vector<ModuleInfo> Modules;
+  StringRef ModuleNamesBuffer;
+
+  bool init(std::unique_ptr<MemoryBuffer> Buf, sys::TimePoint<> ModTime,
+            std::string &Error);
+
+  StringRef getProviderIdentifier() const { return ProviderIdentifier; }
+  StringRef getProviderVersion() const { return ProviderVersion; }
+
+  sys::TimePoint<> getModificationTime() const { return ModTime; }
+  StringRef getWorkingDirectory() const { return WorkingDir; }
+  StringRef getOutputFile() const { return OutputFile; }
+  StringRef getSysrootPath() const { return SysrootPath; }
+  StringRef getTarget() const { return Target; }
+
+  StringRef getModuleName() const { return ModuleName; }
+  StringRef getMainFilePath() const { return MainFilePath.str(); }
+  bool hasMainFile() const { return !MainFilePath.empty(); }
+  bool isSystemUnit() const { return IsSystemUnit; }
+  bool isModuleUnit() const { return IsModuleUnit; }
+  bool isDebugCompilation() const { return IsDebugCompilation; }
+
+  /// Unit dependencies are provided ahead of record ones, record ones
+  /// ahead of the file ones.
+  bool foreachDependency(DependencyReceiver Receiver);
+
+  bool foreachInclude(IncludeReceiver Receiver);
+
+  StringRef getPathFromBuffer(size_t Offset, size_t Size) {
+    return PathsBuffer.substr(Offset, Size);
+  }
+
+  void constructFilePath(SmallVectorImpl<char> &Path, int PathIndex);
+
+  StringRef getModuleName(int ModuleIndex);
+};
+
+class IndexUnitBitstreamVisitor : public BitstreamVisitor<IndexUnitBitstreamVisitor> {
+  IndexUnitReaderImpl &Reader;
+  size_t WorkDirOffset;
+  size_t WorkDirSize;
+  size_t OutputFileOffset;
+  size_t OutputFileSize;
+  size_t SysrootOffset;
+  size_t SysrootSize;
+  int MainPathIndex;
+
+public:
+  IndexUnitBitstreamVisitor(llvm::BitstreamCursor &Stream,
+                            IndexUnitReaderImpl &Reader)
+    : BitstreamVisitor(Stream), Reader(Reader) {}
+
+  StreamVisit visitBlock(unsigned ID) {
+    switch ((UnitBitBlock)ID) {
+    case UNIT_VERSION_BLOCK_ID:
+    case UNIT_INFO_BLOCK_ID:
+    case UNIT_PATHS_BLOCK_ID:
+    case UNIT_MODULES_BLOCK_ID:
+      return StreamVisit::Continue;
+
+    case UNIT_DEPENDENCIES_BLOCK_ID:
+      Reader.DependCursor = Stream;
+      if (Reader.DependCursor.EnterSubBlock(ID)) {
+        *Error = "malformed unit dependencies block record";
+        return StreamVisit::Abort;
+      }
+      readBlockAbbrevs(Reader.DependCursor);
+      return StreamVisit::Skip;
+    case UNIT_INCLUDES_BLOCK_ID:
+      Reader.IncludeCursor = Stream;
+      if (Reader.IncludeCursor.EnterSubBlock(ID)) {
+        *Error = "malformed unit includes block record";
+        return StreamVisit::Abort;
+      }
+      readBlockAbbrevs(Reader.IncludeCursor);
+      return StreamVisit::Skip;
+    }
+
+    // Some newly introduced block in a minor version update that we cannot
+    // handle.
+    return StreamVisit::Skip;
+  }
+
+  StreamVisit visitRecord(unsigned BlockID, unsigned RecID,
+                          RecordDataImpl &Record, StringRef Blob) {
+    switch (BlockID) {
+    case UNIT_VERSION_BLOCK_ID: {
+      unsigned StoreFormatVersion = Record[0];
+      if (StoreFormatVersion != STORE_FORMAT_VERSION) {
+        llvm::raw_string_ostream OS(*Error);
+        OS << "Store format version mismatch: " << StoreFormatVersion;
+        OS << " , expected: " << STORE_FORMAT_VERSION;
+        return StreamVisit::Abort;
+      }
+      break;
+    }
+
+    case UNIT_INFO_BLOCK_ID: {
+      assert(RecID == UNIT_INFO);
+      unsigned I = 0;
+      Reader.IsSystemUnit = Record[I++];
+
+      // Save these to lookup them up after we get the paths buffer.
+      WorkDirOffset = Record[I++];
+      WorkDirSize = Record[I++];
+      OutputFileOffset = Record[I++];
+      OutputFileSize = Record[I++];
+      SysrootOffset = Record[I++];
+      SysrootSize = Record[I++];
+      MainPathIndex = (int)Record[I++] - 1;
+      Reader.IsDebugCompilation = Record[I++];
+      Reader.IsModuleUnit = Record[I++];
+
+      size_t moduleNameSize = Record[I++];
+      size_t providerIdentifierSize = Record[I++];
+      size_t providerVersionSize = Record[I++];
+      I++; // Reserved for ProviderDataVersion.
+      Reader.ModuleName = Blob.substr(0, moduleNameSize);
+      Blob = Blob.drop_front(moduleNameSize);
+      Reader.ProviderIdentifier = Blob.substr(0, providerIdentifierSize);
+      Blob = Blob.drop_front(providerIdentifierSize);
+      Reader.ProviderVersion = Blob.substr(0, providerVersionSize);
+      Reader.Target = Blob.drop_front(providerVersionSize);
+      break;
+    }
+
+    case UNIT_PATHS_BLOCK_ID:
+      switch (RecID) {
+      case UNIT_PATH:
+        {
+          unsigned I = 0;
+          UnitFilePathPrefixKind Kind = (UnitFilePathPrefixKind)Record[I++];
+          size_t DirOffset = Record[I++];
+          size_t DirSize = Record[I++];
+          size_t FilenameOffset = Record[I++];
+          size_t FilenameSize = Record[I++];
+
+          Reader.Paths.emplace_back(Kind, BitPathComponent(DirOffset, DirSize),
+                                  BitPathComponent(FilenameOffset, FilenameSize));
+        }
+        break;
+      case UNIT_PATH_BUFFER:
+        Reader.PathsBuffer = Blob;
+        Reader.WorkingDir = Reader.getPathFromBuffer(WorkDirOffset, WorkDirSize);
+        Reader.OutputFile = Reader.getPathFromBuffer(OutputFileOffset, OutputFileSize);
+        Reader.SysrootPath = Reader.getPathFromBuffer(SysrootOffset, SysrootSize);
+
+        // now we can populate the main file's path
+        Reader.constructFilePath(Reader.MainFilePath, MainPathIndex);
+        break;
+      default:
+          llvm_unreachable("shouldn't visit this record");
+      }
+      break;
+
+    case UNIT_MODULES_BLOCK_ID:
+      switch (RecID) {
+      case UNIT_MODULE:
+        {
+          unsigned I = 0;
+          unsigned NameOffset = Record[I++];
+          unsigned NameSize = Record[I++];
+          Reader.Modules.push_back({NameOffset, NameSize});
+        }
+        break;
+      case UNIT_MODULE_BUFFER:
+        Reader.ModuleNamesBuffer = Blob;
+        break;
+      default:
+          llvm_unreachable("shouldn't visit this record");
+      }
+      break;
+
+    case UNIT_DEPENDENCIES_BLOCK_ID:
+    case UNIT_INCLUDES_BLOCK_ID:
+      llvm_unreachable("shouldn't visit this block'");
+    }
+    return StreamVisit::Continue;
+  }
+};
+
+typedef std::function<bool(RecordDataImpl& Record, StringRef Blob)>
+  BlockVisitorCallback;
+
+class IndexUnitBlockBitstreamVisitor : public BitstreamVisitor<IndexUnitBlockBitstreamVisitor> {
+  unsigned RecID;
+  BlockVisitorCallback Visit;
+
+public:
+  IndexUnitBlockBitstreamVisitor(unsigned RecID,
+                                 llvm::BitstreamCursor &BlockStream,
+                                 BlockVisitorCallback Visit)
+  : BitstreamVisitor(BlockStream), RecID(RecID), Visit(std::move(Visit)) {}
+
+  StreamVisit visitRecord(unsigned BlockID, unsigned RecID,
+                          RecordDataImpl &Record, StringRef Blob) {
+    if (RecID != this->RecID)
+      llvm_unreachable("shouldn't be called with this RecID");
+
+    if (Visit(Record, Blob))
+      return StreamVisit::Continue;
+    return StreamVisit::Abort;
+  }
+};
+
+} // anonymous namespace
+
+bool IndexUnitReaderImpl::init(std::unique_ptr<MemoryBuffer> Buf,
+                               sys::TimePoint<> ModTime, std::string &Error) {
+  this->ModTime = ModTime;
+  this->MemBuf = std::move(Buf);
+  llvm::BitstreamCursor Stream(*MemBuf);
+
+  // Sniff for the signature.
+  if (Stream.Read(8) != 'I' ||
+      Stream.Read(8) != 'D' ||
+      Stream.Read(8) != 'X' ||
+      Stream.Read(8) != 'U') {
+    Error = "not a serialized index unit file";
+    return true;
+  }
+
+  IndexUnitBitstreamVisitor BitVisitor(Stream, *this);
+  return !BitVisitor.visit(Error);
+}
+
+/// Unit dependencies are provided ahead of record ones, record ones
+/// ahead of the file ones.
+bool IndexUnitReaderImpl::foreachDependency(DependencyReceiver Receiver) {
+  store::SavedStreamPosition SavedDepPosition(DependCursor);
+  IndexUnitBlockBitstreamVisitor Visitor(UNIT_DEPENDENCY, DependCursor,
+  [&](RecordDataImpl& Record, StringRef Blob) {
+    unsigned I = 0;
+    UnitDependencyKind UnitDepKind = (UnitDependencyKind)Record[I++];
+    bool IsSystem = Record[I++];
+    int PathIndex = (int)Record[I++] - 1;
+    int ModuleIndex = (int)Record[I++] - 1;
+    time_t ModTime = (time_t)Record[I++];
+    size_t FileSize = Record[I++];
+    StringRef Name = Blob;
+
+    IndexUnitReader::DependencyKind DepKind;
+    switch (UnitDepKind) {
+      case UNIT_DEPEND_KIND_UNIT:
+        DepKind = IndexUnitReader::DependencyKind::Unit; break;
+      case UNIT_DEPEND_KIND_RECORD:
+        DepKind = IndexUnitReader::DependencyKind::Record; break;
+      case UNIT_DEPEND_KIND_FILE:
+        DepKind = IndexUnitReader::DependencyKind::File; break;
+    }
+
+    SmallString<512> PathBuf;
+    this->constructFilePath(PathBuf, PathIndex);
+    StringRef ModuleName = this->getModuleName(ModuleIndex);
+
+    return Receiver(IndexUnitReader::DependencyInfo{DepKind, IsSystem, Name,
+      PathBuf.str(), ModuleName, FileSize, ModTime});
+  });
+
+  std::string Error;
+  return Visitor.visit(Error);
+}
+
+bool IndexUnitReaderImpl::foreachInclude(IncludeReceiver Receiver) {
+  store::SavedStreamPosition SavedIncPosition(IncludeCursor);
+  IndexUnitBlockBitstreamVisitor Visitor(UNIT_INCLUDE, IncludeCursor,
+  [&](RecordDataImpl& Record, StringRef Blob) {
+    unsigned I = 0;
+    int SourcePathIndex = (int)Record[I++] - 1;
+    unsigned Line = Record[I++];
+    int TargetPathIndex = (int)Record[I++] - 1;
+
+    SmallString<512> SourceBuf, TargetBuf;
+    this->constructFilePath(SourceBuf, SourcePathIndex);
+    this->constructFilePath(TargetBuf, TargetPathIndex);
+    return Receiver(IndexUnitReader::IncludeInfo{SourceBuf.str(), Line, TargetBuf.str()});
+  });
+
+  std::string Error;
+  return Visitor.visit(Error);
+}
+
+
+void IndexUnitReaderImpl::constructFilePath(SmallVectorImpl<char> &PathBuf,
+                       int PathIndex) {
+
+  if (PathIndex < 0) return;
+  FileBitPath &Path = Paths[PathIndex];
+  StringRef Prefix;
+  switch (Path.PrefixKind) {
+  case UNIT_PATH_PREFIX_NONE:
+    break;
+  case UNIT_PATH_PREFIX_WORKDIR:
+    Prefix = getWorkingDirectory();
+    break;
+  case UNIT_PATH_PREFIX_SYSROOT:
+    Prefix = getSysrootPath();
+    break;
+  }
+  PathBuf.append(Prefix.begin(), Prefix.end());
+  sys::path::append(PathBuf,
+                    getPathFromBuffer(Path.Dir.Offset, Path.Dir.Size),
+                    getPathFromBuffer(Path.Filename.Offset, Path.Filename.Size));
+}
+
+StringRef IndexUnitReaderImpl::getModuleName(int ModuleIndex) {
+  if (ModuleIndex < 0)
+    return StringRef();
+  auto &ModInfo = Modules[ModuleIndex];
+  return StringRef(ModuleNamesBuffer.data()+ModInfo.NameOffset, ModInfo.NameSize);
+}
+
+
+//===----------------------------------------------------------------------===//
+// IndexUnitReader
+//===----------------------------------------------------------------------===//
+
+std::unique_ptr<IndexUnitReader>
+IndexUnitReader::createWithUnitFilename(StringRef UnitFilename,
+                                        StringRef StorePath,
+                                        std::string &Error) {
+  SmallString<128> PathBuf = StorePath;
+  appendUnitSubDir(PathBuf);
+  sys::path::append(PathBuf, UnitFilename);
+  return createWithFilePath(PathBuf.str(), Error);
+}
+
+std::unique_ptr<IndexUnitReader>
+IndexUnitReader::createWithFilePath(StringRef FilePath, std::string &Error) {
+  int FD;
+  std::error_code EC = sys::fs::openFileForRead(FilePath, FD);
+  if (EC) {
+    raw_string_ostream(Error) << "Failed opening '" << FilePath << "': "
+      << EC.message();
+    return nullptr;
+  }
+
+  assert(FD != -1);
+  struct AutoFDClose {
+    int FD;
+    AutoFDClose(int FD) : FD(FD) {}
+    ~AutoFDClose() {
+      ::close(FD);
+    }
+  } AutoFDClose(FD);
+
+  sys::fs::file_status FileStat;
+  EC = sys::fs::status(FD, FileStat);
+  if (EC) {
+    Error = EC.message();
+    return nullptr;
+  }
+
+  auto ErrOrBuf = MemoryBuffer::getOpenFile(FD, FilePath, /*FileSize=*/-1,
+                                            /*RequiresNullTerminator=*/false);
+  if (!ErrOrBuf) {
+    raw_string_ostream(Error) << "Failed opening '" << FilePath << "': "
+      << ErrOrBuf.getError().message();
+    return nullptr;
+  }
+
+  std::unique_ptr<IndexUnitReaderImpl> Impl(new IndexUnitReaderImpl());
+  bool Err = Impl->init(std::move(*ErrOrBuf), FileStat.getLastModificationTime(),
+                        Error);
+  if (Err)
+    return nullptr;
+
+  std::unique_ptr<IndexUnitReader> Reader;
+  Reader.reset(new IndexUnitReader(Impl.release()));
+  return Reader;
+}
+
+Optional<sys::TimePoint<>>
+IndexUnitReader::getModificationTimeForUnit(StringRef UnitFilename,
+                                            StringRef StorePath,
+                                            std::string &Error) {
+  SmallString<128> PathBuf = StorePath;
+  appendUnitSubDir(PathBuf);
+  sys::path::append(PathBuf, UnitFilename);
+
+  sys::fs::file_status FileStat;
+  std::error_code EC = sys::fs::status(PathBuf.str(), FileStat);
+  if (EC) {
+    Error = EC.message();
+    return None;
+  }
+  return FileStat.getLastModificationTime();
+}
+
+#define IMPL static_cast<IndexUnitReaderImpl*>(Impl)
+
+IndexUnitReader::~IndexUnitReader() {
+  delete IMPL;
+}
+
+StringRef IndexUnitReader::getProviderIdentifier() const {
+  return IMPL->getProviderIdentifier();
+}
+
+StringRef IndexUnitReader::getProviderVersion() const {
+  return IMPL->getProviderVersion();
+}
+
+llvm::sys::TimePoint<> IndexUnitReader::getModificationTime() const {
+  return IMPL->getModificationTime();
+}
+
+StringRef IndexUnitReader::getWorkingDirectory() const {
+  return IMPL->getWorkingDirectory();
+}
+
+StringRef IndexUnitReader::getOutputFile() const {
+  return IMPL->getOutputFile();
+}
+
+StringRef IndexUnitReader::getSysrootPath() const {
+  return IMPL->getSysrootPath();
+}
+
+StringRef IndexUnitReader::getMainFilePath() const {
+  return IMPL->getMainFilePath();
+}
+
+StringRef IndexUnitReader::getModuleName() const {
+  return IMPL->getModuleName();
+}
+
+StringRef IndexUnitReader::getTarget() const {
+  return IMPL->getTarget();
+}
+
+bool IndexUnitReader::hasMainFile() const {
+  return IMPL->hasMainFile();
+}
+
+bool IndexUnitReader::isSystemUnit() const {
+  return IMPL->isSystemUnit();
+}
+
+bool IndexUnitReader::isModuleUnit() const {
+  return IMPL->isModuleUnit();
+}
+
+bool IndexUnitReader::isDebugCompilation() const {
+  return IMPL->isDebugCompilation();
+}
+
+/// \c Index is the index in the \c getDependencies array.
+/// Unit dependencies are provided ahead of record ones.
+bool IndexUnitReader::foreachDependency(DependencyReceiver Receiver) {
+  return IMPL->foreachDependency(std::move(Receiver));
+}
+
+bool IndexUnitReader::foreachInclude(IncludeReceiver Receiver) {
+  return IMPL->foreachInclude(std::move(Receiver));
+}

lib/Index/IndexUnitWriter.cpp

+//===--- IndexUnitWriter.cpp - Index unit serialization -------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Index/IndexUnitWriter.h"
+#include "IndexDataStoreUtils.h"
+#include "clang/Basic/FileManager.h"
+#include "llvm/ADT/APInt.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/Hashing.h"
+#include "llvm/ADT/StringMap.h"
+#include "llvm/Bitcode/BitstreamWriter.h"
+#include "llvm/Support/Allocator.h"
+#include "llvm/Support/Errc.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/raw_ostream.h"
+
+using namespace clang;
+using namespace clang::index;
+using namespace clang::index::store;
+using namespace llvm;
+
+
+class IndexUnitWriter::PathStorage {
+  std::string WorkDir;
+  std::string SysrootPath;
+  SmallString<512> PathsBuf;
+  StringMap<DirBitPath, BumpPtrAllocator> Dirs;
+  std::vector<FileBitPath> FileBitPaths;
+  DenseMap<const FileEntry *, size_t> FileToIndex;
+
+public:
+  PathStorage(StringRef workDir, StringRef sysrootPath) {
+    WorkDir = workDir;
+    if (sysrootPath == "/")
+      sysrootPath = StringRef();
+    SysrootPath = sysrootPath;
+  }
+
+  StringRef getPathsBuffer() const { return PathsBuf.str(); }
+
+  ArrayRef<FileBitPath> getBitPaths() const { return FileBitPaths; }
+
+  int getPathIndex(const FileEntry *FE) {
+    if (!FE)
+      return -1;
+    auto Pair = FileToIndex.insert(std::make_pair(FE, FileBitPaths.size()));
+    bool IsNew = Pair.second;
+    size_t Index = Pair.first->getSecond();
+
+    if (IsNew) {
+      StringRef Filename = sys::path::filename(FE->getName());
+      DirBitPath Dir = getDirBitPath(sys::path::parent_path(FE->getName()));
+      FileBitPaths.emplace_back(Dir.PrefixKind, Dir.Dir,
+                                BitPathComponent(getPathOffset(Filename),
+                                                 Filename.size()));
+    }
+    return Index;
+  }
+
+  size_t getPathOffset(StringRef Path) {
+    if (Path.empty())
+      return 0;
+    size_t offset = PathsBuf.size();
+    PathsBuf += Path;
+    return offset;
+  }
+  
+private:
+  DirBitPath getDirBitPath(StringRef dirStr) {
+    auto pair = Dirs.insert(std::make_pair(dirStr, DirBitPath()));
+    bool isNew = pair.second;
+    auto &dirPath = pair.first->second;
+
+    if (isNew) {
+      if (isPathInDir(SysrootPath, dirStr)) {
+        dirPath.PrefixKind = UNIT_PATH_PREFIX_SYSROOT;
+        dirStr = dirStr.drop_front(SysrootPath.size());
+        while (!dirStr.empty() && dirStr[0] == '/')
+          dirStr = dirStr.drop_front();
+      } else if (isPathInDir(WorkDir, dirStr)) {
+        dirPath.PrefixKind = UNIT_PATH_PREFIX_WORKDIR;
+        dirStr = dirStr.drop_front(WorkDir.size());
+        while (!dirStr.empty() && dirStr[0] == '/')
+          dirStr = dirStr.drop_front();
+      }
+      dirPath.Dir.Offset = getPathOffset(dirStr);
+      dirPath.Dir.Size = dirStr.size();
+    }
+    return dirPath;
+  }
+
+  static bool isPathInDir(StringRef dir, StringRef path) {
+    if (dir.empty() || !path.startswith(dir))
+      return false;
+    StringRef rest = path.drop_front(dir.size());
+    return !rest.empty() && sys::path::is_separator(rest.front());
+  }
+};
+
+IndexUnitWriter::IndexUnitWriter(FileManager &FileMgr,
+                                 StringRef StorePath,
+                                 StringRef ProviderIdentifier,
+                                 StringRef ProviderVersion,
+                                 StringRef OutputFile,
+                                 StringRef ModuleName,
+                                 const FileEntry *MainFile,
+                                 bool IsSystem,
+                                 bool IsModuleUnit,
+                                 bool IsDebugCompilation,
+                                 StringRef TargetTriple,
+                                 StringRef SysrootPath,
+                                 writer::ModuleInfoWriterCallback GetInfoForModule)
+: FileMgr(FileMgr) {
+  this->UnitsPath = StorePath;
+  store::appendUnitSubDir(this->UnitsPath);
+  this->ProviderIdentifier = ProviderIdentifier;
+  this->ProviderVersion = ProviderVersion;
+  this->OutputFile = OutputFile;
+  this->ModuleName = ModuleName;
+  this->MainFile = MainFile;
+  this->IsSystemUnit = IsSystem;
+  this->IsModuleUnit = IsModuleUnit;
+  this->IsDebugCompilation = IsDebugCompilation;
+  this->TargetTriple = TargetTriple;
+  this->SysrootPath = SysrootPath;
+  this->GetInfoForModuleFn = GetInfoForModule;
+}
+
+IndexUnitWriter::~IndexUnitWriter() {}
+
+int IndexUnitWriter::addModule(writer::OpaqueModule Mod) {
+  if (!Mod)
+    return -1;
+
+  auto Pair = IndexByModule.insert(std::make_pair(Mod, Modules.size()));
+  bool WasInserted = Pair.second;
+  if (WasInserted) {
+    Modules.push_back(Mod);
+  }
+  return Pair.first->second;
+}
+
+int IndexUnitWriter::addFileDependency(const FileEntry *File, bool IsSystem,
+                                       writer::OpaqueModule Mod) {
+  assert(File);
+  auto Pair = IndexByFile.insert(std::make_pair(File, Files.size()));
+  bool WasInserted = Pair.second;
+  if (WasInserted) {
+    Files.push_back(FileEntryData{File, IsSystem, addModule(Mod), {}});
+  }
+  return Pair.first->second;
+}
+
+void IndexUnitWriter::addRecordFile(StringRef RecordFile, const FileEntry *File,
+                                    bool IsSystem, writer::OpaqueModule Mod) {
+  int Dep = File ? addFileDependency(File, IsSystem, /*module=*/nullptr) : -1;
+  Records.push_back(RecordOrUnitData{RecordFile, Dep, addModule(Mod), IsSystem});
+}
+
+void IndexUnitWriter::addASTFileDependency(const FileEntry *File, bool IsSystem,
+                                           writer::OpaqueModule Mod,
+                                           bool withoutUnitName) {
+  assert(File);
+  if (!SeenASTFiles.insert(File).second)
+    return;
+
+  SmallString<64> UnitName;
+  if (!withoutUnitName)
+    getUnitNameForOutputFile(File->getName(), UnitName);
+  addUnitDependency(UnitName.str(), File, IsSystem, Mod);
+}
+
+void IndexUnitWriter::addUnitDependency(StringRef UnitFile,
+                                        const FileEntry *File, bool IsSystem,
+                                        writer::OpaqueModule Mod) {
+  int Dep = File ? addFileDependency(File, IsSystem, /*module=*/nullptr) : -1;
+  ASTFileUnits.emplace_back(RecordOrUnitData{UnitFile, Dep, addModule(Mod), IsSystem});
+}
+
+bool IndexUnitWriter::addInclude(const FileEntry *Source, unsigned Line,
+                                 const FileEntry *Target) {
+  // FIXME: This will ignore includes of headers that resolve to module imports
+  // because the 'target' header has not been added as a file dependency earlier
+  // so it is missing from \c IndexByFile.
+
+  auto It = IndexByFile.find(Source);
+  if (It == IndexByFile.end())
+    return false;
+  int SourceIndex = It->getSecond();
+  It = IndexByFile.find(Target);
+  if (It == IndexByFile.end())
+    return false;
+  int TargetIndex = It->getSecond();
+  Files[SourceIndex].Includes.emplace_back(FileInclude{TargetIndex, Line});
+  return true;
+};
+
+void IndexUnitWriter::getUnitNameForOutputFile(StringRef FilePath,
+                                               SmallVectorImpl<char> &Str) {
+  SmallString<256> AbsPath(FilePath);
+  FileMgr.makeAbsolutePath(AbsPath);
+  return getUnitNameForAbsoluteOutputFile(AbsPath, Str);
+}
+
+void IndexUnitWriter::getUnitPathForOutputFile(StringRef FilePath,
+                                               SmallVectorImpl<char> &Str) {
+  Str.append(UnitsPath.begin(), UnitsPath.end());
+  Str.push_back('/');
+  return getUnitNameForOutputFile(FilePath, Str);
+}
+
+Optional<bool> IndexUnitWriter::isUnitUpToDateForOutputFile(StringRef FilePath,
+                                                            Optional<StringRef> TimeCompareFilePath,
+                                                            std::string &Error) {
+  SmallString<256> UnitPath;
+  getUnitPathForOutputFile(FilePath, UnitPath);
+
+  llvm::sys::fs::file_status UnitStat;
+  if (std::error_code EC = llvm::sys::fs::status(UnitPath.c_str(), UnitStat)) {
+    if (EC != llvm::errc::no_such_file_or_directory) {
+      llvm::raw_string_ostream Err(Error);
+      Err << "could not access path '" << UnitPath
+          << "': " << EC.message();
+      return None;
+    }
+    return false;
+  }
+
+  if (!TimeCompareFilePath.hasValue())
+    return true;
+
+  llvm::sys::fs::file_status CompareStat;
+  if (std::error_code EC = llvm::sys::fs::status(*TimeCompareFilePath, CompareStat)) {
+    if (EC != llvm::errc::no_such_file_or_directory) {
+      llvm::raw_string_ostream Err(Error);
+      Err << "could not access path '" << *TimeCompareFilePath
+          << "': " << EC.message();
+      return None;
+    }
+    return true;
+  }
+
+  // Return true (unit is up-to-date) if the file to compare is older than the
+  // unit file.
+  return CompareStat.getLastModificationTime() <= UnitStat.getLastModificationTime();
+}
+
+void IndexUnitWriter::getUnitNameForAbsoluteOutputFile(StringRef FilePath,
+                                                   SmallVectorImpl<char> &Str) {
+  StringRef Fname = sys::path::filename(FilePath);
+  Str.append(Fname.begin(), Fname.end());
+  Str.push_back('-');
+  llvm::hash_code PathHashVal = llvm::hash_value(FilePath);
+  llvm::APInt(64, PathHashVal).toString(Str, 36, /*Signed=*/false);
+}
+
+static void writeBlockInfo(BitstreamWriter &Stream) {
+  RecordData Record;
+
+  Stream.EnterBlockInfoBlock();
+#define BLOCK(X) emitBlockID(X ## _ID, #X, Stream, Record)
+#define RECORD(X) emitRecordID(X, #X, Stream, Record)
+
+  BLOCK(UNIT_VERSION_BLOCK);
+  RECORD(UNIT_VERSION);
+
+  BLOCK(UNIT_INFO_BLOCK);
+  RECORD(UNIT_INFO);
+
+  BLOCK(UNIT_DEPENDENCIES_BLOCK);
+  RECORD(UNIT_DEPENDENCY);
+
+  BLOCK(UNIT_INCLUDES_BLOCK);
+  RECORD(UNIT_INCLUDE);
+
+  BLOCK(UNIT_PATHS_BLOCK);
+  RECORD(UNIT_PATH);
+  RECORD(UNIT_PATH_BUFFER);
+
+  BLOCK(UNIT_MODULES_BLOCK);
+  RECORD(UNIT_MODULE);
+  RECORD(UNIT_MODULE_BUFFER);
+
+#undef RECORD
+#undef BLOCK
+  Stream.ExitBlock();
+}
+
+static void writeVersionInfo(BitstreamWriter &Stream) {
+  using namespace llvm::sys;
+
+  Stream.EnterSubblock(UNIT_VERSION_BLOCK_ID, 3);
+
+  auto Abbrev = std::make_shared<BitCodeAbbrev>();
+  Abbrev->Add(BitCodeAbbrevOp(UNIT_VERSION));
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 6)); // Store format version
+  unsigned AbbrevCode = Stream.EmitAbbrev(std::move(Abbrev));
+
+  RecordData Record;
+  Record.push_back(UNIT_VERSION);
+  Record.push_back(STORE_FORMAT_VERSION);
+  Stream.EmitRecordWithAbbrev(AbbrevCode, Record);
+
+  Stream.ExitBlock();
+}
+
+bool IndexUnitWriter::write(std::string &Error) {
+  using namespace llvm::sys;
+
+  // Determine the working directory.
+  SmallString<128> CWDPath;
+  if (!FileMgr.getFileSystemOpts().WorkingDir.empty()) {
+    CWDPath = FileMgr.getFileSystemOpts().WorkingDir;
+    if (!path::is_absolute(CWDPath)) {
+      fs::make_absolute(CWDPath);
+    }
+  } else {
+    std::error_code EC = sys::fs::current_path(CWDPath);
+    if (EC) {
+      llvm::raw_string_ostream Err(Error);
+      Err << "failed to determine current working directory: " << EC.message();
+      return true;
+    }
+  }
+  WorkDir = CWDPath.str();
+
+  SmallString<512> Buffer;
+  BitstreamWriter Stream(Buffer);
+  Stream.Emit('I', 8);
+  Stream.Emit('D', 8);
+  Stream.Emit('X', 8);
+  Stream.Emit('U', 8);
+
+  PathStorage PathStore(WorkDir, SysrootPath);
+
+  writeBlockInfo(Stream);
+  writeVersionInfo(Stream);
+  writeUnitInfo(Stream, PathStore);
+  writeDependencies(Stream, PathStore);
+  writeIncludes(Stream, PathStore);
+  writePaths(Stream, PathStore);
+  writeModules(Stream);
+
+  SmallString<256> UnitPath;
+  getUnitPathForOutputFile(OutputFile, UnitPath);
+
+  SmallString<128> TempPath;
+  TempPath = path::parent_path(UnitsPath);
+  TempPath += '/';
+  TempPath += path::filename(UnitPath);
+  TempPath += "-%%%%%%%%";
+  int TempFD;
+  if (llvm::sys::fs::createUniqueFile(TempPath.str(), TempFD, TempPath)) {
+    llvm::raw_string_ostream Err(Error);
+    Err << "failed to create temporary file: " << TempPath;
+    return true;
+  }
+
+  raw_fd_ostream OS(TempFD, /*shouldClose=*/true);
+  OS.write(Buffer.data(), Buffer.size());
+  OS.close();
+
+  std::error_code EC = fs::rename(/*from=*/TempPath.c_str(), /*to=*/UnitPath.c_str());
+  if (EC) {
+    llvm::raw_string_ostream Err(Error);
+    Err << "failed to rename '" << TempPath << "' to '" << UnitPath << "': " << EC.message();
+    return true;
+  }
+
+  return false;
+}
+
+void IndexUnitWriter::writeUnitInfo(llvm::BitstreamWriter &Stream,
+                                    PathStorage &PathStore) {
+  Stream.EnterSubblock(UNIT_INFO_BLOCK_ID, 3);
+
+  auto Abbrev = std::make_shared<BitCodeAbbrev>();
+  Abbrev->Add(BitCodeAbbrevOp(UNIT_INFO));
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 1)); // IsSystemUnit
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 10)); // WorkDir offset
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 8)); // WorkDir size
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 10)); // OutputFile offset
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 8)); // OutputFile size
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 10)); // Sysroot offset
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 8)); // Sysroot size
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 10)); // Main path id
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 1)); // IsDebugCompilation
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 1)); // IsModuleUnit
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 5)); // Module name size
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 5)); // ProviderIdentifier size
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 5)); // ProviderVersion size
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 5)); // ProviderDataVersion
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Blob)); // Module name + ProviderIdentifier + ProviderVersion + target triple
+  unsigned AbbrevCode = Stream.EmitAbbrev(std::move(Abbrev));
+
+  RecordData Record;
+  Record.push_back(UNIT_INFO);
+  Record.push_back(IsSystemUnit);
+  Record.push_back(PathStore.getPathOffset(WorkDir));
+  Record.push_back(WorkDir.size());
+  Record.push_back(PathStore.getPathOffset(OutputFile));
+  Record.push_back(OutputFile.size());
+  Record.push_back(PathStore.getPathOffset(SysrootPath));
+  Record.push_back(SysrootPath.size());
+  Record.push_back(PathStore.getPathIndex(MainFile) + 1); // Make 1-based with 0=invalid
+  Record.push_back(IsDebugCompilation);
+  Record.push_back(IsModuleUnit);
+  Record.push_back(ModuleName.size());
+  Record.push_back(ProviderIdentifier.size());
+  Record.push_back(ProviderVersion.size());
+  // ProviderDataVersion is reserved. Not sure it is a good to idea to have
+  // clients consider the specifics of a 'provider data version', but reserving
+  // to avoid store format version change in case there is a use case in the
+  // future.
+  Record.push_back(0); // ProviderDataVersion
+  SmallString<128> InfoStrings;
+  InfoStrings += ModuleName;
+  InfoStrings += ProviderIdentifier;
+  InfoStrings += ProviderVersion;
+  InfoStrings += TargetTriple;
+  Stream.EmitRecordWithBlob(AbbrevCode, Record, InfoStrings);
+
+  Stream.ExitBlock();
+}
+
+void IndexUnitWriter::writeDependencies(llvm::BitstreamWriter &Stream,
+                                        PathStorage &PathStore) {
+  std::vector<bool> FileUsedForRecordOrUnit;
+  FileUsedForRecordOrUnit.resize(Files.size());
+
+  Stream.EnterSubblock(UNIT_DEPENDENCIES_BLOCK_ID, 3);
+
+  auto Abbrev = std::make_shared<BitCodeAbbrev>();
+  Abbrev->Add(BitCodeAbbrevOp(UNIT_DEPENDENCY));
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, UnitDependencyKindBitNum)); // Dependency kind
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 1)); // IsSystem
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 10)); // PathIndex (1-based, 0 = none)
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 8)); // ModuleIndex (1-based, 0 = none)
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 32)); // time_t
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 16)); // file size
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Blob)); // Name
+  unsigned AbbrevCode = Stream.EmitAbbrev(std::move(Abbrev));
+
+  RecordData Record;
+
+  auto addRecordOrUnitData = [&](UnitDependencyKind K, const RecordOrUnitData &Data) {
+    Record.push_back(UNIT_DEPENDENCY);
+    Record.push_back(K);
+    Record.push_back(Data.IsSystem);
+    if (Data.FileIndex != -1) {
+      Record.push_back(PathStore.getPathIndex(Files[Data.FileIndex].File) + 1);
+      FileUsedForRecordOrUnit[Data.FileIndex] = true;
+    } else {
+      Record.push_back(0);
+    }
+    if (Data.ModuleIndex != -1) {
+      Record.push_back(Data.ModuleIndex + 1);
+    } else {
+      Record.push_back(0);
+    }
+    if (Data.FileIndex != -1) {
+      Record.push_back(Files[Data.FileIndex].File->getModificationTime());
+      Record.push_back(Files[Data.FileIndex].File->getSize());
+    } else {
+      Record.push_back(0);
+      Record.push_back(0);
+    }
+    Stream.EmitRecordWithBlob(AbbrevCode, Record, Data.Name);
+  };
+
+  for (auto &ASTData : ASTFileUnits) {
+    Record.clear();
+    addRecordOrUnitData(UNIT_DEPEND_KIND_UNIT, ASTData);
+  }
+  for (auto &recordData : Records) {
+    Record.clear();
+    addRecordOrUnitData(UNIT_DEPEND_KIND_RECORD, recordData);
+  }
+  size_t FileIndex = 0;
+  for (auto &File : Files) {
+    if (FileUsedForRecordOrUnit[FileIndex++])
+      continue;
+    Record.clear();
+    Record.push_back(UNIT_DEPENDENCY);
+    Record.push_back(UNIT_DEPEND_KIND_FILE);
+    Record.push_back(File.IsSystem);
+    Record.push_back(PathStore.getPathIndex(File.File) + 1);
+    if (File.ModuleIndex != -1) {
+      Record.push_back(File.ModuleIndex + 1);
+    } else {
+      Record.push_back(0);
+    }
+    Record.push_back(File.File->getModificationTime());
+    Record.push_back(File.File->getSize());
+    Stream.EmitRecordWithBlob(AbbrevCode, Record, StringRef());
+  }
+
+  Stream.ExitBlock();
+}
+
+void IndexUnitWriter::writeIncludes(llvm::BitstreamWriter &Stream,
+                                    PathStorage &PathStore) {
+  Stream.EnterSubblock(UNIT_INCLUDES_BLOCK_ID, 3);
+
+  auto Abbrev = std::make_shared<BitCodeAbbrev>();
+  Abbrev->Add(BitCodeAbbrevOp(UNIT_INCLUDE));
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 10)); // source path index (1-based, 0 = no path)
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 12)); // source include line
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 10)); // target path index (1-based, 0 = no path)
+  unsigned AbbrevCode = Stream.EmitAbbrev(std::move(Abbrev));
+
+  RecordData Record;
+
+  for (auto &Including : Files) {
+    for(auto &Included: Including.Includes) {
+      Record.clear();
+      Record.push_back(UNIT_INCLUDE);
+      Record.push_back(PathStore.getPathIndex(Including.File) + 1);
+      Record.push_back(Included.Line);
+      Record.push_back(PathStore.getPathIndex(Files[Included.Index].File) + 1);
+      Stream.EmitRecordWithAbbrev(AbbrevCode, Record);
+    }
+  }
+  Stream.ExitBlock();
+}
+
+void IndexUnitWriter::writePaths(llvm::BitstreamWriter &Stream,
+                                 PathStorage &PathStore) {
+  Stream.EnterSubblock(UNIT_PATHS_BLOCK_ID, 3);
+
+  auto PathAbbrev = std::make_shared<BitCodeAbbrev>();
+  PathAbbrev->Add(BitCodeAbbrevOp(UNIT_PATH));
+  PathAbbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, UnitFilePathPrefixKindBitNum)); // Path prefix kind
+  PathAbbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 10)); // DirPath offset
+  PathAbbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 8)); // DirPath size
+  PathAbbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 10)); // Filename offset
+  PathAbbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 6)); // Filename size
+  unsigned PathAbbrevCode = Stream.EmitAbbrev(std::move(PathAbbrev));
+
+  auto PathBufferAbbrev = std::make_shared<BitCodeAbbrev>();
+  PathBufferAbbrev->Add(BitCodeAbbrevOp(UNIT_PATH_BUFFER));
+  PathBufferAbbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Blob)); // Paths buffer
+  unsigned PathBufferAbbrevCode = Stream.EmitAbbrev(PathBufferAbbrev);
+
+  RecordData Record;
+  for(auto &BitPath: PathStore.getBitPaths()) {
+    Record.push_back(UNIT_PATH);
+    Record.push_back(BitPath.PrefixKind);
+    Record.push_back(BitPath.Dir.Offset);
+    Record.push_back(BitPath.Dir.Size);
+    Record.push_back(BitPath.Filename.Offset);
+    Record.push_back(BitPath.Filename.Size);
+    Stream.EmitRecordWithAbbrev(PathAbbrevCode, Record);
+    Record.clear();
+  }
+
+  Record.push_back(UNIT_PATH_BUFFER);
+  Stream.EmitRecordWithBlob(PathBufferAbbrevCode, Record, PathStore.getPathsBuffer());
+
+  Stream.ExitBlock();
+}
+
+void IndexUnitWriter::writeModules(llvm::BitstreamWriter &Stream) {
+  Stream.EnterSubblock(UNIT_MODULES_BLOCK_ID, 3);
+
+  auto Abbrev = std::make_shared<BitCodeAbbrev>();
+  Abbrev->Add(BitCodeAbbrevOp(UNIT_MODULE));
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 9)); // Module name offset
+  Abbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 6)); // Module name size
+  unsigned AbbrevCode = Stream.EmitAbbrev(std::move(Abbrev));
+
+  auto BufferAbbrev = std::make_shared<BitCodeAbbrev>();
+  BufferAbbrev->Add(BitCodeAbbrevOp(UNIT_MODULE_BUFFER));
+  BufferAbbrev->Add(BitCodeAbbrevOp(BitCodeAbbrevOp::Blob)); // Module names buffer
+  unsigned BufferAbbrevCode = Stream.EmitAbbrev(BufferAbbrev);
+
+  SmallString<512> ModuleNamesBuf;
+
+  RecordData Record;
+  for (auto &Mod : Modules) {
+    SmallString<64> ModuleName;
+    StringRef name = GetInfoForModuleFn(Mod, ModuleName).Name;
+    size_t offset = ModuleNamesBuf.size();
+    ModuleNamesBuf += name;
+
+    Record.push_back(UNIT_MODULE);
+    Record.push_back(offset);
+    Record.push_back(name.size());
+    Stream.EmitRecordWithAbbrev(AbbrevCode, Record);
+    Record.clear();
+  }
+
+  Record.push_back(UNIT_MODULE_BUFFER);
+  Stream.EmitRecordWithBlob(BufferAbbrevCode, Record, ModuleNamesBuf.str());
+
+  Stream.ExitBlock();
+}
+
+bool IndexUnitWriter::initIndexDirectory(StringRef StorePath,
+                                         std::string &Error) {
+  using namespace llvm::sys;
+  SmallString<128> SubPath = StorePath;
+  store::appendRecordSubDir(SubPath);
+  std::error_code EC = fs::create_directories(SubPath);
+  if (EC) {
+    llvm::raw_string_ostream Err(Error);
+    Err << "failed to create directory '" << SubPath << "': " << EC.message();
+    return true;
+  }
+
+  SubPath = StorePath;
+  store::appendUnitSubDir(SubPath);
+  EC = fs::create_directory(SubPath);
+  if (EC) {
+    llvm::raw_string_ostream Err(Error);
+    Err << "failed to create directory '" << SubPath << "': " << EC.message();
+    return true;
+  }
+
+  return false;
+}

lib/Index/IndexingAction.cpp

 //===- IndexingAction.cpp - Frontend index action -------------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "clang/Index/IndexingAction.h"
 #include "clang/Index/IndexDataConsumer.h"
+#include "FileIndexRecord.h"
 #include "IndexingContext.h"
+#include "ClangIndexRecordWriter.h"
+#include "IndexDataStoreUtils.h"
+#include "clang/Index/IndexUnitWriter.h"
+#include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/FrontendAction.h"
+#include "clang/Frontend/FrontendDiagnostic.h"
 #include "clang/Frontend/MultiplexConsumer.h"
+#include "clang/Frontend/Utils.h"
 #include "clang/Lex/Preprocessor.h"
 #include "clang/Serialization/ASTReader.h"
 
 using namespace clang;
 using namespace clang::index;
 
 void IndexDataConsumer::_anchor() {}
 
protected:
   std::shared_ptr<IndexDataConsumer> DataConsumer;
   IndexingContext IndexCtx;
 
   IndexActionBase(std::shared_ptr<IndexDataConsumer> dataConsumer,
                   IndexingOptions Opts)
     : DataConsumer(std::move(dataConsumer)),
       IndexCtx(Opts, *DataConsumer) {}
 
-  std::unique_ptr<IndexASTConsumer> createIndexASTConsumer() {
+  std::unique_ptr<IndexASTConsumer> createIndexASTConsumer(CompilerInstance &CI) {
+    IndexCtx.setSysrootPath(CI.getHeaderSearchOpts().Sysroot);
     return llvm::make_unique<IndexASTConsumer>(IndexCtx);
   }
 
   void finish() {
     DataConsumer->finish();
   }
 };
 
 class IndexAction : public ASTFrontendAction, IndexActionBase {
 public:
   IndexAction(std::shared_ptr<IndexDataConsumer> DataConsumer,
               IndexingOptions Opts)
     : IndexActionBase(std::move(DataConsumer), Opts) {}
 
 protected:
   std::unique_ptr<ASTConsumer> CreateASTConsumer(CompilerInstance &CI,
                                                  StringRef InFile) override {
-    return createIndexASTConsumer();
+    return createIndexASTConsumer(CI);
   }
 
   void EndSourceFileAction() override {
     FrontendAction::EndSourceFileAction();
     finish();
   }
 };
 
 class WrappingIndexAction : public WrapperFrontendAction, IndexActionBase {
-  bool IndexActionFailed = false;
+  bool CreatedASTConsumer = false;
 
 public:
   WrappingIndexAction(std::unique_ptr<FrontendAction> WrappedAction,
                       std::shared_ptr<IndexDataConsumer> DataConsumer,
                       IndexingOptions Opts)
     : WrapperFrontendAction(std::move(WrappedAction)),
       IndexActionBase(std::move(DataConsumer), Opts) {}
 
 protected:
   std::unique_ptr<ASTConsumer> CreateASTConsumer(CompilerInstance &CI,
                                                  StringRef InFile) override;
   void EndSourceFileAction() override;
 };
 
 } // anonymous namespace
 
 void WrappingIndexAction::EndSourceFileAction() {
   // Invoke wrapped action's method.
   WrapperFrontendAction::EndSourceFileAction();
-  if (!IndexActionFailed)
+  if (CreatedASTConsumer)
     finish();
 }
 
 std::unique_ptr<ASTConsumer>
 WrappingIndexAction::CreateASTConsumer(CompilerInstance &CI, StringRef InFile) {
   auto OtherConsumer = WrapperFrontendAction::CreateASTConsumer(CI, InFile);
-  if (!OtherConsumer) {
-    IndexActionFailed = true;
+  if (!OtherConsumer)
     return nullptr;
-  }
 
+  CreatedASTConsumer = true;
   std::vector<std::unique_ptr<ASTConsumer>> Consumers;
   Consumers.push_back(std::move(OtherConsumer));
-  Consumers.push_back(createIndexASTConsumer());
+  Consumers.push_back(createIndexASTConsumer(CI));
   return llvm::make_unique<MultiplexConsumer>(std::move(Consumers));
 }
 
 std::unique_ptr<FrontendAction>
 index::createIndexingAction(std::shared_ptr<IndexDataConsumer> DataConsumer,
                             IndexingOptions Opts,
                             std::unique_ptr<FrontendAction> WrappedAction) {
   if (WrappedAction)
     return llvm::make_unique<WrappingIndexAction>(std::move(WrappedAction),
                                                   std::move(DataConsumer),
                                                   Opts);
void index::indexModuleFile(serialization::ModuleFile &Mod,
   IndexCtx.setASTContext(Ctx);
   DataConsumer->initialize(Ctx);
 
   for (const Decl *D :Reader.getModuleFileLevelDecls(Mod)) {
     IndexCtx.indexTopLevelDecl(D);
   }
   DataConsumer->finish();
 }
+
+//===----------------------------------------------------------------------===//
+// Index Data Recording
+//===----------------------------------------------------------------------===//
+
+namespace {
+
+class IndexDataRecorder : public IndexDataConsumer {
+  IndexingContext *IndexCtx = nullptr;
+  const Preprocessor *PP = nullptr;
+  typedef llvm::DenseMap<FileID, std::unique_ptr<FileIndexRecord>> RecordByFileTy;
+  RecordByFileTy RecordByFile;
+
+public:
+  void init(IndexingContext *idxCtx, const CompilerInstance &CI) {
+    IndexCtx = idxCtx;
+    PP = &CI.getPreprocessor();
+    initialize(CI.getASTContext());
+  }
+
+  RecordByFileTy::const_iterator record_begin() const { return RecordByFile.begin(); }
+  RecordByFileTy::const_iterator record_end() const { return RecordByFile.end(); }
+  bool record_empty() const { return RecordByFile.empty(); }
+
+private:
+  bool handleDeclOccurence(const Decl *D, SymbolRoleSet Roles,
+                           ArrayRef<SymbolRelation> Relations,
+                           FileID FID, unsigned Offset,
+                           ASTNodeInfo ASTNode) override {
+    // Ignore the predefines buffer.
+    if (FID == PP->getPredefinesFileID())
+      return true;
+
+    FileIndexRecord &Rec = getFileIndexRecord(FID);
+    Rec.addDeclOccurence(Roles, Offset, D, Relations);
+    return true;
+  }
+
+  FileIndexRecord &getFileIndexRecord(FileID FID) {
+    auto &Entry = RecordByFile[FID];
+    if (!Entry) {
+      Entry.reset(new FileIndexRecord(FID, IndexCtx->isSystemFile(FID)));
+    }
+    return *Entry;
+  }
+};
+
+struct IncludeLocation {
+  const FileEntry *Source;
+  const FileEntry *Target;
+  unsigned Line;
+};
+
+class IncludePPCallbacks : public PPCallbacks {
+  IndexingContext &IndexCtx;
+  RecordingOptions RecordOpts;
+  std::vector<IncludeLocation> &Includes;
+  SourceManager &SourceMgr;
+
+public:
+  IncludePPCallbacks(IndexingContext &indexCtx, RecordingOptions recordOpts,
+                     std::vector<IncludeLocation> &IncludesForFile,
+                     SourceManager &SourceMgr) :
+    IndexCtx(indexCtx), RecordOpts(recordOpts),
+    Includes(IncludesForFile), SourceMgr(SourceMgr) {}
+
+private:
+  void addInclude(SourceLocation From, const FileEntry *To) {
+    assert(To);
+    if (RecordOpts.RecordIncludes == RecordingOptions::IncludesRecordingKind::None)
+      return;
+
+    std::pair<FileID, unsigned> LocInfo = SourceMgr.getDecomposedExpansionLoc(From);
+    switch (RecordOpts.RecordIncludes) {
+      case RecordingOptions::IncludesRecordingKind::None:
+        llvm_unreachable("should have already checked in the beginning");
+      case RecordingOptions::IncludesRecordingKind::UserOnly:
+        if (IndexCtx.isSystemFile(LocInfo.first))
+          return; // Ignore includes of system headers.
+        break;
+      case RecordingOptions::IncludesRecordingKind::All:
+        break;
+    }
+    auto *FE = SourceMgr.getFileEntryForID(LocInfo.first);
+    if (!FE)
+      return;
+    auto lineNo = SourceMgr.getLineNumber(LocInfo.first, LocInfo.second);
+    Includes.push_back({FE, To, lineNo});
+  }
+
+  virtual void InclusionDirective(SourceLocation HashLoc,
+                                  const Token &IncludeTok,
+                                  StringRef FileName,
+                                  bool IsAngled,
+                                  CharSourceRange FilenameRange,
+                                  const FileEntry *File,
+                                  StringRef SearchPath,
+                                  StringRef RelativePath,
+                                  const Module *Imported) override {
+    if (HashLoc.isFileID() && File && File->isValid())
+      addInclude(HashLoc, File);
+  }
+};
+
+class IndexDependencyProvider {
+public:
+  virtual ~IndexDependencyProvider() {}
+
+  virtual void visitFileDependencies(const CompilerInstance &CI,
+      llvm::function_ref<void(const FileEntry *FE, bool isSystem)> visitor) = 0;
+  virtual void visitIncludes(
+                 llvm::function_ref<void(const FileEntry *Source, unsigned Line,
+                                         const FileEntry *Target)> visitor) = 0;
+  virtual void visitModuleImports(const CompilerInstance &CI,
+                 llvm::function_ref<void(serialization::ModuleFile &Mod,
+                                         bool isSystem)> visitor) = 0;
+};
+
+class SourceFilesIndexDependencyCollector : public DependencyCollector, public IndexDependencyProvider {
+  IndexingContext &IndexCtx;
+  RecordingOptions RecordOpts;
+  llvm::SetVector<const FileEntry *> Entries;
+  llvm::BitVector IsSystemByUID;
+  std::vector<IncludeLocation> Includes;
+  SourceManager *SourceMgr = nullptr;
+  std::string SysrootPath;
+
+public:
+  SourceFilesIndexDependencyCollector(IndexingContext &indexCtx, RecordingOptions recordOpts)
+    : IndexCtx(indexCtx), RecordOpts(recordOpts) {}
+
+  virtual void attachToPreprocessor(Preprocessor &PP) override {
+    DependencyCollector::attachToPreprocessor(PP);
+    PP.addPPCallbacks(llvm::make_unique<IncludePPCallbacks>(IndexCtx,
+                                                            RecordOpts,
+                                                            Includes,
+                                                            PP.getSourceManager()));
+  }
+
+  void setSourceManager(SourceManager *SourceMgr) {
+    this->SourceMgr = SourceMgr;
+  }
+  void setSysrootPath(StringRef sysroot) { SysrootPath = sysroot; }
+
+  void visitFileDependencies(const CompilerInstance &CI,
+      llvm::function_ref<void(const FileEntry *FE, bool isSystem)> visitor) override {
+    for (auto *FE : getEntries()) {
+      visitor(FE, isSystemFile(FE));
+    }
+  }
+
+  void visitIncludes(
+                 llvm::function_ref<void(const FileEntry *Source, unsigned Line,
+                                         const FileEntry *Target)> visitor) override {
+    for (auto &Include : Includes) {
+      visitor(Include.Source, Include.Line, Include.Target);
+    }
+  }
+
+  void visitModuleImports(const CompilerInstance &CI,
+                 llvm::function_ref<void(serialization::ModuleFile &Mod,
+                                         bool isSystem)> visitor) override {
+    HeaderSearch &HS = CI.getPreprocessor().getHeaderSearchInfo();
+
+    if (auto Reader = CI.getModuleManager()) {
+      Reader->getModuleManager().visit([&](serialization::ModuleFile &Mod) -> bool {
+        bool isSystemMod = false;
+        if (Mod.isModule()) {
+          if (auto *M = HS.lookupModule(Mod.ModuleName, /*AllowSearch=*/false))
+            isSystemMod = M->IsSystem;
+        }
+        if (!isSystemMod || needSystemDependencies())
+          visitor(Mod, isSystemMod);
+        return true; // skip module dependencies.
+      });
+    }
+  }
+
+private:
+  bool isSystemFile(const FileEntry *FE) {
+    auto UID = FE->getUID();
+    return IsSystemByUID.size() > UID && IsSystemByUID[UID];
+  }
+
+  ArrayRef<const FileEntry *> getEntries() const {
+    return Entries.getArrayRef();
+  }
+
+  bool needSystemDependencies() override {
+    return RecordOpts.RecordSystemDependencies;
+  }
+
+  bool sawDependency(StringRef Filename, bool FromModule,
+                     bool IsSystem, bool IsModuleFile, bool IsMissing) override {
+    bool sawIt = DependencyCollector::sawDependency(Filename, FromModule,
+                                                    IsSystem, IsModuleFile,
+                                                    IsMissing);
+    if (auto *FE = SourceMgr->getFileManager().getFile(Filename)) {
+      if (sawIt)
+        Entries.insert(FE);
+      // Record system-ness for all files that we pass through.
+      if (IsSystemByUID.size() < FE->getUID()+1)
+        IsSystemByUID.resize(FE->getUID()+1);
+        IsSystemByUID[FE->getUID()] = IsSystem || isInSysroot(Filename);
+    }
+    return sawIt;
+  }
+
+  bool isInSysroot(StringRef Filename) {
+    return !SysrootPath.empty() && Filename.startswith(SysrootPath);
+  }
+};
+
+class IndexRecordActionBase {
+protected:
+  RecordingOptions RecordOpts;
+  IndexDataRecorder Recorder;
+  IndexingContext IndexCtx;
+  SourceFilesIndexDependencyCollector DepCollector;
+
+  IndexRecordActionBase(IndexingOptions IndexOpts, RecordingOptions recordOpts)
+    : RecordOpts(std::move(recordOpts)),
+      IndexCtx(IndexOpts, Recorder),
+      DepCollector(IndexCtx, RecordOpts) {
+  }
+
+  std::unique_ptr<IndexASTConsumer>
+  createIndexASTConsumer(CompilerInstance &CI) {
+    IndexCtx.setSysrootPath(CI.getHeaderSearchOpts().Sysroot);
+    Recorder.init(&IndexCtx, CI);
+
+    Preprocessor &PP = CI.getPreprocessor();
+    DepCollector.setSourceManager(&CI.getSourceManager());
+    DepCollector.setSysrootPath(IndexCtx.getSysrootPath());
+    DepCollector.attachToPreprocessor(PP);
+
+    return llvm::make_unique<IndexASTConsumer>(IndexCtx);
+  }
+
+  void finish(CompilerInstance &CI);
+};
+
+class IndexRecordAction : public ASTFrontendAction, IndexRecordActionBase {
+public:
+  IndexRecordAction(IndexingOptions IndexOpts, RecordingOptions RecordOpts)
+    : IndexRecordActionBase(std::move(IndexOpts), std::move(RecordOpts)) {}
+
+protected:
+  std::unique_ptr<ASTConsumer> CreateASTConsumer(CompilerInstance &CI,
+                                                 StringRef InFile) override {
+    return createIndexASTConsumer(CI);
+  }
+
+  void EndSourceFileAction() override {
+    FrontendAction::EndSourceFileAction();
+    finish(getCompilerInstance());
+  }
+};
+
+class WrappingIndexRecordAction : public WrapperFrontendAction, IndexRecordActionBase {
+  bool CreatedASTConsumer = false;
+
+public:
+  WrappingIndexRecordAction(std::unique_ptr<FrontendAction> WrappedAction,
+                            IndexingOptions IndexOpts,
+                            RecordingOptions RecordOpts)
+    : WrapperFrontendAction(std::move(WrappedAction)),
+      IndexRecordActionBase(std::move(IndexOpts), std::move(RecordOpts)) {}
+
+protected:
+  std::unique_ptr<ASTConsumer> CreateASTConsumer(CompilerInstance &CI,
+                                                 StringRef InFile) override {
+    auto OtherConsumer = WrapperFrontendAction::CreateASTConsumer(CI, InFile);
+    if (!OtherConsumer)
+      return nullptr;
+
+    CreatedASTConsumer = true;
+    std::vector<std::unique_ptr<ASTConsumer>> Consumers;
+    Consumers.push_back(std::move(OtherConsumer));
+    Consumers.push_back(createIndexASTConsumer(CI));
+    return llvm::make_unique<MultiplexConsumer>(std::move(Consumers));
+  }
+
+  void EndSourceFileAction() override {
+    // Invoke wrapped action's method.
+    WrapperFrontendAction::EndSourceFileAction();
+    if (CreatedASTConsumer)
+      finish(getCompilerInstance());
+  }
+};
+
+} // anonymous namespace
+
+static std::string getClangVersion() {
+  // Try picking the version from an Apple Clang tag.
+  std::string RepositoryPath = getClangRepositoryPath();
+  StringRef BuildNumber = StringRef(RepositoryPath);
+  size_t DashOffset = BuildNumber.find('-');
+  if (BuildNumber.startswith("clang") && DashOffset != StringRef::npos) {
+    BuildNumber = BuildNumber.substr(DashOffset + 1);
+    return BuildNumber;
+  }
+  // Fallback to the generic version.
+  return CLANG_VERSION_STRING;
+}
+
+static void writeUnitData(const CompilerInstance &CI,
+                          IndexDataRecorder &Recorder,
+                          IndexDependencyProvider &DepProvider,
+                          IndexingOptions IndexOpts,
+                          RecordingOptions RecordOpts,
+                          StringRef OutputFile,
+                          const FileEntry *RootFile,
+                          Module *UnitModule,
+                          StringRef SysrootPath);
+
+void IndexRecordActionBase::finish(CompilerInstance &CI) {
+  // We may emit more diagnostics so do the begin/end source file invocations
+  // on the diagnostic client.
+  // FIXME: FrontendAction::EndSourceFile() should probably not call
+  // CI.getDiagnosticClient().EndSourceFile()' until after it has called
+  // 'EndSourceFileAction()', so that code executing during EndSourceFileAction()
+  // can emit diagnostics. If this is fixed, DiagClientBeginEndRAII can go away.
+  struct DiagClientBeginEndRAII {
+    CompilerInstance &CI;
+    DiagClientBeginEndRAII(CompilerInstance &CI) : CI(CI) {
+      CI.getDiagnosticClient().BeginSourceFile(CI.getLangOpts());
+    }
+    ~DiagClientBeginEndRAII() {
+      CI.getDiagnosticClient().EndSourceFile();
+    }
+  } diagClientBeginEndRAII(CI);
+
+  SourceManager &SM = CI.getSourceManager();
+  DiagnosticsEngine &Diag = CI.getDiagnostics();
+  HeaderSearch &HS = CI.getPreprocessor().getHeaderSearchInfo();
+  StringRef DataPath = RecordOpts.DataDirPath;
+
+  std::string Error;
+  if (IndexUnitWriter::initIndexDirectory(DataPath, Error)) {
+    unsigned DiagID = Diag.getCustomDiagID(
+        DiagnosticsEngine::Error, "failed creating index directory %0");
+    Diag.Report(DiagID) << Error;
+    return;
+  }
+
+  std::string OutputFile = CI.getFrontendOpts().OutputFile;
+  if (OutputFile.empty()) {
+    OutputFile = CI.getFrontendOpts().Inputs[0].getFile();
+    OutputFile += ".o";
+  }
+
+  const FileEntry *RootFile = nullptr;
+  Module *UnitMod = nullptr;
+  bool isModuleGeneration = CI.getLangOpts().isCompilingModule();
+  if (!isModuleGeneration &&
+      CI.getFrontendOpts().ProgramAction != frontend::GeneratePCH) {
+    RootFile = SM.getFileEntryForID(SM.getMainFileID());
+  }
+  if (isModuleGeneration) {
+    UnitMod = HS.lookupModule(CI.getLangOpts().CurrentModule,
+                              /*AllowSearch=*/false);
+  }
+
+  writeUnitData(CI, Recorder, DepCollector, IndexCtx.getIndexOpts(), RecordOpts,
+                OutputFile, RootFile, UnitMod,
+                IndexCtx.getSysrootPath());
+}
+
+/// Checks if the unit file exists for module file, if it doesn't it generates
+/// index data for it.
+static bool produceIndexDataForModuleFile(
+                                      serialization::ModuleFile &Mod,
+                                      const CompilerInstance &CI,
+                                      IndexingOptions IndexOpts,
+                                      RecordingOptions RecordOpts,
+                                      IndexUnitWriter &ParentUnitWriter);
+
+static void writeUnitData(const CompilerInstance &CI,
+                          IndexDataRecorder &Recorder,
+                          IndexDependencyProvider &DepProvider,
+                          IndexingOptions IndexOpts,
+                          RecordingOptions RecordOpts,
+                          StringRef OutputFile,
+                          const FileEntry *RootFile,
+                          Module *UnitModule,
+                          StringRef SysrootPath) {
+
+  SourceManager &SM = CI.getSourceManager();
+  DiagnosticsEngine &Diag = CI.getDiagnostics();
+  HeaderSearch &HS = CI.getPreprocessor().getHeaderSearchInfo();
+  StringRef DataPath = RecordOpts.DataDirPath;
+  bool IsSystemUnit = UnitModule ? UnitModule->IsSystem : false;
+  bool IsModuleUnit = UnitModule != nullptr;
+  bool IsDebugCompilation = CI.getCodeGenOpts().OptimizationLevel == 0;
+  std::string ModuleName = UnitModule ? UnitModule->getFullModuleName() : std::string();
+
+  auto getModuleInfo = [](writer::OpaqueModule mod, SmallVectorImpl<char> &Scratch) -> writer::ModuleInfo {
+    assert(mod);
+    writer::ModuleInfo info;
+    std::string fullName = static_cast<const Module*>(mod)->getFullModuleName();
+    unsigned offset = Scratch.size();
+    Scratch.append(fullName.begin(), fullName.end());
+    info.Name = StringRef(Scratch.data()+offset, fullName.size());
+    return info;
+  };
+
+  auto findModuleForHeader = [&](const FileEntry *FE) -> Module * {
+    if (!UnitModule)
+      return nullptr;
+    if (auto Mod = HS.findModuleForHeader(FE).getModule())
+      if (Mod->isSubModuleOf(UnitModule))
+        return Mod;
+    return nullptr;
+  };
+
+  IndexUnitWriter UnitWriter(CI.getFileManager(),
+                             DataPath,
+                             "clang", getClangVersion(),
+                             OutputFile,
+                             ModuleName,
+                             RootFile,
+                             IsSystemUnit,
+                             IsModuleUnit,
+                             IsDebugCompilation,
+                             CI.getTargetOpts().Triple,
+                             SysrootPath,
+                             getModuleInfo);
+
+  DepProvider.visitFileDependencies(CI, [&](const FileEntry *FE, bool isSystemFile) {
+    UnitWriter.addFileDependency(FE, isSystemFile, findModuleForHeader(FE));
+  });
+  DepProvider.visitIncludes([&](const FileEntry *Source, unsigned Line, const FileEntry *Target) {
+    UnitWriter.addInclude(Source, Line, Target);
+  });
+  DepProvider.visitModuleImports(CI, [&](serialization::ModuleFile &Mod, bool isSystemMod) {
+    Module *UnitMod = HS.lookupModule(Mod.ModuleName, /*AllowSearch=*/false);
+    UnitWriter.addASTFileDependency(Mod.File, isSystemMod, UnitMod);
+    if (Mod.isModule()) {
+      produceIndexDataForModuleFile(Mod, CI, IndexOpts, RecordOpts, UnitWriter);
+    }
+  });
+
+  ClangIndexRecordWriter RecordWriter(CI.getASTContext(), RecordOpts);
+  for (auto I = Recorder.record_begin(), E = Recorder.record_end(); I != E; ++I) {
+    FileID FID = I->first;
+    const FileIndexRecord &Rec = *I->second;
+    const FileEntry *FE = SM.getFileEntryForID(FID);
+    std::string RecordFile;
+    std::string Error;
+
+    if (RecordWriter.writeRecord(FE->getName(), Rec, Error, &RecordFile)) {
+      unsigned DiagID = Diag.getCustomDiagID(DiagnosticsEngine::Error,
+                                             "failed writing record '%0': %1");
+      Diag.Report(DiagID) << RecordFile << Error;
+      return;
+    }
+    UnitWriter.addRecordFile(RecordFile, FE, Rec.isSystem(),
+                             findModuleForHeader(FE));
+  }
+
+  std::string Error;
+  if (UnitWriter.write(Error)) {
+    unsigned DiagID = Diag.getCustomDiagID(DiagnosticsEngine::Error,
+                                           "failed writing unit data: %0");
+    Diag.Report(DiagID) << Error;
+    return;
+  }
+}
+
+namespace {
+class ModuleFileIndexDependencyCollector : public IndexDependencyProvider {
+  serialization::ModuleFile &ModFile;
+  RecordingOptions RecordOpts;
+
+public:
+  ModuleFileIndexDependencyCollector(serialization::ModuleFile &Mod,
+                                     RecordingOptions recordOpts)
+  : ModFile(Mod), RecordOpts(recordOpts) {}
+
+  void visitFileDependencies(const CompilerInstance &CI,
+      llvm::function_ref<void(const FileEntry *FE, bool isSystem)> visitor) override {
+    auto Reader = CI.getModuleManager();
+    Reader->visitInputFiles(ModFile, RecordOpts.RecordSystemDependencies,
+                            /*Complain=*/false,
+                        [&](const serialization::InputFile &IF, bool isSystem) {
+      auto *FE = IF.getFile();
+      if (!FE)
+        return;
+      // Ignore module map files, they are not as important to track as source
+      // files and they may be auto-generated which would create an undesirable
+      // dependency on an intermediate build byproduct.
+      if (FE->getName().endswith("module.modulemap"))
+        return;
+
+      visitor(FE, isSystem);
+    });
+  }
+
+  void visitIncludes(
+           llvm::function_ref<void(const FileEntry *Source, unsigned Line,
+                                   const FileEntry *Target)> visitor) override {
+   // FIXME: Module files without a preprocessing record do not have info about
+   // include locations. Serialize enough data to be able to retrieve such info.
+  }
+
+  void visitModuleImports(const CompilerInstance &CI,
+                 llvm::function_ref<void(serialization::ModuleFile &Mod,
+                                         bool isSystem)> visitor) override {
+    HeaderSearch &HS = CI.getPreprocessor().getHeaderSearchInfo();
+    for (auto *Mod : ModFile.Imports) {
+      bool isSystemMod = false;
+      if (auto *M = HS.lookupModule(Mod->ModuleName, /*AllowSearch=*/false))
+        isSystemMod = M->IsSystem;
+      if (!isSystemMod || RecordOpts.RecordSystemDependencies)
+        visitor(*Mod, isSystemMod);
+    }
+  }
+};
+} // anonymous namespace.
+
+static void indexModule(serialization::ModuleFile &Mod,
+                        const CompilerInstance &CI,
+                        IndexingOptions IndexOpts,
+                        RecordingOptions RecordOpts) {
+  DiagnosticsEngine &Diag = CI.getDiagnostics();
+  Diag.Report(Mod.ImportLoc, diag::remark_index_producing_module_file_data)
+    << Mod.FileName;
+
+  StringRef SysrootPath = CI.getHeaderSearchOpts().Sysroot;
+  HeaderSearch &HS = CI.getPreprocessor().getHeaderSearchInfo();
+  Module *UnitMod = HS.lookupModule(Mod.ModuleName, /*AllowSearch=*/false);
+
+  IndexDataRecorder Recorder;
+  IndexingContext IndexCtx(IndexOpts, Recorder);
+
+  IndexCtx.setASTContext(CI.getASTContext());
+  IndexCtx.setSysrootPath(SysrootPath);
+  Recorder.init(&IndexCtx, CI);
+
+  for (const Decl *D : CI.getModuleManager()->getModuleFileLevelDecls(Mod)) {
+    IndexCtx.indexTopLevelDecl(D);
+  }
+  Recorder.finish();
+
+  ModuleFileIndexDependencyCollector DepCollector(Mod, RecordOpts);
+  writeUnitData(CI, Recorder, DepCollector, IndexOpts, RecordOpts,
+                Mod.FileName, /*RootFile=*/nullptr, UnitMod, SysrootPath);
+
+}
+
+static bool produceIndexDataForModuleFile(
+                                      serialization::ModuleFile &Mod,
+                                      const CompilerInstance &CI,
+                                      IndexingOptions IndexOpts,
+                                      RecordingOptions RecordOpts,
+                                      IndexUnitWriter &ParentUnitWriter) {
+  DiagnosticsEngine &Diag = CI.getDiagnostics();
+  std::string Error;
+  // We don't do timestamp check with the PCM file, on purpose. The PCM may get
+  // touched for various reasons which would cause unnecessary work to emit
+  // index data. User modules normally will get rebuilt and their index data
+  // re-emitted, and system modules are generally stable (and they can also can
+  // get rebuilt along with their index data).
+  auto IsUptodateOpt = ParentUnitWriter.isUnitUpToDateForOutputFile(Mod.FileName, None, Error);
+  if (!IsUptodateOpt.hasValue()) {
+    unsigned DiagID = Diag.getCustomDiagID(DiagnosticsEngine::Error,
+                                           "failed file status check: %0");
+    Diag.Report(DiagID) << Error;
+    return false;
+  }
+  if (*IsUptodateOpt)
+    return false;
+
+  indexModule(Mod, CI, IndexOpts, RecordOpts);
+  return true;
+}
+
+static std::unique_ptr<FrontendAction>
+createIndexDataRecordingAction(IndexingOptions IndexOpts,
+                               RecordingOptions RecordOpts,
+                               std::unique_ptr<FrontendAction> WrappedAction) {
+  if (WrappedAction)
+    return llvm::make_unique<WrappingIndexRecordAction>(std::move(WrappedAction),
+                                                        std::move(IndexOpts),
+                                                        std::move(RecordOpts));
+  return llvm::make_unique<IndexRecordAction>(std::move(IndexOpts),
+                                              std::move(RecordOpts));
+}
+
+static std::pair<IndexingOptions, RecordingOptions>
+getIndexOptionsFromFrontendOptions(const FrontendOptions &FEOpts) {
+  index::IndexingOptions IndexOpts;
+  index::RecordingOptions RecordOpts;
+  RecordOpts.DataDirPath = FEOpts.IndexStorePath;
+  if (FEOpts.IndexIgnoreSystemSymbols) {
+    IndexOpts.SystemSymbolFilter =
+    index::IndexingOptions::SystemSymbolFilterKind::None;
+  }
+  RecordOpts.RecordSymbolCodeGenName = FEOpts.IndexRecordCodegenName;
+  return { IndexOpts, RecordOpts };
+}
+
+std::unique_ptr<FrontendAction>
+index::createIndexDataRecordingAction(const FrontendOptions &FEOpts,
+                                std::unique_ptr<FrontendAction> WrappedAction) {
+  index::IndexingOptions IndexOpts;
+  index::RecordingOptions RecordOpts;
+  std::tie(IndexOpts, RecordOpts) = getIndexOptionsFromFrontendOptions(FEOpts);
+  return ::createIndexDataRecordingAction(IndexOpts, RecordOpts,
+                                          std::move(WrappedAction));
+}
+
+bool index::emitIndexDataForModuleFile(const Module *Mod,
+                                       const CompilerInstance &CI,
+                                       IndexUnitWriter &ParentUnitWriter) {
+  index::IndexingOptions IndexOpts;
+  index::RecordingOptions RecordOpts;
+  std::tie(IndexOpts, RecordOpts) = getIndexOptionsFromFrontendOptions(CI.getFrontendOpts());
+
+  auto astReader = CI.getModuleManager();
+  serialization::ModuleFile *ModFile = astReader->getModuleManager().lookup(Mod->getASTFile());
+  assert(ModFile && "no module file loaded for module ?");
+  return produceIndexDataForModuleFile(*ModFile, CI, IndexOpts, RecordOpts, ParentUnitWriter);
+}

lib/Index/IndexingContext.h

 //===- IndexingContext.h - Indexing context data ----------------*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_LIB_INDEX_INDEXINGCONTEXT_H
 #define LLVM_CLANG_LIB_INDEX_INDEXINGCONTEXT_H
 
 #include "clang/Basic/LLVM.h"
+#include "clang/Basic/SourceLocation.h"
 #include "clang/Index/IndexSymbol.h"
 #include "clang/Index/IndexingAction.h"
 #include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseMap.h"
 
 namespace clang {
   class ASTContext;
   class Decl;
   class DeclGroupRef;
   class ImportDecl;
   class TagDecl;
   class TypeSourceInfo;
   class NamedDecl;
   class ObjCMethodDecl;
   class DeclContext;
   class NestedNameSpecifierLoc;
   class Stmt;
   class Expr;
   class TypeLoc;
-  class SourceLocation;
+  class DirectoryEntry;
 
 namespace index {
   class IndexDataConsumer;
 
 class IndexingContext {
   IndexingOptions IndexOpts;
   IndexDataConsumer &DataConsumer;
   ASTContext *Ctx = nullptr;
+  std::string SysrootPath;
+
+  // Records whether a directory entry is system or not.
+  llvm::DenseMap<const DirectoryEntry *, bool> DirEntries;
+  // Keeps track of the last check for whether a FileID is system or
+  // not. This is used to speed up isSystemFile() call.
+  std::pair<FileID, bool> LastFileCheck;
 
 public:
   IndexingContext(IndexingOptions IndexOpts, IndexDataConsumer &DataConsumer)
     : IndexOpts(IndexOpts), DataConsumer(DataConsumer) {}
 
   const IndexingOptions &getIndexOpts() const { return IndexOpts; }
   IndexDataConsumer &getDataConsumer() { return DataConsumer; }
 
   void setASTContext(ASTContext &ctx) { Ctx = &ctx; }
+  void setSysrootPath(StringRef path);
+  StringRef getSysrootPath() const { return SysrootPath; }
+
+  bool isSystemFile(FileID FID);
 
   bool shouldIndex(const Decl *D);
 
   const LangOptions &getLangOpts() const;
 
   bool shouldSuppressRefs() const {
     return false;
   }

lib/Index/IndexingContext.cpp

if (Loc.isInvalid())
     return true;
 
   FileID FID;
   unsigned Offset;
   std::tie(FID, Offset) = SM.getDecomposedLoc(Loc);
   if (FID.isInvalid())
     return true;
 
-  bool Invalid = false;
-  const SrcMgr::SLocEntry &SEntry = SM.getSLocEntry(FID, &Invalid);
-  if (Invalid || !SEntry.isFile())
-    return true;
-
-  if (SEntry.getFile().getFileCharacteristic() != SrcMgr::C_User) {
+  if (isSystemFile(FID)) {
     switch (IndexOpts.SystemSymbolFilter) {
     case IndexingOptions::SystemSymbolFilterKind::None:
       return true;
     case IndexingOptions::SystemSymbolFilterKind::DeclarationsOnly:
     case IndexingOptions::SystemSymbolFilterKind::All:
       break;
     }
   }
if (isa<ObjCIvarDecl>(D))
     return false;
   if (isa<ObjCMethodDecl>(D))
     return false;
   if (isa<ImportDecl>(D))
     return false;
   return true;
 }
 
+void IndexingContext::setSysrootPath(StringRef path) {
+  // Ignore sysroot path if it points to root, otherwise every header will be
+  // treated as system one.
+  if (path == "/")
+    path = StringRef();
+  SysrootPath = path;
+}
+
+bool IndexingContext::isSystemFile(FileID FID) {
+  if (LastFileCheck.first == FID)
+    return LastFileCheck.second;
+
+  auto result = [&](bool res) -> bool {
+    LastFileCheck = { FID, res };
+    return res;
+  };
+
+  bool Invalid = false;
+  const SrcMgr::SLocEntry &SEntry =
+    Ctx->getSourceManager().getSLocEntry(FID, &Invalid);
+  if (Invalid || !SEntry.isFile())
+    return result(false);
+
+  const SrcMgr::FileInfo &FI = SEntry.getFile();
+  if (FI.getFileCharacteristic() != SrcMgr::C_User)
+    return result(true);
+
+  auto *CC = FI.getContentCache();
+  if (!CC)
+    return result(false);
+  auto *FE = CC->OrigEntry;
+  if (!FE)
+    return result(false);
+
+  if (SysrootPath.empty())
+    return result(false);
+
+  // Check if directory is in sysroot so that we can consider system headers
+  // even the headers found via a user framework search path, pointing inside
+  // sysroot.
+  auto dirEntry = FE->getDir();
+  auto pair = DirEntries.insert(std::make_pair(dirEntry, false));
+  bool &isSystemDir = pair.first->second;
+  bool wasInserted = pair.second;
+  if (wasInserted) {
+    isSystemDir = StringRef(dirEntry->getName()).startswith(SysrootPath);
+  }
+  return result(isSystemDir);
+}
+
 static const CXXRecordDecl *
 getDeclContextForTemplateInstationPattern(const Decl *D) {
   if (const auto *CTSD =
           dyn_cast<ClassTemplateSpecializationDecl>(D->getDeclContext()))
     return CTSD->getTemplateInstantiationPattern();
   else if (const auto *RD = dyn_cast<CXXRecordDecl>(D->getDeclContext()))
     return RD->getInstantiatedFromMemberClass();
   return nullptr;

 
 bool IndexingContext::handleDeclOccurrence(const Decl *D, SourceLocation Loc,
                                            bool IsRef, const Decl *Parent,
                                            SymbolRoleSet Roles,
                                            ArrayRef<SymbolRelation> Relations,
                                            const Expr *OrigE,
                                            const Decl *OrigD,
                                            const DeclContext *ContainerDC) {
-  if (D->isImplicit() && !isa<ObjCMethodDecl>(D))
+  if (D->isImplicit() && !(isa<ObjCMethodDecl>(D) || isa<ObjCIvarDecl>(D)))
     return true;
   if (!isa<NamedDecl>(D) || shouldSkipNamelessDecl(cast<NamedDecl>(D)))
     return true;
 
   SourceManager &SM = Ctx->getSourceManager();
   Loc = SM.getFileLoc(Loc);
   if (Loc.isInvalid())
     return true;
 
   FileID FID;
   unsigned Offset;
   std::tie(FID, Offset) = SM.getDecomposedLoc(Loc);
   if (FID.isInvalid())
     return true;
 
-  bool Invalid = false;
-  const SrcMgr::SLocEntry &SEntry = SM.getSLocEntry(FID, &Invalid);
-  if (Invalid || !SEntry.isFile())
-    return true;
-
-  if (SEntry.getFile().getFileCharacteristic() != SrcMgr::C_User) {
+  if (isSystemFile(FID)) {
     switch (IndexOpts.SystemSymbolFilter) {
     case IndexingOptions::SystemSymbolFilterKind::None:
       return true;
     case IndexingOptions::SystemSymbolFilterKind::DeclarationsOnly:
       if (!shouldReportOccurrenceForSystemDeclOnlyMode(IsRef, Roles, Relations))
         return true;
       break;
     case IndexingOptions::SystemSymbolFilterKind::All:

test/Index/Store/Inputs/head.h

+
+extern void test1_func(void);
+extern void test2_func(void);

test/Index/Store/Inputs/json.c.json

+{
+  "files": [
+    "/test1.o",
+    "/Inputs/test1.c",
+    "/Inputs/head.h",
+    "/test2.o",
+    "/Inputs/test2.c",
+    "/test3.o",
+    "/Inputs/test3.cpp"
+  ],
+  "symbols": [
+    {
+      "kind": "function",
+      "lang": "C",
+      "usr": "c:@F@test1_func",
+      "name": "test1_func",
+      "codegen": "_test1_func",
+      "roles": "Decl,Def"
+    },
+    {
+      "kind": "function",
+      "lang": "C",
+      "usr": "c:@F@test2_func",
+      "name": "test2_func",
+      "codegen": "_test2_func",
+      "roles": "Decl,Def"
+    },
+    {
+      "kind": "class",
+      "lang": "C++",
+      "usr": "c:@S@Base",
+      "name": "Base",
+      "roles": "Def,Ref,RelBase,RelCont"
+    },
+    {
+      "kind": "class",
+      "lang": "C++",
+      "usr": "c:@S@Sub",
+      "name": "Sub",
+      "roles": "Def",
+      "rel-roles": "RelBase,RelCont"
+    }
+  ],
+  "records": [
+    {
+      "occurrences": [
+        {
+          "symbol": 0,
+          "line": 3,
+          "col": 6,
+          "roles": "Def"
+        }
+      ]
+    },
+    {
+      "occurrences": [
+        {
+          "symbol": 0,
+          "line": 2,
+          "col": 13,
+          "roles": "Decl"
+        },
+        {
+          "symbol": 1,
+          "line": 3,
+          "col": 13,
+          "roles": "Decl"
+        }
+      ]
+    },
+    {
+      "occurrences": [
+        {
+          "symbol": 1,
+          "line": 3,
+          "col": 6,
+          "roles": "Def"
+        }
+      ]
+    },
+    {
+      "occurrences": [
+        {
+          "symbol": 2,
+          "line": 1,
+          "col": 7,
+          "roles": "Def"
+        },
+        {
+          "symbol": 3,
+          "line": 2,
+          "col": 7,
+          "roles": "Def"
+        },
+        {
+          "symbol": 2,
+          "line": 2,
+          "col": 20,
+          "roles": "Ref,RelBase,RelCont",
+          "relations": [
+            {
+              "symbol": 3,
+              "rel-roles": "RelBase,RelCont"
+            }
+          ]
+
+        }
+      ]
+    }
+  ],
+  "units": [
+    {
+      "triple": "x86_64-apple-macosx10.7.0",
+      "out-file": 0,
+      "sources": [
+        {
+          "file": 1,
+          "records": [0]
+        },
+        {
+          "file": 2,
+          "records": [1]
+        }
+      ]
+    },
+    {
+      "triple": "x86_64-apple-macosx10.7.0",
+      "out-file": 3,
+      "sources": [
+        {
+          "file": 4,
+          "records": [2]
+        },
+        {
+          "file": 2,
+          "records": [1]
+        }
+      ]
+    },
+    {
+      "triple": "x86_64-apple-macosx10.7.0",
+      "out-file": 5,
+      "sources": [
+        {
+          "file": 6,
+          "records": [3]
+        }
+      ]
+    }
+  ]
+}

test/Index/Store/Inputs/module/ModDep.h

+#include "ModTop.h"
+
+void ModDep_func(ModTopStruct s);

test/Index/Store/Inputs/module/ModSystem.h

+
+typedef struct {} ModSystemStruct;
+
+void ModSystem_func(void);

test/Index/Store/Inputs/module/ModTop.h

+
+typedef struct {} ModTopStruct;
+
+void ModTop_func(void);

test/Index/Store/Inputs/module/ModTopSub1.h

+void ModTopSub1_func(void);

test/Index/Store/Inputs/module/ModTopSub2.h

+// This header has no symbols, intended to show up as file dependency.

test/Index/Store/Inputs/module/module.modulemap

+module ModTop {
+  header "ModTop.h"
+  export *
+  module Sub1 {
+    header "ModTopSub1.h"
+  }
+  module Sub2 {
+    header "ModTopSub2.h"
+  }
+}
+module ModDep { header "ModDep.h" export * }
+module ModSystem [system] { header "ModSystem.h" export * }

test/Index/Store/Inputs/overlay.yaml

+{
+  'version': 0,
+  'roots': [{ 'type': 'file', 'name': 'OUT_DIR/using-overlay.h',
+             'external-contents': 'INPUT_DIR/using-overlay.h'
+           }]
+}

test/Index/Store/Inputs/print-unit.h

+#include "head.h"
+#include "using-overlay.h"

test/Index/Store/Inputs/sys/another.h

+
+extern void sys_another_func(void);

test/Index/Store/Inputs/sys/syshead.h

+
+#include "another.h"
+
+extern void sys_test1_func(void);

test/Index/Store/Inputs/test1.c

+#include "head.h"
+
+void test1_func(void) {}

test/Index/Store/Inputs/test2.c

+#include "head.h"
+
+void test2_func(void) {}

test/Index/Store/Inputs/test3.cpp

+class Base {};
+class Sub : public Base {};

test/Index/Store/Inputs/using-overlay.h

+void using_overlay(void);

test/Index/Store/assembly-invocation.c

+// Make sure it doesn't crash.
+// RUN: %clang -target x86_64-apple-macosx10.7 -S %s -o %t.s
+// RUN: env CLANG_PROJECT_INDEX_PATH=%t.idx %clang -target x86_64-apple-macosx10.7 -c %t.s -o %t.o

test/Index/Store/external-source-symbol-hash.m

+// RUN: rm -rf %t.idx
+// RUN: %clang_cc1 %s -index-store-path %t.idx -D USE_EXTERNAL
+// RUN: c-index-test core -print-record %t.idx | FileCheck %s
+// RUN: %clang_cc1 %s -index-store-path %t.idx
+// RUN: find %t.idx/*/records -name "external-source-symbol-hash*" | count 2
+
+#ifdef USE_EXTERNAL
+#  define EXT_DECL(mod_name) __attribute__((external_source_symbol(language="Swift", defined_in=mod_name)))
+#else
+#  define EXT_DECL(mod_name)
+#endif
+
+#define NS_ENUM(_name, _type) enum _name:_type _name; enum _name : _type
+
+// Forward declarations should pick up the attribute from later decls
+@protocol P1;
+// CHECK: [[@LINE-1]]:11 | protocol/Swift | c:@M@some_module@objc(pl)P1 | Ref | rel: 0
+@class I2;
+// CHECK: [[@LINE-1]]:8 | class/Swift | c:@M@other_module@objc(cs)I2 | Ref | rel: 0
+enum E3: int;
+// CHECK: [[@LINE-1]]:6 | enum/Swift | c:@M@third_module@E@E3 | Ref | rel: 0
+
+void test(id<P1> first, I2 *second, enum E3 third) {}
+// CHECK: [[@LINE-1]]:14 | protocol/Swift | c:@M@some_module@objc(pl)P1 | Ref,RelCont | rel: 1
+// CHECK: [[@LINE-2]]:25 | class/Swift | c:@M@other_module@objc(cs)I2 | Ref,RelCont | rel: 1
+// CHECK: [[@LINE-3]]:42 | enum/Swift | c:@M@third_module@E@E3 | Ref,RelCont | rel: 1
+
+EXT_DECL("some_module")
+@protocol P1
+// CHECK: [[@LINE-1]]:11 | protocol/Swift | c:@M@some_module@objc(pl)P1 | Decl | rel: 0
+-(void)method;
+// CHECK: [[@LINE-1]]:8 | instance-method/Swift | c:@M@some_module@objc(pl)P1(im)method | Decl,Dyn,RelChild | rel: 1
+@end
+
+EXT_DECL("other_module")
+@interface I2
+// CHECK: [[@LINE-1]]:12 | class/Swift | c:@M@other_module@objc(cs)I2 | Decl | rel: 0
+-(void)method;
+// CHECK: [[@LINE-1]]:8 | instance-method/Swift | c:@M@other_module@objc(cs)I2(im)method | Decl,Dyn,RelChild | rel: 1
+@end
+
+
+typedef NS_ENUM(E3, int) {
+// CHECK: [[@LINE-1]]:17 | enum/Swift | c:@M@third_module@E@E3 | Def | rel: 0
+  firstCase = 1,
+  // CHECK: [[@LINE-1]]:3 | enumerator/Swift | c:@M@third_module@E@E3@firstCase | Def,RelChild | rel: 1
+} EXT_DECL("third_module");

test/Index/Store/handle-prebuilt-module.m

+// XFAIL: linux
+
+// RUN: rm -rf %t
+// RUN: mkdir %t
+// RUN: %clang -arch x86_64 -mmacosx-version-min=10.7 -fsyntax-only %s -o %t.o -index-store-path %t/idx1 -fmodules -fmodules-cache-path=%t/mcp -I %S/Inputs/module -Rindex-store 2> %t.err1
+// RUN: %clang -arch x86_64 -mmacosx-version-min=10.7 -fsyntax-only %s -o %t.o -index-store-path %t/idx2 -fmodules -fmodules-cache-path=%t/mcp -I %S/Inputs/module -Rindex-store 2> %t.err2
+// RUN: %clang -arch x86_64 -mmacosx-version-min=10.7 -fsyntax-only %s -o %t.o -index-store-path %t/idx2 -fmodules -fmodules-cache-path=%t/mcp -I %S/Inputs/module -Rindex-store 2> %t.err3
+// RUN: FileCheck -input-file=%t.err1 -check-prefix=CREATING_MODULES %s -allow-empty
+// RUN: FileCheck -input-file=%t.err2 -check-prefix=CREATING_INDEX_DATA_FROM_MODULE_FILES %s
+// RUN: FileCheck -input-file=%t.err3 -check-prefix=EXISTING_INDEX_DATA_FROM_MODULE_FILES %s -allow-empty
+// RUN: c-index-test core -print-unit %t/idx1 > %t/all-units1.txt
+// RUN: c-index-test core -print-unit %t/idx2 > %t/all-units2.txt
+// RUN: c-index-test core -print-record %t/idx1 > %t/all-records1.txt
+// RUN: c-index-test core -print-record %t/idx2 > %t/all-records2.txt
+// RUN: diff -u %t/all-units1.txt %t/all-units2.txt
+// RUN: diff -u %t/all-records1.txt %t/all-records2.txt
+
+@import ModDep;
+
+// CREATING_MODULES-NOT: remark:
+
+// CREATING_INDEX_DATA_FROM_MODULE_FILES: remark: producing index data for module file {{.*}}ModDep{{.*}}.pcm
+// CREATING_INDEX_DATA_FROM_MODULE_FILES: remark: producing index data for module file {{.*}}ModTop{{.*}}.pcm
+
+// EXISTING_INDEX_DATA_FROM_MODULE_FILES-NOT: remark:

test/Index/Store/json-with-module.m

+// RUN: rm -rf %t.idx %t.mcp
+// RUN: %clang -arch x86_64 -mmacosx-version-min=10.7 -c %s -o %t.o -index-store-path %t.idx -fmodules -fmodules-cache-path=%t.mcp -Xclang -fdisable-module-hash -I %S/Inputs/module
+// RUN: c-index-test core -aggregate-json %t.idx -o %t.json
+// RUN: sed -e "s:%S::g" -e "s:%T::g" %t.json > %t.final.json
+// RUN: diff -u %s.json %t.final.json
+
+// XFAIL: linux
+
+@import ModDep;

test/Index/Store/json-with-module.m.json

+{
+  "files": [
+    "/json-with-module.m.tmp.mcp/ModDep.pcm",
+    "/json-with-module.m.tmp.mcp/ModTop.pcm",
+    "/Inputs/module/ModDep.h",
+    "/Inputs/module/ModTop.h",
+    "/Inputs/module/ModTopSub1.h",
+    "/Inputs/module/ModTopSub2.h",
+    "/json-with-module.m.tmp.o",
+    "/json-with-module.m",
+    "/Inputs/module/module.modulemap"
+  ],
+  "symbols": [
+    {
+      "kind": "function",
+      "lang": "C",
+      "usr": "c:@F@ModDep_func",
+      "name": "ModDep_func",
+      "roles": "Decl",
+      "rel-roles": "RelCont"
+    },
+    {
+      "kind": "type-alias",
+      "lang": "C",
+      "usr": "c:@T@ModTopStruct",
+      "name": "ModTopStruct",
+      "roles": "Def,Ref,RelCont"
+    },
+    {
+      "kind": "struct",
+      "lang": "C",
+      "usr": "c:@SA@ModTopStruct",
+      "name": "",
+      "roles": "Def"
+    },
+    {
+      "kind": "function",
+      "lang": "C",
+      "usr": "c:@F@ModTop_func",
+      "name": "ModTop_func",
+      "roles": "Decl"
+    },
+    {
+      "kind": "function",
+      "lang": "C",
+      "usr": "c:@F@ModTopSub1_func",
+      "name": "ModTopSub1_func",
+      "roles": "Decl"
+    }
+  ],
+  "records": [
+    {
+      "occurrences": [
+        {
+          "symbol": 0,
+          "line": 3,
+          "col": 6,
+          "roles": "Decl"
+        },
+        {
+          "symbol": 1,
+          "line": 3,
+          "col": 18,
+          "roles": "Ref,RelCont",
+          "relations": [
+            {
+              "symbol": 0,
+              "rel-roles": "RelCont"
+            }
+          ]
+
+        }
+      ]
+    },
+    {
+      "occurrences": [
+        {
+          "symbol": 2,
+          "line": 2,
+          "col": 9,
+          "roles": "Def"
+        },
+        {
+          "symbol": 1,
+          "line": 2,
+          "col": 19,
+          "roles": "Def"
+        },
+        {
+          "symbol": 3,
+          "line": 4,
+          "col": 6,
+          "roles": "Decl"
+        }
+      ]
+    },
+    {
+      "occurrences": [
+        {
+          "symbol": 4,
+          "line": 1,
+          "col": 6,
+          "roles": "Decl"
+        }
+      ]
+    }
+  ],
+  "units": [
+    {
+      "triple": "x86_64-apple-macosx10.7.0",
+      "out-file": 0,
+      "unit-dependencies": [1],
+      "sources": [
+        {
+          "file": 2,
+          "records": [0]
+        }
+      ]
+    },
+    {
+      "triple": "x86_64-apple-macosx10.7.0",
+      "out-file": 1,
+      "sources": [
+        {
+          "file": 3,
+          "records": [1]
+        },
+        {
+          "file": 4,
+          "records": [2]
+        },
+        {
+          "file": 5
+        }
+      ]
+    },
+    {
+      "triple": "x86_64-apple-macosx10.7.0",
+      "out-file": 6,
+      "unit-dependencies": [0],
+      "sources": [
+        {
+          "file": 7
+        },
+        {
+          "file": 8
+        }
+      ]
+    }
+  ]
+}

test/Index/Store/json-with-pch.c

+// RUN: rm -rf %t.idx
+// RUN: %clang -arch x86_64 -mmacosx-version-min=10.7 -x c-header %S/Inputs/head.h -o %t.h.pch -index-store-path %t.idx
+// RUN: %clang -arch x86_64 -mmacosx-version-min=10.7 -c %s -o %t.o -index-store-path %t.idx -include %t.h -Werror
+// RUN: c-index-test core -aggregate-json %t.idx -o %t.json
+// RUN: sed -e "s:%S::g" -e "s:%T::g" %t.json > %t.final.json
+// RUN: diff -u %s.json %t.final.json
+// XFAIL: linux
+int main() {
+  test1_func();
+}

test/Index/Store/json-with-pch.c.json

+{
+  "files": [
+    "/json-with-pch.c.tmp.h.pch",
+    "/Inputs/head.h",
+    "/json-with-pch.c.tmp.o",
+    "/json-with-pch.c"
+  ],
+  "symbols": [
+    {
+      "kind": "function",
+      "lang": "C",
+      "usr": "c:@F@test1_func",
+      "name": "test1_func",
+      "roles": "Decl,Ref,Call,RelCall,RelCont"
+    },
+    {
+      "kind": "function",
+      "lang": "C",
+      "usr": "c:@F@test2_func",
+      "name": "test2_func",
+      "roles": "Decl"
+    },
+    {
+      "kind": "function",
+      "lang": "C",
+      "usr": "c:@F@main",
+      "name": "main",
+      "roles": "Def",
+      "rel-roles": "RelCall,RelCont"
+    }
+  ],
+  "records": [
+    {
+      "occurrences": [
+        {
+          "symbol": 0,
+          "line": 2,
+          "col": 13,
+          "roles": "Decl"
+        },
+        {
+          "symbol": 1,
+          "line": 3,
+          "col": 13,
+          "roles": "Decl"
+        }
+      ]
+    },
+    {
+      "occurrences": [
+        {
+          "symbol": 2,
+          "line": 8,
+          "col": 5,
+          "roles": "Def"
+        },
+        {
+          "symbol": 0,
+          "line": 9,
+          "col": 3,
+          "roles": "Ref,Call,RelCall,RelCont",
+          "relations": [
+            {
+              "symbol": 2,
+              "rel-roles": "RelCall,RelCont"
+            }
+          ]
+
+        }
+      ]
+    }
+  ],
+  "units": [
+    {
+      "triple": "x86_64-apple-macosx10.7.0",
+      "out-file": 0,
+      "sources": [
+        {
+          "file": 1,
+          "records": [0]
+        }
+      ]
+    },
+    {
+      "triple": "x86_64-apple-macosx10.7.0",
+      "out-file": 2,
+      "unit-dependencies": [0],
+      "sources": [
+        {
+          "file": 3,
+          "records": [1]
+        }
+      ]
+    }
+  ]
+}

test/Index/Store/json.c

+// RUN: rm -rf %t.idx
+// RUN: mkdir -p %t.o
+// RUN: env CLANG_PROJECT_INDEX_PATH=%t.idx %clang -target x86_64-apple-macosx10.7 -arch x86_64 -mmacosx-version-min=10.7 -c %S/Inputs/test1.c -o %t.o/test1.o
+// RUN: env CLANG_PROJECT_INDEX_PATH=%t.idx %clang -target x86_64-apple-macosx10.7 -arch x86_64 -mmacosx-version-min=10.7 -c %S/Inputs/test2.c -o %t.o/test2.o
+// RUN: env CLANG_PROJECT_INDEX_PATH=%t.idx %clang -target x86_64-apple-macosx10.7 -arch x86_64 -mmacosx-version-min=10.7 -c %S/Inputs/test3.cpp -o %t.o/test3.o
+// RUN: c-index-test core -aggregate-json %t.idx -o %t.json
+// RUN: sed -e "s:%S::g" -e "s:%t.o::g" %t.json > %t.final.json
+// RUN: diff -u %S/Inputs/json.c.json %t.final.json
+
+// XFAIL: linux

test/Index/Store/print-record.mm

+// RUN: rm -rf %t.idx
+// RUN: %clang_cc1 %s -index-store-path %t.idx
+// RUN: c-index-test core -print-record %t.idx | FileCheck %s
+
+// XFAIL: linux
+
+@class MyCls;
+
+@interface MyCls
+@end
+
+// CHECK: [[@LINE+2]]:6 | function/C | c:@F@foo#*$objc(cs)MyCls# | Decl | rel: 0
+// CHECK: [[@LINE+1]]:10 | class/ObjC | c:objc(cs)MyCls | Ref,RelCont | rel: 1
+void foo(MyCls *p);
+
+
+// RANGE-NOT: before_range
+void before_range();
+
+// RANGE: [[@LINE+1]]:6 | function/C | c:@F@in_range1# | Decl
+void in_range1();
+// RANGE: [[@LINE+1]]:6 | function/C | c:@F@in_range2# | Decl
+void in_range2();
+
+// RANGE-NOT: after_range
+void after_range();
+
+// RUN: c-index-test core -print-record %t.idx -filepath %s:21:23 | FileCheck -check-prefix=RANGE %s

test/Index/Store/print-unit.c

+// XFAIL: linux
+
+#include "print-unit.h"
+#include "syshead.h"
+
+void foo(int i);
+
+// RUN: rm -rf %t
+// RUN: %clang_cc1 -I %S/Inputs -isystem %S/Inputs/sys -index-store-path %t/idx %s -triple x86_64-apple-macosx10.8
+// RUN: c-index-test core -print-unit %t/idx | FileCheck %s
+// RUN: %clang_cc1 -I %S/Inputs -isystem %S/Inputs/sys -index-store-path %t/idx_opt1 %s -triple x86_64-apple-macosx10.8 -O2
+// RUN: c-index-test core -print-unit %t/idx_opt1 | FileCheck %s -check-prefix=OPT
+// RUN: %clang_cc1 -I %S/Inputs -isystem %S/Inputs/sys -index-store-path %t/idx_opt2 %s -triple x86_64-apple-macosx10.8 -Os
+// RUN: c-index-test core -print-unit %t/idx_opt2 | FileCheck %s -check-prefix=OPT
+
+// CHECK: print-unit.c.o
+// CHECK: provider: clang-
+// CHECK: is-system: 0
+// CHECK: has-main: 1
+// CHECK: main-path: {{.*}}/print-unit.c
+// CHECK: out-file: {{.*}}/print-unit.c.o
+// CHECK: target: x86_64-apple-macosx10.8
+// CHECK: is-debug: 1
+// CHECK: DEPEND START
+// CHECK: Record | user | {{.*}}/print-unit.c | print-unit.c-
+// CHECK: Record | user | {{.*}}/Inputs/head.h | head.h-
+// CHECK: Record | user | {{.*}}/Inputs/using-overlay.h | using-overlay.h-
+// CHECK: Record | system | {{.*}}/Inputs/sys/syshead.h | syshead.h-
+// CHECK: Record | system | {{.*}}/Inputs/sys/another.h | another.h-
+// CHECK: File | user | {{.*}}/Inputs/print-unit.h | | {{[0-9]*$}}
+// CHECK: DEPEND END (6)
+// CHECK: INCLUDE START
+// CHECK: {{.*}}/print-unit.c:3 | {{.*}}/Inputs/print-unit.h
+// CHECK: {{.*}}/print-unit.c:4 | {{.*}}/Inputs/sys/syshead.h
+// CHECK: {{.*}}/Inputs/print-unit.h:1 | {{.*}}/Inputs/head.h
+// CHECK: {{.*}}/Inputs/print-unit.h:2 | {{.*}}/Inputs/using-overlay.h
+// CHECK: INCLUDE END (4)
+
+// OPT: is-debug: 0

test/Index/Store/print-units-with-modules.m

+// RUN: rm -rf %t.idx %t.mcp
+// RUN: %clang -arch x86_64 -mmacosx-version-min=10.7 -c %s -o %t.o -index-store-path %t.idx -fmodules -fmodules-cache-path=%t.mcp -Xclang -fdisable-module-hash -I %S/Inputs/module
+// RUN: c-index-test core -print-unit %t.idx | FileCheck %s
+
+// XFAIL: linux
+
+@import ModDep;
+@import ModSystem;
+
+// CHECK: ModDep.pcm
+// CHECK: provider: clang-
+// CHECK: is-system: 0
+// CHECK: is-module: 1
+// CHECK: module-name: ModDep
+// CHECK: has-main: 0
+// CHECK: main-path: {{$}}
+// CHECK: out-file: {{.*}}/ModDep.pcm
+// CHECK: DEPEND START
+// CHECK: Unit | user | ModTop | {{.*}}/ModTop.pcm | ModTop.pcm
+// CHECK: Record | user | ModDep | {{.*}}/Inputs/module/ModDep.h | ModDep.h
+// CHECK: DEPEND END (2)
+
+// CHECK: ModSystem.pcm
+// CHECK: is-system: 1
+// CHECK: is-module: 1
+// CHECK: module-name: ModSystem
+// CHECK: has-main: 0
+// CHECK: main-path: {{$}}
+// CHECK: out-file: {{.*}}/ModSystem.pcm
+// CHECK: DEPEND START
+// CHECK: Record | system | ModSystem | {{.*}}/Inputs/module/ModSystem.h | ModSystem.h
+// CHECK: DEPEND END (1)
+
+// CHECK: ModTop.pcm
+// CHECK: is-system: 0
+// CHECK: is-module: 1
+// CHECK: module-name: ModTop
+// CHECK: has-main: 0
+// CHECK: main-path: {{$}}
+// CHECK: out-file: {{.*}}/ModTop.pcm
+// CHECK: DEPEND START
+// CHECK: Record | user | ModTop | {{.*}}/Inputs/module/ModTop.h | ModTop.h
+// CHECK: Record | user | ModTop.Sub1 | {{.*}}/Inputs/module/ModTopSub1.h | ModTopSub1.h
+// CHECK: File | user | ModTop.Sub2 | {{.*}}/Inputs/module/ModTopSub2.h | | {{[0-9]*$}}
+// CHECK: DEPEND END (3)
+
+// CHECK: print-units-with-modules.m.tmp.o
+// CHECK: is-system: 0
+// CHECK: is-module: 0
+// CHECK: module-name: <none>
+// CHECK: has-main: 1
+// CHECK: main-path: {{.*}}/print-units-with-modules.m
+// CHECK: out-file: {{.*}}/print-units-with-modules.m.tmp.o
+// CHECK: DEPEND START
+// CHECK: Unit | user | ModDep | {{.*}}/ModDep.pcm | ModDep.pcm
+// CHECK: Unit | system | ModSystem | {{.*}}/ModSystem.pcm | ModSystem.pcm
+// CHECK: File | user | {{.*}}/print-units-with-modules.m | | {{[0-9]*$}}
+// CHECK: File | user | {{.*}}/Inputs/module/module.modulemap | | {{[0-9]*$}}
+// CHECK: DEPEND END (4)

test/Index/Store/print-units-with-pch.c

+// RUN: rm -rf %t.idx
+// RUN: %clang -arch x86_64 -mmacosx-version-min=10.7 -x c-header %S/Inputs/head.h -o %t.h.pch -index-store-path %t.idx
+// RUN: %clang -arch x86_64 -mmacosx-version-min=10.7 -c %s -o %t.o -index-store-path %t.idx -include %t.h -Werror
+// RUN: c-index-test core -print-unit %t.idx | FileCheck %s
+
+// XFAIL: linux
+
+int main() {
+  test1_func();
+}
+
+// CHECK: print-units-with-pch.c.tmp.h.pch
+// CHECK: is-system: 0
+// CHECK: has-main: 0
+// CHECK: main-path: {{$}}
+// CHECK: out-file: {{.*}}/print-units-with-pch.c.tmp.h.pch
+// CHECK: DEPEND START
+// CHECK: Record | user | {{.*}}/Inputs/head.h | head.h
+// CHECK: DEPEND END (1)
+
+// CHECK: print-units-with-pch.c.tmp.o
+// CHECK: is-system: 0
+// CHECK: has-main: 1
+// CHECK: main-path: {{.*}}/print-units-with-pch.c
+// CHECK: out-file: {{.*}}/print-units-with-pch.c.tmp.o
+// CHECK: DEPEND START
+// CHECK: Unit | user | {{.*}}/print-units-with-pch.c.tmp.h.pch | print-units-with-pch.c.tmp.h.pch
+// CHECK: Record | user | {{.*}}/print-units-with-pch.c | print-units-with-pch.c
+// CHECK: DEPEND END (2)

test/Index/Store/record-hash-crash-invalid-name.cpp

+// Makes sure it doesn't crash.
+
+// XFAIL: linux
+
+// RUN: rm -rf %t
+// RUN: not %clang_cc1 %s -index-store-path %t/idx -std=c++14
+// RUN: c-index-test core -print-record %t/idx | FileCheck %s
+
+namespace rdar32474406 {
+void foo();
+typedef void (*Func_t)();
+// CHECK: [[@LINE+4]]:1 | type-alias/C | c:record-hash-crash-invalid-name.cpp@N@rdar32474406@T@Func_t | Ref,RelCont | rel: 1
+// CHECK-NEXT: RelCont | c:@N@rdar32474406
+// CHECK: [[@LINE+2]]:14 | function/C | c:@N@rdar32474406@F@foo# | Ref,RelCont | rel: 1
+// CHECK-NEXT: RelCont | c:@N@rdar32474406
+Func_t[] = { foo }; // invalid decomposition
+}

test/Index/Store/record-hash-crash.cpp

+// Makes sure it doesn't crash.
+
+// XFAIL: linux
+
+// RUN: rm -rf %t
+// RUN: %clang_cc1 %s -index-store-path %t/idx -std=c++14
+// RUN: c-index-test core -print-record %t/idx | FileCheck %s
+
+namespace crash1 {
+// CHECK: [[@LINE+1]]:6 | function/C
+auto getit() { return []() {}; }
+}
+
+namespace crash2 {
+// CHECK: [[@LINE+2]]:7 | class(Gen)/C++ | c:@N@crash2@ST>1#T@Foo | Ref,RelCont | rel: 1
+template <typename T>
+class Foo; // canonical decl
+
+// CHECK: [[@LINE+2]]:7 | class(Gen)/C++ | c:@N@crash2@ST>1#T@Foo | Def,RelChild | rel: 1
+template <typename T>
+class Foo {};
+
+// CHECK: [[@LINE+2]]:8 | struct(Gen)/C++ | c:@N@crash2@ST>1#t>1#pT@Wrapper | Def,RelChild | rel: 1
+template <template <typename... ARGS> class TYPE>
+struct Wrapper {};
+
+// CHECK: [[@LINE+3]]:8 | struct(Gen,TS)/C++ | c:@N@crash2@S@Wrapper>#@N@crash2@ST>1#T@Foo | Def,RelChild,RelSpecialization | rel: 2
+// CHECK: RelSpecialization | c:@N@crash2@ST>1#t>1#pT@Wrapper
+template <>
+struct Wrapper<Foo> {};
+}

test/Index/Store/record-hash-using.cpp

+// XFAIL: linux
+
+// RUN: rm -rf %t
+// RUN: %clang_cc1 %s -std=c++11 -index-store-path %t/idx -DTYPE1=A -DTYPE2=A -DTYPE3=T -DTYPE4=T
+// RUN: %clang_cc1 %s -std=c++11 -index-store-path %t/idx -DTYPE1=B -DTYPE2=A -DTYPE3=T -DTYPE4=T
+// RUN: %clang_cc1 %s -std=c++11 -index-store-path %t/idx -DTYPE1=A -DTYPE2=B -DTYPE3=T -DTYPE4=T
+// RUN: %clang_cc1 %s -std=c++11 -index-store-path %t/idx -DTYPE1=B -DTYPE2=B -DTYPE3=T -DTYPE4=T
+// RUN: find %t/idx/*/records -name "record-hash*" | count 4
+//
+// RUN: rm -rf %t
+// RUN: %clang_cc1 %s -std=c++11 -index-store-path %t/idx -DTYPE1=A -DTYPE2=A -DTYPE3=T -DTYPE4=T
+// RUN: %clang_cc1 %s -std=c++11 -index-store-path %t/idx -DTYPE1=A -DTYPE2=A -DTYPE3=U -DTYPE4=T
+// RUN: %clang_cc1 %s -std=c++11 -index-store-path %t/idx -DTYPE1=A -DTYPE2=A -DTYPE3=T -DTYPE4=U
+// RUN: %clang_cc1 %s -std=c++11 -index-store-path %t/idx -DTYPE1=A -DTYPE2=A -DTYPE3=U -DTYPE4=U
+// RUN: find %t/idx/*/records -name "record-hash*" | count 4
+
+template<typename T>
+struct A {
+  typedef int X;
+  void foo();
+};
+
+template<typename T>
+struct B : public A<T> {
+  typedef float X;
+  void foo(int);
+};
+
+template<typename T>
+struct C : public B<T> {
+// This should result in different records, due to the different types.
+  using TYPE1<T>::X;
+  using TYPE2<T>::foo;
+};
+
+template <typename T>
+struct D {
+  typedef T X;
+  void foo(T);
+};
+template <typename T, typename U>
+struct E : public D<T>, public D<U> {
+// This should result in different records, due to the different template parameter.
+  using D<TYPE3>::X;
+  using D<TYPE4>::foo;
+};

test/Index/Store/record-hash.cpp

+// XFAIL: linux
+
+// RUN: rm -rf %t
+// RUN: %clang_cc1 %s -index-store-path %t/idx -D THE_TYPE=long
+// RUN: %clang_cc1 %s -index-store-path %t/idx -D THE_TYPE=char
+// RUN: find %t/idx/*/records -name "record-hash*" | count 2
+
+template<typename T>
+class TC {};
+
+// This should result in different records, due to the different template parameter type.
+void some_func(TC<THE_TYPE>);

test/Index/Store/relative-out-path.c

+// Needs 'find'.
+// REQUIRES: shell
+
+// RUN: rm -rf %t
+// RUN: mkdir -p %t
+// RUN: %clang %s -index-store-path %t/idx1 -c -o %t/outfile.o
+// RUN: cd %t
+// RUN: %clang %s -index-store-path %t/idx2 -c -o outfile.o
+// RUN: cd ..
+// RUN: %clang %s -index-store-path %t/idx3 -fsyntax-only -o outfile.o -working-directory=%t
+// RUN: diff -r -u %t/idx2 %t/idx3
+
+// RUN: find %t/idx1 -name '*outfile.o*' > %t/hashes.txt
+// RUN: find %t/idx3 -name '*outfile.o*' >> %t/hashes.txt
+// RUN: FileCheck %s --input-file=%t/hashes.txt
+// CHECK:      outfile.o[[OUT_HASH:.*$]]
+// CHECK-NEXT: outfile.o[[OUT_HASH]]
+
+void foo();

test/Index/Store/syntax-only.c

+// RUN: rm -rf %t.idx
+// RUN: %clang -fsyntax-only %s -index-store-path %t.idx -o %T/syntax-only.c.myoutfile
+// RUN: c-index-test core -print-unit %t.idx | FileCheck %s -check-prefix=CHECK-UNIT
+// RUN: c-index-test core -print-record %t.idx | FileCheck %s -check-prefix=CHECK-RECORD
+
+// XFAIL: linux
+
+// CHECK-UNIT: out-file: {{.*}}/syntax-only.c.myoutfile
+// CHECK-RECORD: function/C | foo | c:@F@foo
+
+void foo();

test/Index/Store/unit-with-vfs.c

+// RUN: sed -e "s:INPUT_DIR:%S/Inputs:g" -e "s:OUT_DIR:%t:g" %S/Inputs/overlay.yaml > %t.yaml
+// REQUIRES: shell
+
+#include "using-overlay.h"
+
+// RUN: rm -rf %t.idx
+// RUN: %clang_cc1 %s -index-store-path %t.idx -I %t -ivfsoverlay %t.yaml
+// RUN: c-index-test core -print-unit %t.idx | FileCheck %s
+
+// XFAIL: linux
+
+// CHECK: Record | user | {{.*}}test/Index/Store/Inputs/using-overlay.h

test/Index/Store/unit-workdir-prefix.c

+// XFAIL: linux
+
+#include "header.h"
+
+void foo(void) {
+  bar();
+}
+
+// RUN: rm -rf %t
+// RUN: mkdir -p %t/Directory
+// RUN: mkdir -p %t/Directory.surprise
+// RUN: mkdir -p %t/sdk
+// RUN: mkdir -p %t/sdk_other
+// RUN: echo "void bar(void);" > %t/sdk_other/header.h
+// RUN: cp %s %t/Directory.surprise/main.c
+//
+// RUN: %clang_cc1 -isystem %t/sdk_other -isysroot %t/sdk -index-store-path %t/idx %t/Directory.surprise/main.c -triple x86_64-apple-macosx10.8 -working-directory %t/Directory
+// RUN: c-index-test core -print-unit %t/idx | FileCheck %s
+
+// CHECK: main.c.o
+// CHECK: provider: clang-
+// CHECK: is-system: 0
+// CHECK: has-main: 1
+// CHECK: main-path: {{.*}}Directory.surprise{{/|\\}}main.c
+// CHECK: out-file: {{.*}}Directory.surprise{{/|\\}}main.c.o
+// CHECK: target: x86_64-apple-macosx10.8
+// CHECK: is-debug: 1
+// CHECK: DEPEND START
+// CHECK: Record | user | {{.*}}Directory.surprise{{/|\\}}main.c | main.c-
+// CHECK: Record | system | {{.*}}sdk_other{{/|\\}}header.h | header.h-

test/Index/Store/using-libstdcpp-arc.mm

+// Test to make sure we don't crash, rdar://30816887.
+
+// RUN: rm -rf %t.idx
+// RUN: %clang_cc1 %s -index-store-path %t.idx -fobjc-arc -fobjc-arc-cxxlib=libstdc++
+// RUN: c-index-test core -print-record %t.idx | FileCheck %s
+
+// XFAIL: linux
+
+// CHECK: [[@LINE+1]]:6 | function/C
+void test1(void);

tools/CMakeLists.txt

 create_subdirectory_options(CLANG TOOL)
 
 add_clang_subdirectory(diagtool)
 add_clang_subdirectory(driver)
 add_clang_subdirectory(clang-diff)
 add_clang_subdirectory(clang-format)
 add_clang_subdirectory(clang-format-vs)
 add_clang_subdirectory(clang-fuzzer)
 add_clang_subdirectory(clang-import-test)
 add_clang_subdirectory(clang-offload-bundler)
 
 add_clang_subdirectory(c-index-test)
+add_clang_subdirectory(IndexStore)
 
 add_clang_subdirectory(clang-rename)
 add_clang_subdirectory(clang-refactor)
 
 if(CLANG_ENABLE_ARCMT)
   add_clang_subdirectory(arcmt-test)
   add_clang_subdirectory(c-arcmt-test)
 endif()

tools/IndexStore/CMakeLists.txt

+include(CheckIncludeFiles)
+
+set(SOURCES
+  IndexStore.cpp
+
+  ADDITIONAL_HEADERS
+  ../../include/indexstore/indexstore.h
+  ../../include/indexstore/IndexStoreCXX.h
+  )
+
+set(LIBS
+  clangDirectoryWatcher
+  clangIndex
+)
+
+set(LLVM_EXPORTED_SYMBOL_FILE ${CMAKE_CURRENT_SOURCE_DIR}/IndexStore.exports)
+
+set(ENABLE_SHARED SHARED)
+
+if(WIN32)
+  set(output_name "libIndexStore")
+else()
+  set(output_name "IndexStore")
+endif()
+
+# FIXME: needs to be ported to non-Apple platforms.
+if(APPLE)
+
+add_clang_library(IndexStore ${ENABLE_SHARED} ${ENABLE_STATIC}
+  OUTPUT_NAME ${output_name}
+  ${SOURCES}
+
+  LINK_LIBS
+  ${LIBS}
+
+  LINK_COMPONENTS
+  ${LLVM_TARGETS_TO_BUILD}
+  Core
+  Support
+  )
+
+set(INDEXSTORE_LIBRARY_VERSION "${LLVM_VERSION_MAJOR}.${LLVM_VERSION_MINOR}.${LLVM_VERSION_PATCH}")
+
+if(ENABLE_SHARED)
+  if(WIN32)
+    set_target_properties(IndexStore
+      PROPERTIES
+      VERSION ${INDEXSTORE_LIBRARY_VERSION}
+      DEFINE_SYMBOL _CINDEX_LIB_)
+  elseif(APPLE)
+    set(INDEXSTORE_LINK_FLAGS " -Wl,-compatibility_version -Wl,1")
+    set(INDEXSTORE_LINK_FLAGS "${INDEXSTORE_LINK_FLAGS} -Wl,-current_version -Wl,${INDEXSTORE_LIBRARY_VERSION}")
+
+    check_include_files("CoreServices/CoreServices.h" HAVE_CORESERVICES_H)
+    if(HAVE_CORESERVICES_H)
+      set(INDEXSTORE_LINK_FLAGS "${INDEXSTORE_LINK_FLAGS} -framework CoreServices")
+    endif()
+
+    set_property(TARGET IndexStore APPEND_STRING PROPERTY
+                 LINK_FLAGS ${INDEXSTORE_LINK_FLAGS})
+  else()
+    set_target_properties(IndexStore
+      PROPERTIES
+      VERSION ${INDEXSTORE_LIBRARY_VERSION}
+      DEFINE_SYMBOL _CINDEX_LIB_)
+  endif()
+endif()
+
+if (LLVM_INSTALL_TOOLCHAIN_ONLY)
+  install(TARGETS IndexStore
+    COMPONENT IndexStore
+    LIBRARY DESTINATION lib${LLVM_LIBDIR_SUFFIX}
+    ARCHIVE DESTINATION lib${LLVM_LIBDIR_SUFFIX}
+    RUNTIME DESTINATION bin)
+
+  if (NOT CMAKE_CONFIGURATION_TYPES)
+    add_custom_target(install-IndexStore
+                      DEPENDS IndexStore
+                      COMMAND "${CMAKE_COMMAND}"
+                              -DCMAKE_INSTALL_COMPONENT=IndexStore
+                              -P "${CMAKE_BINARY_DIR}/cmake_install.cmake")
+  endif()
+endif()
+
+set(INDEXSTORE_HEADERS_INSTALL_DESTINATION "local/include")
+
+install(DIRECTORY ../../include/indexstore
+  COMPONENT IndexStore
+  DESTINATION "${INDEXSTORE_HEADERS_INSTALL_DESTINATION}"
+  FILES_MATCHING
+  PATTERN "*.h"
+  PATTERN ".svn" EXCLUDE
+  )
+endif()

tools/IndexStore/IndexStore.cpp

+//===- IndexStore.cpp - Index store API -----------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements the API for the index store.
+//
+//===----------------------------------------------------------------------===//
+
+#include "indexstore/indexstore.h"
+#include "clang/Index/IndexDataStore.h"
+#include "clang/Index/IndexDataStoreSymbolUtils.h"
+#include "clang/Index/IndexRecordReader.h"
+#include "clang/Index/IndexUnitReader.h"
+#include "clang/Index/IndexUnitWriter.h"
+#include "clang/DirectoryWatcher/DirectoryWatcher.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/Optional.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/Support/Chrono.h"
+#include <Block.h>
+
+using namespace clang;
+using namespace clang::index;
+using namespace llvm;
+
+static indexstore_string_ref_t toIndexStoreString(StringRef str) {
+  return indexstore_string_ref_t{ str.data(), str.size() };
+}
+
+static timespec toTimeSpec(sys::TimePoint<> tp) {
+  std::chrono::seconds sec = std::chrono::time_point_cast<std::chrono::seconds>(
+                 tp).time_since_epoch();
+  std::chrono::nanoseconds nsec =
+    std::chrono::time_point_cast<std::chrono::nanoseconds>(tp - sec)
+      .time_since_epoch();
+  timespec ts;
+  ts.tv_sec = sec.count();
+  ts.tv_nsec = nsec.count();
+  return ts;
+}
+
+namespace {
+
+struct IndexStoreError {
+  std::string Error;
+};
+
+} // anonymous namespace
+
+const char *
+indexstore_error_get_description(indexstore_error_t err) {
+  return static_cast<IndexStoreError*>(err)->Error.c_str();
+}
+
+void
+indexstore_error_dispose(indexstore_error_t err) {
+  delete static_cast<IndexStoreError*>(err);
+}
+
+unsigned
+indexstore_format_version(void) {
+  return IndexDataStore::getFormatVersion();
+}
+
+indexstore_t
+indexstore_store_create(const char *store_path, indexstore_error_t *c_error) {
+  std::unique_ptr<IndexDataStore> store;
+  std::string error;
+  store = IndexDataStore::create(store_path, error);
+  if (!store) {
+    if (c_error)
+      *c_error = new IndexStoreError{ error };
+    return nullptr;
+  }
+  return store.release();
+}
+
+void
+indexstore_store_dispose(indexstore_t store) {
+  delete static_cast<IndexDataStore*>(store);
+}
+
+#if INDEXSTORE_HAS_BLOCKS
+bool
+indexstore_store_units_apply(indexstore_t c_store, unsigned sorted,
+                            bool(^applier)(indexstore_string_ref_t unit_name)) {
+  IndexDataStore *store = static_cast<IndexDataStore*>(c_store);
+  return store->foreachUnitName(sorted, [&](StringRef unitName) -> bool {
+    return applier(toIndexStoreString(unitName));
+  });
+}
+
+size_t
+indexstore_unit_event_notification_get_events_count(indexstore_unit_event_notification_t c_evtnote) {
+  auto *evtnote = static_cast<IndexDataStore::UnitEventNotification*>(c_evtnote);
+  return evtnote->Events.size();
+}
+
+indexstore_unit_event_t
+indexstore_unit_event_notification_get_event(indexstore_unit_event_notification_t c_evtnote, size_t index) {
+  auto *evtnote = static_cast<IndexDataStore::UnitEventNotification*>(c_evtnote);
+  return (indexstore_unit_event_t)&evtnote->Events[index];
+}
+
+bool
+indexstore_unit_event_notification_is_initial(indexstore_unit_event_notification_t c_evtnote) {
+  auto *evtnote = static_cast<IndexDataStore::UnitEventNotification*>(c_evtnote);
+  return evtnote->IsInitial;
+}
+
+indexstore_unit_event_kind_t
+indexstore_unit_event_get_kind(indexstore_unit_event_t c_evt) {
+  auto *evt = static_cast<IndexDataStore::UnitEvent*>(c_evt);
+  indexstore_unit_event_kind_t k;
+  switch (evt->Kind) {
+    case IndexDataStore::UnitEventKind::Added:
+      k = INDEXSTORE_UNIT_EVENT_ADDED; break;
+    case IndexDataStore::UnitEventKind::Removed:
+      k = INDEXSTORE_UNIT_EVENT_REMOVED; break;
+    case IndexDataStore::UnitEventKind::Modified:
+      k = INDEXSTORE_UNIT_EVENT_MODIFIED; break;
+    case IndexDataStore::UnitEventKind::DirectoryDeleted:
+      k = INDEXSTORE_UNIT_EVENT_DIRECTORY_DELETED; break;
+  }
+  return k;
+}
+
+indexstore_string_ref_t
+indexstore_unit_event_get_unit_name(indexstore_unit_event_t c_evt) {
+  auto *evt = static_cast<IndexDataStore::UnitEvent*>(c_evt);
+  return toIndexStoreString(evt->UnitName);
+}
+
+timespec
+indexstore_unit_event_get_modification_time(indexstore_unit_event_t c_evt) {
+  auto *evt = static_cast<IndexDataStore::UnitEvent*>(c_evt);
+  return evt->ModTime;
+}
+
+void
+indexstore_store_set_unit_event_handler(indexstore_t c_store,
+                                    indexstore_unit_event_handler_t blk_handler) {
+  IndexDataStore *store = static_cast<IndexDataStore*>(c_store);
+  if (!blk_handler) {
+    store->setUnitEventHandler(nullptr);
+    return;
+  }
+
+  class BlockWrapper {
+    indexstore_unit_event_handler_t blk_handler;
+  public:
+    BlockWrapper(indexstore_unit_event_handler_t handler) {
+      blk_handler = Block_copy(handler);
+    }
+    BlockWrapper(const BlockWrapper &other) {
+      blk_handler = Block_copy(other.blk_handler);
+    }
+    ~BlockWrapper() {
+      Block_release(blk_handler);
+    }
+
+    void operator()(indexstore_unit_event_notification_t evt_note) const {
+      blk_handler(evt_note);
+    }
+  };
+
+  BlockWrapper handler(blk_handler);
+
+  store->setUnitEventHandler([handler](IndexDataStore::UnitEventNotification evtNote) {
+    handler(&evtNote);
+  });
+}
+#endif
+
+bool
+indexstore_store_start_unit_event_listening(indexstore_t c_store,
+                                            indexstore_unit_event_listen_options_t *client_opts,
+                                            size_t listen_options_struct_size,
+                                            indexstore_error_t *c_error) {
+  IndexDataStore *store = static_cast<IndexDataStore*>(c_store);
+  indexstore_unit_event_listen_options_t listen_opts;
+  memset(&listen_opts, 0, sizeof(listen_opts));
+  unsigned clientOptSize = listen_options_struct_size < sizeof(listen_opts)
+                             ? listen_options_struct_size : sizeof(listen_opts);
+  memcpy(&listen_opts, client_opts, clientOptSize);
+
+  std::string error;
+  auto createFn = [](StringRef Path, AbstractDirectoryWatcher::EventReceiver Receiver, bool waitInitialSync, std::string &Error)
+      -> std::unique_ptr<AbstractDirectoryWatcher> {
+    return DirectoryWatcher::create(Path, std::move(Receiver), waitInitialSync, Error);
+  };
+  bool err = store->startEventListening(createFn, listen_opts.wait_initial_sync, error);
+  if (err && c_error)
+    *c_error = new IndexStoreError{ error };
+  return err;
+}
+
+void
+indexstore_store_stop_unit_event_listening(indexstore_t c_store) {
+  IndexDataStore *store = static_cast<IndexDataStore*>(c_store);
+  store->stopEventListening();
+}
+
+void
+indexstore_store_discard_unit(indexstore_t c_store, const char *unit_name) {
+  IndexDataStore *store = static_cast<IndexDataStore*>(c_store);
+  store->discardUnit(unit_name);
+}
+
+void
+indexstore_store_discard_record(indexstore_t c_store, const char *record_name) {
+  IndexDataStore *store = static_cast<IndexDataStore*>(c_store);
+  store->discardRecord(record_name);
+}
+
+void
+indexstore_store_purge_stale_data(indexstore_t c_store) {
+  IndexDataStore *store = static_cast<IndexDataStore*>(c_store);
+  store->purgeStaleData();
+}
+
+indexstore_symbol_kind_t
+indexstore_symbol_get_kind(indexstore_symbol_t sym) {
+  return getIndexStoreKind(static_cast<IndexRecordDecl *>(sym)->SymInfo.Kind);
+}
+
+indexstore_symbol_subkind_t
+indexstore_symbol_get_subkind(indexstore_symbol_t sym) {
+  return getIndexStoreSubKind(static_cast<IndexRecordDecl *>(sym)->SymInfo.SubKind);
+}
+
+indexstore_symbol_language_t
+indexstore_symbol_get_language(indexstore_symbol_t sym) {
+  return getIndexStoreLang(static_cast<IndexRecordDecl *>(sym)->SymInfo.Lang);
+}
+
+uint64_t
+indexstore_symbol_get_properties(indexstore_symbol_t sym) {
+  return getIndexStoreProperties(static_cast<IndexRecordDecl *>(sym)->SymInfo.Properties);
+}
+
+uint64_t
+indexstore_symbol_get_roles(indexstore_symbol_t sym) {
+  return getIndexStoreRoles(static_cast<IndexRecordDecl *>(sym)->Roles);
+}
+
+uint64_t
+indexstore_symbol_get_related_roles(indexstore_symbol_t sym) {
+  return getIndexStoreRoles(static_cast<IndexRecordDecl *>(sym)->RelatedRoles);
+}
+
+indexstore_string_ref_t
+indexstore_symbol_get_name(indexstore_symbol_t sym) {
+  auto *D = static_cast<IndexRecordDecl*>(sym);
+  return toIndexStoreString(D->Name);
+}
+
+indexstore_string_ref_t
+indexstore_symbol_get_usr(indexstore_symbol_t sym) {
+  auto *D = static_cast<IndexRecordDecl*>(sym);
+  return toIndexStoreString(D->USR);
+}
+
+indexstore_string_ref_t
+indexstore_symbol_get_codegen_name(indexstore_symbol_t sym) {
+  auto *D = static_cast<IndexRecordDecl*>(sym);
+  return toIndexStoreString(D->CodeGenName);
+}
+
+uint64_t
+indexstore_symbol_relation_get_roles(indexstore_symbol_relation_t sym_rel) {
+  return getIndexStoreRoles(static_cast<IndexRecordRelation *>(sym_rel)->Roles);
+}
+
+indexstore_symbol_t
+indexstore_symbol_relation_get_symbol(indexstore_symbol_relation_t sym_rel) {
+  return (indexstore_symbol_t)static_cast<IndexRecordRelation*>(sym_rel)->Dcl;
+}
+
+indexstore_symbol_t
+indexstore_occurrence_get_symbol(indexstore_occurrence_t occur) {
+  return (indexstore_symbol_t)static_cast<IndexRecordOccurrence*>(occur)->Dcl;
+}
+
+#if INDEXSTORE_HAS_BLOCKS
+bool
+indexstore_occurrence_relations_apply(indexstore_occurrence_t occur,
+                      bool(^applier)(indexstore_symbol_relation_t symbol_rel)) {
+  auto *recOccur = static_cast<IndexRecordOccurrence*>(occur);
+  for (auto &rel : recOccur->Relations) {
+    if (!applier(&rel))
+      return false;
+  }
+  return true;
+}
+#endif
+
+uint64_t
+indexstore_occurrence_get_roles(indexstore_occurrence_t occur) {
+  return static_cast<IndexRecordOccurrence*>(occur)->Roles;
+}
+
+void
+indexstore_occurrence_get_line_col(indexstore_occurrence_t occur,
+                              unsigned *line, unsigned *column) {
+  auto *recOccur = static_cast<IndexRecordOccurrence*>(occur);
+  if (line)
+    *line = recOccur->Line;
+  if (column)
+    *column = recOccur->Column;
+}
+
+typedef void *indexstore_record_reader_t;
+
+indexstore_record_reader_t
+indexstore_record_reader_create(indexstore_t c_store, const char *record_name,
+                                indexstore_error_t *c_error) {
+  IndexDataStore *store = static_cast<IndexDataStore*>(c_store);
+  std::unique_ptr<IndexRecordReader> reader;
+  std::string error;
+  reader = IndexRecordReader::createWithRecordFilename(record_name,
+                                                       store->getFilePath(),
+                                                       error);
+  if (!reader) {
+    if (c_error)
+      *c_error = new IndexStoreError{ error };
+    return nullptr;
+  }
+  return reader.release();
+}
+
+void
+indexstore_record_reader_dispose(indexstore_record_reader_t rdr) {
+  auto *reader = static_cast<IndexRecordReader *>(rdr);
+  delete reader;
+}
+
+#if INDEXSTORE_HAS_BLOCKS
+/// Goes through the symbol data and passes symbols to \c receiver, for the
+/// symbol data that \c filter returns true on.
+///
+/// This allows allocating memory only for the record symbols that the caller is
+/// interested in.
+bool
+indexstore_record_reader_search_symbols(indexstore_record_reader_t rdr,
+    bool(^filter)(indexstore_symbol_t symbol, bool *stop),
+    void(^receiver)(indexstore_symbol_t symbol)) {
+  auto *reader = static_cast<IndexRecordReader *>(rdr);
+
+  auto filterFn = [&](const IndexRecordDecl &D) -> IndexRecordReader::DeclSearchReturn {
+    bool stop = false;
+    bool accept = filter((indexstore_symbol_t)&D, &stop);
+    return { accept, !stop };
+  };
+  auto receiverFn = [&](const IndexRecordDecl *D) {
+    receiver((indexstore_symbol_t)D);
+  };
+
+  return reader->searchDecls(filterFn, receiverFn);
+}
+
+bool
+indexstore_record_reader_symbols_apply(indexstore_record_reader_t rdr,
+                                        bool nocache,
+                                   bool(^applier)(indexstore_symbol_t symbol)) {
+  auto *reader = static_cast<IndexRecordReader *>(rdr);
+  auto receiverFn = [&](const IndexRecordDecl *D) -> bool {
+    return applier((indexstore_symbol_t)D);
+  };
+  return reader->foreachDecl(nocache, receiverFn);
+}
+
+bool
+indexstore_record_reader_occurrences_apply(indexstore_record_reader_t rdr,
+                                bool(^applier)(indexstore_occurrence_t occur)) {
+  auto *reader = static_cast<IndexRecordReader *>(rdr);
+  auto receiverFn = [&](const IndexRecordOccurrence &RO) -> bool {
+    return applier((indexstore_occurrence_t)&RO);
+  };
+  return reader->foreachOccurrence(receiverFn);
+}
+
+bool
+indexstore_record_reader_occurrences_in_line_range_apply(indexstore_record_reader_t rdr,
+                                                         unsigned line_start,
+                                                         unsigned line_count,
+                                bool(^applier)(indexstore_occurrence_t occur)) {
+  auto *reader = static_cast<IndexRecordReader *>(rdr);
+  auto receiverFn = [&](const IndexRecordOccurrence &RO) -> bool {
+    return applier((indexstore_occurrence_t)&RO);
+  };
+  return reader->foreachOccurrenceInLineRange(line_start, line_count, receiverFn);
+}
+
+/// \param symbols if non-zero \c symbols_count, indicates the list of symbols
+/// that we want to get occurrences for. An empty array indicates that we want
+/// occurrences for all symbols.
+/// \param related_symbols Same as \c symbols but for related symbols.
+bool
+indexstore_record_reader_occurrences_of_symbols_apply(indexstore_record_reader_t rdr,
+        indexstore_symbol_t *symbols, size_t symbols_count,
+        indexstore_symbol_t *related_symbols, size_t related_symbols_count,
+        bool(^applier)(indexstore_occurrence_t occur)) {
+  auto *reader = static_cast<IndexRecordReader *>(rdr);
+  auto receiverFn = [&](const IndexRecordOccurrence &RO) -> bool {
+    return applier((indexstore_occurrence_t)&RO);
+  };
+  return reader->foreachOccurrence({(IndexRecordDecl**)symbols, symbols_count},
+                                   {(IndexRecordDecl**)related_symbols, related_symbols_count},
+                                   receiverFn);
+}
+#endif
+
+size_t
+indexstore_store_get_unit_name_from_output_path(indexstore_t store,
+                                                const char *output_path,
+                                                char *name_buf,
+                                                size_t buf_size) {
+  SmallString<256> unitName;
+  IndexUnitWriter::getUnitNameForAbsoluteOutputFile(output_path, unitName);
+  size_t nameLen = unitName.size();
+  strlcpy(name_buf, unitName.c_str(), buf_size);
+  return nameLen;
+}
+
+bool
+indexstore_store_get_unit_modification_time(indexstore_t c_store,
+                                            const char *unit_name,
+                                            int64_t *seconds,
+                                            int64_t *nanoseconds,
+                                            indexstore_error_t *c_error) {
+  IndexDataStore *store = static_cast<IndexDataStore*>(c_store);
+  std::string error;
+  // FIXME: This provides mod time with second-only accuracy.
+  auto optModTime = IndexUnitReader::getModificationTimeForUnit(unit_name,
+                                              store->getFilePath(), error);
+  if (!optModTime) {
+    if (c_error)
+      *c_error = new IndexStoreError{ error };
+    return true;
+  }
+
+  timespec ts = toTimeSpec(*optModTime);
+  if (seconds)
+    *seconds = ts.tv_sec;
+  if (nanoseconds)
+    *nanoseconds = ts.tv_nsec;
+
+  return false;
+}
+
+indexstore_unit_reader_t
+indexstore_unit_reader_create(indexstore_t c_store, const char *unit_name,
+                              indexstore_error_t *c_error) {
+  IndexDataStore *store = static_cast<IndexDataStore*>(c_store);
+  std::unique_ptr<IndexUnitReader> reader;
+  std::string error;
+  reader = IndexUnitReader::createWithUnitFilename(unit_name,
+                                                   store->getFilePath(), error);
+  if (!reader) {
+    if (c_error)
+      *c_error = new IndexStoreError{ error };
+    return nullptr;
+  }
+  return reader.release();
+}
+
+void
+indexstore_unit_reader_dispose(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader*>(rdr);
+  delete reader;
+}
+
+indexstore_string_ref_t
+indexstore_unit_reader_get_provider_identifier(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader*>(rdr);
+  return toIndexStoreString(reader->getProviderIdentifier());
+}
+
+indexstore_string_ref_t
+indexstore_unit_reader_get_provider_version(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader*>(rdr);
+  return toIndexStoreString(reader->getProviderVersion());
+}
+
+void
+indexstore_unit_reader_get_modification_time(indexstore_unit_reader_t rdr,
+                                             int64_t *seconds,
+                                             int64_t *nanoseconds) {
+  auto reader = static_cast<IndexUnitReader*>(rdr);
+  // FIXME: This provides mod time with second-only accuracy.
+  sys::TimePoint<> timeVal = reader->getModificationTime();
+  timespec ts = toTimeSpec(timeVal);
+  if (seconds)
+    *seconds = ts.tv_sec;
+  if (nanoseconds)
+    *nanoseconds = ts.tv_nsec;
+}
+
+bool
+indexstore_unit_reader_is_system_unit(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader*>(rdr);
+  return reader->isSystemUnit();
+}
+
+bool
+indexstore_unit_reader_is_module_unit(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader*>(rdr);
+  return reader->isModuleUnit();
+}
+
+bool
+indexstore_unit_reader_is_debug_compilation(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader*>(rdr);
+  return reader->isDebugCompilation();
+}
+
+bool
+indexstore_unit_reader_has_main_file(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader*>(rdr);
+  return reader->hasMainFile();
+}
+
+indexstore_string_ref_t
+indexstore_unit_reader_get_main_file(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader*>(rdr);
+  return toIndexStoreString(reader->getMainFilePath());
+}
+
+indexstore_string_ref_t
+indexstore_unit_reader_get_module_name(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader*>(rdr);
+  return toIndexStoreString(reader->getModuleName());
+}
+
+indexstore_string_ref_t
+indexstore_unit_reader_get_working_dir(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader*>(rdr);
+  return toIndexStoreString(reader->getWorkingDirectory());
+}
+
+indexstore_string_ref_t
+indexstore_unit_reader_get_output_file(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader*>(rdr);
+  return toIndexStoreString(reader->getOutputFile());
+}
+
+indexstore_string_ref_t
+indexstore_unit_reader_get_sysroot_path(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader*>(rdr);
+  return toIndexStoreString(reader->getSysrootPath());
+}
+
+indexstore_string_ref_t
+indexstore_unit_reader_get_target(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader*>(rdr);
+  return toIndexStoreString(reader->getTarget());
+}
+
+indexstore_unit_dependency_kind_t
+indexstore_unit_dependency_get_kind(indexstore_unit_dependency_t c_dep) {
+  auto dep = static_cast<const IndexUnitReader::DependencyInfo*>(c_dep);
+  switch (dep->Kind) {
+  case IndexUnitReader::DependencyKind::Unit: return INDEXSTORE_UNIT_DEPENDENCY_UNIT;
+  case IndexUnitReader::DependencyKind::Record: return INDEXSTORE_UNIT_DEPENDENCY_RECORD;
+  case IndexUnitReader::DependencyKind::File: return INDEXSTORE_UNIT_DEPENDENCY_FILE;
+  }
+}
+
+bool
+indexstore_unit_dependency_is_system(indexstore_unit_dependency_t c_dep) {
+  auto dep = static_cast<const IndexUnitReader::DependencyInfo*>(c_dep);
+  return dep->IsSystem;
+}
+
+indexstore_string_ref_t
+indexstore_unit_dependency_get_filepath(indexstore_unit_dependency_t c_dep) {
+  auto dep = static_cast<const IndexUnitReader::DependencyInfo*>(c_dep);
+  return toIndexStoreString(dep->FilePath);
+}
+
+indexstore_string_ref_t
+indexstore_unit_dependency_get_modulename(indexstore_unit_dependency_t c_dep) {
+  auto dep = static_cast<const IndexUnitReader::DependencyInfo*>(c_dep);
+  return toIndexStoreString(dep->ModuleName);
+}
+
+indexstore_string_ref_t
+indexstore_unit_dependency_get_name(indexstore_unit_dependency_t c_dep) {
+  auto dep = static_cast<const IndexUnitReader::DependencyInfo*>(c_dep);
+  return toIndexStoreString(dep->UnitOrRecordName);
+}
+
+time_t
+indexstore_unit_dependency_get_modification_time(indexstore_unit_dependency_t c_dep) {
+  auto dep = static_cast<const IndexUnitReader::DependencyInfo*>(c_dep);
+  return dep->ModTime;
+}
+
+size_t
+indexstore_unit_dependency_get_file_size(indexstore_unit_dependency_t c_dep) {
+  auto dep = static_cast<const IndexUnitReader::DependencyInfo*>(c_dep);
+  return dep->FileSize;
+}
+
+indexstore_string_ref_t
+indexstore_unit_include_get_source_path(indexstore_unit_include_t c_inc) {
+  auto inc = static_cast<const IndexUnitReader::IncludeInfo*>(c_inc);
+  return toIndexStoreString(inc->SourcePath);
+}
+
+indexstore_string_ref_t
+indexstore_unit_include_get_target_path(indexstore_unit_include_t c_inc) {
+  auto inc = static_cast<const IndexUnitReader::IncludeInfo*>(c_inc);
+  return toIndexStoreString(inc->TargetPath);
+}
+
+unsigned
+indexstore_unit_include_get_source_line(indexstore_unit_include_t c_inc) {
+  auto inc = static_cast<const IndexUnitReader::IncludeInfo*>(c_inc);
+  return inc->SourceLine;
+}
+
+#if INDEXSTORE_HAS_BLOCKS
+bool
+indexstore_unit_reader_dependencies_apply(indexstore_unit_reader_t rdr,
+                             bool(^applier)(indexstore_unit_dependency_t)) {
+  auto reader = static_cast<IndexUnitReader*>(rdr);
+  return reader->foreachDependency([&](const IndexUnitReader::DependencyInfo &depInfo) -> bool {
+    return applier((void*)&depInfo);
+  });
+}
+
+bool
+indexstore_unit_reader_includes_apply(indexstore_unit_reader_t rdr,
+                             bool(^applier)(indexstore_unit_include_t)) {
+  auto reader = static_cast<IndexUnitReader*>(rdr);
+  return reader->foreachInclude([&](const IndexUnitReader::IncludeInfo &incInfo) -> bool {
+    return applier((void*)&incInfo);
+  });
+}
+#endif

tools/IndexStore/IndexStore.exports

+indexstore_error_get_description
+indexstore_error_dispose
+indexstore_format_version
+indexstore_store_create
+indexstore_store_dispose
+indexstore_store_get_unit_modification_time
+indexstore_store_get_unit_name_from_output_path
+indexstore_store_units_apply
+indexstore_store_set_unit_event_handler
+indexstore_store_start_unit_event_listening
+indexstore_store_stop_unit_event_listening
+indexstore_store_discard_unit
+indexstore_store_discard_record
+indexstore_store_purge_stale_data
+indexstore_symbol_get_kind
+indexstore_symbol_get_language
+indexstore_symbol_get_properties
+indexstore_symbol_get_roles
+indexstore_symbol_get_related_roles
+indexstore_symbol_get_subkind
+indexstore_symbol_get_name
+indexstore_symbol_get_usr
+indexstore_symbol_get_codegen_name
+indexstore_symbol_relation_get_roles
+indexstore_symbol_relation_get_symbol
+indexstore_occurrence_get_symbol
+indexstore_occurrence_get_roles
+indexstore_occurrence_get_line_col
+indexstore_occurrence_relations_apply
+indexstore_record_reader_create
+indexstore_record_reader_dispose
+indexstore_record_reader_search_symbols
+indexstore_record_reader_symbols_apply
+indexstore_record_reader_occurrences_apply
+indexstore_record_reader_occurrences_in_line_range_apply
+indexstore_record_reader_occurrences_of_symbols_apply
+indexstore_unit_dependency_get_kind
+indexstore_unit_dependency_get_filepath
+indexstore_unit_dependency_get_file_size
+indexstore_unit_dependency_get_modification_time
+indexstore_unit_dependency_get_modulename
+indexstore_unit_dependency_get_name
+indexstore_unit_dependency_is_system
+indexstore_unit_event_get_kind
+indexstore_unit_event_get_modification_time
+indexstore_unit_event_get_unit_name
+indexstore_unit_event_notification_get_event
+indexstore_unit_event_notification_get_events_count
+indexstore_unit_event_notification_is_initial
+indexstore_unit_reader_create
+indexstore_unit_reader_dispose
+indexstore_unit_reader_get_main_file
+indexstore_unit_reader_get_modification_time
+indexstore_unit_reader_get_module_name
+indexstore_unit_reader_get_provider_identifier
+indexstore_unit_reader_get_provider_version
+indexstore_unit_reader_get_working_dir
+indexstore_unit_reader_get_output_file
+indexstore_unit_reader_get_sysroot_path
+indexstore_unit_reader_get_target
+indexstore_unit_reader_dependencies_apply
+indexstore_unit_reader_includes_apply
+indexstore_unit_reader_has_main_file
+indexstore_unit_reader_is_debug_compilation
+indexstore_unit_reader_is_module_unit
+indexstore_unit_reader_is_system_unit
+indexstore_unit_include_get_source_path
+indexstore_unit_include_get_target_path
+indexstore_unit_include_get_source_line

tools/c-index-test/CMakeLists.txt

+include(CheckIncludeFiles)
+
 set(LLVM_LINK_COMPONENTS
   support
 )
 
 add_clang_executable(c-index-test
   c-index-test.c
   core_main.cpp
+  JSONAggregation.cpp
   )
 
+set(INDEXSTORE_LIB)
+set(CINDEXTEST_LIBS)
+if(APPLE)
+  set(INDEXSTORE_LIB IndexStore)
+endif()
+
 if(NOT MSVC)
   set_property(
     SOURCE c-index-test.c
     PROPERTY COMPILE_FLAGS "-std=gnu89"
     )
 endif()
 
 if (LLVM_BUILD_STATIC)
   target_link_libraries(c-index-test
     libclang_static
     clangCodeGen
     clangIndex
+    ${CINDEXTEST_LIBS}
   )
 else()
   target_link_libraries(c-index-test
     libclang
+    ${INDEXSTORE_LIB}
     clangAST
     clangBasic
     clangCodeGen
+    clangDirectoryWatcher
     clangFrontend
     clangIndex
     clangSerialization
+    ${CINDEXTEST_LIBS}
   )
 endif()
 
 set_target_properties(c-index-test
   PROPERTIES
   LINKER_LANGUAGE CXX)
 
 # If libxml2 is available, make it available for c-index-test.
 if (CLANG_HAVE_LIBXML)
   include_directories(SYSTEM ${LIBXML2_INCLUDE_DIR})
   target_link_libraries(c-index-test ${LIBXML2_LIBRARIES})
 endif()
 
+if(APPLE)
+  check_include_files("CoreServices/CoreServices.h" HAVE_CORESERVICES_H)
+  if(HAVE_CORESERVICES_H)
+    target_link_libraries(c-index-test "-framework CoreServices")
+  endif()
+endif()
+
 if (NOT LLVM_INSTALL_TOOLCHAIN_ONLY)
   if(INTERNAL_INSTALL_PREFIX)
     set(INSTALL_DESTINATION "${INTERNAL_INSTALL_PREFIX}/bin")
     set_property(TARGET c-index-test APPEND PROPERTY INSTALL_RPATH
        "@executable_path/../../lib")
   else()
     set(INSTALL_DESTINATION bin)
   endif()

tools/c-index-test/JSONAggregation.h

+//===--- JSONAggregation.h - Index data aggregation in JSON format --------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_TOOLS_CINDEXTEST_JSONAGGREGATION_H
+#define LLVM_CLANG_TOOLS_CINDEXTEST_JSONAGGREGATION_H
+
+#include "clang/Basic/LLVM.h"
+
+namespace clang {
+namespace index {
+
+/// Returns true if an error occurred, false otherwise.
+bool aggregateDataAsJSON(StringRef StorePath, raw_ostream &OS);
+
+} // end namespace index
+} // end namespace clang
+
+#endif

tools/c-index-test/JSONAggregation.cpp

+//===--- JSONAggregation.cpp - Index data aggregation in JSON format ------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "JSONAggregation.h"
+#include "indexstore/IndexStoreCXX.h"
+#include "clang/Index/IndexDataStoreSymbolUtils.h"
+#include "llvm/ADT/StringMap.h"
+#include "llvm/Support/Allocator.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/raw_ostream.h"
+
+using namespace clang;
+using namespace clang::index;
+using namespace indexstore;
+using namespace llvm;
+
+#if INDEXSTORE_HAS_BLOCKS
+
+namespace {
+
+typedef size_t FilePathIndex;
+typedef size_t RecordIndex;
+typedef size_t SymbolIndex;
+
+struct UnitSourceInfo {
+  FilePathIndex FilePath;
+  SmallVector<RecordIndex, 2> AssociatedRecords;
+};
+
+struct UnitInfo {
+  std::string Name;
+  SmallVector<UnitSourceInfo, 8> Sources;
+  SmallVector<std::string, 3> UnitDepends;
+  FilePathIndex OutFile;
+  StringRef Triple;
+};
+
+struct SymbolInfo {
+  SymbolKind Kind;
+  SymbolLanguage Lang;
+  StringRef USR;
+  StringRef Name;
+  StringRef CodegenName;
+  SymbolRoleSet Roles = 0;
+  SymbolRoleSet RelatedRoles = 0;
+};
+
+struct SymbolRelationInfo {
+  SymbolIndex RelatedSymbol;
+  SymbolRoleSet Roles;
+  SymbolRelationInfo(SymbolIndex relSymbol, SymbolRoleSet roles)
+    : RelatedSymbol(relSymbol), Roles(roles) {}
+};
+
+struct SymbolOccurrenceInfo {
+  SymbolIndex Symbol;
+  SymbolRoleSet Roles = 0;
+  std::vector<SymbolRelationInfo> Relations;
+  unsigned Line;
+  unsigned Column;
+};
+
+struct RecordInfo {
+  SmallVector<SymbolOccurrenceInfo, 8> Occurrences;
+};
+
+class Aggregator {
+  IndexStore Store;
+
+  BumpPtrAllocator Allocator;
+
+  StringMap<FilePathIndex, BumpPtrAllocator &> FilePathIndices;
+  std::vector<StringRef> FilePaths;
+  StringMap<char, BumpPtrAllocator &> Triples;
+
+  std::vector<std::unique_ptr<UnitInfo>> Units;
+
+  StringMap<RecordIndex, BumpPtrAllocator &> RecordIndices;
+  std::vector<std::unique_ptr<RecordInfo>> Records;
+
+  StringMap<SymbolIndex, BumpPtrAllocator &> SymbolIndices;
+  std::vector<SymbolInfo> Symbols;
+
+public:
+  explicit Aggregator(IndexStore store)
+  : Store(std::move(store)),
+    FilePathIndices(Allocator),
+    Triples(Allocator),
+    RecordIndices(Allocator),
+    SymbolIndices(Allocator) {}
+
+  bool process();
+  void processUnit(StringRef name, IndexUnitReader &UnitReader);
+  void dumpJSON(raw_ostream &OS);
+
+private:
+  StringRef copyStr(StringRef str) {
+    if (str.empty())
+      return StringRef();
+    char *buf = Allocator.Allocate<char>(str.size());
+    std::copy(str.begin(), str.end(), buf);
+    return StringRef(buf, str.size());
+  }
+
+  StringRef getTripleString(StringRef inputTriple) {
+    return Triples.insert(std::make_pair(inputTriple, 0)).first->first();
+  }
+
+  FilePathIndex getFilePathIndex(StringRef path, StringRef workingDir);
+  RecordIndex getRecordIndex(StringRef recordFile);
+  SymbolIndex getSymbolIndex(IndexRecordSymbol sym);
+  std::unique_ptr<RecordInfo> processRecord(StringRef recordFile);
+};
+
+} // anonymous namespace
+
+bool Aggregator::process() {
+  bool succ = Store.foreachUnit(/*sorted=*/true, [&](StringRef unitName) -> bool {
+    std::string error;
+    auto unitReader = IndexUnitReader(Store, unitName, error);
+    if (!unitReader) {
+      errs() << "error opening unit file '" << unitName << "': " << error << '\n';
+      return false;
+    }
+
+    processUnit(unitName, unitReader);
+    return true;
+  });
+
+  return !succ;
+}
+
+void Aggregator::processUnit(StringRef name, IndexUnitReader &UnitReader) {
+  auto workDir = UnitReader.getWorkingDirectory();
+  auto unit = llvm::make_unique<UnitInfo>();
+  unit->Name = name;
+  unit->Triple = getTripleString(UnitReader.getTarget());
+  unit->OutFile = getFilePathIndex(UnitReader.getOutputFile(), workDir);
+
+  struct DepInfo {
+    UnitSourceInfo source;
+    std::string unitName;
+  };
+  SmallVector<DepInfo, 32> Deps;
+  UnitReader.foreachDependency([&](IndexUnitDependency dep) -> bool {
+    Deps.resize(Deps.size()+1);
+    auto &depInfo = Deps.back();
+    switch (dep.getKind()) {
+      case IndexUnitDependency::DependencyKind::Unit: {
+        depInfo.unitName = dep.getName();
+        StringRef filePath = dep.getFilePath();
+        if (!filePath.empty())
+          depInfo.source.FilePath = getFilePathIndex(filePath, workDir);
+        break;
+      }
+      case IndexUnitDependency::DependencyKind::Record: {
+        depInfo.source.FilePath = getFilePathIndex(dep.getFilePath(), workDir);
+        RecordIndex recIndex = getRecordIndex(dep.getName());
+        depInfo.source.AssociatedRecords.push_back(recIndex);
+        break;
+      }
+      case IndexUnitDependency::DependencyKind::File:
+        depInfo.source.FilePath = getFilePathIndex(dep.getFilePath(), workDir);
+    }
+    return true;
+  });
+
+  unit->Sources.reserve(Deps.size());
+  for (auto &dep : Deps) {
+    if (!dep.unitName.empty()) {
+      unit->UnitDepends.emplace_back(std::move(dep.unitName));
+    } else {
+      unit->Sources.push_back(std::move(dep.source));
+    }
+  }
+
+  Units.push_back(std::move(unit));
+}
+
+FilePathIndex Aggregator::getFilePathIndex(StringRef path, StringRef workingDir) {
+  StringRef absPath;
+  SmallString<128> absPathBuf;
+  if (sys::path::is_absolute(path) || workingDir.empty()) {
+    absPath = path;
+  } else {
+    absPathBuf = workingDir;
+    sys::path::append(absPathBuf, path);
+    absPath = absPathBuf.str();
+  }
+
+  auto pair = FilePathIndices.insert(std::make_pair(absPath, FilePaths.size()));
+  bool wasInserted = pair.second;
+  if (wasInserted) {
+    FilePaths.push_back(pair.first->first());
+  }
+  return pair.first->second;
+}
+
+RecordIndex Aggregator::getRecordIndex(StringRef recordFile) {
+  auto pair = RecordIndices.insert(std::make_pair(recordFile, Records.size()));
+  bool wasInserted = pair.second;
+  if (wasInserted) {
+    Records.push_back(processRecord(recordFile));
+  }
+  return pair.first->second;
+}
+
+std::unique_ptr<RecordInfo> Aggregator::processRecord(StringRef recordFile) {
+  std::string error;
+  auto recordReader = IndexRecordReader(Store, recordFile, error);
+  if (!recordReader) {
+    errs() << "failed reading record file: " << recordFile << '\n';
+    ::exit(1);
+  }
+  auto record = llvm::make_unique<RecordInfo>();
+  recordReader.foreachOccurrence([&](IndexRecordOccurrence idxOccur) -> bool {
+    SymbolIndex symIdx = getSymbolIndex(idxOccur.getSymbol());
+    SymbolInfo &symInfo = Symbols[symIdx];
+    symInfo.Roles |= idxOccur.getRoles();
+    SymbolOccurrenceInfo occurInfo;
+    occurInfo.Symbol = symIdx;
+    idxOccur.foreachRelation([&](IndexSymbolRelation rel) -> bool {
+      SymbolIndex relsymIdx = getSymbolIndex(rel.getSymbol());
+      SymbolInfo &relsymInfo = Symbols[relsymIdx];
+      relsymInfo.RelatedRoles |= rel.getRoles();
+      occurInfo.Relations.emplace_back(relsymIdx, rel.getRoles());
+      return true;
+    });
+    occurInfo.Roles = idxOccur.getRoles();
+    std::tie(occurInfo.Line, occurInfo.Column) = idxOccur.getLineCol();
+    record->Occurrences.push_back(std::move(occurInfo));
+    return true;
+  });
+  return record;
+}
+
+SymbolIndex Aggregator::getSymbolIndex(IndexRecordSymbol sym) {
+  auto pair = SymbolIndices.insert(std::make_pair(sym.getUSR(), Symbols.size()));
+  bool wasInserted = pair.second;
+  if (wasInserted) {
+    SymbolInfo symInfo;
+    symInfo.Kind = getSymbolKind(sym.getKind());
+    symInfo.Lang = getSymbolLanguage(sym.getLanguage());
+    symInfo.USR = pair.first->first();
+    symInfo.Name = copyStr(sym.getName());
+    symInfo.CodegenName = copyStr(sym.getCodegenName());
+    Symbols.push_back(std::move(symInfo));
+  }
+  return pair.first->second;
+}
+
+
+void Aggregator::dumpJSON(raw_ostream &OS) {
+  OS << "{\n";
+  OS.indent(2) << "\"files\": [\n";
+  for (unsigned i = 0, e = FilePaths.size(); i != e; ++i) {
+    OS.indent(4) << '\"' << FilePaths[i] << '\"';
+    if (i < e-1) OS << ',';
+    OS << '\n';
+  }
+  OS.indent(2) << "],\n";
+
+  OS.indent(2) << "\"symbols\": [\n";
+  for (unsigned i = 0, e = Symbols.size(); i != e; ++i) {
+    OS.indent(4) << "{\n";
+    SymbolInfo &symInfo = Symbols[i];
+    OS.indent(6) << "\"kind\": \"" << getSymbolKindString(symInfo.Kind) << "\",\n";
+    OS.indent(6) << "\"lang\": \"" << getSymbolLanguageString(symInfo.Lang) << "\",\n";
+    OS.indent(6) << "\"usr\": \"" << symInfo.USR << "\",\n";
+    OS.indent(6) << "\"name\": \"" << symInfo.Name << "\",\n";
+    if (!symInfo.CodegenName.empty())
+      OS.indent(6) << "\"codegen\": \"" << symInfo.CodegenName << "\",\n";
+    OS.indent(6) << "\"roles\": \"";
+    printSymbolRoles(symInfo.Roles, OS);
+    OS << '\"';
+    if (symInfo.RelatedRoles != 0) {
+      OS << ",\n";
+      OS.indent(6) << "\"rel-roles\": \"";
+      printSymbolRoles(symInfo.RelatedRoles, OS);
+      OS << '\"';
+    }
+    OS << '\n';
+    OS.indent(4) << "}";
+    if (i < e-1) OS << ',';
+    OS << '\n';
+  }
+  OS.indent(2) << "],\n";
+
+  OS.indent(2) << "\"records\": [\n";
+  for (unsigned i = 0, e = Records.size(); i != e; ++i) {
+    OS.indent(4) << "{\n";
+    RecordInfo &recInfo = *Records[i];
+    OS.indent(6) << "\"occurrences\": [\n";
+    for (unsigned oi = 0, oe = recInfo.Occurrences.size(); oi != oe; ++oi) {
+      OS.indent(8) << "{\n";
+      SymbolOccurrenceInfo &occurInfo = recInfo.Occurrences[oi];
+      OS.indent(10) << "\"symbol\": " << occurInfo.Symbol << ",\n";
+      OS.indent(10) << "\"line\": " << occurInfo.Line << ",\n";
+      OS.indent(10) << "\"col\": " << occurInfo.Column << ",\n";
+      OS.indent(10) << "\"roles\": \"";
+      printSymbolRoles(occurInfo.Roles, OS);
+      OS << '\"';
+      if (!occurInfo.Relations.empty()) {
+        OS << ",\n";
+        OS.indent(10) << "\"relations\": [\n";
+        for (unsigned ri = 0, re = occurInfo.Relations.size(); ri != re; ++ri) {
+          OS.indent(12) << "{\n";
+          SymbolRelationInfo &relInfo = occurInfo.Relations[ri];
+          OS.indent(14) << "\"symbol\": " << relInfo.RelatedSymbol << ",\n";
+          OS.indent(14) << "\"rel-roles\": \"";
+          printSymbolRoles(relInfo.Roles, OS);
+          OS << "\"\n";
+          OS.indent(12) << "}";
+          if (ri < re-1) OS << ',';
+          OS << '\n';
+        }
+        OS.indent(10) << "]\n";
+      }
+      OS << '\n';
+      OS.indent(8) << "}";
+      if (oi < oe-1) OS << ',';
+      OS << '\n';
+    }
+    OS.indent(6) << "]\n";
+    OS.indent(4) << "}";
+    if (i < e-1) OS << ',';
+    OS << '\n';
+  }
+  OS.indent(2) << "],\n";
+
+  StringMap<size_t> UnitIndicesByName;
+  for (unsigned i = 0, e = Units.size(); i != e; ++i) {
+    UnitInfo &unit = *Units[i];
+    UnitIndicesByName[unit.Name] = i;
+  }
+
+  OS.indent(2) << "\"units\": [\n";
+  for (unsigned i = 0, e = Units.size(); i != e; ++i) {
+    OS.indent(4) << "{\n";
+    UnitInfo &unit = *Units[i];
+    OS.indent(6) << "\"triple\": \"" << unit.Triple << "\",\n";
+    OS.indent(6) << "\"out-file\": " << unit.OutFile << ",\n";
+    if (!unit.UnitDepends.empty()) {
+      OS.indent(6) << "\"unit-dependencies\": [";
+      for (unsigned ui = 0, ue = unit.UnitDepends.size(); ui != ue; ++ui) {
+        OS << UnitIndicesByName[unit.UnitDepends[ui]];
+        if (ui < ue-1) OS << ", ";
+      }
+      OS << "],\n";
+    }
+    OS.indent(6) << "\"sources\": [\n";
+    for (unsigned si = 0, se = unit.Sources.size(); si != se; ++si) {
+      OS.indent(8) << "{\n";
+      UnitSourceInfo &source = unit.Sources[si];
+      OS.indent(10) << "\"file\": " << source.FilePath;
+      if (!source.AssociatedRecords.empty()) {
+        OS << ",\n";
+        OS.indent(10) << "\"records\": [";
+        for (unsigned ri = 0, re = source.AssociatedRecords.size(); ri != re; ++ri) {
+          OS << source.AssociatedRecords[ri];
+          if (ri < re-1) OS << ", ";
+        }
+        OS << ']';
+      }
+      OS << '\n';
+      OS.indent(8) << "}";
+      if (si < se-1) OS << ',';
+      OS << '\n';
+    }
+    OS.indent(6) << "]\n";
+    OS.indent(4) << "}";
+    if (i < e-1) OS << ',';
+    OS << '\n';
+  }
+  OS.indent(2) << "]\n";
+  OS << "}\n";
+}
+
+
+bool index::aggregateDataAsJSON(StringRef StorePath, raw_ostream &OS) {
+  std::string error;
+  auto dataStore = IndexStore(StorePath, error);
+  if (!dataStore) {
+    errs() << "error opening store path '" << StorePath << "': " << error << '\n';
+    return true;
+  }
+
+  // Explicitely avoid doing any memory cleanup for aggregator since the process
+  // is going to exit when we are done.
+  Aggregator *aggregator = new Aggregator(std::move(dataStore));
+  bool err = aggregator->process();
+  if (err)
+    return true;
+  aggregator->dumpJSON(OS);
+  return false;
+}
+
+#else
+
+bool index::aggregateDataAsJSON(StringRef StorePath, raw_ostream &OS) {
+  return true;
+}
+#endif

tools/c-index-test/core_main.cpp

 //===-- core_main.cpp - Core Index Tool testbed ---------------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
+#include "JSONAggregation.h"
+#include "indexstore/IndexStoreCXX.h"
+#include "clang/DirectoryWatcher/DirectoryWatcher.h"
 #include "clang/CodeGen/ObjectFilePCHContainerOperations.h"
 #include "clang/Frontend/ASTUnit.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/CompilerInvocation.h"
 #include "clang/Frontend/FrontendAction.h"
 #include "clang/Index/IndexingAction.h"
 #include "clang/Index/IndexDataConsumer.h"
+#include "clang/Index/IndexDataStoreSymbolUtils.h"
+#include "clang/Index/IndexRecordReader.h"
+#include "clang/Index/IndexUnitReader.h"
 #include "clang/Index/USRGeneration.h"
 #include "clang/Index/CodegenNameGenerator.h"
 #include "clang/Serialization/ASTReader.h"
 #include "llvm/Support/CommandLine.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/Path.h"
 #include "llvm/Support/Signals.h"
 #include "llvm/Support/raw_ostream.h"
 #include "llvm/Support/PrettyStackTrace.h"
 
+#define HAVE_CORESERVICES 0
+
+#if defined(__has_include)
+#if __has_include(<CoreServices/CoreServices.h>)
+
+#include <CoreServices/CoreServices.h>
+#undef HAVE_CORESERVICES
+#define HAVE_CORESERVICES 1
+
+#endif
+#endif
+
 using namespace clang;
 using namespace clang::index;
 using namespace llvm;
 
 extern "C" int indextest_core_main(int argc, const char **argv);
 
 namespace {
 
 enum class ActionType {
   None,
   PrintSourceSymbols,
+  PrintRecord,
+  PrintUnit,
+  PrintStoreFormatVersion,
+  AggregateAsJSON,
+  WatchDir,
 };
 
 namespace options {
 
 static cl::OptionCategory IndexTestCoreCategory("index-test-core options");
 
 static cl::opt<ActionType>
 Action(cl::desc("Action:"), cl::init(ActionType::None),
        cl::values(
           clEnumValN(ActionType::PrintSourceSymbols,
-                     "print-source-symbols", "Print symbols from source")),
+                     "print-source-symbols", "Print symbols from source"),
+          clEnumValN(ActionType::PrintRecord,
+                     "print-record", "Print record info"),
+          clEnumValN(ActionType::PrintUnit,
+                     "print-unit", "Print unit info"),
+          clEnumValN(ActionType::PrintStoreFormatVersion,
+                     "print-store-format-version", "Print store format version"),
+          clEnumValN(ActionType::AggregateAsJSON,
+                     "aggregate-json", "Aggregate index data in JSON format"),
+          clEnumValN(ActionType::WatchDir,
+                     "watch-dir", "Watch directory for file events")),
        cl::cat(IndexTestCoreCategory));
 
+static cl::opt<std::string>
+OutputFile("o", cl::desc("output file"),
+           cl::cat(IndexTestCoreCategory));
+
+static cl::list<std::string>
+InputFiles(cl::Positional, cl::desc("<filename>..."));
+
 static cl::extrahelp MoreHelp(
   "\nAdd \"-- <compiler arguments>\" at the end to setup the compiler "
   "invocation\n"
 );
 
 static cl::opt<bool>
 DumpModuleImports("dump-imported-module-files",
                cl::desc("Print symbols and input files from imported modules"));
 
 static cl::opt<bool>
 IncludeLocals("include-locals", cl::desc("Print local symbols"));
 
 static cl::opt<std::string>
 ModuleFilePath("module-file",
                cl::desc("Path to module file to print symbols from"));
 static cl::opt<std::string>
   ModuleFormat("fmodule-format", cl::init("raw"),
         cl::desc("Container format for clang modules and PCH, 'raw' or 'obj'"));
 
+static cl::opt<std::string>
+FilePathAndRange("filepath",
+               cl::desc("File path that can optionally include a line range"));
+
 }
 } // anonymous namespace
 
 static void printSymbolInfo(SymbolInfo SymInfo, raw_ostream &OS);
 static void printSymbolNameAndUSR(const Decl *D, ASTContext &Ctx,
                                   raw_ostream &OS);
 
 namespace {
static bool printSourceSymbolsFromModule(StringRef modulePath,
 
   auto DataConsumer = std::make_shared<PrintIndexDataConsumer>(outs());
   IndexingOptions IndexOpts;
   indexASTUnit(*AU, DataConsumer, IndexOpts);
 
   return false;
 }
 
+#if INDEXSTORE_HAS_BLOCKS
+
+//===----------------------------------------------------------------------===//
+// Print Record
+//===----------------------------------------------------------------------===//
+
+static void printSymbol(const IndexRecordDecl &Rec, raw_ostream &OS);
+static void printSymbol(const IndexRecordOccurrence &Rec, raw_ostream &OS);
+
+static int printRecord(StringRef Filename, raw_ostream &OS) {
+  std::string Error;
+  auto Reader = IndexRecordReader::createWithFilePath(Filename, Error);
+  if (!Reader) {
+    errs() << Error << '\n';
+    return true;
+  }
+
+  Reader->foreachDecl(/*noCache=*/true, [&](const IndexRecordDecl *Rec)->bool {
+    printSymbol(*Rec, OS);
+    return true;
+  });
+  OS << "------------\n";
+  Reader->foreachOccurrence([&](const IndexRecordOccurrence &Rec)->bool {
+    printSymbol(Rec, OS);
+    return true;
+  });
+
+  return false;
+};
+
+//===----------------------------------------------------------------------===//
+// Print Store Records
+//===----------------------------------------------------------------------===//
+
+static void printSymbol(indexstore::IndexRecordSymbol Sym, raw_ostream &OS);
+static void printSymbol(indexstore::IndexRecordOccurrence Occur, raw_ostream &OS);
+
+static bool printStoreRecord(indexstore::IndexStore &Store, StringRef RecName,
+                             StringRef FilePath, raw_ostream &OS) {
+  std::string Error;
+  indexstore::IndexRecordReader Reader(Store, RecName, Error);
+  if (!Reader) {
+    errs() << "error loading record: " << Error << "\n";
+    return true;
+  }
+
+  StringRef Filename = sys::path::filename(FilePath);
+  OS << Filename << '\n';
+  OS << "------------\n";
+  Reader.foreachSymbol(/*noCache=*/true, [&](indexstore::IndexRecordSymbol Sym) -> bool {
+    printSymbol(Sym, OS);
+    return true;
+  });
+  OS << "------------\n";
+  Reader.foreachOccurrence([&](indexstore::IndexRecordOccurrence Occur)->bool {
+    printSymbol(Occur, OS);
+    return true;
+  });
+
+  return false;
+}
+
+static int printStoreRecords(StringRef StorePath, raw_ostream &OS) {
+  std::string Error;
+  indexstore::IndexStore Store(StorePath, Error);
+  if (!Store) {
+    errs() << "error loading store: " << Error << "\n";
+    return 1;
+  }
+
+  bool Success = Store.foreachUnit(/*sorted=*/true, [&](StringRef UnitName) -> bool {
+    indexstore::IndexUnitReader Reader(Store, UnitName, Error);
+    if (!Reader) {
+      errs() << "error loading unit: " << Error << "\n";
+      return false;
+    }
+    return Reader.foreachDependency([&](indexstore::IndexUnitDependency Dep) -> bool {
+      if (Dep.getKind() == indexstore::IndexUnitDependency::DependencyKind::Record) {
+        bool Err = printStoreRecord(Store, Dep.getName(), Dep.getFilePath(), OS);
+        OS << '\n';
+        return !Err;
+      }
+      return true;
+    });
+  });
+
+  return !Success;
+}
+
+static std::string findRecordNameForFile(indexstore::IndexStore &store,
+                                         StringRef filePath) {
+  std::string recName;
+  store.foreachUnit(/*sorted=*/false, [&](StringRef unitName) -> bool {
+    std::string error;
+    indexstore::IndexUnitReader Reader(store, unitName, error);
+    if (!Reader) {
+      errs() << "error loading unit: " << error << "\n";
+      return false;
+    }
+    Reader.foreachDependency([&](indexstore::IndexUnitDependency Dep) -> bool {
+      if (Dep.getKind() == indexstore::IndexUnitDependency::DependencyKind::Record) {
+        if (Dep.getFilePath() == filePath) {
+          recName = Dep.getName();
+          return false;
+        }
+        return true;
+      }
+      return true;
+    });
+    return true;
+  });
+  return recName;
+}
+
+static int printStoreFileRecord(StringRef storePath, StringRef filePath,
+                                Optional<unsigned> lineStart, unsigned lineCount,
+                                raw_ostream &OS) {
+  std::string error;
+  indexstore::IndexStore store(storePath, error);
+  if (!store) {
+    errs() << "error loading store: " << error << "\n";
+    return 1;
+  }
+
+  std::string recName = findRecordNameForFile(store, filePath);
+  if (recName.empty()) {
+    errs() << "could not find record for '" << filePath << "'\n";
+    return 1;
+  }
+
+  if (!lineStart.hasValue())
+    return printStoreRecord(store, recName, filePath, OS);
+
+  indexstore::IndexRecordReader Reader(store, recName, error);
+  if (!Reader) {
+    errs() << "error loading record: " << error << "\n";
+    return 1;
+  }
+
+  Reader.foreachOccurrenceInLineRange(*lineStart, lineCount, [&](indexstore::IndexRecordOccurrence Occur)->bool {
+    printSymbol(Occur, OS);
+    return true;
+  });
+
+  return 0;
+}
+
+
+//===----------------------------------------------------------------------===//
+// Print Unit
+//===----------------------------------------------------------------------===//
+
+static int printUnit(StringRef Filename, raw_ostream &OS) {
+  std::string Error;
+  auto Reader = IndexUnitReader::createWithFilePath(Filename, Error);
+  if (!Reader) {
+    errs() << Error << '\n';
+    return true;
+  }
+
+  OS << "provider: " << Reader->getProviderIdentifier() << '-' << Reader->getProviderVersion() << '\n';
+  OS << "is-system: " << Reader->isSystemUnit() << '\n';
+  OS << "is-module: " << Reader->isModuleUnit() << '\n';
+  OS << "module-name: " << (Reader->getModuleName().empty() ? "<none>" : Reader->getModuleName()) << '\n';
+  OS << "has-main: " << Reader->hasMainFile() << '\n';
+  OS << "main-path: " << Reader->getMainFilePath() << '\n';
+  OS << "work-dir: " << Reader->getWorkingDirectory() << '\n';
+  OS << "out-file: " << Reader->getOutputFile() << '\n';
+  OS << "target: " << Reader->getTarget() << '\n';
+  OS << "is-debug: " << Reader->isDebugCompilation() << '\n';
+  OS << "DEPEND START\n";
+  unsigned NumDepends = 0;
+  Reader->foreachDependency([&](const IndexUnitReader::DependencyInfo &Dep) -> bool {
+    switch (Dep.Kind) {
+    case IndexUnitReader::DependencyKind::Unit:
+      OS << "Unit | "; break;
+    case IndexUnitReader::DependencyKind::Record:
+      OS << "Record | "; break;
+    case IndexUnitReader::DependencyKind::File:
+      OS << "File | "; break;
+    }
+    OS << (Dep.IsSystem ? "system" : "user");
+    OS << " | ";
+    if (!Dep.ModuleName.empty())
+      OS << Dep.ModuleName << " | ";
+    OS << Dep.FilePath << " | " << Dep.UnitOrRecordName << " | ";
+    OS << Dep.ModTime << " | " << Dep.FileSize << '\n';
+    ++NumDepends;
+    return true;
+  });
+  OS << "DEPEND END (" << NumDepends << ")\n";
+  OS << "INCLUDE START\n";
+  unsigned NumIncludes = 0;
+  Reader->foreachInclude([&](const IndexUnitReader::IncludeInfo &Inc) -> bool {
+    OS << Inc.SourcePath << ":" << Inc.SourceLine << " | ";
+    OS << Inc.TargetPath << '\n';
+    ++NumIncludes;
+    return true;
+  });
+  OS << "INCLUDE END (" << NumIncludes << ")\n";
+
+  return false;
+};
+
+//===----------------------------------------------------------------------===//
+// Print Store Units
+//===----------------------------------------------------------------------===//
+
+static bool printStoreUnit(indexstore::IndexStore &Store, StringRef UnitName,
+                           raw_ostream &OS) {
+  std::string Error;
+  indexstore::IndexUnitReader Reader(Store, UnitName, Error);
+  if (!Reader) {
+    errs() << "error loading unit: " << Error << "\n";
+    return true;
+  }
+
+  OS << "provider: " << Reader.getProviderIdentifier() << '-' << Reader.getProviderVersion() << '\n';
+  OS << "is-system: " << Reader.isSystemUnit() << '\n';
+  OS << "is-module: " << Reader.isModuleUnit() << '\n';
+  OS << "module-name: " << (Reader.getModuleName().empty() ? "<none>" : Reader.getModuleName()) << '\n';
+  OS << "has-main: " << Reader.hasMainFile() << '\n';
+  OS << "main-path: " << Reader.getMainFilePath() << '\n';
+  OS << "work-dir: " << Reader.getWorkingDirectory() << '\n';
+  OS << "out-file: " << Reader.getOutputFile() << '\n';
+  OS << "target: " << Reader.getTarget() << '\n';
+  OS << "is-debug: " << Reader.isDebugCompilation() << '\n';
+  OS << "DEPEND START\n";
+  unsigned NumDepends = 0;
+  Reader.foreachDependency([&](indexstore::IndexUnitDependency Dep) -> bool {
+    switch (Dep.getKind()) {
+    case indexstore::IndexUnitDependency::DependencyKind::Unit:
+      OS << "Unit | "; break;
+    case indexstore::IndexUnitDependency::DependencyKind::Record:
+      OS << "Record | "; break;
+    case indexstore::IndexUnitDependency::DependencyKind::File:
+      OS << "File | "; break;
+    }
+    OS << (Dep.isSystem() ? "system" : "user");
+    OS << " | ";
+    if (!Dep.getModuleName().empty())
+      OS << Dep.getModuleName() << " | ";
+    OS << Dep.getFilePath() << " | " << Dep.getName() << " | ";
+    OS << Dep.getModificationTime() << '\n';
+    ++NumDepends;
+    return true;
+  });
+  OS << "DEPEND END (" << NumDepends << ")\n";
+  OS << "INCLUDE START\n";
+  unsigned NumIncludes = 0;
+  Reader.foreachInclude([&](indexstore::IndexUnitInclude Inc) -> bool {
+    OS << Inc.getSourcePath() << ":" << Inc.getSourceLine() << " | ";
+    OS << Inc.getTargetPath() << '\n';
+    ++NumIncludes;
+    return true;
+  });
+  OS << "INCLUDE END (" << NumIncludes << ")\n";
+
+  return false;
+}
+
+static int printStoreUnits(StringRef StorePath, raw_ostream &OS) {
+  std::string Error;
+  indexstore::IndexStore Store(StorePath, Error);
+  if (!Store) {
+    errs() << "error loading store: " << Error << "\n";
+    return 1;
+  }
+
+  bool Success = Store.foreachUnit(/*sorted=*/true, [&](StringRef UnitName) -> bool {
+    OS << UnitName << '\n';
+    OS << "--------\n";
+    bool err = printStoreUnit(Store, UnitName, OS);
+    OS << '\n';
+    return !err;
+  });
+
+  return !Success;
+}
+
+
+#else
+
+static int printUnit(StringRef Filename, raw_ostream &OS) {
+  return 1;
+}
+
+static int printStoreUnits(StringRef StorePath, raw_ostream &OS) {
+  return 1;
+}
+
+static int printStoreFileRecord(StringRef storePath, StringRef filePath,
+                                Optional<unsigned> lineStart, unsigned lineCount,
+                                raw_ostream &OS) {
+  return 1;
+}
+
+#endif
+
 //===----------------------------------------------------------------------===//
 // Helper Utils
 //===----------------------------------------------------------------------===//
 
 static void printSymbolInfo(SymbolInfo SymInfo, raw_ostream &OS) {
   OS << getSymbolKindString(SymInfo.Kind);
   if (SymInfo.SubKind != SymbolSubKind::None)
     OS << '/' << getSymbolSubKindString(SymInfo.SubKind);
static void printSymbolNameAndUSR(const Decl *D, ASTContext &Ctx,
   SmallString<256> USRBuf;
   if (generateUSRForDecl(D, USRBuf)) {
     OS << "<no-usr>";
   } else {
     OS << USRBuf;
   }
 }
 
+#if INDEXSTORE_HAS_BLOCKS
+
+static void printSymbol(const IndexRecordDecl &Rec, raw_ostream &OS) {
+  printSymbolInfo(Rec.SymInfo, OS);
+  OS << " | ";
+
+  if (Rec.Name.empty())
+    OS << "<no-name>";
+  else
+    OS << Rec.Name;
+  OS << " | ";
+
+  if (Rec.USR.empty())
+    OS << "<no-usr>";
+  else
+    OS << Rec.USR;
+  OS << " | ";
+
+  if (Rec.CodeGenName.empty())
+    OS << "<no-cgname>";
+  else
+    OS << Rec.CodeGenName;
+  OS << " | ";
+
+  printSymbolRoles(Rec.Roles, OS);
+  OS << " - ";
+  printSymbolRoles(Rec.RelatedRoles, OS);
+  OS << '\n';
+}
+
+static void printSymbol(const IndexRecordOccurrence &Rec, raw_ostream &OS) {
+  OS << Rec.Line << ':' << Rec.Column << " | ";
+  printSymbolInfo(Rec.Dcl->SymInfo, OS);
+  OS << " | ";
+
+  if (Rec.Dcl->USR.empty())
+    OS << "<no-usr>";
+  else
+    OS << Rec.Dcl->USR;
+  OS << " | ";
+
+  printSymbolRoles(Rec.Roles, OS);
+  OS << " | ";
+  OS << "rel: " << Rec.Relations.size() << '\n';
+  for (auto &Rel : Rec.Relations) {
+    OS << '\t';
+    printSymbolRoles(Rel.Roles, OS);
+    OS << " | ";
+    if (Rel.Dcl->USR.empty())
+      OS << "<no-usr>";
+    else
+      OS << Rel.Dcl->USR;
+    OS << '\n';
+  }
+}
+
+static void printSymbol(indexstore::IndexRecordSymbol Sym, raw_ostream &OS) {
+  SymbolInfo SymInfo{getSymbolKind(Sym.getKind()),
+                     getSymbolSubKind(Sym.getSubKind()),
+                     SymbolPropertySet(Sym.getProperties()),
+                     getSymbolLanguage(Sym.getLanguage())};
+
+  printSymbolInfo(SymInfo, OS);
+  OS << " | ";
+
+  if (Sym.getName().empty())
+    OS << "<no-name>";
+  else
+    OS << Sym.getName();
+  OS << " | ";
+
+  if (Sym.getUSR().empty())
+    OS << "<no-usr>";
+  else
+    OS << Sym.getUSR();
+  OS << " | ";
+
+  if (Sym.getCodegenName().empty())
+    OS << "<no-cgname>";
+  else
+    OS << Sym.getCodegenName();
+  OS << " | ";
+
+  printSymbolRoles(Sym.getRoles(), OS);
+  OS << " - ";
+  printSymbolRoles(Sym.getRelatedRoles(), OS);
+  OS << '\n';
+}
+
+static void printSymbol(indexstore::IndexRecordOccurrence Occur, raw_ostream &OS) {
+  OS << Occur.getLineCol().first << ':' << Occur.getLineCol().second << " | ";
+  auto Sym = Occur.getSymbol();
+  SymbolInfo SymInfo{getSymbolKind(Sym.getKind()),
+                     getSymbolSubKind(Sym.getSubKind()),
+                     SymbolPropertySet(Sym.getProperties()),
+                     getSymbolLanguage(Sym.getLanguage())};
+
+  printSymbolInfo(SymInfo, OS);
+  OS << " | ";
+
+  if (Sym.getUSR().empty())
+    OS << "<no-usr>";
+  else
+    OS << Sym.getUSR();
+  OS << " | ";
+
+  unsigned NumRelations = 0;
+  Occur.foreachRelation([&](indexstore::IndexSymbolRelation) {
+    ++NumRelations;
+    return true;
+  });
+
+  printSymbolRoles(Occur.getRoles(), OS);
+  OS << " | ";
+  OS << "rel: " << NumRelations << '\n';
+  Occur.foreachRelation([&](indexstore::IndexSymbolRelation Rel) {
+    OS << '\t';
+    printSymbolRoles(Rel.getRoles(), OS);
+    OS << " | ";
+    auto Sym = Rel.getSymbol();
+    if (Sym.getUSR().empty())
+      OS << "<no-usr>";
+    else
+      OS << Sym.getUSR();
+    OS << '\n';
+    return true;
+  });
+}
+
+#else
+
+static int printRecord(StringRef Filename, raw_ostream &OS) {
+  return 1;
+}
+static int printStoreRecords(StringRef StorePath, raw_ostream &OS) {
+  return 1;
+}
+
+#endif
+
+static int watchDirectory(StringRef dirPath) {
+  raw_ostream &OS = outs();
+  auto receiver = [&](ArrayRef<DirectoryWatcher::Event> Events, bool isInitial) {
+    for (auto evt : Events) {
+      switch (evt.Kind) {
+        case DirectoryWatcher::EventKind::Added:
+          OS << "added: "; break;
+        case DirectoryWatcher::EventKind::Modified:
+          OS << "modified: "; break;
+        case DirectoryWatcher::EventKind::Removed:
+          OS << "removed: "; break;
+        case DirectoryWatcher::EventKind::DirectoryDeleted:
+          OS << "dir deleted: "; break;
+
+      }
+      OS << evt.Filename << '\n';
+    }
+  };
+  std::string Error;
+  auto watcher = DirectoryWatcher::create(dirPath, receiver,
+                                          /*waitInitialSync=*/true, Error);
+  if (!watcher) {
+    errs() << "failed creating directory watcher: " << Error << '\n';
+    return 1;
+  }
+#if HAVE_CORESERVICES
+  dispatch_main();
+#else
+  return 1;
+#endif
+}
+
 //===----------------------------------------------------------------------===//
 // Command line processing.
 //===----------------------------------------------------------------------===//
 
+bool deconstructPathAndRange(StringRef input,
+                             std::string &filepath,
+                             Optional<unsigned> &lineStart,
+                             unsigned &lineCount) {
+  StringRef path, range;
+  std::tie(path, range) = input.split(':');
+  StringRef start, end;
+  std::tie(start, end) = range.split(':');
+  filepath = path;
+  lineCount = 0;
+  if (start.empty())
+    return false;
+  unsigned num;
+  if (start.getAsInteger(10, num)) {
+    errs() << "couldn't convert to integer: " << start << '\n';
+    return true;
+  }
+  lineStart = num;
+  if (end.empty())
+    return false;
+  if (end.getAsInteger(10, num)) {
+    errs() << "couldn't convert to integer: " << end << '\n';
+    return true;
+  }
+  lineCount = num-lineStart.getValue();
+  return false;
+}
+
 int indextest_core_main(int argc, const char **argv) {
   sys::PrintStackTraceOnErrorSignal(argv[0]);
   PrettyStackTraceProgram X(argc, argv);
 
   assert(argv[1] == StringRef("core"));
   ++argv;
   --argc;
 
if (options::Action == ActionType::PrintSourceSymbols) {
     }
     if (CompArgs.empty()) {
       errs() << "error: missing compiler args; pass '-- <compiler arguments>'\n";
       return 1;
     }
     return printSourceSymbols(CompArgs, options::DumpModuleImports, options::IncludeLocals);
   }
 
+  if (options::Action == ActionType::PrintRecord) {
+    if (!options::FilePathAndRange.empty()) {
+      std::string filepath;
+      Optional<unsigned> lineStart;
+      unsigned lineCount;
+      if (deconstructPathAndRange(options::FilePathAndRange,
+                                  filepath, lineStart, lineCount))
+        return 1;
+
+      if (options::InputFiles.empty()) {
+        errs() << "error: missing index store path\n";
+        return 1;
+      }
+      return printStoreFileRecord(options::InputFiles[0], filepath, lineStart, lineCount, outs());
+    }
+
+    if (options::InputFiles.empty()) {
+      errs() << "error: missing input file or directory\n";
+      return 1;
+    }
+
+    if (sys::fs::is_directory(options::InputFiles[0]))
+      return printStoreRecords(options::InputFiles[0], outs());
+    else
+      return printRecord(options::InputFiles[0], outs());
+  }
+
+  if (options::Action == ActionType::PrintUnit) {
+    if (options::InputFiles.empty()) {
+      errs() << "error: missing input file or directory\n";
+      return 1;
+    }
+
+    if (sys::fs::is_directory(options::InputFiles[0]))
+      return printStoreUnits(options::InputFiles[0], outs());
+    else
+      return printUnit(options::InputFiles[0], outs());
+  }
+
+#if INDEXSTORE_HAS_BLOCKS
+  if (options::Action == ActionType::PrintStoreFormatVersion) {
+    outs() << indexstore::IndexStore::formatVersion() << '\n';
+  }
+#endif
+
+  if (options::Action == ActionType::AggregateAsJSON) {
+    if (options::InputFiles.empty()) {
+      errs() << "error: missing input data store directory\n";
+      return 1;
+    }
+    StringRef storePath = options::InputFiles[0];
+    if (options::OutputFile.empty())
+      return aggregateDataAsJSON(storePath, outs());
+    std::error_code EC;
+    raw_fd_ostream OS(options::OutputFile, EC, llvm::sys::fs::F_None);
+    if (EC) {
+      errs() << "failed to open output file: " << EC.message() << '\n';
+      return 1;
+    }
+    return aggregateDataAsJSON(storePath, OS);
+  }
+
+  if (options::Action == ActionType::WatchDir) {
+    if (options::InputFiles.empty()) {
+      errs() << "error: missing directory path\n";
+      return 1;
+    }
+    return watchDirectory(options::InputFiles[0]);
+  }
+
   return 0;
 }