[WIP] Index-while-building

[sammccall]

Thanks for sharing this! I hope I will have time to dig into more detail.

At a high level, my main question is around layering.
Until now, lib/Index has mostly been a pretty abstract framework: it gathers information from ASTs and exposes it to callers, who then decide what to do with it.
This has served clangd pretty well: we've been able to hook into it, use our own storage, fix the occasional bug, and avoid a bunch of custom tree traversals.
My understanding is that historically XCode has used it in similar ways by out-of-tree consumers (possibly via libclang?)

This patch introduces another concrete consumer in-tree, which is great - making uses more visible helps everyone understand and change the API.
It's also well-layered! But much of it's in the Index library, which makes the layering more fragile:

• it's unclear to me whether strict layering is intended here - there seem to be a couple of violations (though this is preview code)
• it's not going to be clear to people arriving at the code which dependencies are OK, and whether IWB is special or one client of many
• as it stands, there's no technical enforcement of layering (so it may rot), nor a way to depend on the abstract part but not the concrete part. This makes changes to the concrete part scarier.

If the currently-abstract parts Index becomes coupled to IWB over time, it'll be a regression for how clangd is using the library. Maybe we can converge on the concrete consumers too, but carefully (and not necessarily!)

Can we move the concrete parts to IndexStore directories (both header and impls)? (EmitIndexAction, DeclOccurrenceGenerator, IndexDataFormat, IndexDataStore, IndexRecordWriter, PathStorage, half of IndexOptions, BitstreamUtil, FSUtil, FileIndexRecord)

clang/include/clang/Index/GenerateIndexAction.h

35

(this seems like suspicious layering, but also seems to be unused)

clang/include/clang/Index/IndexOptions.h

40

This file mixes abstract indexing stuff (IndexingOptions) with concrete (RecordingOptions)

clang/lib/Index/GenerateIndexAction.cpp

12

I can't tell where this is used - it's suspicious if required

[jkorous]

Thanks for taking a look Sam!

I finished porting of all the changes on top of llvm.org master in the meantime. The updated patch contains all the code we'd like to land. Build (on macOS) and tests are fine.
I'll leave it as a single patch for now so we can finish discussion about layering and I'll split it into couple smaller patches once this is done.

I've put the actual index-while-building part into separate subdirectories for now. For now I went with IndexWhileBuilding name but I'm open to discussion.

clang/include/clang/Index/GenerateIndexAction.h

35

Right. Removed.

clang/lib/Index/GenerateIndexAction.cpp

12

Removed. Thanks.

[jkorous]

Uploaded the full diff.

[akyrtzi]

In D64384#1596521, @jkorous wrote:

For now I went with IndexWhileBuilding name but I'm open to discussion.

I'd prefer 'IndexSerialization'.

[jkorous]

clang-format

[gribozavr]

Not a full review, just some comments that would help reduce the size of the patch, plus code organization concerns.

Could we avoid nested libraries and rename clang/include/clang/Index/IndexWhileBuilding to clang/include/clang/IndexWhileBuilding ?

clang-tools-extra/clangd/ClangdUnit.cpp

30

Please submit changes like these this #include removal separately to reduce the size of the patch.

clang-tools-extra/clangd/FindSymbols.cpp

20

Please submit changes like these this #include removal separately to reduce the size of the patch.

clang-tools-extra/clangd/index/IndexAction.cpp

128

Could you move index action into a separate header and inline index::createIndexingAction into its callers in a separate commit?

clang/include/clang/Index/IndexAction.h

22

+1. Instead of WrapperFrontendAction, could you implement a general frontend action multiplexer in clang/include/clang/Frontend/FrontendAction.h ? You can do that as a separate commit.

clang/include/clang/Index/IndexDataConsumer.h

64

Changes to this file can be submitted as a separate patch.

clang/include/clang/Index/IndexOptions.h

19

Are these the declarations moved from clang/include/clang/Index/IndexingAction.h? I'd suggest to commit this header splitting separately to reduce the size of the patch.

clang/include/clang/Index/IndexWhileBuilding/PathStorage.h

74

Please add newlines (everywhere in the patch).

clang/include/clang/Index/USRGeneration.h

100

Unintended edit?

[gribozavr]

clang/include/clang/Driver/Options.td

331

Could you add a more detailed description? Even as a compiler engineer, I can't really tell what index-record-codegen-name really does or why I would want to turn it on.

clang/include/clang/Frontend/CompilerInstance.h

180

No need to write "\brief", it is the default.

180

Please don't write what something is used for, write what something is.

"A factory for wrappers around FrontendActions that compile imported modules."

810

I don't see anything actually using GenModuleActionWrapper.

clang/lib/Frontend/CompilerInstance.cpp

1110

Please end comments with a period.

[gribozavr]

clang/include/clang/Frontend/FrontendOptions.h

391

Could we change "ignore system symbols" into a positive option ("record system symbols")? Throughout the patch please.

clang/include/clang/Index/CodegenNameGenerator.h

46

Why change to a raw pointer?

We can use a unique_ptr to an incomplete type, as long as the destructor ~CodegenNameGenerator is defined in a file where the type is complete.

clang/lib/Index/CodegenNameGenerator.cpp

31

This code looks like a straight copy from clang/lib/AST/Mangle.cpp -- can it be de-duplicated?

clang/unittests/Index/IndexTests.cpp

74

This change can be a separate patch.

106

Unclear why this test overrides these methods, if the implementation is identical to the base class.

112

No need to add virtual when we already have override.

[jkorous]

clang/include/clang/Index/IndexAction.h

22

Sent for review: https://reviews.llvm.org/D66125
I'll rebase this patch once/if that lands.

[jkorous]

Thanks for the review! I addressed just some minor details so far.

clang/include/clang/Frontend/CompilerInstance.h

810

IIUC it's both used (lambda on CompilerInstance.cpp:L1136) and passed (CompilerInstance.cpp:L1112) to CompilerInstance of module dependencies in compileModuleImpl.

clang/unittests/Index/IndexTests.cpp

106

Removed these overrides.

[jkorous]

Addressed some of the comments.

[gribozavr]

clang/lib/Frontend/CompilerInstance.cpp

1132

My suggestion would be to change GenerateModuleAction::CreateASTConsumer to perform the indexing. Doing so will avoid the problem of having to compose FrontendActions (which we found in another patch to be tricky), and also will make the code a lot more straightforward, I think.

WDYT?

[jkorous]

Added ASTConsumerInjector and changed `Emit|Generate-IndexAction to -ASTConsumer.

[jkorous]

clang/lib/Index/CodegenNameGenerator.cpp

31

Seems like you're right: https://reviews.llvm.org/D63535

[jkorous]

• addressed more comments
• rebased
• clang-format

[jkorous]

clang/include/clang/Frontend/CompilerInstance.h

810

I was wrong! I thought we are passing some data in the wrapper. Turned out we aren't and I was able to completely remove it. Thanks!

[jkorous]

• addressed some comments
• removed GenModuleActionWrapper

[jkorous]

clang/include/clang/Frontend/FrontendOptions.h

391

Surprisingly it seems that the name is actually mostly determined by the fact that we want to index system symbols by default and that natural CLI for boolean argument with default positive value is optional flag with negative semantics.

Specifically - the semantics/name of the flag OPT_index_ignore_system_symbols is pretty much given (or ?).

I could rename just IndexIgnoreSystemSymbols and keep OPT_index_ignore_system_symbols but it feels less clear than current state.

WDYT?

[jkorous]

• renamed "codegen name generator" -> "mangled name generator"
• ranemed flag index-record-codegen-name -> index-record-mangled-name
• removed wrapper over ASTNameGenerator from libIndex

[jkorous]

I addressed the rest of the comments (excluding comments about splitting the patch).
Next step is to follow up on the discussion about FrontendActions/ASTConsumers from https://reviews.llvm.org/D66125 and then I'd like to start splitting off separate commits.
I'll keep this patch rebased on top of what has already landed to have some sense what still needs to be done.

[jkorous]

I'm now taking a look at suggestions from https://reviews.llvm.org/D66125 regarding IndexingFrontendAction in libclang.

IndexingFrontendAction is a lot more complex -- not only it needs a complex factory that creates the ASTConsumer, but it also overrides getTranslationUnitKind, which changes the behavior of Sema and Preprocessor in significant ways. So it does not seem like IndexFrontendAction can become an ASTConsumer.

However, IndexFrontendAction does not seem like a standalone ASTConsumer anyway. Its primary reason to exist is the logic in IndexingConsumer::shouldSkipFunctionBody. That logic seems interesting for general-purpose indexing, not just for libclang, so it would make sense to fold it into IndexASTConsumer.

To summarize:

Fold shouldSkipBody from IndexingConsumer in libclang into IndexASTConsumer in libIndex.
Make libIndex expose an ASTConsumer factory function, that encapsulates the logic currently in IndexingFrontendAction::CreateASTConsumer. Clients who need to combine indexing with some other FrontendAction can use that factory to create a consumer that will index whatever comes out of Sema.
Make a FrontendAction that implements the getTranslationUnitKind function to implement the corresponding optimization. If this FrontendAction is only needed in libclang, it can live there, otherwise, it can be in libIndex.
What do you think?

IIUC the goal was to move as much of interesting logic to libIndex as possible and hopefully implement the rest of the IndexingFrontendAction as an ASTConsumer instead of FrontendAction.

Now, it seems to me:

• IndexingFrontendAction::getTranslationUnitKind() method just returns two different results based on configuration, there's no logic - we just pass the configuration via DataConsumer. I am not quite sure this is interesting enough to create a self-standing FrontendAction. (To be completely honest - this actually seems more like a hack but I am not familiar enough with how does TUKind affect Preprocessor and Sema to assess this.)
• We could factor out an ASTConsumer factory from IndexingFrontendAction::CreateASTConsumer but there are also preprocessor callbacks created, DataConsumer set up with CompilerInstance and CXIndexDataConsumer::importedPCH called. This means we'd probably end up with having a FrontendAction with CXIndexDataConsumer in libclang anyway as this doesn't seem like a code that would quite fit into the hypothetical ASTConsumer factory.

Overall - it doesn't seem to me we could make IndexingFrontendAction significantly simpler or could reuse much of its code elsewhere so I suggest we leave it as it is.

WDYT?

[jkorous]

I'll start splitting off some self-standing parts that won't be affected by hypothetical changes at the FrontendAction/ASTConsumer level.

clang/include/clang/Index/IndexDataConsumer.h

64

Done in cfd641d84a5 [clang][Index][NFC] Move IndexDataConsumer default implementation

[jkorous]

• Split off change to IndexDataConsumer.
• Rebase.

[jkorous]

Rebased on TOT and removed noise from the patch as I've already landed couple refactoring patches.

[jkorous]

Rebased on top of current master and also tentatively on top of SerializablePathCollection (previously PathStorage, PathIndexer) from https://reviews.llvm.org/D66854

[jkorous]

Rebased on recent master + https://reviews.llvm.org/D66854

clang-tools-extra/clangd/ClangdUnit.cpp

 #include "clang/Basic/LangOptions.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Basic/TokenKinds.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/CompilerInvocation.h"
 #include "clang/Frontend/FrontendActions.h"
 #include "clang/Frontend/Utils.h"
 #include "clang/Index/IndexDataConsumer.h"
-#include "clang/Index/IndexingAction.h"
 #include "clang/Lex/Lexer.h"
 #include "clang/Lex/MacroInfo.h"
 #include "clang/Lex/Preprocessor.h"
 #include "clang/Lex/PreprocessorOptions.h"
 #include "clang/Sema/Sema.h"
 #include "clang/Serialization/ASTWriter.h"
 #include "clang/Serialization/PCHContainerOperations.h"
 #include "clang/Tooling/CompilationDatabase.h"

clang-tools-extra/clangd/FindSymbols.cpp

 #include "FuzzyMatch.h"
 #include "Logger.h"
 #include "Quality.h"
 #include "SourceCode.h"
 #include "index/Index.h"
 #include "clang/AST/DeclTemplate.h"
 #include "clang/Index/IndexDataConsumer.h"
 #include "clang/Index/IndexSymbol.h"
-#include "clang/Index/IndexingAction.h"
 #include "llvm/Support/FormatVariadic.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/ScopedPrinter.h"
 
 #define DEBUG_TYPE "FindSymbols"
 
 namespace clang {
 namespace clangd {

clang-tools-extra/clangd/XRefs.cpp

 #include "clang/AST/DeclTemplate.h"
 #include "clang/AST/ExprCXX.h"
 #include "clang/AST/PrettyPrinter.h"
 #include "clang/AST/RecursiveASTVisitor.h"
 #include "clang/AST/Type.h"
 #include "clang/Basic/LLVM.h"
 #include "clang/Basic/SourceLocation.h"
 #include "clang/Basic/SourceManager.h"
+#include "clang/Index/GenerateIndexAction.h"
 #include "clang/Index/IndexDataConsumer.h"
+#include "clang/Index/IndexOptions.h"
 #include "clang/Index/IndexSymbol.h"
-#include "clang/Index/IndexingAction.h"
 #include "clang/Index/USRGeneration.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/None.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/FormatVariadic.h"

clang-tools-extra/clangd/index/FileIndex.cpp

 #include "Logger.h"
 #include "SymbolCollector.h"
 #include "index/CanonicalIncludes.h"
 #include "index/Index.h"
 #include "index/MemIndex.h"
 #include "index/Merge.h"
 #include "index/SymbolOrigin.h"
 #include "index/dex/Dex.h"
-#include "clang/Index/IndexingAction.h"
+#include "clang/Index/GenerateIndexAction.h"
+#include "clang/Index/IndexOptions.h"
 #include "clang/Lex/MacroInfo.h"
 #include "clang/Lex/Preprocessor.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/DenseSet.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/StringRef.h"
 #include <memory>
 

clang-tools-extra/clangd/index/IndexAction.cpp

 //===--- IndexAction.cpp -----------------------------------------*- C++-*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "IndexAction.h"
 #include "Headers.h"
 #include "Logger.h"
 #include "index/Relation.h"
 #include "index/SymbolOrigin.h"
 #include "clang/Frontend/CompilerInstance.h"
-#include "clang/Index/IndexingAction.h"
+#include "clang/Index/GenerateIndexAction.h"
+#include "clang/Index/IndexOptions.h"
 #include "clang/Tooling/Tooling.h"
 #include <utility>
 
 namespace clang {
 namespace clangd {
 namespace {
 
 llvm::Optional<std::string> toURI(const FileEntry *File) {

 public:
   IndexAction(std::shared_ptr<SymbolCollector> C,
               std::unique_ptr<CanonicalIncludes> Includes,
               const index::IndexingOptions &Opts,
               std::function<void(SymbolSlab)> SymbolsCallback,
               std::function<void(RefSlab)> RefsCallback,
               std::function<void(RelationSlab)> RelationsCallback,
               std::function<void(IncludeGraph)> IncludeGraphCallback)
-      : WrapperFrontendAction(index::createIndexingAction(C, Opts, nullptr)),
+      : WrapperFrontendAction(
+            llvm::make_unique<index::GenerateIndexAction>(C, Opts)),
         SymbolsCallback(SymbolsCallback), RefsCallback(RefsCallback),
         RelationsCallback(RelationsCallback),
         IncludeGraphCallback(IncludeGraphCallback), Collector(C),
         Includes(std::move(Includes)),
         PragmaHandler(collectIWYUHeaderMaps(this->Includes.get())) {}
 
   std::unique_ptr<ASTConsumer>
   CreateASTConsumer(CompilerInstance &CI, llvm::StringRef InFile) override {

clang-tools-extra/clangd/index/SymbolCollector.cpp

 #include "clang/AST/Decl.h"
 #include "clang/AST/DeclBase.h"
 #include "clang/AST/DeclCXX.h"
 #include "clang/AST/DeclTemplate.h"
 #include "clang/Basic/SourceLocation.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Basic/Specifiers.h"
 #include "clang/Index/IndexSymbol.h"
-#include "clang/Index/IndexingAction.h"
 #include "clang/Index/USRGeneration.h"
 #include "clang/Lex/Preprocessor.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Path.h"
 
 namespace clang {

clang-tools-extra/clangd/unittests/SymbolCollectorTests.cpp

 //===-- SymbolCollectorTests.cpp  -------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "Annotations.h"
 #include "TestFS.h"
 #include "TestTU.h"
 #include "index/SymbolCollector.h"
 #include "clang/Basic/FileManager.h"
 #include "clang/Basic/FileSystemOptions.h"
 #include "clang/Frontend/CompilerInstance.h"
-#include "clang/Index/IndexingAction.h"
+#include "clang/Index/GenerateIndexAction.h"
+#include "clang/Index/IndexOptions.h"
 #include "clang/Tooling/Tooling.h"
 #include "llvm/ADT/IntrusiveRefCntPtr.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/VirtualFileSystem.h"
 #include "gmock/gmock-matchers.h"
 #include "gmock/gmock-more-matchers.h"
public:
 
   clang::FrontendAction *create() override {
     class WrappedIndexAction : public WrapperFrontendAction {
     public:
       WrappedIndexAction(std::shared_ptr<SymbolCollector> C,
                          const index::IndexingOptions &Opts,
                          CommentHandler *PragmaHandler)
           : WrapperFrontendAction(
-                index::createIndexingAction(C, Opts, nullptr)),
+                llvm::make_unique<index::GenerateIndexAction>(C, Opts)),
             PragmaHandler(PragmaHandler) {}
 
       std::unique_ptr<ASTConsumer>
       CreateASTConsumer(CompilerInstance &CI, llvm::StringRef InFile) override {
         if (PragmaHandler)
           CI.getPreprocessor().addCommentHandler(PragmaHandler);
         return WrapperFrontendAction::CreateASTConsumer(CI, InFile);
       }

clang-tools-extra/clangd/unittests/SymbolInfoTests.cpp

 #include "Compiler.h"
 #include "Matchers.h"
 #include "SyncAPI.h"
 #include "TestFS.h"
 #include "TestTU.h"
 #include "XRefs.h"
 #include "index/FileIndex.h"
 #include "index/SymbolCollector.h"
-#include "clang/Index/IndexingAction.h"
 #include "llvm/Support/Path.h"
 #include "gmock/gmock.h"
 #include "gtest/gtest.h"
 
 namespace clang {
 namespace clangd {
 namespace {
 

clang-tools-extra/clangd/unittests/TypeHierarchyTests.cpp

 #include "SyncAPI.h"
 #include "TestFS.h"
 #include "TestTU.h"
 #include "XRefs.h"
 #include "index/FileIndex.h"
 #include "index/SymbolCollector.h"
 #include "clang/AST/DeclCXX.h"
 #include "clang/AST/DeclTemplate.h"
-#include "clang/Index/IndexingAction.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/ScopedPrinter.h"
 #include "gmock/gmock.h"
 #include "gtest/gtest.h"
 
 namespace clang {
 namespace clangd {
 namespace {

clang-tools-extra/clangd/unittests/XRefsTests.cpp

 #include "SyncAPI.h"
 #include "TestFS.h"
 #include "TestIndex.h"
 #include "TestTU.h"
 #include "XRefs.h"
 #include "index/FileIndex.h"
 #include "index/MemIndex.h"
 #include "index/SymbolCollector.h"
-#include "clang/Index/IndexingAction.h"
 #include "llvm/ADT/None.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/ScopedPrinter.h"
 #include "gmock/gmock.h"
 #include "gtest/gtest.h"
 #include <string>
 
 namespace clang {

clang/include/clang/Basic/DiagnosticFrontendKinds.td

 def err_modules_embed_file_not_found :
   Error<"file '%0' specified by '-fmodules-embed-file=' not found">,
   DefaultFatal;
 def err_module_header_file_not_found :
   Error<"module header file '%0' not found">, DefaultFatal;
 def err_module_header_file_invalid :
   Error<"unexpected module header file input '%0'">, DefaultFatal;
 
+def remark_index_producing_module_file_data : Remark<"producing index data for "
+  "module file '%0'">,
+    InGroup<IndexStore>;
+
 def err_interface_stubs : Error<"clang-ifs (-emit-iterface-stubs): %0">;
 
 def err_test_module_file_extension_version : Error<
   "test module file extension '%0' has different version (%1.%2) than expected "
   "(%3.%4)">;
 
 def err_missing_vfs_overlay_file : Error<
   "virtual filesystem overlay file '%0' not found">, DefaultFatal;

clang/include/clang/Basic/DiagnosticGroups.td

 def MismatchedParameterTypes : DiagGroup<"mismatched-parameter-types">;
 def MismatchedReturnTypes : DiagGroup<"mismatched-return-types">;
 def MismatchedTags : DiagGroup<"mismatched-tags">;
 def MissingFieldInitializers : DiagGroup<"missing-field-initializers">;
 def ModuleBuild : DiagGroup<"module-build">;
 def ModuleImport : DiagGroup<"module-import">;
 def ModuleConflict : DiagGroup<"module-conflict">;
 def ModuleFileExtension : DiagGroup<"module-file-extension">;
+def IndexStore : DiagGroup<"index-store">;
 def NewlineEOF : DiagGroup<"newline-eof">;
 def Nullability : DiagGroup<"nullability">;
 def NullabilityDeclSpec : DiagGroup<"nullability-declspec">;
 def NullabilityInferredOnNestedType : DiagGroup<"nullability-inferred-on-nested-type">;
 def NullableToNonNullConversion : DiagGroup<"nullable-to-nonnull-conversion">;
 def NullabilityCompletenessOnArrays : DiagGroup<"nullability-completeness-on-arrays">;
 def NullabilityCompleteness : DiagGroup<"nullability-completeness",
                                         [NullabilityCompletenessOnArrays]>;

clang/include/clang/Driver/Job.h

 
 class Action;
 class InputInfo;
 class Tool;
 
 struct CrashReportInfo {
   StringRef Filename;
   StringRef VFSPath;
+  StringRef IndexStorePath;
 
-  CrashReportInfo(StringRef Filename, StringRef VFSPath)
-      : Filename(Filename), VFSPath(VFSPath) {}
+  CrashReportInfo(StringRef Filename, StringRef VFSPath,
+                  StringRef IndexStorePath)
+      : Filename(Filename), VFSPath(VFSPath), IndexStorePath(IndexStorePath) {}
 };
 
 /// Command - An executable path/name and argument vector to
 /// execute.
 class Command {
   /// Source - The action which caused the creation of this job.
   const Action &Source;
 

clang/include/clang/Driver/Options.td

 def objcmt_migrate_designated_init : Flag<["-"], "objcmt-migrate-designated-init">, Flags<[CC1Option]>,
   HelpText<"Enable migration to infer NS_DESIGNATED_INITIALIZER for initializer methods">;
 def objcmt_whitelist_dir_path: Joined<["-"], "objcmt-whitelist-dir-path=">, Flags<[CC1Option]>,
   HelpText<"Only modify files with a filename contained in the provided directory path">;
 // The misspelt "white-list" [sic] alias is due for removal.
 def : Joined<["-"], "objcmt-white-list-dir-path=">, Flags<[CC1Option]>,
     Alias<objcmt_whitelist_dir_path>;
 
+def index_store_path : Separate<["-"], "index-store-path">, Flags<[CC1Option]>,
+  HelpText<"Enable indexing with the specified data store path">;
+def index_ignore_system_symbols : Flag<["-"], "index-ignore-system-symbols">, Flags<[CC1Option]>,
+  HelpText<"Ignore symbols from system headers">;
+def index_record_codegen_name : Flag<["-"], "index-record-codegen-name">, Flags<[CC1Option]>,
+  HelpText<"Record the codegen name for symbols">;
+
 // Make sure all other -ccc- options are rejected.
 def ccc_ : Joined<["-"], "ccc-">, Group<internal_Group>, Flags<[Unsupported]>;
 
 // Standard Options
 
 def _HASH_HASH_HASH : Flag<["-"], "###">, Flags<[DriverOption, CoreOption]>,
     HelpText<"Print (but do not run) the commands to run for this compilation">;
 def _DASH_DASH : Option<["--"], "", KIND_REMAINING_ARGS>,

clang/include/clang/Frontend/CompilerInstance.h

class CompilerInstance : public ModuleLoader {
   /// If the output doesn't support seeking (terminal, pipe). we switch
   /// the stream to a buffer_ostream. These are the buffer and the original
   /// stream.
   std::unique_ptr<llvm::raw_fd_ostream> NonSeekStream;
 
   /// The list of active output files.
   std::list<OutputFile> OutputFiles;
 
+  /// \brief An optional callback function used to wrap all FrontendActions
+  /// produced to generate imported modules before they are executed.
+  std::function<std::unique_ptr<FrontendAction>(
+      const FrontendOptions &opts, std::unique_ptr<FrontendAction> action)>
+      GenModuleActionWrapper;
+
   /// Force an output buffer.
   std::unique_ptr<llvm::raw_pwrite_stream> OutputStream;
 
   CompilerInstance(const CompilerInstance &) = delete;
   void operator=(const CompilerInstance &) = delete;
 public:
   explicit CompilerInstance(
       std::shared_ptr<PCHContainerOperations> PCHContainerOps =
public:
   bool hadModuleLoaderFatalFailure() const {
     return ModuleLoader::HadFatalFailure;
   }
 
   GlobalModuleIndex *loadGlobalModuleIndex(SourceLocation TriggerLoc) override;
 
   bool lookupMissingImports(StringRef Name, SourceLocation TriggerLoc) override;
 
+  void setGenModuleActionWrapper(
+      std::function<std::unique_ptr<FrontendAction>(
+          const FrontendOptions &Opts, std::unique_ptr<FrontendAction> Action)>
+          Wrapper) {
+    GenModuleActionWrapper = Wrapper;
+  }
+
+  std::function<std::unique_ptr<FrontendAction>(
+      const FrontendOptions &Opts, std::unique_ptr<FrontendAction> Action)>
+  getGenModuleActionWrapper() const {
+    return GenModuleActionWrapper;
+  }
+
   void addDependencyCollector(std::shared_ptr<DependencyCollector> Listener) {
     DependencyCollectors.push_back(std::move(Listener));
   }
 
   void setExternalSemaSource(IntrusiveRefCntPtr<ExternalSemaSource> ESS);
 
   InMemoryModuleCache &getModuleCache() const { return *ModuleCache; }
 };
 
 } // end namespace clang
 
 #endif

clang/include/clang/Frontend/FrontendOptions.h

ObjCMT_MigrateAll = (ObjCMT_Literals | ObjCMT_Subscripting |
                          ObjCMT_MigrateDecls | ObjCMT_PropertyDotSyntax)
   };
   unsigned ObjCMTAction = ObjCMT_None;
   std::string ObjCMTWhiteListPath;
 
   std::string MTMigrateDir;
   std::string ARCMTMigrateReportOut;
 
+  std::string IndexStorePath;
+  unsigned IndexIgnoreSystemSymbols : 1;
+  unsigned IndexRecordCodegenName : 1;
+
   /// The input files and their types.
   std::vector<FrontendInputFile> Inputs;
 
   /// When the input is a module map, the original module map file from which
   /// that map was inferred, if any (for umbrella modules).
   std::string OriginalModuleMap;
 
   /// The output file, if any.
FrontendOptions()
       : DisableFree(false), RelocatablePCH(false), ShowHelp(false),
         ShowStats(false), ShowTimers(false), TimeTrace(false),
         ShowVersion(false), FixWhatYouCan(false), FixOnlyWarnings(false),
         FixAndRecompile(false), FixToTemporaries(false),
         ARCMTMigrateEmitARCErrors(false), SkipFunctionBodies(false),
         UseGlobalModuleIndex(true), GenerateGlobalModuleIndex(true),
         ASTDumpDecls(false), ASTDumpLookups(false),
         BuildingImplicitModule(false), ModulesEmbedAllFiles(false),
-        IncludeTimestamps(true) {}
+        IncludeTimestamps(true), IndexIgnoreSystemSymbols(false),
+        IndexRecordCodegenName(false) {}
 
   /// getInputKindForExtension - Return the appropriate input kind for a file
   /// extension. For example, "c" would return InputKind::C.
   ///
   /// \return The input kind for the extension, or InputKind::Unknown if the
   /// extension is not recognized.
   static InputKind getInputKindForExtension(StringRef Extension);
 };
 
 } // namespace clang
 
 #endif // LLVM_CLANG_FRONTEND_FRONTENDOPTIONS_H

clang/include/clang/Index/CodegenNameGenerator.h

 
 namespace index {
 
 class CodegenNameGenerator {
 public:
   explicit CodegenNameGenerator(ASTContext &Ctx);
   ~CodegenNameGenerator();
 
-  /// \returns true on failure to produce a name for the given decl, false on
-  /// success.
+  /// Writes name for \p D to \p OS.
+  /// \returns true on failure, false on success.
   bool writeName(const Decl *D, raw_ostream &OS);
 
-  /// Version of \c writeName function that returns a string.
+  /// \returns name for \p D
   std::string getName(const Decl *D);
 
-  /// This can return multiple mangled names when applicable, e.g. for C++
-  /// constructors/destructors.
+  /// \returns all applicable mangled names.
+  /// For example C++ constructors/destructors can have multiple.
   std::vector<std::string> getAllManglings(const Decl *D);
 
 private:
   struct Implementation;
-  std::unique_ptr<ASTNameGenerator> Impl;
+  Implementation *Impl;
 };
 
 } // namespace index
 } // namespace clang
 
-#endif // LLVM_CLANG_INDEX_CODEGENNAMEGENERATOR_H
+#endif

clang/include/clang/Index/DeclOccurrence.h

-//===- DeclOccurrence.h - An occurrence of a decl within a file -*- C++ -*-===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-
-#ifndef LLVM_CLANG_INDEX_DECLOCCURRENCE_H
-#define LLVM_CLANG_INDEX_DECLOCCURRENCE_H
-
-#include "clang/Basic/LLVM.h"
-#include "clang/Index/IndexSymbol.h"
-#include "llvm/ADT/ArrayRef.h"
-#include "llvm/ADT/SmallVector.h"
-
-namespace clang {
-class Decl;
-
-namespace index {
-
-struct DeclOccurrence {
-  SymbolRoleSet Roles;
-  unsigned Offset;
-  const Decl *Dcl;
-  SmallVector<SymbolRelation, 3> Relations;
-
-  DeclOccurrence(SymbolRoleSet R, unsigned Offset, const Decl *D,
-                 ArrayRef<SymbolRelation> Relations)
-      : Roles(R), Offset(Offset), Dcl(D),
-        Relations(Relations.begin(), Relations.end()) {}
-
-  friend bool operator<(const DeclOccurrence &LHS, const DeclOccurrence &RHS) {
-    return LHS.Offset < RHS.Offset;
-  }
-};
-
-} // namespace index
-} // namespace clang
-
-#endif // LLVM_CLANG_INDEX_DECLOCCURRENCE_H

clang/include/clang/Index/GenerateIndexAction.h

+//===--- GenerateIndexAction.h ----------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEX_INDEXINGACTION_H
+#define LLVM_CLANG_INDEX_INDEXINGACTION_H
+
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Index/IndexAction.h"
+#include "clang/Index/IndexDataConsumer.h"
+#include "clang/Index/IndexingContext.h"
+#include "llvm/ADT/StringRef.h"
+#include <memory>
+
+namespace clang {
+class ASTContext;
+class ASTReader;
+class ASTUnit;
+class CompilerInstance;
+class Decl;
+
+namespace serialization {
+class ModuleFile;
+}
+
+namespace index {
+class IndexDataConsumer;
+
+/// Recursively indexes all decls in the \p Unit and feeds them to \p
+/// DataConsumer.
+void indexASTUnit(ASTUnit &Unit, IndexDataConsumer &DataConsumer,
+                  const IndexingOptions &Opts);
+
+/// Recursively indexes \p Decls and feeds them to \p DataConsumer.
+void indexTopLevelDecls(ASTContext &Ctx, Preprocessor &PP,
+                        ArrayRef<const Decl *> Decls,
+                        IndexDataConsumer &DataConsumer,
+                        const IndexingOptions &Opts);
+
+/// Creates a PPCallbacks that indexes macros and feeds them to \p Consumer.
+/// The caller is responsible for calling `Consumer.setPreprocessor()`.
+std::unique_ptr<PPCallbacks> indexMacrosCallback(IndexDataConsumer &Consumer,
+                                                 const IndexingOptions &Opts);
+
+/// Recursively indexes all top-level decls in the \p Mod and feeds them to \p
+/// DataConsumer.
+void indexModuleFile(serialization::ModuleFile &Mod, ASTReader &Reader,
+                     IndexDataConsumer &DataConsumer,
+                     const IndexingOptions &Opts);
+
+/// Generates index data and uses IndexDataConsumer to consume it.
+/// Handles both index units and index records.
+class GenerateIndexAction : public IndexAction {
+  std::shared_ptr<IndexDataConsumer> DataConsumer;
+  std::shared_ptr<IndexingContext> IndexCtx;
+
+public:
+  GenerateIndexAction(std::shared_ptr<IndexDataConsumer> DataConsumer,
+                      const IndexingOptions &Opts);
+
+  std::unique_ptr<ASTConsumer> CreateASTConsumer(CompilerInstance &CI,
+                                                 StringRef InFile) override;
+
+  void EndSourceFileAction() override;
+};
+
+} // namespace index
+} // namespace clang
+
+#endif

clang/include/clang/Index/IndexAction.h

+//===--- IndexAction.h - Interface for index actions ------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEX_INDEXACTION_H
+#define LLVM_CLANG_INDEX_INDEXACTION_H
+
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Frontend/FrontendAction.h"
+#include "llvm/ADT/StringRef.h"
+#include <memory>
+
+namespace clang {
+namespace index {
+
+class IndexActionWrapper;
+
+/// This is primarily a hack - poking hole through FrontendAction encapsulation
+/// in order to access protected methods from IndexActionWrapper. FIXME?
+/// Auxiliary purpose is type distinction for two different actions in
+/// IndexActionWrapper in order to quarantee their methods are called in correct
+/// order.
+class IndexAction : public ASTFrontendAction {
+  friend class IndexActionWrapper;
+};
+
+/// Combines implementation of WrappedActionParam and IndexingActionParam.
+class IndexActionWrapper : public WrapperFrontendAction {
+  std::unique_ptr<IndexAction> IndexingAction;
+
+public:
+  IndexActionWrapper(std::unique_ptr<IndexAction> IndexingActionParam,
+                     std::unique_ptr<FrontendAction> WrappedActionParam);
+
+  /// \returns a consumer that first invokes consumer created by
+  /// WrappedActionParam and then consumer created by IndexingActionParam.
+  std::unique_ptr<ASTConsumer> CreateASTConsumer(CompilerInstance &CI,
+                                                 StringRef InFile) override;
+
+  /// Invokes implementation of EndSourceFileAction in WrappedActionParam first,
+  /// then implementation in IndexingActionParam.
+  void EndSourceFileAction() override;
+};
+
+} // namespace index
+} // namespace clang
+
+#endif

clang/include/clang/Index/IndexDataConsumer.h

 public:
   struct ASTNodeInfo {
     const Expr *OrigE;
     const Decl *OrigD;
     const Decl *Parent;
     const DeclContext *ContainerDC;
   };
 
-  virtual ~IndexDataConsumer() {}
+  virtual ~IndexDataConsumer() = default;
 
   virtual void initialize(ASTContext &Ctx) {}
 
   virtual void setPreprocessor(std::shared_ptr<Preprocessor> PP) {}
 
   /// \returns true to continue indexing, or false to abort.
   virtual bool handleDeclOccurence(const Decl *D, SymbolRoleSet Roles,
                                    ArrayRef<SymbolRelation> Relations,
-                                   SourceLocation Loc, ASTNodeInfo ASTNode);
+                                   SourceLocation Loc, ASTNodeInfo ASTNode) {
+    return true;
+  }
 
   /// \returns true to continue indexing, or false to abort.
   virtual bool handleMacroOccurence(const IdentifierInfo *Name,
                                     const MacroInfo *MI, SymbolRoleSet Roles,
-                                    SourceLocation Loc);
+                                    SourceLocation Loc) {
+    return true;
+  }
 
   /// \returns true to continue indexing, or false to abort.
   ///
   /// This will be called for each module reference in an import decl.
   /// For "@import MyMod.SubMod", there will be a call for 'MyMod' with the
   /// 'reference' role, and a call for 'SubMod' with the 'declaration' role.
   virtual bool handleModuleOccurence(const ImportDecl *ImportD,
-                                     const Module *Mod,
-                                     SymbolRoleSet Roles, SourceLocation Loc);
+                                     const Module *Mod, SymbolRoleSet Roles,
+                                     SourceLocation Loc) {
+    return true;
+  }
 
   virtual void finish() {}
 };
 
 } // namespace index
 } // namespace clang
 
 #endif

clang/include/clang/Index/IndexOptions.h

+//===--- IndexOptions.h - Options for indexing ------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEX_INDEXOPTIONS_H
+#define LLVM_CLANG_INDEX_INDEXOPTIONS_H
+
+#include "clang/Frontend/FrontendOptions.h"
+#include <memory>
+#include <string>
+
+namespace clang {
+namespace index {
+
+struct IndexingOptions {
+  enum class SystemSymbolFilterKind {
+    None,
+    DeclarationsOnly,
+    All,
+  };
+
+  SystemSymbolFilterKind SystemSymbolFilter =
+      SystemSymbolFilterKind::DeclarationsOnly;
+  bool IndexFunctionLocals = false;
+  bool IndexImplicitInstantiation = false;
+  // Whether to index macro definitions in the Preprocesor when preprocessor
+  // callback is not available (e.g. after parsing has finished). Note that
+  // macro references are not available in Proprocessor.
+  bool IndexMacrosInPreprocessor = false;
+  // Has no effect if IndexFunctionLocals are false.
+  bool IndexParametersInDeclarations = false;
+  bool IndexTemplateParameters = false;
+};
+
+IndexingOptions parseIndexGenerationOptions(const FrontendOptions &FEOpts);
+
+} // namespace index
+} // namespace clang
+
+#endif

clang/include/clang/Index/IndexPPCallbacks.h

+//===--- IndexPPCallbacks.h - Gathering index record data -----------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEX_INDEXPPCALLBACKS_H
+#define LLVM_CLANG_INDEX_INDEXPPCALLBACKS_H
+
+#include "clang/Index/IndexingContext.h"
+#include "clang/Lex/PPCallbacks.h"
+
+#include <memory>
+
+namespace llvm {
+class BitstreamWriter;
+}
+
+namespace clang {
+class FileEntry;
+class FileManager;
+
+namespace index {
+
+class IndexPPCallbacks : public PPCallbacks {
+  std::shared_ptr<IndexingContext> IndexCtx;
+
+public:
+  IndexPPCallbacks(std::shared_ptr<IndexingContext> IndexCtx)
+      : IndexCtx(IndexCtx) {}
+
+  void MacroExpands(const Token &MacroNameTok, const MacroDefinition &MD,
+                    SourceRange Range, const MacroArgs *Args) override {
+    IndexCtx->handleMacroReference(*MacroNameTok.getIdentifierInfo(),
+                                   Range.getBegin(), *MD.getMacroInfo());
+  }
+
+  void MacroDefined(const Token &MacroNameTok,
+                    const MacroDirective *MD) override {
+    IndexCtx->handleMacroDefined(*MacroNameTok.getIdentifierInfo(),
+                                 MacroNameTok.getLocation(),
+                                 *MD->getMacroInfo());
+  }
+
+  void MacroUndefined(const Token &MacroNameTok, const MacroDefinition &MD,
+                      const MacroDirective *Undef) override {
+    if (!MD.getMacroInfo()) // Ignore noop #undef.
+      return;
+    IndexCtx->handleMacroUndefined(*MacroNameTok.getIdentifierInfo(),
+                                   MacroNameTok.getLocation(),
+                                   *MD.getMacroInfo());
+  }
+};
+
+} // end namespace index
+} // end namespace clang
+
+#endif

clang/include/clang/Index/IndexWhileBuilding/DeclOccurrenceCollector.h

+//===--- DeclOccurrenceCollector.h - Collecting data for index records --*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEX_DECLOCCURENCECOLLECTOR_H
+#define LLVM_CLANG_INDEX_DECLOCCURENCECOLLECTOR_H
+
+#include "clang/Index/IndexDataConsumer.h"
+#include "clang/Index/IndexWhileBuilding/IndexRecordWriter.h"
+#include "clang/Index/IndexingContext.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/StringRef.h"
+#include <memory>
+
+namespace clang {
+namespace index {
+
+/// Collects occurences of declarations.
+class DeclOccurrenceCollector : public IndexDataConsumer {
+  SourceManager &SM;
+  // This exists because there's cyclic dependency between IndexingContext and
+  // DeclOccurrenceCollector.
+  IndexingContext *IndexCtx = nullptr;
+
+  llvm::DenseMap<FileID, IndexRecordWriter> RecordByFile;
+
+public:
+  explicit DeclOccurrenceCollector(SourceManager &SM) : SM(SM) {}
+  void setIndexingContext(IndexingContext &IC) { IndexCtx = &IC; }
+  bool handleDeclOccurence(const Decl *D, SymbolRoleSet Roles,
+                           ArrayRef<SymbolRelation> Relations,
+                           SourceLocation Loc, ASTNodeInfo ASTNode) override;
+  const llvm::DenseMap<FileID, IndexRecordWriter> &getRecordByFile() const {
+    return RecordByFile;
+  }
+};
+
+} // namespace index
+} // namespace clang
+
+#endif

clang/include/clang/Index/IndexWhileBuilding/EmitIndexAction.h

+//===--- EmitIndexAction.h - Index emitting action --------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEX_EMITINDEXACTION_H
+#define LLVM_CLANG_INDEX_EMITINDEXACTION_H
+
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Frontend/FrontendAction.h"
+#include "clang/Index/IndexAction.h"
+#include "clang/Index/IndexDataConsumer.h"
+#include "clang/Index/IndexWhileBuilding/DeclOccurrenceCollector.h"
+#include "clang/Index/IndexWhileBuilding/IndexRecordWriter.h"
+#include "clang/Index/IndexWhileBuilding/IndexUnitWriter.h"
+#include "clang/Index/IndexingContext.h"
+#include "llvm/ADT/StringRef.h"
+#include <memory>
+
+namespace clang {
+namespace index {
+
+class SourceFilesDependencyCollector : public DependencyCollector {
+  const CompilerInstance &CI;
+  IndexingContext &IndexCtx;
+  EmitIndexOptions RecordOpts;
+
+public:
+  IndexUnitWriter UnitWriter;
+
+  SourceFilesDependencyCollector(const CompilerInstance &CI,
+                                 IndexingContext &indexCtx,
+                                 EmitIndexOptions recordOpts);
+
+  virtual void attachToPreprocessor(Preprocessor &PP) override;
+
+private:
+  bool sawDependency(StringRef Filename, bool FromModule, bool IsSystem,
+                     bool IsModuleFile, bool IsMissing) override;
+  bool isInSysroot(StringRef Filename);
+};
+
+// FIXME:
+// This isn't nice - just a glue.
+// The reasons for this to exist are:
+// - We need access to CompilerInstance to create collectors which we get only
+// in CreateASTConsumer method, not in FrontendAction implementation's
+// constructor.
+// - createIndexRecordingASTConsumer returns unique_ptr but we need to preserve
+// some data until EndSourceFileAction.
+struct ActionContext {
+  const CompilerInstance &CI;
+  DeclOccurrenceCollector DeclsDataCollector;
+  IndexingContext Ctx;
+  SourceFilesDependencyCollector DepCollector;
+  EmitIndexOptions RecordOpts;
+  IndexingOptions IndexOpts;
+
+  ActionContext(const IndexingOptions &IndexOpts, const CompilerInstance &CI,
+                const EmitIndexOptions &RecordOpts);
+
+  void emitCollectedData();
+};
+
+/// Generates index data and serializes them to disk.
+/// Handles both index units and index records.
+class EmitIndexAction : public IndexAction {
+  EmitIndexOptions RecordOpts;
+  IndexingOptions IndexOpts;
+  std::shared_ptr<ActionContext> Context = nullptr;
+
+public:
+  EmitIndexAction(const IndexingOptions &IndexOpts,
+                  const EmitIndexOptions &RecordOpts);
+
+  std::unique_ptr<ASTConsumer> CreateASTConsumer(CompilerInstance &CI,
+                                                 StringRef) override;
+
+  void EndSourceFileAction() override;
+};
+
+/// Checks if the unit file exists for \p Mod and emits index data in case it
+/// doesn't. Handles both index units and index records. \returns true if the
+/// index data were generated, false otherwise.
+bool emitIndexDataForModule(const Module *Mod, const CompilerInstance &CI);
+
+/// Emits index data for \p Mod.
+/// Handles both index units and index records.
+void emitIndexDataForModuleFile(serialization::ModuleFile &Mod,
+                                const CompilerInstance &CI,
+                                const IndexingOptions &IndexOpts,
+                                const EmitIndexOptions &RecordOpts);
+
+} // namespace index
+} // namespace clang
+
+#endif

clang/include/clang/Index/IndexWhileBuilding/EmitIndexOptions.h

+//===--- IndexOptions.h - Options for indexing ------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEX_EMITINDEXOPTIONS_H
+#define LLVM_CLANG_INDEX_EMITINDEXOPTIONS_H
+
+#include "clang/Frontend/FrontendOptions.h"
+#include <memory>
+#include <string>
+
+namespace clang {
+namespace index {
+
+struct EmitIndexOptions {
+  enum class IncludesRecordingKind {
+    None,
+    UserOnly, // only record includes inside non-system files.
+    All,
+  };
+
+  const std::string RootDir;
+  // Caching the full paths.
+  const std::string IndexDir;
+  const std::string UnitDir;
+  const std::string RecordDir;
+
+  const bool RecordSymbolCodeGenName;
+  const bool RecordSystemDependencies;
+  const IncludesRecordingKind RecordIncludes;
+
+  EmitIndexOptions(const StringRef RootDir,
+                   const bool RecordSymbolCodeGenName = false,
+                   const bool RecordSystemDependencies = true,
+                   const IncludesRecordingKind RecordIncludes =
+                       IncludesRecordingKind::UserOnly);
+};
+
+EmitIndexOptions parseEmitIndexOptions(const FrontendOptions &FEOpts);
+
+} // namespace index
+} // namespace clang
+
+#endif

clang/include/clang/Index/IndexWhileBuilding/IndexDataFormat.h

+//===--- IndexDataFormat.h - Layout of index on a filesystem --------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_LIB_INDEX_INDEXDATAFORMAT_H
+#define LLVM_CLANG_LIB_INDEX_INDEXDATAFORMAT_H
+
+#include "llvm/ADT/SmallString.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Bitstream/BitCodes.h"
+
+namespace clang {
+namespace index {
+namespace store {
+
+/// Index is organized like this:
+/// (root of index dir)
+///   /<IndexDir>
+///     /<IndexRecordSubdir>
+///       /AA
+///         /(records)
+///       /AB
+///         /(records)
+///       /AC
+///         /(records)
+///       ...
+///     /<IndexUnitSubdir>
+///       /(units)
+
+/// \returns version of index format
+unsigned getIndexFormatVersion();
+
+/// \returns name of directory where index data lives inside the root directory.
+const char *getIndexDir();
+
+/// \returns name of directory where unit data lives inside the index directory.
+const char *getIndexUnitSubdir();
+
+/// \returns name of directory where record data lives inside the index
+/// directory.
+const char *getIndexRecordSubdir();
+
+/// \returns name of index record related to \p SourceFilename.
+/// Index record filename is the original source filename with hash of the index
+/// data as a suffix.
+std::string
+generateFilenameForIndexRecord(llvm::StringRef SourceFilename,
+                               const llvm::SmallString<512> &IndexRecordBuffer);
+
+/// Appends name of index unit related to \p OutputFilePath to \p Str.
+/// \p OutputFilePath is absolute path to output file.
+void getUnitNameForOutputFile(llvm::StringRef OutputFilePath,
+                              llvm::SmallVectorImpl<char> &Str);
+
+/// Appends index directory and unit subdirectory to \p Buffer without erasing
+/// it.
+void appendUnitSubDir(llvm::SmallVectorImpl<char> &Buffer);
+
+/// Appends index directory and record subdirectory to \p Buffer without erasing
+/// it.
+void appendRecordSubDir(llvm::SmallVectorImpl<char> &Buffer);
+
+/// This is currently a placeholder - just appends \p UnitName to \p Buffer.
+void appendUnitBucketDir(llvm::StringRef UnitName,
+                         llvm::SmallVectorImpl<char> &Buffer);
+
+/// We're bucketing record files according to the last two characters in their
+/// names in order to keep number of files in a single directory reasonable.
+/// Appends bucket subdirectory and delimiter.
+void appendRecordBucketDir(llvm::StringRef RecordName,
+                           llvm::SmallVectorImpl<char> &Buffer);
+
+/// Appends \p RecordName to \p Buffer.
+void appendRecordFilename(llvm::StringRef RecordName,
+                          llvm::SmallVectorImpl<char> &Buffer);
+
+enum RecordBitRecord {
+  REC_VERSION = 0,
+  REC_DECLINFO = 1,
+  REC_DECLOFFSETS = 2,
+  REC_DECLOCCURRENCE = 3,
+};
+
+enum RecordBitBlock {
+  REC_VERSION_BLOCK_ID = llvm::bitc::FIRST_APPLICATION_BLOCKID,
+  REC_DECLS_BLOCK_ID,
+  REC_DECLOFFSETS_BLOCK_ID,
+  REC_DECLOCCURRENCES_BLOCK_ID,
+};
+
+enum UnitBitRecord {
+  UNIT_VERSION = 0,
+  UNIT_INFO = 1,
+  UNIT_DEPENDENCY = 2,
+  UNIT_INCLUDE = 3,
+  UNIT_PATH = 4,
+  UNIT_PATH_BUFFER = 5,
+  UNIT_MODULE = 6,
+  UNIT_MODULE_BUFFER = 7,
+};
+
+enum UnitBitBlock {
+  UNIT_VERSION_BLOCK_ID = llvm::bitc::FIRST_APPLICATION_BLOCKID,
+  UNIT_INFO_BLOCK_ID,
+  UNIT_DEPENDENCIES_BLOCK_ID,
+  UNIT_INCLUDES_BLOCK_ID,
+  UNIT_PATHS_BLOCK_ID,
+  UNIT_MODULES_BLOCK_ID,
+};
+
+enum UnitDependencyKind {
+  UNIT_DEPEND_KIND_FILE = 0,
+  UNIT_DEPEND_KIND_RECORD = 1,
+  UNIT_DEPEND_KIND_UNIT = 2,
+};
+static const unsigned UnitDependencyKindBitNum = 2;
+static const unsigned UnitFilePathPrefixKindBitNum = 2;
+
+} // end namespace store
+} // end namespace index
+} // end namespace clang
+
+#endif

clang/include/clang/Index/IndexWhileBuilding/IndexRecordReader.h

+//===--- IndexRecordReader.h - Index record deserialization ---------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEX_INDEXRECORDREADER_H
+#define LLVM_CLANG_INDEX_INDEXRECORDREADER_H
+
+#include "clang/Index/IndexSymbol.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Bitstream/BitstreamReader.h"
+#include <memory>
+
+namespace llvm {
+class MemoryBuffer;
+}
+
+namespace clang {
+namespace index {
+
+struct IndexRecordDecl {
+  unsigned DeclID;
+  SymbolInfo SymInfo;
+  SymbolRoleSet Roles;
+  SymbolRoleSet RelatedRoles;
+  StringRef Name;
+  StringRef USR;
+  StringRef CodeGenName;
+};
+
+struct IndexRecordRelation {
+  SymbolRoleSet Roles;
+  const IndexRecordDecl *Dcl = nullptr;
+
+  IndexRecordRelation() = default;
+  IndexRecordRelation(SymbolRoleSet Roles, const IndexRecordDecl *Dcl)
+      : Roles(Roles), Dcl(Dcl) {}
+};
+
+struct IndexRecordOccurrence {
+  const IndexRecordDecl *Dcl;
+  SmallVector<IndexRecordRelation, 4> Relations;
+  SymbolRoleSet Roles;
+  unsigned Line;
+  unsigned Column;
+};
+
+/// Interface for reading index record data backed by a memory buffer.
+class IndexRecordReader {
+public:
+  static std::unique_ptr<IndexRecordReader>
+  createWithRecordFilename(StringRef RecordFilename, StringRef StorePath,
+                           std::string &Error);
+  static std::unique_ptr<IndexRecordReader>
+  createWithFilePath(StringRef FilePath, std::string &Error);
+  static std::unique_ptr<IndexRecordReader>
+  createWithBuffer(std::unique_ptr<llvm::MemoryBuffer> Buffer,
+                   std::string &Error);
+
+  struct DeclSearchReturn {
+    bool AcceptDecl;
+    bool ContinueSearch;
+  };
+  typedef DeclSearchReturn(DeclSearchCheck)(const IndexRecordDecl &);
+  /// Iterates over declarations and calls first \p Checker and conditionally
+  /// also \p Receiver for each of them. Receiver is called for those
+  /// declarations for which \p Checker returns AcceptDecl == true. Iteration is
+  /// terminated early if \p Checker returns ContinueSearch == false. Resulting
+  /// decls can be used as filter for \c foreachOccurrence. This allows
+  /// allocating memory only for the record decls that the caller is interested
+  /// in.
+  bool searchDecls(llvm::function_ref<DeclSearchCheck> Checker,
+                   llvm::function_ref<void(const IndexRecordDecl *)> Receiver);
+
+  /// Iterates over declarations and calls \p Receiver for each of them.
+  /// Avoids caching the declaration data if \p NoCache is true.
+  bool foreachDecl(bool NoCache,
+                   llvm::function_ref<bool(const IndexRecordDecl *)> Receiver);
+
+  /// Iterates over occurences of declarations and conditionally calls \p
+  /// Receiver for occurences of declarations. If \p WhitelistDecls is non-empty
+  /// \p Receiver is called only for occurence of declarations that it contains.
+  /// If \p WhitelistRelated is non-empty \p Receiver is called only for
+  /// occurences that are related to any of the declarations it contains. In
+  /// case \p WhitelistDecls is non-empty and a particular declaration isn't
+  /// present, it is skipped and isn't further inspected in regards to \p
+  /// WhitelistRelated.
+  bool foreachOccurrence(
+      llvm::function_ref<bool(const IndexRecordOccurrence &)> Receiver,
+      ArrayRef<const IndexRecordDecl *> WhitelistDecls = None,
+      ArrayRef<const IndexRecordDecl *> WhitelistRelated = None);
+
+  /// Iterates over occurences of declarations and conditionally calls \p
+  /// Receiver for occurences of declarations. \p Receiver will be called only
+  /// for occurences between \p lineStart and \p lineStart + \p lineCount (both
+  /// ends are included).
+  bool foreachOccurrenceInLineRange(
+      unsigned lineStart, unsigned lineCount,
+      llvm::function_ref<bool(const IndexRecordOccurrence &)> Receiver);
+
+  // TODO order
+private:
+  friend class IndexBitstreamVisitor;
+  llvm::BumpPtrAllocator Allocator;
+  std::unique_ptr<llvm::MemoryBuffer> Buffer;
+  llvm::BitstreamCursor DeclCursor;
+  llvm::BitstreamCursor OccurCursor;
+  llvm::ArrayRef<uint32_t> DeclOffsets;
+  const IndexRecordDecl **Decls;
+
+  void setDeclOffsets(ArrayRef<uint32_t> Offs);
+
+  unsigned getNumDecls() const;
+
+  const IndexRecordDecl *getDeclByID(unsigned DeclID);
+
+  const IndexRecordDecl *getDecl(unsigned Index);
+
+  /// Reads occurrence data.
+  /// \param WhitelistDecls if non-empty indicates the list of decls that we
+  /// want to get occurrences for. If empty then indicates that we want
+  /// occurrences for all decls. \param WhitelistRelated Same as \c
+  /// WhitelistDecls but for related decls. \returns true if the occurrence info
+  /// was filled out, false if occurrence was ignored.
+  bool readOccurrence(SmallVectorImpl<uint64_t> &Record, StringRef Blob,
+                      ArrayRef<const IndexRecordDecl *> WhitelistDecls,
+                      ArrayRef<const IndexRecordDecl *> WhitelistRelated,
+                      IndexRecordOccurrence &RecOccur);
+
+  void readDecl(unsigned Index, IndexRecordDecl &RecD);
+};
+
+} // namespace index
+} // namespace clang
+
+#endif

clang/include/clang/Index/IndexWhileBuilding/IndexRecordWriter.h

+//===--- IndexRecordWriter.h - Index record serialization-------- ---------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEX_INDEXRECORDWRITER_H
+#define LLVM_CLANG_INDEX_INDEXRECORDWRITER_H
+
+#include "clang/AST/DeclBase.h"
+#include "clang/Basic/SourceLocation.h"
+#include "clang/Index/CodegenNameGenerator.h"
+#include "clang/Index/IndexSymbol.h"
+#include "clang/Index/IndexWhileBuilding/EmitIndexOptions.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/Bitstream/BitstreamWriter.h"
+#include <map>
+#include <vector>
+
+namespace clang {
+namespace index {
+
+struct DeclInfo {
+  const Decl *D;
+  SymbolRoleSet Roles;
+  SymbolRoleSet RelatedRoles;
+};
+
+/// Collects declaration occurences in a header or source file and serializes
+/// them.
+class IndexRecordWriter {
+private:
+  const bool IsSystem;
+
+  llvm::DenseMap<const Decl *, unsigned> IndexForDecl;
+  std::vector<DeclInfo> Decls;
+  std::multimap<unsigned, SmallVector<uint64_t, 10>> Occurrences;
+
+public:
+  IndexRecordWriter(bool IsSystem) : IsSystem(IsSystem) {}
+
+  bool isSystem() const { return IsSystem; }
+
+  /// Adds an occurrence of the canonical declaration \c D at the supplied
+  /// \c Offset
+  ///
+  /// \param Roles the roles the occurrence fulfills in this position.
+  /// \param Offset the offset in the file of this occurrence.
+  /// \param D the canonical declaration this is an occurrence of.
+  /// \param Relations the set of symbols related to this occurrence.
+  void addDeclOccurence(SymbolRoleSet Roles, FileID FID, unsigned Offset,
+                        const Decl *D, ArrayRef<SymbolRelation> Relations);
+
+  bool isRecordEmpty() const { return Decls.empty(); }
+
+  /// \returns filename of the index record created or None in case of failure.
+  llvm::Optional<std::string>
+  writeToDisk(StringRef SourceFileName, const EmitIndexOptions &RecOpts,
+              const PrintingPolicy &Policy,
+              std::shared_ptr<CodegenNameGenerator> CGNameGen,
+              DiagnosticsEngine &Diag) const;
+
+private:
+  void getBitcodeRepresentation(
+      llvm::SmallString<512> &Buffer, const clang::PrintingPolicy &Policy,
+      std::shared_ptr<CodegenNameGenerator> CGNameGen) const;
+};
+
+} // end namespace index
+} // end namespace clang
+
+#endif

clang/include/clang/Index/IndexWhileBuilding/IndexUnitReader.h

+//===--- IndexUnitReader.h - Index unit deserialization -------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEX_INDEXUNITREADER_H
+#define LLVM_CLANG_INDEX_INDEXUNITREADER_H
+
+#include "clang/Basic/LLVM.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Bitstream/BitstreamReader.h"
+#include "llvm/Support/Chrono.h"
+#include "llvm/Support/MemoryBuffer.h"
+
+#include "clang/Index/IndexWhileBuilding/PathStorage.h"
+#include <memory>
+#include <vector>
+
+namespace clang {
+namespace index {
+
+struct ModuleInfo {
+  unsigned NameOffset;
+  unsigned NameSize;
+};
+
+/// Interface for reading index unit data backed by a memory buffer.
+class IndexUnitReader {
+public:
+  enum class DependencyKind {
+    Unit,
+    Record,
+    File,
+  };
+
+  IndexUnitReader(std::unique_ptr<llvm::MemoryBuffer> Buf,
+                  llvm::BitstreamCursor &&DependenciesBlockCursor,
+                  llvm::BitstreamCursor &&IncludeCursor,
+                  StringRef ProviderIdentifier, // < reference to the Buf data
+                  StringRef ProviderVersion,    // < reference to the Buf data
+                  bool IsSystemUnit, bool IsModuleUnit, bool IsDebugCompilation,
+                  StringRef ModuleName, // < reference to the Buf data
+                  StringRef Target,     // < reference to the Buf data
+                  int MainPathIndex, std::vector<FileBitPath> &&Paths,
+                  StringRef PathsBuffer,
+                  // Offset-size pairs to the PathsBuffer.
+                  size_t WorkDirOffset, size_t WorkDirSize,
+                  size_t OutputFileOffset, size_t OutputFileSize,
+                  size_t SysrootOffset, size_t SysrootSize,
+                  std::vector<ModuleInfo> &&Modules,
+                  StringRef ModuleNamesBuffer, // < reference to the Buf data
+                  llvm::sys::TimePoint<> ModTime);
+
+  /// Creates IndexUnitReader from \p Buf.
+  /// \returns nullptr in case of any failure.
+  static std::unique_ptr<IndexUnitReader>
+  create(std::unique_ptr<llvm::MemoryBuffer> Buf,
+         llvm::sys::TimePoint<> ModTime, std::string &Error);
+
+  /// Creates IndexUnitReader from a file.
+  /// File location is derived from \p IndexUnitFilename and \p IndexRootDir.
+  static std::unique_ptr<IndexUnitReader>
+  createWithUnitFilename(StringRef IndexUnitFilename, StringRef IndexRootDir,
+                         std::string &Error);
+
+  /// Creates IndexUnitReader from a file at \p IndexUnitPath.
+  static std::unique_ptr<IndexUnitReader>
+  createWithFilePath(StringRef IndexUnitPath, std::string &Error);
+
+  /// Get Modification time of index unit file by getting it's status on
+  /// filesystem. File location is derived from \p IndexUnitFilename and \p
+  /// IndexRootDir. \returns nullptr in case of failure to get status of the
+  /// file.
+  static Optional<llvm::sys::TimePoint<>>
+  getModificationTimeForUnit(StringRef IndexUnitFilename,
+                             StringRef IndexRootDir, std::string &Error);
+
+  StringRef getProviderIdentifier() const;
+  StringRef getProviderVersion() const;
+
+  llvm::sys::TimePoint<> getModificationTime() const;
+  StringRef getWorkingDirectory() const;
+  StringRef getOutputFile() const;
+  StringRef getSysrootPath() const;
+  StringRef getMainFilePath() const;
+  StringRef getModuleName() const;
+  StringRef getTarget() const;
+  bool hasMainFile() const;
+  bool isSystemUnit() const;
+  bool isModuleUnit() const;
+  bool isDebugCompilation() const;
+
+  struct DependencyInfo {
+    DependencyKind Kind;
+    bool IsSystem;
+    StringRef UnitOrRecordName;
+    StringRef FilePath;
+    StringRef ModuleName;
+  };
+  /// Iterates over dependencies of this index unit and calls \p Receiver for
+  /// each of them. Order of dependencies is: units, records, files.
+  bool foreachDependency(
+      llvm::function_ref<bool(const DependencyInfo &Info)> Receiver);
+
+  struct IncludeInfo {
+    StringRef SourcePath;
+    unsigned SourceLine;
+    StringRef TargetPath;
+  };
+  /// Iterates over files included into this index unit and calls \p Receiver
+  /// for each of them.
+  bool
+  foreachInclude(llvm::function_ref<bool(const IncludeInfo &Info)> Receiver);
+
+private:
+  const std::unique_ptr<llvm::MemoryBuffer> Buf;
+
+  llvm::BitstreamCursor DependenciesBlockCursor;
+  llvm::BitstreamCursor IncludedFilesCursor;
+  StringRef ProviderIdentifier;
+  StringRef ProviderVersion;
+  const bool IsSystemUnit;
+  const bool IsModuleUnit;
+  const bool IsDebugCompilation;
+  StringRef ModuleName;
+  StringRef Target;
+
+  SmallString<128> MainFilePath;
+  const std::vector<FileBitPath> Paths;
+
+  StringRef PathsBuffer;
+  // Offset-size pairs to the PathsBuffer.
+  const size_t WorkDirOffset;
+  const size_t WorkDirSize;
+  const size_t OutputFileOffset;
+  const size_t OutputFileSize;
+  const size_t SysrootOffset;
+  const size_t SysrootSize;
+
+  const std::vector<ModuleInfo> Modules;
+  StringRef ModuleNamesBuffer;
+  const llvm::sys::TimePoint<> ModTime;
+
+private:
+  /// Appends the full path related to \p PathIndex to \p PathBuf.
+  void constructFilePath(SmallVectorImpl<char> &PathBuf, int PathIndex) const;
+};
+
+} // namespace index
+} // namespace clang
+
+#endif

clang/include/clang/Index/IndexWhileBuilding/IndexUnitWriter.h

+//===--- IndexUnitWriter.h - Index unit serialization ---------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEX_INDEXUNITWRITER_H
+#define LLVM_CLANG_INDEX_INDEXUNITWRITER_H
+
+#include "clang/Basic/LLVM.h"
+#include "clang/Index/IndexWhileBuilding/EmitIndexOptions.h"
+#include "clang/Index/IndexWhileBuilding/PathStorage.h"
+#include "llvm/ADT/DenseSet.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/ADT/StringMap.h"
+#include "llvm/Support/Allocator.h"
+#include <set>
+#include <string>
+#include <vector>
+
+namespace llvm {
+class BitstreamWriter;
+}
+
+namespace clang {
+class CompilerInstance;
+class FileEntry;
+class FileManager;
+class Module;
+
+namespace index {
+
+namespace writer {
+/// An opaque pointer to a module used by the IndexUnitWriter to associate
+/// record and file dependencies with a module, and as a token for getting
+/// information about the module from the caller.
+typedef const void *OpaqueModule;
+
+struct ModuleInfo {
+  /// Full dot-separated module name.
+  StringRef Name;
+};
+
+typedef llvm::function_ref<ModuleInfo(OpaqueModule,
+                                      SmallVectorImpl<char> &Scratch)>
+    ModuleInfoWriterCallback;
+} // end namespace writer
+
+struct ModuleIndexData {
+  const std::string Name;
+  const bool IsSystem;
+};
+
+class IndexUnitWriter {
+private:
+  const EmitIndexOptions RecOpts;
+
+  // index unit metadata
+  const std::string ProviderIdentifier;
+  const std::string ProviderVersion;
+  const std::string OutputFile;
+  const std::string AbsPathToOutputFile;
+  const FileEntry *MainFile;
+
+  // Absence of value means we're not indexing a module.
+  const llvm::Optional<ModuleIndexData> ModuleData;
+
+  const bool IsDebugCompilation;
+  const std::string TargetTriple;
+  const std::string WorkDir;
+  // FIXME
+  // IIUC we're serializing this as a source of truth of what is considered a
+  // system part of directory tree but we should probably be using values of all
+  // of -isysroot, -isystem and -F options.
+  const std::string SysrootPath;
+
+  mutable PathStorage PathStore;
+  // Module dependencies
+  SmallString<512> ModuleNamesBuf;
+  std::vector<std::array<uint64_t, 3>> Modules;
+  // Maps module to one-based index in the Modules container.
+  llvm::DenseMap<writer::OpaqueModule, unsigned> ModToIdx;
+
+  // Source file dependencies - mapping from UID to data.
+  llvm::DenseMap<unsigned, std::array<uint64_t, 7>> SourceFiles;
+  // Source files that has a corresponding IndexRecord aren't written into
+  // index.
+  llvm::DenseSet<unsigned> SourceFileHasARecord;
+  std::vector<std::array<uint64_t, 4>> IncludedFiles;
+
+  std::vector<std::pair<SmallString<64>, std::array<uint64_t, 7>>>
+      IndexUnitDependencies;
+  llvm::DenseSet<unsigned> ASTFilesOfSeenIndexUnits;
+
+  std::vector<std::pair<std::string, std::array<uint64_t, 7>>>
+      IndexRecordDependencies;
+
+  std::function<writer::ModuleInfo(writer::OpaqueModule,
+                                   SmallVectorImpl<char> &Scratch)>
+      GetInfoForModuleFn;
+
+public:
+  /// Creates IndexUnitWriter for index unit represented either by \p MainFile
+  /// or \p ModuleData. \p MainFile the main file for a compiled source file.
+  /// This should be null for PCH and module units.
+  IndexUnitWriter(FileManager &FileMgr, const EmitIndexOptions &RecOpts,
+                  StringRef ProviderIdentifier, StringRef ProviderVersion,
+                  StringRef OutputFile, StringRef AbsPathToOutputFile,
+                  const FileEntry *MainFile,
+                  const llvm::Optional<ModuleIndexData> &ModuleData,
+                  bool IsDebugCompilation, StringRef TargetTriple,
+                  StringRef WorkDir, StringRef SysrootPath,
+                  writer::ModuleInfoWriterCallback GetInfoForModule);
+
+  unsigned addModule(writer::OpaqueModule Mod);
+
+  /// Stores dependency of this index unit on source file \p File.
+  /// If \p Mod != nullptr it means \p File is compiled as part of module \p
+  /// Mod.
+  void addSourceFile(const FileEntry *File, bool IsSystem,
+                     writer::OpaqueModule Mod);
+
+  /// Stores dependency of this index unit on index record.
+  /// Index record \p RecordFile contains index data of \p SourceFile.
+  /// If \p Mod != nullptr it means \p SourceFile is compiled as part of module
+  /// \p Mod.
+  void addIndexRecord(StringRef RecordFile, const FileEntry *SourceFile,
+                      bool IsSystem, writer::OpaqueModule Mod);
+
+  /// Stores dependency of this index unit on another index unit - \p File.
+  /// If \p Mod != nullptr it means \p File contains index data of module \p
+  /// Mod.
+  void addIndexUnit(const FileEntry *File, StringRef AbsolutePath,
+                    bool IsSystem, writer::OpaqueModule Mod,
+                    bool withoutUnitName = false);
+
+  /// Stores dependency of this index unit on \p IncludedFile.
+  void addIncludedFile(const FileEntry *IncludingFile, unsigned Line,
+                       const FileEntry *IncludedFile);
+
+  bool writeToDisk(std::string &Error) const;
+
+private:
+  void writeUnitInfo(llvm::BitstreamWriter &Stream) const;
+  void writeDependencies(llvm::BitstreamWriter &Stream) const;
+  void writeIncludes(llvm::BitstreamWriter &Stream) const;
+  void writePaths(llvm::BitstreamWriter &Stream) const;
+  void writeModules(llvm::BitstreamWriter &Stream) const;
+
+  void getBitcodeRepresentation(llvm::SmallString<512> &Buffer,
+                                std::string &Error) const;
+};
+
+IndexUnitWriter makeIndexUnitWriter(const CompilerInstance &CI,
+                                    const EmitIndexOptions &RecordOpts,
+                                    StringRef OutputFile,
+                                    const FileEntry *MainFile,
+                                    Module *UnitModule);
+
+bool initIndexDirectory(const EmitIndexOptions &RecordingOpts,
+                        std::string &Error);
+
+// \returns true in case index unit for \p ModuleFilePath exists, false in case
+// it doesn't and llvm::None in case it couldn't be stat()-ed.
+llvm::Optional<bool>
+doesIndexUnitForModuleExist(StringRef ModuleFilePath,
+                            const FileManager &FileMgr,
+                            const EmitIndexOptions &RecOpts);
+
+} // end namespace index
+} // end namespace clang
+
+#endif

clang/include/clang/Index/IndexWhileBuilding/PathStorage.h

+//===--- PathStorage.h -- Index of paths ------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEX_PATHSTORAGE_H
+#define LLVM_CLANG_INDEX_PATHSTORAGE_H
+
+#include "clang/Basic/FileManager.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/ADT/StringMap.h"
+#include "llvm/ADT/StringRef.h"
+
+#include <string>
+#include <vector>
+
+enum UnitFilePathPrefixKind {
+  UNIT_PATH_PREFIX_NONE = 0,
+  UNIT_PATH_PREFIX_WORKDIR = 1,
+  UNIT_PATH_PREFIX_SYSROOT = 2,
+};
+
+struct BitPathComponent {
+  size_t Offset = 0;
+  size_t Size = 0;
+  BitPathComponent(size_t Offset, size_t Size) : Offset(Offset), Size(Size) {}
+  BitPathComponent() = default;
+};
+
+struct DirBitPath {
+  UnitFilePathPrefixKind PrefixKind;
+  BitPathComponent Dir;
+
+  DirBitPath(UnitFilePathPrefixKind Kind, BitPathComponent Dir);
+  DirBitPath();
+};
+
+struct FileBitPath : DirBitPath {
+  BitPathComponent Filename;
+  FileBitPath(UnitFilePathPrefixKind Kind, BitPathComponent Dir,
+              BitPathComponent Filename)
+      : DirBitPath(Kind, Dir), Filename(Filename) {}
+  FileBitPath() = default;
+};
+
+class PathStorage {
+  std::string WorkDir;
+  std::string SysrootPath;
+  llvm::SmallString<512> PathsBuf;
+  llvm::StringMap<DirBitPath, llvm::BumpPtrAllocator> Dirs;
+  std::vector<FileBitPath> FileBitPaths;
+  llvm::DenseMap<const clang::FileEntry *, size_t> FileToIndex;
+
+public:
+  PathStorage(llvm::StringRef workDir, llvm::StringRef sysrootPath);
+
+  llvm::StringRef getPathsBuffer() const { return PathsBuf.str(); }
+
+  llvm::ArrayRef<FileBitPath> getBitPaths() const { return FileBitPaths; }
+
+  int getPathIndex(const clang::FileEntry *FE);
+
+  size_t getPathOffset(llvm::StringRef Path);
+
+private:
+  DirBitPath getDirBitPath(llvm::StringRef dirStr);
+};
+
+#endif
 No newline at end of file

clang/include/clang/Index/IndexingAction.h

-//===--- IndexingAction.h - Frontend index action ---------------*- C++ -*-===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-
-#ifndef LLVM_CLANG_INDEX_INDEXINGACTION_H
-#define LLVM_CLANG_INDEX_INDEXINGACTION_H
-
-#include "clang/Basic/LLVM.h"
-#include "clang/Lex/PPCallbacks.h"
-#include "clang/Lex/Preprocessor.h"
-#include "llvm/ADT/ArrayRef.h"
-#include <memory>
-
-namespace clang {
-  class ASTContext;
-  class ASTReader;
-  class ASTUnit;
-  class Decl;
-  class FrontendAction;
-
-namespace serialization {
-  class ModuleFile;
-}
-
-namespace index {
-  class IndexDataConsumer;
-
-struct IndexingOptions {
-  enum class SystemSymbolFilterKind {
-    None,
-    DeclarationsOnly,
-    All,
-  };
-
-  SystemSymbolFilterKind SystemSymbolFilter
-    = SystemSymbolFilterKind::DeclarationsOnly;
-  bool IndexFunctionLocals = false;
-  bool IndexImplicitInstantiation = false;
-  // Whether to index macro definitions in the Preprocesor when preprocessor
-  // callback is not available (e.g. after parsing has finished). Note that
-  // macro references are not available in Proprocessor.
-  bool IndexMacrosInPreprocessor = false;
-  // Has no effect if IndexFunctionLocals are false.
-  bool IndexParametersInDeclarations = false;
-  bool IndexTemplateParameters = false;
-};
-
-/// Creates a frontend action that indexes all symbols (macros and AST decls).
-/// \param WrappedAction another frontend action to wrap over or null.
-std::unique_ptr<FrontendAction>
-createIndexingAction(std::shared_ptr<IndexDataConsumer> DataConsumer,
-                     IndexingOptions Opts,
-                     std::unique_ptr<FrontendAction> WrappedAction);
-
-/// Recursively indexes all decls in the AST.
-void indexASTUnit(ASTUnit &Unit, IndexDataConsumer &DataConsumer,
-                  IndexingOptions Opts);
-
-/// Recursively indexes \p Decls.
-void indexTopLevelDecls(ASTContext &Ctx, Preprocessor &PP,
-                        ArrayRef<const Decl *> Decls,
-                        IndexDataConsumer &DataConsumer, IndexingOptions Opts);
-
-/// Creates a PPCallbacks that indexes macros and feeds macros to \p Consumer.
-/// The caller is responsible for calling `Consumer.setPreprocessor()`.
-std::unique_ptr<PPCallbacks> indexMacrosCallback(IndexDataConsumer &Consumer,
-                                                 IndexingOptions Opts);
-
-/// Recursively indexes all top-level decls in the module.
-void indexModuleFile(serialization::ModuleFile &Mod, ASTReader &Reader,
-                     IndexDataConsumer &DataConsumer, IndexingOptions Opts);
-
-} // namespace index
-} // namespace clang
-
-#endif

clang/include/clang/Index/IndexingContext.h

 //===- IndexingContext.h - Indexing context data ----------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_LIB_INDEX_INDEXINGCONTEXT_H
 #define LLVM_CLANG_LIB_INDEX_INDEXINGCONTEXT_H
 
 #include "clang/Basic/IdentifierTable.h"
 #include "clang/Basic/LLVM.h"
+#include "clang/Basic/SourceLocation.h"
+#include "clang/Index/IndexOptions.h"
 #include "clang/Index/IndexSymbol.h"
-#include "clang/Index/IndexingAction.h"
 #include "clang/Lex/MacroInfo.h"
 #include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseMap.h"
 
 namespace clang {
   class ASTContext;
   class Decl;
   class DeclGroupRef;
   class ImportDecl;
   class TagDecl;
   class TypeSourceInfo;
   class NamedDecl;
   class ObjCMethodDecl;
   class DeclContext;
   class NestedNameSpecifierLoc;
   class Stmt;
   class Expr;
   class TypeLoc;
-  class SourceLocation;
+  class DirectoryEntry;
 
   namespace index {
   class IndexDataConsumer;
 
 class IndexingContext {
   IndexingOptions IndexOpts;
   IndexDataConsumer &DataConsumer;
   ASTContext *Ctx = nullptr;
+  std::string SysrootPath;
+
+  // Records whether a directory entry is system or not.
+  llvm::DenseMap<const DirectoryEntry *, bool> DirEntries;
+  // Keeps track of the last check for whether a FileID is system or
+  // not. This is used to speed up isSystemFile() call.
+  std::pair<FileID, bool> LastFileCheck;
 
 public:
   IndexingContext(IndexingOptions IndexOpts, IndexDataConsumer &DataConsumer)
     : IndexOpts(IndexOpts), DataConsumer(DataConsumer) {}
 
   const IndexingOptions &getIndexOpts() const { return IndexOpts; }
   IndexDataConsumer &getDataConsumer() { return DataConsumer; }
 
   void setASTContext(ASTContext &ctx) { Ctx = &ctx; }
+  void setSysrootPath(StringRef path);
+  StringRef getSysrootPath() const { return SysrootPath; }
+
+  bool isSystemFile(FileID FID);
 
   bool shouldIndex(const Decl *D);
 
   const LangOptions &getLangOpts() const;
 
   bool shouldSuppressRefs() const {
     return false;
   }

clang/include/clang/Index/USRGeneration.h

 /// Generate a USR fragment for a module name.
 /// \returns true on error, false on success.
 bool generateUSRFragmentForModuleName(StringRef ModName, raw_ostream &OS);
 
 
 } // namespace index
 } // namespace clang
 
-#endif // LLVM_CLANG_INDEX_USRGENERATION_H
-
+#endif

clang/include/clang/IndexDataStore/IndexDataStore.h

+//===--- IndexDataStore.h - Index data store info -------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEX_INDEXDATASTORE_H
+#define LLVM_CLANG_INDEX_INDEXDATASTORE_H
+
+#include "clang/Basic/LLVM.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/Chrono.h"
+#include <functional>
+#include <memory>
+#include <string>
+
+namespace clang {
+namespace index {
+
+class IndexDataStore {
+public:
+  ~IndexDataStore();
+
+  static std::unique_ptr<IndexDataStore> create(StringRef IndexStorePath,
+                                                std::string &Error);
+
+  StringRef getFilePath() const;
+  bool foreachUnitName(bool sorted,
+                       llvm::function_ref<bool(StringRef unitName)> receiver);
+
+  static unsigned getFormatVersion();
+
+  enum class UnitEventKind {
+    Removed,
+    Modified,
+    /// The directory got deleted. No more events will follow.
+    DirectoryDeleted,
+    Failure
+  };
+  struct UnitEvent {
+    UnitEventKind Kind;
+    StringRef UnitName;
+  };
+  struct UnitEventNotification {
+    bool IsInitial;
+    ArrayRef<UnitEvent> Events;
+  };
+  typedef std::function<void(UnitEventNotification)> UnitEventHandler;
+
+  void setUnitEventHandler(UnitEventHandler Handler);
+  /// \returns true if an error occurred.
+  bool startEventListening(bool waitInitialSync, std::string &Error);
+  void stopEventListening();
+
+  void discardUnit(StringRef UnitName);
+  void discardRecord(StringRef RecordName);
+
+  void purgeStaleData();
+
+private:
+  IndexDataStore(void *Impl) : Impl(Impl) {}
+
+  void *Impl; // An IndexDataStoreImpl.
+};
+
+} // namespace index
+} // namespace clang
+
+#endif

clang/include/indexstore/Conversions.h

+//===--- IndexDataStoreSymbolUtils.h - Utilities for indexstore symbols ---===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEXSTORE_CONVERSIONS_H
+#define LLVM_CLANG_INDEXSTORE_CONVERSIONS_H
+
+#include "indexstore/indexstore.h"
+#include "clang/Index/IndexSymbol.h"
+
+namespace clang {
+namespace index {
+
+inline SymbolKind getSymbolKind(indexstore_symbol_kind_t K) {
+  switch ((uint64_t)K) {
+  default:
+  case INDEXSTORE_SYMBOL_KIND_UNKNOWN:
+    return SymbolKind::Unknown;
+  case INDEXSTORE_SYMBOL_KIND_MODULE:
+    return SymbolKind::Module;
+  case INDEXSTORE_SYMBOL_KIND_NAMESPACE:
+    return SymbolKind::Namespace;
+  case INDEXSTORE_SYMBOL_KIND_NAMESPACEALIAS:
+    return SymbolKind::NamespaceAlias;
+  case INDEXSTORE_SYMBOL_KIND_MACRO:
+    return SymbolKind::Macro;
+  case INDEXSTORE_SYMBOL_KIND_ENUM:
+    return SymbolKind::Enum;
+  case INDEXSTORE_SYMBOL_KIND_STRUCT:
+    return SymbolKind::Struct;
+  case INDEXSTORE_SYMBOL_KIND_CLASS:
+    return SymbolKind::Class;
+  case INDEXSTORE_SYMBOL_KIND_PROTOCOL:
+    return SymbolKind::Protocol;
+  case INDEXSTORE_SYMBOL_KIND_EXTENSION:
+    return SymbolKind::Extension;
+  case INDEXSTORE_SYMBOL_KIND_UNION:
+    return SymbolKind::Union;
+  case INDEXSTORE_SYMBOL_KIND_TYPEALIAS:
+    return SymbolKind::TypeAlias;
+  case INDEXSTORE_SYMBOL_KIND_FUNCTION:
+    return SymbolKind::Function;
+  case INDEXSTORE_SYMBOL_KIND_VARIABLE:
+    return SymbolKind::Variable;
+  case INDEXSTORE_SYMBOL_KIND_FIELD:
+    return SymbolKind::Field;
+  case INDEXSTORE_SYMBOL_KIND_ENUMCONSTANT:
+    return SymbolKind::EnumConstant;
+  case INDEXSTORE_SYMBOL_KIND_INSTANCEMETHOD:
+    return SymbolKind::InstanceMethod;
+  case INDEXSTORE_SYMBOL_KIND_CLASSMETHOD:
+    return SymbolKind::ClassMethod;
+  case INDEXSTORE_SYMBOL_KIND_STATICMETHOD:
+    return SymbolKind::StaticMethod;
+  case INDEXSTORE_SYMBOL_KIND_INSTANCEPROPERTY:
+    return SymbolKind::InstanceProperty;
+  case INDEXSTORE_SYMBOL_KIND_CLASSPROPERTY:
+    return SymbolKind::ClassProperty;
+  case INDEXSTORE_SYMBOL_KIND_STATICPROPERTY:
+    return SymbolKind::StaticProperty;
+  case INDEXSTORE_SYMBOL_KIND_CONSTRUCTOR:
+    return SymbolKind::Constructor;
+  case INDEXSTORE_SYMBOL_KIND_DESTRUCTOR:
+    return SymbolKind::Destructor;
+  case INDEXSTORE_SYMBOL_KIND_CONVERSIONFUNCTION:
+    return SymbolKind::ConversionFunction;
+  case INDEXSTORE_SYMBOL_KIND_PARAMETER:
+    return SymbolKind::Parameter;
+  case INDEXSTORE_SYMBOL_KIND_USING:
+    return SymbolKind::Using;
+  }
+}
+
+inline SymbolSubKind getSymbolSubKind(indexstore_symbol_subkind_t K) {
+  switch ((uint64_t)K) {
+  default:
+  case INDEXSTORE_SYMBOL_SUBKIND_NONE:
+    return SymbolSubKind::None;
+  case INDEXSTORE_SYMBOL_SUBKIND_CXXCOPYCONSTRUCTOR:
+    return SymbolSubKind::CXXCopyConstructor;
+  case INDEXSTORE_SYMBOL_SUBKIND_CXXMOVECONSTRUCTOR:
+    return SymbolSubKind::CXXMoveConstructor;
+  case INDEXSTORE_SYMBOL_SUBKIND_ACCESSORGETTER:
+    return SymbolSubKind::AccessorGetter;
+  case INDEXSTORE_SYMBOL_SUBKIND_ACCESSORSETTER:
+    return SymbolSubKind::AccessorSetter;
+  case INDEXSTORE_SYMBOL_SUBKIND_USINGTYPENAME:
+    return SymbolSubKind::UsingTypename;
+  case INDEXSTORE_SYMBOL_SUBKIND_USINGVALUE:
+    return SymbolSubKind::UsingValue;
+  }
+}
+
+inline SymbolLanguage getSymbolLanguage(indexstore_symbol_language_t L) {
+  switch ((uint64_t)L) {
+  default: // FIXME: add an unknown language?
+  case INDEXSTORE_SYMBOL_LANG_C:
+    return SymbolLanguage::C;
+  case INDEXSTORE_SYMBOL_LANG_OBJC:
+    return SymbolLanguage::ObjC;
+  case INDEXSTORE_SYMBOL_LANG_CXX:
+    return SymbolLanguage::CXX;
+  case INDEXSTORE_SYMBOL_LANG_SWIFT:
+    return SymbolLanguage::Swift;
+  }
+}
+
+inline SymbolPropertySet getSymbolProperties(uint64_t Props) {
+  SymbolPropertySet SymbolProperties = 0;
+  if (Props & INDEXSTORE_SYMBOL_PROPERTY_GENERIC)
+    SymbolProperties |= (SymbolPropertySet)SymbolProperty::Generic;
+  if (Props & INDEXSTORE_SYMBOL_PROPERTY_TEMPLATE_PARTIAL_SPECIALIZATION)
+    SymbolProperties |=
+        (SymbolPropertySet)SymbolProperty::TemplatePartialSpecialization;
+  if (Props & INDEXSTORE_SYMBOL_PROPERTY_TEMPLATE_SPECIALIZATION)
+    SymbolProperties |=
+        (SymbolPropertySet)SymbolProperty::TemplateSpecialization;
+  if (Props & INDEXSTORE_SYMBOL_PROPERTY_UNITTEST)
+    SymbolProperties |= (SymbolPropertySet)SymbolProperty::UnitTest;
+  if (Props & INDEXSTORE_SYMBOL_PROPERTY_IBANNOTATED)
+    SymbolProperties |= (SymbolPropertySet)SymbolProperty::IBAnnotated;
+  if (Props & INDEXSTORE_SYMBOL_PROPERTY_IBOUTLETCOLLECTION)
+    SymbolProperties |= (SymbolPropertySet)SymbolProperty::IBOutletCollection;
+  if (Props & INDEXSTORE_SYMBOL_PROPERTY_GKINSPECTABLE)
+    SymbolProperties |= (SymbolPropertySet)SymbolProperty::GKInspectable;
+  if (Props & INDEXSTORE_SYMBOL_PROPERTY_LOCAL)
+    SymbolProperties |= (SymbolPropertySet)SymbolProperty::Local;
+  if (Props & INDEXSTORE_SYMBOL_PROPERTY_PROTOCOL_INTERFACE)
+    SymbolProperties |= (SymbolPropertySet)SymbolProperty::ProtocolInterface;
+
+  return SymbolProperties;
+}
+
+inline SymbolRoleSet getSymbolRoles(uint64_t Roles) {
+  SymbolRoleSet SymbolRoles = 0;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_DECLARATION)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::Declaration;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_DEFINITION)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::Definition;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_REFERENCE)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::Reference;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_READ)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::Read;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_WRITE)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::Write;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_CALL)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::Call;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_DYNAMIC)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::Dynamic;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_ADDRESSOF)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::AddressOf;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_IMPLICIT)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::Implicit;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_UNDEFINITION)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::Undefinition;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_REL_CHILDOF)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::RelationChildOf;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_REL_BASEOF)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::RelationBaseOf;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_REL_OVERRIDEOF)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::RelationOverrideOf;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_REL_RECEIVEDBY)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::RelationReceivedBy;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_REL_CALLEDBY)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::RelationCalledBy;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_REL_EXTENDEDBY)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::RelationExtendedBy;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_REL_ACCESSOROF)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::RelationAccessorOf;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_REL_CONTAINEDBY)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::RelationContainedBy;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_REL_IBTYPEOF)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::RelationIBTypeOf;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_REL_SPECIALIZATIONOF)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::RelationSpecializationOf;
+
+  return SymbolRoles;
+}
+
+inline indexstore_symbol_kind_t getIndexStoreKind(SymbolKind K) {
+  switch (K) {
+  case SymbolKind::Unknown:
+    return INDEXSTORE_SYMBOL_KIND_UNKNOWN;
+  case SymbolKind::Module:
+    return INDEXSTORE_SYMBOL_KIND_MODULE;
+  case SymbolKind::Namespace:
+    return INDEXSTORE_SYMBOL_KIND_NAMESPACE;
+  case SymbolKind::NamespaceAlias:
+    return INDEXSTORE_SYMBOL_KIND_NAMESPACEALIAS;
+  case SymbolKind::Macro:
+    return INDEXSTORE_SYMBOL_KIND_MACRO;
+  case SymbolKind::Enum:
+    return INDEXSTORE_SYMBOL_KIND_ENUM;
+  case SymbolKind::Struct:
+    return INDEXSTORE_SYMBOL_KIND_STRUCT;
+  case SymbolKind::Class:
+    return INDEXSTORE_SYMBOL_KIND_CLASS;
+  case SymbolKind::Protocol:
+    return INDEXSTORE_SYMBOL_KIND_PROTOCOL;
+  case SymbolKind::Extension:
+    return INDEXSTORE_SYMBOL_KIND_EXTENSION;
+  case SymbolKind::Union:
+    return INDEXSTORE_SYMBOL_KIND_UNION;
+  case SymbolKind::TypeAlias:
+    return INDEXSTORE_SYMBOL_KIND_TYPEALIAS;
+  case SymbolKind::Function:
+    return INDEXSTORE_SYMBOL_KIND_FUNCTION;
+  case SymbolKind::Variable:
+    return INDEXSTORE_SYMBOL_KIND_VARIABLE;
+  case SymbolKind::Field:
+    return INDEXSTORE_SYMBOL_KIND_FIELD;
+  case SymbolKind::EnumConstant:
+    return INDEXSTORE_SYMBOL_KIND_ENUMCONSTANT;
+  case SymbolKind::InstanceMethod:
+    return INDEXSTORE_SYMBOL_KIND_INSTANCEMETHOD;
+  case SymbolKind::ClassMethod:
+    return INDEXSTORE_SYMBOL_KIND_CLASSMETHOD;
+  case SymbolKind::StaticMethod:
+    return INDEXSTORE_SYMBOL_KIND_STATICMETHOD;
+  case SymbolKind::InstanceProperty:
+    return INDEXSTORE_SYMBOL_KIND_INSTANCEPROPERTY;
+  case SymbolKind::ClassProperty:
+    return INDEXSTORE_SYMBOL_KIND_CLASSPROPERTY;
+  case SymbolKind::StaticProperty:
+    return INDEXSTORE_SYMBOL_KIND_STATICPROPERTY;
+  case SymbolKind::Constructor:
+    return INDEXSTORE_SYMBOL_KIND_CONSTRUCTOR;
+  case SymbolKind::Destructor:
+    return INDEXSTORE_SYMBOL_KIND_DESTRUCTOR;
+  case SymbolKind::ConversionFunction:
+    return INDEXSTORE_SYMBOL_KIND_CONVERSIONFUNCTION;
+  case SymbolKind::Parameter:
+    return INDEXSTORE_SYMBOL_KIND_PARAMETER;
+  case SymbolKind::Using:
+    return INDEXSTORE_SYMBOL_KIND_USING;
+  }
+  llvm_unreachable("unexpected symbol kind");
+}
+
+inline indexstore_symbol_subkind_t getIndexStoreSubKind(SymbolSubKind K) {
+  switch (K) {
+  case SymbolSubKind::None:
+    return INDEXSTORE_SYMBOL_SUBKIND_NONE;
+  case SymbolSubKind::CXXCopyConstructor:
+    return INDEXSTORE_SYMBOL_SUBKIND_CXXCOPYCONSTRUCTOR;
+  case SymbolSubKind::CXXMoveConstructor:
+    return INDEXSTORE_SYMBOL_SUBKIND_CXXMOVECONSTRUCTOR;
+  case SymbolSubKind::AccessorGetter:
+    return INDEXSTORE_SYMBOL_SUBKIND_ACCESSORGETTER;
+  case SymbolSubKind::AccessorSetter:
+    return INDEXSTORE_SYMBOL_SUBKIND_ACCESSORSETTER;
+  case SymbolSubKind::UsingTypename:
+    return INDEXSTORE_SYMBOL_SUBKIND_USINGTYPENAME;
+  case SymbolSubKind::UsingValue:
+    return INDEXSTORE_SYMBOL_SUBKIND_USINGVALUE;
+  }
+  llvm_unreachable("unexpected symbol subkind");
+}
+
+inline indexstore_symbol_language_t getIndexStoreLang(SymbolLanguage L) {
+  switch (L) {
+  case SymbolLanguage::C:
+    return INDEXSTORE_SYMBOL_LANG_C;
+  case SymbolLanguage::ObjC:
+    return INDEXSTORE_SYMBOL_LANG_OBJC;
+  case SymbolLanguage::CXX:
+    return INDEXSTORE_SYMBOL_LANG_CXX;
+  case SymbolLanguage::Swift:
+    return INDEXSTORE_SYMBOL_LANG_SWIFT;
+  }
+  llvm_unreachable("unexpected symbol language");
+}
+
+inline uint64_t getIndexStoreProperties(SymbolPropertySet Props) {
+  uint64_t storeProp = 0;
+  applyForEachSymbolProperty(Props, [&](SymbolProperty prop) {
+    switch (prop) {
+    case SymbolProperty::Generic:
+      storeProp |= INDEXSTORE_SYMBOL_PROPERTY_GENERIC;
+      break;
+    case SymbolProperty::TemplatePartialSpecialization:
+      storeProp |= INDEXSTORE_SYMBOL_PROPERTY_TEMPLATE_PARTIAL_SPECIALIZATION;
+      break;
+    case SymbolProperty::TemplateSpecialization:
+      storeProp |= INDEXSTORE_SYMBOL_PROPERTY_TEMPLATE_SPECIALIZATION;
+      break;
+    case SymbolProperty::UnitTest:
+      storeProp |= INDEXSTORE_SYMBOL_PROPERTY_UNITTEST;
+      break;
+    case SymbolProperty::IBAnnotated:
+      storeProp |= INDEXSTORE_SYMBOL_PROPERTY_IBANNOTATED;
+      break;
+    case SymbolProperty::IBOutletCollection:
+      storeProp |= INDEXSTORE_SYMBOL_PROPERTY_IBOUTLETCOLLECTION;
+      break;
+    case SymbolProperty::GKInspectable:
+      storeProp |= INDEXSTORE_SYMBOL_PROPERTY_GKINSPECTABLE;
+      break;
+    case SymbolProperty::Local:
+      storeProp |= INDEXSTORE_SYMBOL_PROPERTY_LOCAL;
+      break;
+    case SymbolProperty::ProtocolInterface:
+      storeProp |= INDEXSTORE_SYMBOL_PROPERTY_PROTOCOL_INTERFACE;
+      break;
+    }
+  });
+  return storeProp;
+}
+
+inline uint64_t getIndexStoreRoles(SymbolRoleSet Roles) {
+  uint64_t storeRoles = 0;
+  applyForEachSymbolRole(Roles, [&](SymbolRole role) {
+    switch (role) {
+    case SymbolRole::Declaration:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_DECLARATION;
+      break;
+    case SymbolRole::Definition:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_DEFINITION;
+      break;
+    case SymbolRole::Reference:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REFERENCE;
+      break;
+    case SymbolRole::Read:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_READ;
+      break;
+    case SymbolRole::Write:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_WRITE;
+      break;
+    case SymbolRole::Call:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_CALL;
+      break;
+    case SymbolRole::Dynamic:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_DYNAMIC;
+      break;
+    case SymbolRole::AddressOf:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_ADDRESSOF;
+      break;
+    case SymbolRole::Implicit:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_IMPLICIT;
+      break;
+    case SymbolRole::Undefinition:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_UNDEFINITION;
+      break;
+    case SymbolRole::RelationChildOf:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REL_CHILDOF;
+      break;
+    case SymbolRole::RelationBaseOf:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REL_BASEOF;
+      break;
+    case SymbolRole::RelationOverrideOf:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REL_OVERRIDEOF;
+      break;
+    case SymbolRole::RelationReceivedBy:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REL_RECEIVEDBY;
+      break;
+    case SymbolRole::RelationCalledBy:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REL_CALLEDBY;
+      break;
+    case SymbolRole::RelationExtendedBy:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REL_EXTENDEDBY;
+      break;
+    case SymbolRole::RelationAccessorOf:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REL_ACCESSOROF;
+      break;
+    case SymbolRole::RelationContainedBy:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REL_CONTAINEDBY;
+      break;
+    case SymbolRole::RelationIBTypeOf:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REL_IBTYPEOF;
+      break;
+    case SymbolRole::RelationSpecializationOf:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REL_SPECIALIZATIONOF;
+      break;
+    }
+  });
+  return storeRoles;
+}
+
+} // end namespace index
+} // namespace clang
+
+#endif

clang/include/indexstore/IndexStoreCXX.h

+//===--- IndexStoreCXX.h - C++ wrapper for the Index Store C API. ---------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// Header-only C++ wrapper for the Index Store C API.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEXSTORE_INDEXSTORECXX_H
+#define LLVM_CLANG_INDEXSTORE_INDEXSTORECXX_H
+
+#include "indexstore/indexstore.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/Optional.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/SmallString.h"
+
+namespace indexstore {
+using llvm::ArrayRef;
+using llvm::Optional;
+using llvm::StringRef;
+
+static inline StringRef
+stringFromIndexStoreStringRef(indexstore_string_ref_t str) {
+  return StringRef(str.data, str.length);
+}
+
+template <typename Ret, typename... Params>
+static inline Ret functionPtrFromFunctionRef(void *ctx, Params... params) {
+  auto fn = (llvm::function_ref<Ret(Params...)> *)ctx;
+  return (*fn)(std::forward<Params>(params)...);
+}
+
+class IndexRecordSymbol {
+  indexstore_symbol_t obj;
+  friend class IndexRecordReader;
+
+public:
+  IndexRecordSymbol(indexstore_symbol_t obj) : obj(obj) {}
+
+  indexstore_symbol_language_t getLanguage() {
+    return indexstore_symbol_get_language(obj);
+  }
+  indexstore_symbol_kind_t getKind() { return indexstore_symbol_get_kind(obj); }
+  indexstore_symbol_subkind_t getSubKind() {
+    return indexstore_symbol_get_subkind(obj);
+  }
+  uint64_t getProperties() { return indexstore_symbol_get_properties(obj); }
+  uint64_t getRoles() { return indexstore_symbol_get_roles(obj); }
+  uint64_t getRelatedRoles() {
+    return indexstore_symbol_get_related_roles(obj);
+  }
+  StringRef getName() {
+    return stringFromIndexStoreStringRef(indexstore_symbol_get_name(obj));
+  }
+  StringRef getUSR() {
+    return stringFromIndexStoreStringRef(indexstore_symbol_get_usr(obj));
+  }
+  StringRef getCodegenName() {
+    return stringFromIndexStoreStringRef(
+        indexstore_symbol_get_codegen_name(obj));
+  }
+};
+
+class IndexSymbolRelation {
+  indexstore_symbol_relation_t obj;
+
+public:
+  IndexSymbolRelation(indexstore_symbol_relation_t obj) : obj(obj) {}
+
+  uint64_t getRoles() { return indexstore_symbol_relation_get_roles(obj); }
+  IndexRecordSymbol getSymbol() {
+    return indexstore_symbol_relation_get_symbol(obj);
+  }
+};
+
+class IndexRecordOccurrence {
+  indexstore_occurrence_t obj;
+
+public:
+  IndexRecordOccurrence(indexstore_occurrence_t obj) : obj(obj) {}
+
+  IndexRecordSymbol getSymbol() {
+    return indexstore_occurrence_get_symbol(obj);
+  }
+  uint64_t getRoles() { return indexstore_occurrence_get_roles(obj); }
+
+  bool foreachRelation(llvm::function_ref<bool(IndexSymbolRelation)> receiver) {
+#if INDEXSTORE_HAS_BLOCKS
+    return indexstore_occurrence_relations_apply(
+        obj, ^bool(indexstore_symbol_relation_t sym_rel) {
+          return receiver(sym_rel);
+        });
+#else
+    return indexstore_occurrence_relations_apply_f(obj, &receiver,
+                                                   functionPtrFromFunctionRef);
+#endif
+  }
+
+  std::pair<unsigned, unsigned> getLineCol() {
+    unsigned line, col;
+    indexstore_occurrence_get_line_col(obj, &line, &col);
+    return std::make_pair(line, col);
+  }
+};
+
+class IndexStore;
+typedef std::shared_ptr<IndexStore> IndexStoreRef;
+
+class IndexStore {
+  indexstore_t obj;
+  friend class IndexRecordReader;
+  friend class IndexUnitReader;
+
+public:
+  IndexStore(StringRef path, std::string &error) {
+    llvm::SmallString<64> buf = path;
+    indexstore_error_t c_err = nullptr;
+    obj = indexstore_store_create(buf.c_str(), &c_err);
+    if (c_err) {
+      error = indexstore_error_get_description(c_err);
+      indexstore_error_dispose(c_err);
+    }
+  }
+
+  IndexStore(IndexStore &&other) : obj(other.obj) { other.obj = nullptr; }
+
+  ~IndexStore() { indexstore_store_dispose(obj); }
+
+  static IndexStoreRef create(StringRef path, std::string &error) {
+    auto storeRef = std::make_shared<IndexStore>(path, error);
+    if (storeRef->isInvalid())
+      return nullptr;
+    return storeRef;
+  }
+
+  static unsigned formatVersion() { return indexstore_format_version(); }
+
+  bool isValid() const { return obj; }
+  bool isInvalid() const { return !isValid(); }
+  explicit operator bool() const { return isValid(); }
+
+  bool foreachUnit(bool sorted,
+                   llvm::function_ref<bool(StringRef unitName)> receiver) {
+#if INDEXSTORE_HAS_BLOCKS
+    return indexstore_store_units_apply(
+        obj, sorted, ^bool(indexstore_string_ref_t unit_name) {
+          return receiver(stringFromIndexStoreStringRef(unit_name));
+        });
+#else
+    return indexstore_store_units_apply_f(obj, sorted, &receiver,
+                                          functionPtrFromFunctionRef);
+#endif
+  }
+
+  class UnitEvent {
+    indexstore_unit_event_t obj;
+
+  public:
+    UnitEvent(indexstore_unit_event_t obj) : obj(obj) {}
+
+    enum class Kind { Removed, Modified, DirectoryDeleted, Failure };
+    Kind getKind() const {
+      indexstore_unit_event_kind_t c_k = indexstore_unit_event_get_kind(obj);
+      Kind K;
+      switch (c_k) {
+      case INDEXSTORE_UNIT_EVENT_REMOVED:
+        K = Kind::Removed;
+        break;
+      case INDEXSTORE_UNIT_EVENT_MODIFIED:
+        K = Kind::Modified;
+        break;
+      case INDEXSTORE_UNIT_EVENT_DIRECTORY_DELETED:
+        K = Kind::DirectoryDeleted;
+        break;
+      case INDEXSTORE_UNIT_EVENT_FAILURE:
+        K = Kind::Failure;
+        break;
+      }
+      return K;
+    }
+
+    StringRef getUnitName() const {
+      return stringFromIndexStoreStringRef(
+          indexstore_unit_event_get_unit_name(obj));
+    }
+  };
+
+  class UnitEventNotification {
+    indexstore_unit_event_notification_t obj;
+
+  public:
+    UnitEventNotification(indexstore_unit_event_notification_t obj)
+        : obj(obj) {}
+
+    bool isInitial() const {
+      return indexstore_unit_event_notification_is_initial(obj);
+    }
+    size_t getEventsCount() const {
+      return indexstore_unit_event_notification_get_events_count(obj);
+    }
+    UnitEvent getEvent(size_t index) const {
+      return indexstore_unit_event_notification_get_event(obj, index);
+    }
+  };
+
+  typedef std::function<void(UnitEventNotification)> UnitEventHandler;
+
+  void setUnitEventHandler(UnitEventHandler handler) {
+#if INDEXSTORE_HAS_BLOCKS
+    if (!handler) {
+      indexstore_store_set_unit_event_handler(obj, nullptr);
+      return;
+    }
+
+    indexstore_store_set_unit_event_handler(
+        obj, ^(indexstore_unit_event_notification_t evt_note) {
+          handler(UnitEventNotification(evt_note));
+        });
+#else
+    if (!handler) {
+      indexstore_store_set_unit_event_handler_f(obj, nullptr, nullptr, nullptr);
+      return;
+    }
+
+    auto fnPtr = new UnitEventHandler(handler);
+    indexstore_store_set_unit_event_handler_f(obj, fnPtr, event_handler,
+                                              event_handler_finalizer);
+#endif
+  }
+
+private:
+  static void event_handler(void *ctx,
+                            indexstore_unit_event_notification_t evt) {
+    auto fnPtr = (UnitEventHandler *)ctx;
+    (*fnPtr)(evt);
+  }
+  static void event_handler_finalizer(void *ctx) {
+    auto fnPtr = (UnitEventHandler *)ctx;
+    delete fnPtr;
+  }
+
+public:
+  bool startEventListening(bool waitInitialSync, std::string &error) {
+    indexstore_unit_event_listen_options_t opts;
+    opts.wait_initial_sync = waitInitialSync;
+    indexstore_error_t c_err = nullptr;
+    bool ret = indexstore_store_start_unit_event_listening(
+        obj, &opts, sizeof(opts), &c_err);
+    if (c_err) {
+      error = indexstore_error_get_description(c_err);
+      indexstore_error_dispose(c_err);
+    }
+    return ret;
+  }
+
+  void stopEventListening() {
+    return indexstore_store_stop_unit_event_listening(obj);
+  }
+
+  void discardUnit(StringRef UnitName) {
+    llvm::SmallString<64> buf = UnitName;
+    indexstore_store_discard_unit(obj, buf.c_str());
+  }
+
+  void discardRecord(StringRef RecordName) {
+    llvm::SmallString<64> buf = RecordName;
+    indexstore_store_discard_record(obj, buf.c_str());
+  }
+
+  void getUnitNameFromOutputPath(StringRef outputPath,
+                                 llvm::SmallVectorImpl<char> &nameBuf) {
+    llvm::SmallString<256> buf = outputPath;
+    llvm::SmallString<64> unitName;
+    unitName.resize(64);
+    size_t nameLen = indexstore_store_get_unit_name_from_output_path(
+        obj, buf.c_str(), unitName.data(), unitName.size());
+    if (nameLen + 1 > unitName.size()) {
+      unitName.resize(nameLen + 1);
+      indexstore_store_get_unit_name_from_output_path(
+          obj, buf.c_str(), unitName.data(), unitName.size());
+    }
+    nameBuf.append(unitName.begin(), unitName.begin() + nameLen);
+  }
+
+  void purgeStaleData() { indexstore_store_purge_stale_data(obj); }
+};
+
+class IndexRecordReader {
+  indexstore_record_reader_t obj;
+
+public:
+  IndexRecordReader(IndexStore &store, StringRef recordName,
+                    std::string &error) {
+    llvm::SmallString<64> buf = recordName;
+    indexstore_error_t c_err = nullptr;
+    obj = indexstore_record_reader_create(store.obj, buf.c_str(), &c_err);
+    if (c_err) {
+      error = indexstore_error_get_description(c_err);
+      indexstore_error_dispose(c_err);
+    }
+  }
+
+  IndexRecordReader(IndexRecordReader &&other) : obj(other.obj) {
+    other.obj = nullptr;
+  }
+
+  ~IndexRecordReader() { indexstore_record_reader_dispose(obj); }
+
+  bool isValid() const { return obj; }
+  bool isInvalid() const { return !isValid(); }
+  explicit operator bool() const { return isValid(); }
+
+  /// Goes through and passes record decls, after filtering using a \c Checker
+  /// function.
+  ///
+  /// Resulting decls can be used as filter for \c foreachOccurrence. This
+  /// allows allocating memory only for the record decls that the caller is
+  /// interested in.
+  bool
+  searchSymbols(llvm::function_ref<bool(IndexRecordSymbol, bool &stop)> filter,
+                llvm::function_ref<void(IndexRecordSymbol)> receiver) {
+#if INDEXSTORE_HAS_BLOCKS
+    return indexstore_record_reader_search_symbols(
+        obj,
+        ^bool(indexstore_symbol_t symbol, bool *stop) {
+          return filter(symbol, *stop);
+        },
+        ^(indexstore_symbol_t symbol) {
+          receiver(symbol);
+        });
+#else
+    return indexstore_record_reader_search_symbols_f(
+        obj, &filter, functionPtrFromFunctionRef, &receiver,
+        functionPtrFromFunctionRef);
+#endif
+  }
+
+  bool foreachSymbol(bool noCache,
+                     llvm::function_ref<bool(IndexRecordSymbol)> receiver) {
+#if INDEXSTORE_HAS_BLOCKS
+    return indexstore_record_reader_symbols_apply(
+        obj, noCache, ^bool(indexstore_symbol_t sym) {
+          return receiver(sym);
+        });
+#else
+    return indexstore_record_reader_symbols_apply_f(obj, noCache, &receiver,
+                                                    functionPtrFromFunctionRef);
+#endif
+  }
+
+  /// \param DeclsFilter if non-empty indicates the list of decls that we want
+  /// to get occurrences for. An empty array indicates that we want occurrences
+  /// for all decls.
+  /// \param RelatedDeclsFilter Same as \c DeclsFilter but for related decls.
+  bool
+  foreachOccurrence(ArrayRef<IndexRecordSymbol> symbolsFilter,
+                    ArrayRef<IndexRecordSymbol> relatedSymbolsFilter,
+                    llvm::function_ref<bool(IndexRecordOccurrence)> receiver) {
+    llvm::SmallVector<indexstore_symbol_t, 16> c_symbolsFilter;
+    c_symbolsFilter.reserve(symbolsFilter.size());
+    for (IndexRecordSymbol sym : symbolsFilter) {
+      c_symbolsFilter.push_back(sym.obj);
+    }
+    llvm::SmallVector<indexstore_symbol_t, 16> c_relatedSymbolsFilter;
+    c_relatedSymbolsFilter.reserve(relatedSymbolsFilter.size());
+    for (IndexRecordSymbol sym : relatedSymbolsFilter) {
+      c_relatedSymbolsFilter.push_back(sym.obj);
+    }
+#if INDEXSTORE_HAS_BLOCKS
+    return indexstore_record_reader_occurrences_of_symbols_apply(
+        obj, c_symbolsFilter.data(), c_symbolsFilter.size(),
+        c_relatedSymbolsFilter.data(), c_relatedSymbolsFilter.size(),
+        ^bool(indexstore_occurrence_t occur) {
+          return receiver(occur);
+        });
+#else
+    return indexstore_record_reader_occurrences_of_symbols_apply_f(
+        obj, c_symbolsFilter.data(), c_symbolsFilter.size(),
+        c_relatedSymbolsFilter.data(), c_relatedSymbolsFilter.size(), &receiver,
+        functionPtrFromFunctionRef);
+#endif
+  }
+
+  bool
+  foreachOccurrence(llvm::function_ref<bool(IndexRecordOccurrence)> receiver) {
+#if INDEXSTORE_HAS_BLOCKS
+    return indexstore_record_reader_occurrences_apply(
+        obj, ^bool(indexstore_occurrence_t occur) {
+          return receiver(occur);
+        });
+#else
+    return indexstore_record_reader_occurrences_apply_f(
+        obj, &receiver, functionPtrFromFunctionRef);
+#endif
+  }
+
+  bool foreachOccurrenceInLineRange(
+      unsigned lineStart, unsigned lineEnd,
+      llvm::function_ref<bool(IndexRecordOccurrence)> receiver) {
+#if INDEXSTORE_HAS_BLOCKS
+    return indexstore_record_reader_occurrences_in_line_range_apply(
+        obj, lineStart, lineEnd, ^bool(indexstore_occurrence_t occur) {
+          return receiver(occur);
+        });
+#else
+    return indexstore_record_reader_occurrences_in_line_range_apply_f(
+        obj, lineStart, lineEnd, &receiver, functionPtrFromFunctionRef);
+#endif
+  }
+};
+
+class IndexUnitDependency {
+  indexstore_unit_dependency_t obj;
+  friend class IndexUnitReader;
+
+public:
+  IndexUnitDependency(indexstore_unit_dependency_t obj) : obj(obj) {}
+
+  enum class DependencyKind {
+    Unit,
+    Record,
+    File,
+  };
+  DependencyKind getKind() {
+    switch (indexstore_unit_dependency_get_kind(obj)) {
+    case INDEXSTORE_UNIT_DEPENDENCY_UNIT:
+      return DependencyKind::Unit;
+    case INDEXSTORE_UNIT_DEPENDENCY_RECORD:
+      return DependencyKind::Record;
+    case INDEXSTORE_UNIT_DEPENDENCY_FILE:
+      return DependencyKind::File;
+    }
+  }
+  bool isSystem() { return indexstore_unit_dependency_is_system(obj); }
+  StringRef getName() {
+    return stringFromIndexStoreStringRef(
+        indexstore_unit_dependency_get_name(obj));
+  }
+  StringRef getFilePath() {
+    return stringFromIndexStoreStringRef(
+        indexstore_unit_dependency_get_filepath(obj));
+  }
+  StringRef getModuleName() {
+    return stringFromIndexStoreStringRef(
+        indexstore_unit_dependency_get_modulename(obj));
+  }
+};
+
+class IndexUnitInclude {
+  indexstore_unit_include_t obj;
+  friend class IndexUnitReader;
+
+public:
+  IndexUnitInclude(indexstore_unit_include_t obj) : obj(obj) {}
+
+  StringRef getSourcePath() {
+    return stringFromIndexStoreStringRef(
+        indexstore_unit_include_get_source_path(obj));
+  }
+  StringRef getTargetPath() {
+    return stringFromIndexStoreStringRef(
+        indexstore_unit_include_get_target_path(obj));
+  }
+  unsigned getSourceLine() {
+    return indexstore_unit_include_get_source_line(obj);
+  }
+};
+
+class IndexUnitReader {
+  indexstore_unit_reader_t obj;
+
+public:
+  IndexUnitReader(IndexStore &store, StringRef unitName, std::string &error) {
+    llvm::SmallString<64> buf = unitName;
+    indexstore_error_t c_err = nullptr;
+    obj = indexstore_unit_reader_create(store.obj, buf.c_str(), &c_err);
+    if (c_err) {
+      error = indexstore_error_get_description(c_err);
+      indexstore_error_dispose(c_err);
+    }
+  }
+
+  IndexUnitReader(IndexUnitReader &&other) : obj(other.obj) {
+    other.obj = nullptr;
+  }
+
+  ~IndexUnitReader() { indexstore_unit_reader_dispose(obj); }
+
+  bool isValid() const { return obj; }
+  bool isInvalid() const { return !isValid(); }
+  explicit operator bool() const { return isValid(); }
+
+  StringRef getProviderIdentifier() {
+    return stringFromIndexStoreStringRef(
+        indexstore_unit_reader_get_provider_identifier(obj));
+  }
+  StringRef getProviderVersion() {
+    return stringFromIndexStoreStringRef(
+        indexstore_unit_reader_get_provider_version(obj));
+  }
+
+  timespec getModificationTime() {
+    int64_t seconds, nanoseconds;
+    indexstore_unit_reader_get_modification_time(obj, &seconds, &nanoseconds);
+    timespec ts;
+    ts.tv_sec = seconds;
+    ts.tv_nsec = nanoseconds;
+    return ts;
+  }
+
+  bool isSystemUnit() { return indexstore_unit_reader_is_system_unit(obj); }
+  bool isModuleUnit() { return indexstore_unit_reader_is_module_unit(obj); }
+  bool isDebugCompilation() {
+    return indexstore_unit_reader_is_debug_compilation(obj);
+  }
+  bool hasMainFile() { return indexstore_unit_reader_has_main_file(obj); }
+
+  StringRef getMainFilePath() {
+    return stringFromIndexStoreStringRef(
+        indexstore_unit_reader_get_main_file(obj));
+  }
+  StringRef getModuleName() {
+    return stringFromIndexStoreStringRef(
+        indexstore_unit_reader_get_module_name(obj));
+  }
+  StringRef getWorkingDirectory() {
+    return stringFromIndexStoreStringRef(
+        indexstore_unit_reader_get_working_dir(obj));
+  }
+  StringRef getOutputFile() {
+    return stringFromIndexStoreStringRef(
+        indexstore_unit_reader_get_output_file(obj));
+  }
+  StringRef getSysrootPath() {
+    return stringFromIndexStoreStringRef(
+        indexstore_unit_reader_get_sysroot_path(obj));
+  }
+  StringRef getTarget() {
+    return stringFromIndexStoreStringRef(
+        indexstore_unit_reader_get_target(obj));
+  }
+
+  bool
+  foreachDependency(llvm::function_ref<bool(IndexUnitDependency)> receiver) {
+#if INDEXSTORE_HAS_BLOCKS
+    return indexstore_unit_reader_dependencies_apply(
+        obj, ^bool(indexstore_unit_dependency_t dep) {
+          return receiver(dep);
+        });
+#else
+    return indexstore_unit_reader_dependencies_apply_f(
+        obj, &receiver, functionPtrFromFunctionRef);
+#endif
+  }
+
+  bool foreachInclude(llvm::function_ref<bool(IndexUnitInclude)> receiver) {
+#if INDEXSTORE_HAS_BLOCKS
+    return indexstore_unit_reader_includes_apply(
+        obj, ^bool(indexstore_unit_include_t inc) {
+          return receiver(inc);
+        });
+#else
+    return indexstore_unit_reader_includes_apply_f(obj, &receiver,
+                                                   functionPtrFromFunctionRef);
+#endif
+  }
+};
+
+} // namespace indexstore
+
+#endif

clang/include/indexstore/indexstore.h

+/*===-- indexstore/indexstore.h - Index Store C API ----------------- C -*-===*\
+|*                                                                            *|
+|*                     The LLVM Compiler Infrastructure                       *|
+|*                                                                            *|
+|* This file is distributed under the University of Illinois Open Source      *|
+|* License. See LICENSE.TXT for details.                                      *|
+|*                                                                            *|
+|*===----------------------------------------------------------------------===*|
+|*                                                                            *|
+|* This header provides a C API for the index store.                          *|
+|*                                                                            *|
+\*===----------------------------------------------------------------------===*/
+
+#ifndef LLVM_CLANG_C_INDEXSTORE_INDEXSTORE_H
+#define LLVM_CLANG_C_INDEXSTORE_INDEXSTORE_H
+
+#include <stdbool.h>
+#include <stddef.h>
+#include <stdint.h>
+#include <time.h>
+
+/**
+ * \brief The version constants for the Index Store C API.
+ * INDEXSTORE_VERSION_MINOR should increase when there are API additions.
+ * INDEXSTORE_VERSION_MAJOR is intended for "major" source/ABI breaking changes.
+ */
+#define INDEXSTORE_VERSION_MAJOR 0
+#define INDEXSTORE_VERSION_MINOR 11
+
+#define INDEXSTORE_VERSION_ENCODE(major, minor) (((major)*10000) + ((minor)*1))
+
+#define INDEXSTORE_VERSION                                                     \
+  INDEXSTORE_VERSION_ENCODE(INDEXSTORE_VERSION_MAJOR, INDEXSTORE_VERSION_MINOR)
+
+#define INDEXSTORE_VERSION_STRINGIZE_(major, minor) #major "." #minor
+#define INDEXSTORE_VERSION_STRINGIZE(major, minor)                             \
+  INDEXSTORE_VERSION_STRINGIZE_(major, minor)
+
+#define INDEXSTORE_VERSION_STRING                                              \
+  INDEXSTORE_VERSION_STRINGIZE(INDEXSTORE_VERSION_MAJOR,                       \
+                               INDEXSTORE_VERSION_MINOR)
+
+#ifdef __cplusplus
+#define INDEXSTORE_BEGIN_DECLS extern "C" {
+#define INDEXSTORE_END_DECLS }
+#else
+#define INDEXSTORE_BEGIN_DECLS
+#define INDEXSTORE_END_DECLS
+#endif
+
+#ifndef INDEXSTORE_PUBLIC
+#ifdef _WIN32
+#ifdef IndexStore_EXPORTS
+#define INDEXSTORE_PUBLIC __declspec(dllexport)
+#else
+#define INDEXSTORE_PUBLIC __declspec(dllimport)
+#endif
+#else
+#define INDEXSTORE_PUBLIC
+#endif
+#endif
+
+#ifndef __has_feature
+#define __has_feature(x) 0
+#endif
+
+#ifndef __has_attribute
+#define __has_attribute(x) 0
+#endif
+
+#if __has_feature(blocks)
+#define INDEXSTORE_HAS_BLOCKS 1
+#else
+#define INDEXSTORE_HAS_BLOCKS 0
+#endif
+
+#if __has_attribute(noescape)
+#define INDEXSTORE_NOESCAPE __attribute__((noescape))
+#else
+#define INDEXSTORE_NOESCAPE
+#endif
+
+INDEXSTORE_BEGIN_DECLS
+
+typedef void *indexstore_error_t;
+
+INDEXSTORE_PUBLIC const char *
+    indexstore_error_get_description(indexstore_error_t);
+
+INDEXSTORE_PUBLIC void indexstore_error_dispose(indexstore_error_t);
+
+typedef struct {
+  const char *data;
+  size_t length;
+} indexstore_string_ref_t;
+
+INDEXSTORE_PUBLIC unsigned indexstore_format_version(void);
+
+typedef void *indexstore_t;
+
+INDEXSTORE_PUBLIC indexstore_t
+indexstore_store_create(const char *store_path, indexstore_error_t *error);
+
+INDEXSTORE_PUBLIC void indexstore_store_dispose(indexstore_t);
+
+#if INDEXSTORE_HAS_BLOCKS
+INDEXSTORE_PUBLIC bool indexstore_store_units_apply(
+    indexstore_t, unsigned sorted,
+    INDEXSTORE_NOESCAPE bool (^applier)(indexstore_string_ref_t unit_name));
+#endif
+
+INDEXSTORE_PUBLIC bool indexstore_store_units_apply_f(
+    indexstore_t, unsigned sorted, void *context,
+    INDEXSTORE_NOESCAPE bool (*applier)(void *context,
+                                        indexstore_string_ref_t unit_name));
+
+typedef void *indexstore_unit_event_notification_t;
+typedef void *indexstore_unit_event_t;
+
+INDEXSTORE_PUBLIC size_t indexstore_unit_event_notification_get_events_count(
+    indexstore_unit_event_notification_t);
+
+INDEXSTORE_PUBLIC indexstore_unit_event_t
+indexstore_unit_event_notification_get_event(
+    indexstore_unit_event_notification_t, size_t index);
+
+INDEXSTORE_PUBLIC bool indexstore_unit_event_notification_is_initial(
+    indexstore_unit_event_notification_t);
+
+typedef enum {
+  INDEXSTORE_UNIT_EVENT_REMOVED = 1,
+  INDEXSTORE_UNIT_EVENT_MODIFIED = 2,
+  INDEXSTORE_UNIT_EVENT_DIRECTORY_DELETED = 3,
+  INDEXSTORE_UNIT_EVENT_FAILURE = 4,
+} indexstore_unit_event_kind_t;
+
+INDEXSTORE_PUBLIC indexstore_unit_event_kind_t
+    indexstore_unit_event_get_kind(indexstore_unit_event_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+    indexstore_unit_event_get_unit_name(indexstore_unit_event_t);
+
+#if INDEXSTORE_HAS_BLOCKS
+typedef void (^indexstore_unit_event_handler_t)(
+    indexstore_unit_event_notification_t);
+
+INDEXSTORE_PUBLIC void indexstore_store_set_unit_event_handler(
+    indexstore_t, indexstore_unit_event_handler_t handler);
+#endif
+
+INDEXSTORE_PUBLIC void indexstore_store_set_unit_event_handler_f(
+    indexstore_t, void *context,
+    void (*handler)(void *context, indexstore_unit_event_notification_t),
+    void (*finalizer)(void *context));
+
+typedef struct {
+  /// If true, \c indexstore_store_start_unit_event_listening will block until
+  /// the initial set of units is passed to the unit event handler, otherwise
+  /// the function will return and the initial set will be passed
+  /// asynchronously.
+  bool wait_initial_sync;
+} indexstore_unit_event_listen_options_t;
+
+INDEXSTORE_PUBLIC bool indexstore_store_start_unit_event_listening(
+    indexstore_t, indexstore_unit_event_listen_options_t *,
+    size_t listen_options_struct_size, indexstore_error_t *error);
+
+INDEXSTORE_PUBLIC void indexstore_store_stop_unit_event_listening(indexstore_t);
+
+INDEXSTORE_PUBLIC void indexstore_store_discard_unit(indexstore_t,
+                                                     const char *unit_name);
+
+INDEXSTORE_PUBLIC void indexstore_store_discard_record(indexstore_t,
+                                                       const char *record_name);
+
+INDEXSTORE_PUBLIC void indexstore_store_purge_stale_data(indexstore_t);
+
+/// Determines the unit name from the \c output_path and writes it out in the
+/// \c name_buf buffer. It doesn't write more than \c buf_size.
+/// \returns the length of the name. If this is larger than \c buf_size, the
+/// caller should call the function again with a buffer of the appropriate size.
+INDEXSTORE_PUBLIC size_t indexstore_store_get_unit_name_from_output_path(
+    indexstore_t store, const char *output_path, char *name_buf,
+    size_t buf_size);
+
+/// \returns true if an error occurred, false otherwise.
+INDEXSTORE_PUBLIC bool indexstore_store_get_unit_modification_time(
+    indexstore_t store, const char *unit_name, int64_t *seconds,
+    int64_t *nanoseconds, indexstore_error_t *error);
+
+typedef void *indexstore_symbol_t;
+
+typedef enum {
+  INDEXSTORE_SYMBOL_KIND_UNKNOWN = 0,
+  INDEXSTORE_SYMBOL_KIND_MODULE = 1,
+  INDEXSTORE_SYMBOL_KIND_NAMESPACE = 2,
+  INDEXSTORE_SYMBOL_KIND_NAMESPACEALIAS = 3,
+  INDEXSTORE_SYMBOL_KIND_MACRO = 4,
+  INDEXSTORE_SYMBOL_KIND_ENUM = 5,
+  INDEXSTORE_SYMBOL_KIND_STRUCT = 6,
+  INDEXSTORE_SYMBOL_KIND_CLASS = 7,
+  INDEXSTORE_SYMBOL_KIND_PROTOCOL = 8,
+  INDEXSTORE_SYMBOL_KIND_EXTENSION = 9,
+  INDEXSTORE_SYMBOL_KIND_UNION = 10,
+  INDEXSTORE_SYMBOL_KIND_TYPEALIAS = 11,
+  INDEXSTORE_SYMBOL_KIND_FUNCTION = 12,
+  INDEXSTORE_SYMBOL_KIND_VARIABLE = 13,
+  INDEXSTORE_SYMBOL_KIND_FIELD = 14,
+  INDEXSTORE_SYMBOL_KIND_ENUMCONSTANT = 15,
+  INDEXSTORE_SYMBOL_KIND_INSTANCEMETHOD = 16,
+  INDEXSTORE_SYMBOL_KIND_CLASSMETHOD = 17,
+  INDEXSTORE_SYMBOL_KIND_STATICMETHOD = 18,
+  INDEXSTORE_SYMBOL_KIND_INSTANCEPROPERTY = 19,
+  INDEXSTORE_SYMBOL_KIND_CLASSPROPERTY = 20,
+  INDEXSTORE_SYMBOL_KIND_STATICPROPERTY = 21,
+  INDEXSTORE_SYMBOL_KIND_CONSTRUCTOR = 22,
+  INDEXSTORE_SYMBOL_KIND_DESTRUCTOR = 23,
+  INDEXSTORE_SYMBOL_KIND_CONVERSIONFUNCTION = 24,
+  INDEXSTORE_SYMBOL_KIND_PARAMETER = 25,
+  INDEXSTORE_SYMBOL_KIND_USING = 26,
+
+  INDEXSTORE_SYMBOL_KIND_COMMENTTAG = 1000,
+} indexstore_symbol_kind_t;
+
+typedef enum {
+  INDEXSTORE_SYMBOL_SUBKIND_NONE = 0,
+  INDEXSTORE_SYMBOL_SUBKIND_CXXCOPYCONSTRUCTOR = 1,
+  INDEXSTORE_SYMBOL_SUBKIND_CXXMOVECONSTRUCTOR = 2,
+  INDEXSTORE_SYMBOL_SUBKIND_ACCESSORGETTER = 3,
+  INDEXSTORE_SYMBOL_SUBKIND_ACCESSORSETTER = 4,
+  INDEXSTORE_SYMBOL_SUBKIND_USINGTYPENAME = 5,
+  INDEXSTORE_SYMBOL_SUBKIND_USINGVALUE = 6,
+
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTACCESSORWILLSET = 1000,
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTACCESSORDIDSET = 1001,
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTACCESSORADDRESSOR = 1002,
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTACCESSORMUTABLEADDRESSOR = 1003,
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTEXTENSIONOFSTRUCT = 1004,
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTEXTENSIONOFCLASS = 1005,
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTEXTENSIONOFENUM = 1006,
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTEXTENSIONOFPROTOCOL = 1007,
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTPREFIXOPERATOR = 1008,
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTPOSTFIXOPERATOR = 1009,
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTINFIXOPERATOR = 1010,
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTSUBSCRIPT = 1011,
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTASSOCIATEDTYPE = 1012,
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTGENERICTYPEPARAM = 1013,
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTACCESSORREAD = 1014,
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTACCESSORMODIFY = 1015,
+} indexstore_symbol_subkind_t;
+
+typedef enum {
+  INDEXSTORE_SYMBOL_PROPERTY_GENERIC = 1 << 0,
+  INDEXSTORE_SYMBOL_PROPERTY_TEMPLATE_PARTIAL_SPECIALIZATION = 1 << 1,
+  INDEXSTORE_SYMBOL_PROPERTY_TEMPLATE_SPECIALIZATION = 1 << 2,
+  INDEXSTORE_SYMBOL_PROPERTY_UNITTEST = 1 << 3,
+  INDEXSTORE_SYMBOL_PROPERTY_IBANNOTATED = 1 << 4,
+  INDEXSTORE_SYMBOL_PROPERTY_IBOUTLETCOLLECTION = 1 << 5,
+  INDEXSTORE_SYMBOL_PROPERTY_GKINSPECTABLE = 1 << 6,
+  INDEXSTORE_SYMBOL_PROPERTY_LOCAL = 1 << 7,
+  INDEXSTORE_SYMBOL_PROPERTY_PROTOCOL_INTERFACE = 1 << 8,
+} indexstore_symbol_property_t;
+
+typedef enum {
+  INDEXSTORE_SYMBOL_LANG_C = 0,
+  INDEXSTORE_SYMBOL_LANG_OBJC = 1,
+  INDEXSTORE_SYMBOL_LANG_CXX = 2,
+
+  INDEXSTORE_SYMBOL_LANG_SWIFT = 100,
+} indexstore_symbol_language_t;
+
+typedef enum {
+  INDEXSTORE_SYMBOL_ROLE_DECLARATION = 1 << 0,
+  INDEXSTORE_SYMBOL_ROLE_DEFINITION = 1 << 1,
+  INDEXSTORE_SYMBOL_ROLE_REFERENCE = 1 << 2,
+  INDEXSTORE_SYMBOL_ROLE_READ = 1 << 3,
+  INDEXSTORE_SYMBOL_ROLE_WRITE = 1 << 4,
+  INDEXSTORE_SYMBOL_ROLE_CALL = 1 << 5,
+  INDEXSTORE_SYMBOL_ROLE_DYNAMIC = 1 << 6,
+  INDEXSTORE_SYMBOL_ROLE_ADDRESSOF = 1 << 7,
+  INDEXSTORE_SYMBOL_ROLE_IMPLICIT = 1 << 8,
+  INDEXSTORE_SYMBOL_ROLE_UNDEFINITION = 1 << 19,
+
+  // Relation roles.
+  INDEXSTORE_SYMBOL_ROLE_REL_CHILDOF = 1 << 9,
+  INDEXSTORE_SYMBOL_ROLE_REL_BASEOF = 1 << 10,
+  INDEXSTORE_SYMBOL_ROLE_REL_OVERRIDEOF = 1 << 11,
+  INDEXSTORE_SYMBOL_ROLE_REL_RECEIVEDBY = 1 << 12,
+  INDEXSTORE_SYMBOL_ROLE_REL_CALLEDBY = 1 << 13,
+  INDEXSTORE_SYMBOL_ROLE_REL_EXTENDEDBY = 1 << 14,
+  INDEXSTORE_SYMBOL_ROLE_REL_ACCESSOROF = 1 << 15,
+  INDEXSTORE_SYMBOL_ROLE_REL_CONTAINEDBY = 1 << 16,
+  INDEXSTORE_SYMBOL_ROLE_REL_IBTYPEOF = 1 << 17,
+  INDEXSTORE_SYMBOL_ROLE_REL_SPECIALIZATIONOF = 1 << 18,
+} indexstore_symbol_role_t;
+
+INDEXSTORE_PUBLIC indexstore_symbol_language_t
+    indexstore_symbol_get_language(indexstore_symbol_t);
+
+INDEXSTORE_PUBLIC
+    indexstore_symbol_kind_t indexstore_symbol_get_kind(indexstore_symbol_t);
+
+INDEXSTORE_PUBLIC indexstore_symbol_subkind_t
+    indexstore_symbol_get_subkind(indexstore_symbol_t);
+
+INDEXSTORE_PUBLIC
+    uint64_t indexstore_symbol_get_properties(indexstore_symbol_t);
+
+INDEXSTORE_PUBLIC uint64_t indexstore_symbol_get_roles(indexstore_symbol_t);
+
+INDEXSTORE_PUBLIC
+    uint64_t indexstore_symbol_get_related_roles(indexstore_symbol_t);
+
+INDEXSTORE_PUBLIC
+    indexstore_string_ref_t indexstore_symbol_get_name(indexstore_symbol_t);
+
+INDEXSTORE_PUBLIC
+    indexstore_string_ref_t indexstore_symbol_get_usr(indexstore_symbol_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+    indexstore_symbol_get_codegen_name(indexstore_symbol_t);
+
+typedef void *indexstore_symbol_relation_t;
+
+INDEXSTORE_PUBLIC
+    uint64_t indexstore_symbol_relation_get_roles(indexstore_symbol_relation_t);
+
+INDEXSTORE_PUBLIC indexstore_symbol_t
+    indexstore_symbol_relation_get_symbol(indexstore_symbol_relation_t);
+
+typedef void *indexstore_occurrence_t;
+
+INDEXSTORE_PUBLIC indexstore_symbol_t
+    indexstore_occurrence_get_symbol(indexstore_occurrence_t);
+
+#if INDEXSTORE_HAS_BLOCKS
+INDEXSTORE_PUBLIC bool indexstore_occurrence_relations_apply(
+    indexstore_occurrence_t, INDEXSTORE_NOESCAPE bool (^applier)(
+                                 indexstore_symbol_relation_t symbol_rel));
+#endif
+
+INDEXSTORE_PUBLIC bool indexstore_occurrence_relations_apply_f(
+    indexstore_occurrence_t, void *context,
+    INDEXSTORE_NOESCAPE bool (*applier)(
+        void *context, indexstore_symbol_relation_t symbol_rel));
+
+INDEXSTORE_PUBLIC
+    uint64_t indexstore_occurrence_get_roles(indexstore_occurrence_t);
+
+INDEXSTORE_PUBLIC void
+indexstore_occurrence_get_line_col(indexstore_occurrence_t, unsigned *line,
+                                   unsigned *column);
+
+typedef void *indexstore_record_reader_t;
+
+INDEXSTORE_PUBLIC indexstore_record_reader_t indexstore_record_reader_create(
+    indexstore_t store, const char *record_name, indexstore_error_t *error);
+
+INDEXSTORE_PUBLIC void
+    indexstore_record_reader_dispose(indexstore_record_reader_t);
+
+#if INDEXSTORE_HAS_BLOCKS
+/// Goes through the symbol data and passes symbols to \c receiver, for the
+/// symbol data that \c filter returns true on.
+///
+/// This allows allocating memory only for the record symbols that the caller is
+/// interested in.
+INDEXSTORE_PUBLIC bool indexstore_record_reader_search_symbols(
+    indexstore_record_reader_t,
+    INDEXSTORE_NOESCAPE bool (^filter)(indexstore_symbol_t symbol, bool *stop),
+    INDEXSTORE_NOESCAPE void (^receiver)(indexstore_symbol_t symbol));
+
+/// \param nocache if true, avoids allocating memory for the symbols.
+/// Useful when the caller does not intend to keep \c indexstore_record_reader_t
+/// for more queries.
+INDEXSTORE_PUBLIC bool indexstore_record_reader_symbols_apply(
+    indexstore_record_reader_t, bool nocache,
+    INDEXSTORE_NOESCAPE bool (^applier)(indexstore_symbol_t symbol));
+
+INDEXSTORE_PUBLIC bool indexstore_record_reader_occurrences_apply(
+    indexstore_record_reader_t,
+    INDEXSTORE_NOESCAPE bool (^applier)(indexstore_occurrence_t occur));
+
+INDEXSTORE_PUBLIC bool indexstore_record_reader_occurrences_in_line_range_apply(
+    indexstore_record_reader_t, unsigned line_start, unsigned line_count,
+    INDEXSTORE_NOESCAPE bool (^applier)(indexstore_occurrence_t occur));
+
+/// \param symbols if non-zero \c symbols_count, indicates the list of symbols
+/// that we want to get occurrences for. An empty array indicates that we want
+/// occurrences for all symbols.
+/// \param related_symbols Same as \c symbols but for related symbols.
+INDEXSTORE_PUBLIC bool indexstore_record_reader_occurrences_of_symbols_apply(
+    indexstore_record_reader_t, indexstore_symbol_t *symbols,
+    size_t symbols_count, indexstore_symbol_t *related_symbols,
+    size_t related_symbols_count,
+    INDEXSTORE_NOESCAPE bool (^applier)(indexstore_occurrence_t occur));
+#endif
+
+INDEXSTORE_PUBLIC bool indexstore_record_reader_search_symbols_f(
+    indexstore_record_reader_t, void *filter_ctx,
+    INDEXSTORE_NOESCAPE bool (*filter)(void *filter_ctx,
+                                       indexstore_symbol_t symbol, bool *stop),
+    void *receiver_ctx,
+    INDEXSTORE_NOESCAPE void (*receiver)(void *receiver_ctx,
+                                         indexstore_symbol_t symbol));
+
+INDEXSTORE_PUBLIC bool indexstore_record_reader_symbols_apply_f(
+    indexstore_record_reader_t, bool nocache, void *context,
+    INDEXSTORE_NOESCAPE bool (*applier)(void *context,
+                                        indexstore_symbol_t symbol));
+
+INDEXSTORE_PUBLIC bool indexstore_record_reader_occurrences_apply_f(
+    indexstore_record_reader_t, void *context,
+    INDEXSTORE_NOESCAPE bool (*applier)(void *context,
+                                        indexstore_occurrence_t occur));
+
+INDEXSTORE_PUBLIC bool
+indexstore_record_reader_occurrences_in_line_range_apply_f(
+    indexstore_record_reader_t, unsigned line_start, unsigned line_count,
+    void *context,
+    INDEXSTORE_NOESCAPE bool (*applier)(void *context,
+                                        indexstore_occurrence_t occur));
+
+INDEXSTORE_PUBLIC bool indexstore_record_reader_occurrences_of_symbols_apply_f(
+    indexstore_record_reader_t, indexstore_symbol_t *symbols,
+    size_t symbols_count, indexstore_symbol_t *related_symbols,
+    size_t related_symbols_count, void *context,
+    INDEXSTORE_NOESCAPE bool (*applier)(void *context,
+                                        indexstore_occurrence_t occur));
+
+typedef void *indexstore_unit_reader_t;
+
+INDEXSTORE_PUBLIC indexstore_unit_reader_t indexstore_unit_reader_create(
+    indexstore_t store, const char *unit_name, indexstore_error_t *error);
+
+INDEXSTORE_PUBLIC void indexstore_unit_reader_dispose(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+    indexstore_unit_reader_get_provider_identifier(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+    indexstore_unit_reader_get_provider_version(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC void indexstore_unit_reader_get_modification_time(
+    indexstore_unit_reader_t, int64_t *seconds, int64_t *nanoseconds);
+
+INDEXSTORE_PUBLIC bool
+    indexstore_unit_reader_is_system_unit(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC bool
+    indexstore_unit_reader_is_module_unit(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC bool
+    indexstore_unit_reader_is_debug_compilation(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC bool
+    indexstore_unit_reader_has_main_file(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+    indexstore_unit_reader_get_main_file(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+    indexstore_unit_reader_get_module_name(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+    indexstore_unit_reader_get_working_dir(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+    indexstore_unit_reader_get_output_file(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+    indexstore_unit_reader_get_sysroot_path(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+    indexstore_unit_reader_get_target(indexstore_unit_reader_t);
+
+typedef void *indexstore_unit_dependency_t;
+typedef void *indexstore_unit_include_t;
+
+typedef enum {
+  INDEXSTORE_UNIT_DEPENDENCY_UNIT = 1,
+  INDEXSTORE_UNIT_DEPENDENCY_RECORD = 2,
+  INDEXSTORE_UNIT_DEPENDENCY_FILE = 3,
+} indexstore_unit_dependency_kind_t;
+
+INDEXSTORE_PUBLIC indexstore_unit_dependency_kind_t
+    indexstore_unit_dependency_get_kind(indexstore_unit_dependency_t);
+
+INDEXSTORE_PUBLIC bool
+    indexstore_unit_dependency_is_system(indexstore_unit_dependency_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+    indexstore_unit_dependency_get_filepath(indexstore_unit_dependency_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+    indexstore_unit_dependency_get_modulename(indexstore_unit_dependency_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+    indexstore_unit_dependency_get_name(indexstore_unit_dependency_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+    indexstore_unit_include_get_source_path(indexstore_unit_include_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+    indexstore_unit_include_get_target_path(indexstore_unit_include_t);
+
+INDEXSTORE_PUBLIC unsigned
+    indexstore_unit_include_get_source_line(indexstore_unit_include_t);
+
+#if INDEXSTORE_HAS_BLOCKS
+INDEXSTORE_PUBLIC bool indexstore_unit_reader_dependencies_apply(
+    indexstore_unit_reader_t,
+    INDEXSTORE_NOESCAPE bool (^applier)(indexstore_unit_dependency_t));
+
+INDEXSTORE_PUBLIC bool indexstore_unit_reader_includes_apply(
+    indexstore_unit_reader_t,
+    INDEXSTORE_NOESCAPE bool (^applier)(indexstore_unit_include_t));
+#endif
+
+INDEXSTORE_PUBLIC bool indexstore_unit_reader_dependencies_apply_f(
+    indexstore_unit_reader_t, void *context,
+    INDEXSTORE_NOESCAPE bool (*applier)(void *context,
+                                        indexstore_unit_dependency_t));
+
+INDEXSTORE_PUBLIC bool indexstore_unit_reader_includes_apply_f(
+    indexstore_unit_reader_t, void *context,
+    INDEXSTORE_NOESCAPE bool (*applier)(void *context,
+                                        indexstore_unit_include_t));
+
+INDEXSTORE_END_DECLS
+
+#endif

clang/lib/CMakeLists.txt

 endif()
 add_subdirectory(Driver)
 add_subdirectory(Serialization)
 add_subdirectory(Frontend)
 add_subdirectory(FrontendTool)
 add_subdirectory(Tooling)
 add_subdirectory(DirectoryWatcher)
 add_subdirectory(Index)
+add_subdirectory(IndexDataStore)
 if(CLANG_ENABLE_STATIC_ANALYZER)
   add_subdirectory(StaticAnalyzer)
 endif()
 add_subdirectory(Format)

clang/lib/Driver/Driver.cpp

if (StringRef(TempFile).endswith(".cache")) {
       // In some cases (modules) we'll dump extra data to help with reproducing
       // the crash into a directory next to the output.
       VFS = llvm::sys::path::filename(TempFile);
       llvm::sys::path::append(VFS, "vfs", "vfs.yaml");
     }
   }
 
   // Assume associated files are based off of the first temporary file.
-  CrashReportInfo CrashInfo(TempFiles[0], VFS);
+  CrashReportInfo CrashInfo(
+      TempFiles[0], VFS,
+      C.getArgs().getLastArgValue(options::OPT_index_store_path));
 
   llvm::SmallString<128> Script(CrashInfo.Filename);
   llvm::sys::path::replace_extension(Script, "sh");
   std::error_code EC;
   llvm::raw_fd_ostream ScriptOS(Script, EC, llvm::sys::fs::CD_CreateNew);
   if (EC) {
     Diag(clang::diag::note_drv_command_failed_diag_msg)
         << "Error generating run script: " << Script << " " << EC.message();

clang/lib/Driver/Job.cpp

IsInclude = llvm::StringSwitch<bool>(Flag)
     .Cases("-idirafter", "-internal-isystem", "-iwithprefix", true)
     .Cases("-internal-externc-isystem", "-iprefix", true)
     .Cases("-iwithprefixbefore", "-isystem", "-iquote", true)
     .Cases("-isysroot", "-I", "-F", "-resource-dir", true)
     .Cases("-iframework", "-include-pch", true)
     .Default(false);
   if (IsInclude)
     return !HaveCrashVFS;
+  if (StringRef(Flag).startswith("-index-store-path"))
+    return true;
 
   // The remaining flags are treated as a single argument.
 
   // These flags are all of the form -Flag and have no second argument.
   ShouldSkip = llvm::StringSwitch<bool>(Flag)
     .Cases("-M", "-MM", "-MG", "-MP", "-MD", true)
     .Case("-MMD", true)
     .Default(false);
void Command::Print(raw_ostream &OS, const char *Terminator, bool Quote,
   ArrayRef<const char *> Args = Arguments;
   SmallVector<const char *, 128> ArgsRespFile;
   if (ResponseFile != nullptr) {
     buildArgvForResponseFile(ArgsRespFile);
     Args = ArrayRef<const char *>(ArgsRespFile).slice(1); // no executable name
   }
 
   bool HaveCrashVFS = CrashInfo && !CrashInfo->VFSPath.empty();
+  bool HaveIndexStorePath = CrashInfo && !CrashInfo->IndexStorePath.empty();
   for (size_t i = 0, e = Args.size(); i < e; ++i) {
     const char *const Arg = Args[i];
 
     if (CrashInfo) {
       int NumArgs = 0;
       bool IsInclude = false;
       if (skipArgs(Arg, HaveCrashVFS, NumArgs, IsInclude)) {
         i += NumArgs - 1;
if (CrashInfo && HaveCrashVFS) {
 
     std::string ModCachePath = "-fmodules-cache-path=";
     ModCachePath.append(RelModCacheDir.c_str());
 
     OS << ' ';
     printArg(OS, ModCachePath, Quote);
   }
 
+  if (CrashInfo && HaveIndexStorePath) {
+    SmallString<128> IndexStoreDir;
+
+    if (HaveCrashVFS) {
+      IndexStoreDir = llvm::sys::path::parent_path(
+          llvm::sys::path::parent_path(CrashInfo->VFSPath));
+      llvm::sys::path::append(IndexStoreDir, "index-store");
+    } else {
+      IndexStoreDir = "index-store";
+    }
+
+    OS << ' ';
+    printArg(OS, "-index-store-path", Quote);
+    OS << ' ';
+    printArg(OS, IndexStoreDir.c_str(), Quote);
+  }
+
   if (ResponseFile != nullptr) {
     OS << "\n Arguments passed via response file:\n";
     writeResponseFile(OS);
     // Avoiding duplicated newline terminator, since FileLists are
     // newline-separated.
     if (Creator.getResponseFilesSupport() != Tool::RF_FileList)
       OS << "\n";
     OS << " (end of response file)";

clang/lib/Driver/ToolChains/Clang.cpp

#endif
   // Pass the path to compiler resource files.
   CmdArgs.push_back("-resource-dir");
   CmdArgs.push_back(D.ResourceDir.c_str());
 
   Args.AddLastArg(CmdArgs, options::OPT_working_directory);
 
   RenderARCMigrateToolOptions(D, Args, CmdArgs);
 
+  if (Args.hasArg(options::OPT_index_store_path)) {
+    Args.AddLastArg(CmdArgs, options::OPT_index_store_path);
+    Args.AddLastArg(CmdArgs, options::OPT_index_ignore_system_symbols);
+    Args.AddLastArg(CmdArgs, options::OPT_index_record_codegen_name);
+
+    // If '-o' is passed along with '-fsyntax-only' pass it along the cc1
+    // invocation so that the index action knows what the out file is.
+    if (isa<CompileJobAction>(JA) && JA.getType() == types::TY_Nothing) {
+      Args.AddLastArg(CmdArgs, options::OPT_o);
+    }
+  }
+
+  if (const char *IdxStorePath = ::getenv("CLANG_PROJECT_INDEX_PATH")) {
+    CmdArgs.push_back("-index-store-path");
+    CmdArgs.push_back(IdxStorePath);
+    CmdArgs.push_back("-index-ignore-system-symbols");
+    CmdArgs.push_back("-index-record-codegen-name");
+  }
+
   // Add preprocessing options like -I, -D, etc. if we are using the
   // preprocessor.
   //
   // FIXME: Support -fpreprocessed
   if (types::getPreprocessedType(InputType) != types::TY_INVALID)
     AddPreprocessingOptions(C, JA, D, Args, CmdArgs, Output, Inputs);
 
   // Don't warn about "clang -c -DPIC -fPIC test.i" because libtool.m4 assumes

clang/lib/Frontend/CompilerInstance.cpp

compileModuleImpl(CompilerInstance &ImportingInstance, SourceLocation ImportLoc,
   Instance.setFileManager(&ImportingInstance.getFileManager());
   Instance.createSourceManager(Instance.getFileManager());
   SourceManager &SourceMgr = Instance.getSourceManager();
   SourceMgr.setModuleBuildStack(
     ImportingInstance.getSourceManager().getModuleBuildStack());
   SourceMgr.pushModuleBuildStack(ModuleName,
     FullSourceLoc(ImportLoc, ImportingInstance.getSourceManager()));
 
+  // Pass along the GenModuleActionWrapper callback
+  auto wrapGenModuleAction = ImportingInstance.getGenModuleActionWrapper();
+  Instance.setGenModuleActionWrapper(wrapGenModuleAction);
+
   // If we're collecting module dependencies, we need to share a collector
   // between all of the module CompilerInstances. Other than that, we don't
   // want to produce any dependency output from the module build.
   Instance.setModuleDepCollector(ImportingInstance.getModuleDepCollector());
   Inv.getDependencyOutputOpts() = DependencyOutputOptions();
 
   ImportingInstance.getDiagnostics().Report(ImportLoc,
                                             diag::remark_module_build)
     << ModuleName << ModuleFileName;
 
   PreBuildStep(Instance);
 
   // Execute the action to actually build the module in-place. Use a separate
   // thread so that we get a stack large enough.
   llvm::CrashRecoveryContext CRC;
   CRC.RunSafelyOnThread(
       [&]() {
-        GenerateModuleFromModuleMapAction Action;
-        Instance.ExecuteAction(Action);
+        // FIXME: I have no idea what the best way to do this is, but it's
+        // probably not this. Interfaces changed upstream.
+        std::unique_ptr<FrontendAction> Action(
+            new GenerateModuleFromModuleMapAction);
+        if (wrapGenModuleAction) {
+          Action = wrapGenModuleAction(FrontendOpts, std::move(Action));
+        }
+        Instance.ExecuteAction(*Action);
       },
       DesiredStackSize);
 
   PostBuildStep(Instance);
 
   ImportingInstance.getDiagnostics().Report(ImportLoc,
                                             diag::remark_module_build_done)
     << ModuleName;

clang/lib/Frontend/CompilerInvocation.cpp

static InputKind ParseFrontendArgs(FrontendOptions &Opts, ArgList &Args,
   Opts.ObjCMTWhiteListPath = Args.getLastArgValue(OPT_objcmt_whitelist_dir_path);
 
   if (Opts.ARCMTAction != FrontendOptions::ARCMT_None &&
       Opts.ObjCMTAction != FrontendOptions::ObjCMT_None) {
     Diags.Report(diag::err_drv_argument_not_allowed_with)
       << "ARC migration" << "ObjC migration";
   }
 
+  Opts.IndexStorePath = Args.getLastArgValue(OPT_index_store_path);
+  Opts.IndexIgnoreSystemSymbols = Args.hasArg(OPT_index_ignore_system_symbols);
+  Opts.IndexRecordCodegenName = Args.hasArg(OPT_index_record_codegen_name);
+
   InputKind DashX(InputKind::Unknown);
   if (const Arg *A = Args.getLastArg(OPT_x)) {
     StringRef XValue = A->getValue();
 
     // Parse suffixes: '<lang>(-header|[-module-map][-cpp-output])'.
     // FIXME: Supporting '<lang>-header-cpp-output' would be useful.
     bool Preprocessed = XValue.consume_back("-cpp-output");
     bool ModuleMap = XValue.consume_back("-module-map");

clang/lib/FrontendTool/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   Option
   Support
   )
 
 set(link_libs
   clangBasic
   clangCodeGen
   clangDriver
   clangFrontend
+  clangIndex
+  clangIndexWhileBuilding
   clangRewriteFrontend
   )
 
 if(CLANG_ENABLE_ARCMT)
   list(APPEND link_libs
     clangARCMigrate
     )
 endif()

clang/lib/FrontendTool/ExecuteCompilerInvocation.cpp

 #include "clang/Driver/Options.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/CompilerInvocation.h"
 #include "clang/Frontend/FrontendActions.h"
 #include "clang/Frontend/FrontendDiagnostic.h"
 #include "clang/Frontend/FrontendPluginRegistry.h"
 #include "clang/Frontend/Utils.h"
 #include "clang/FrontendTool/Utils.h"
+#include "clang/Index/GenerateIndexAction.h"
+#include "clang/Index/IndexWhileBuilding/EmitIndexAction.h"
 #include "clang/Rewrite/Frontend/FrontendActions.h"
 #include "clang/StaticAnalyzer/Frontend/FrontendActions.h"
 #include "llvm/Option/OptTable.h"
 #include "llvm/Option/Option.h"
 #include "llvm/Support/BuryPointer.h"
 #include "llvm/Support/DynamicLibrary.h"
 #include "llvm/Support/ErrorHandling.h"
 using namespace clang;
if (CI.getFrontendOpts().ProgramAction != frontend::MigrateSource &&
     if (FEOpts.ObjCMTAction != FrontendOptions::ObjCMT_None) {
       Act = llvm::make_unique<arcmt::ObjCMigrateAction>(std::move(Act),
                                                         FEOpts.MTMigrateDir,
                                                         FEOpts.ObjCMTAction);
     }
   }
 #endif
 
+  if (!FEOpts.IndexStorePath.empty()) {
+    auto ActionFactory = [](const FrontendOptions &Opts,
+                            std::unique_ptr<FrontendAction> Action) {
+      return llvm::make_unique<index::IndexActionWrapper>(
+          llvm::make_unique<index::EmitIndexAction>(
+              index::parseIndexGenerationOptions(Opts),
+              index::parseEmitIndexOptions(Opts)),
+          std::move(Action));
+    };
+
+    Act = ActionFactory(FEOpts, std::move(Act));
+
+    CI.setGenModuleActionWrapper(ActionFactory);
+  }
+
   // If there are any AST files to merge, create a frontend action
   // adaptor to perform the merge.
   if (!FEOpts.ASTMergeFiles.empty())
     Act = llvm::make_unique<ASTMergeAction>(std::move(Act),
                                             FEOpts.ASTMergeFiles);
 
   return Act;
 }

clang/lib/Index/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
+  BitReader
   Core
   Support
   )
 
 add_clang_library(clangIndex
   CodegenNameGenerator.cpp
   CommentToXML.cpp
-  FileIndexRecord.cpp
+  GenerateIndexAction.cpp
+  IndexAction.cpp
   IndexBody.cpp
   IndexDecl.cpp
-  IndexingAction.cpp
   IndexingContext.cpp
+  IndexOptions.cpp
   IndexSymbol.cpp
   IndexTypeSourceInfo.cpp
   USRGeneration.cpp
 
   ADDITIONAL_HEADERS
-  IndexingContext.h
   SimpleFormatContext.h
 
   LINK_LIBS
   clangAST
   clangBasic
   clangFormat
   clangFrontend
   clangLex
   clangRewrite
   clangSerialization
   clangToolingCore
   )
+
+add_subdirectory(IndexWhileBuilding)
 No newline at end of file

clang/lib/Index/CodegenNameGenerator.cpp

 //===- CodegenNameGenerator.cpp - Codegen name generation -----------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Determines the name that the symbol will get for code generation.
 //
 //===----------------------------------------------------------------------===//
 
 #include "clang/Index/CodegenNameGenerator.h"
 #include "clang/AST/ASTContext.h"
+#include "clang/AST/DeclCXX.h"
+#include "clang/AST/DeclObjC.h"
+#include "clang/AST/VTableBuilder.h"
+#include "llvm/IR/Mangler.h"
 
 using namespace clang;
 using namespace clang::index;
 
-CodegenNameGenerator::CodegenNameGenerator(ASTContext &Ctx)
-  : Impl(new ASTNameGenerator(Ctx)) {
+struct CodegenNameGenerator::Implementation {
+  std::unique_ptr<MangleContext> MC;
+  llvm::DataLayout DL;
+
+  Implementation(ASTContext &Ctx)
+      : MC(Ctx.createMangleContext()), DL(Ctx.getTargetInfo().getDataLayout()) {
+  }
+
+  bool writeName(const Decl *D, raw_ostream &OS) {
+    // First apply frontend mangling.
+    SmallString<128> FrontendBuf;
+    llvm::raw_svector_ostream FrontendBufOS(FrontendBuf);
+    if (auto *FD = dyn_cast<FunctionDecl>(D)) {
+      if (FD->isDependentContext())
+        return true;
+      if (writeFuncOrVarName(FD, FrontendBufOS))
+        return true;
+    } else if (auto *VD = dyn_cast<VarDecl>(D)) {
+      if (writeFuncOrVarName(VD, FrontendBufOS))
+        return true;
+    } else if (auto *MD = dyn_cast<ObjCMethodDecl>(D)) {
+      MC->mangleObjCMethodNameWithoutSize(MD, OS);
+      return false;
+    } else if (auto *ID = dyn_cast<ObjCInterfaceDecl>(D)) {
+      writeObjCClassName(ID, FrontendBufOS);
+    } else {
+      return true;
+    }
+
+    // Now apply backend mangling.
+    llvm::Mangler::getNameWithPrefix(OS, FrontendBufOS.str(), DL);
+    return false;
+  }
+
+  std::string getName(const Decl *D) {
+    std::string Name;
+    {
+      llvm::raw_string_ostream OS(Name);
+      // FIXME error handling
+      writeName(D, OS);
+    }
+    return Name;
+  }
+
+  enum ObjCKind {
+    ObjCClass,
+    ObjCMetaclass,
+  };
+
+  std::vector<std::string> getAllManglings(const ObjCContainerDecl *OCD) {
+    StringRef ClassName;
+    if (const auto *OID = dyn_cast<ObjCInterfaceDecl>(OCD))
+      ClassName = OID->getObjCRuntimeNameAsString();
+    else if (const auto *OID = dyn_cast<ObjCImplementationDecl>(OCD))
+      ClassName = OID->getObjCRuntimeNameAsString();
+
+    if (ClassName.empty())
+      return {};
+
+    auto Mangle = [&](ObjCKind Kind, StringRef ClassName) -> std::string {
+      SmallString<40> Mangled;
+      auto Prefix = getClassSymbolPrefix(Kind, OCD->getASTContext());
+      llvm::Mangler::getNameWithPrefix(Mangled, Prefix + ClassName, DL);
+      return Mangled.str();
+    };
+
+    return {
+        Mangle(ObjCClass, ClassName),
+        Mangle(ObjCMetaclass, ClassName),
+    };
+  }
+
+  std::vector<std::string> getAllManglings(const Decl *D) {
+    if (const auto *OCD = dyn_cast<ObjCContainerDecl>(D))
+      return getAllManglings(OCD);
+
+    if (!(isa<CXXRecordDecl>(D) || isa<CXXMethodDecl>(D)))
+      return {};
+
+    const NamedDecl *ND = cast<NamedDecl>(D);
+
+    ASTContext &Ctx = ND->getASTContext();
+    std::unique_ptr<MangleContext> M(Ctx.createMangleContext());
+
+    std::vector<std::string> Manglings;
+
+    auto hasDefaultCXXMethodCC = [](ASTContext &C, const CXXMethodDecl *MD) {
+      auto DefaultCC = C.getDefaultCallingConvention(/*IsVariadic=*/false,
+                                                     /*IsCSSMethod=*/true);
+      auto CC = MD->getType()->getAs<FunctionProtoType>()->getCallConv();
+      return CC == DefaultCC;
+    };
+
+    if (const auto *CD = dyn_cast_or_null<CXXConstructorDecl>(ND)) {
+      Manglings.emplace_back(getMangledStructor(CD, Ctor_Base));
+
+      if (Ctx.getTargetInfo().getCXXABI().isItaniumFamily())
+        if (!CD->getParent()->isAbstract())
+          Manglings.emplace_back(getMangledStructor(CD, Ctor_Complete));
+
+      if (Ctx.getTargetInfo().getCXXABI().isMicrosoft())
+        if (CD->hasAttr<DLLExportAttr>() && CD->isDefaultConstructor())
+          if (!(hasDefaultCXXMethodCC(Ctx, CD) && CD->getNumParams() == 0))
+            Manglings.emplace_back(getMangledStructor(CD, Ctor_DefaultClosure));
+    } else if (const auto *DD = dyn_cast_or_null<CXXDestructorDecl>(ND)) {
+      Manglings.emplace_back(getMangledStructor(DD, Dtor_Base));
+      if (Ctx.getTargetInfo().getCXXABI().isItaniumFamily()) {
+        Manglings.emplace_back(getMangledStructor(DD, Dtor_Complete));
+        if (DD->isVirtual())
+          Manglings.emplace_back(getMangledStructor(DD, Dtor_Deleting));
+      }
+    } else if (const auto *MD = dyn_cast_or_null<CXXMethodDecl>(ND)) {
+      Manglings.emplace_back(getName(ND));
+      if (MD->isVirtual())
+        if (const auto *TIV = Ctx.getVTableContext()->getThunkInfo(MD))
+          for (const auto &T : *TIV)
+            Manglings.emplace_back(getMangledThunk(MD, T));
+    }
+
+    return Manglings;
+  }
+
+private:
+  bool writeFuncOrVarName(const NamedDecl *D, raw_ostream &OS) {
+    if (MC->shouldMangleDeclName(D)) {
+      if (const auto *CtorD = dyn_cast<CXXConstructorDecl>(D))
+        MC->mangleCXXCtor(CtorD, Ctor_Complete, OS);
+      else if (const auto *DtorD = dyn_cast<CXXDestructorDecl>(D))
+        MC->mangleCXXDtor(DtorD, Dtor_Complete, OS);
+      else
+        MC->mangleName(D, OS);
+      return false;
+    } else {
+      IdentifierInfo *II = D->getIdentifier();
+      if (!II)
+        return true;
+      OS << II->getName();
+      return false;
     }
+  }
+
+  void writeObjCClassName(const ObjCInterfaceDecl *D, raw_ostream &OS) {
+    OS << getClassSymbolPrefix(ObjCClass, D->getASTContext());
+    OS << D->getObjCRuntimeNameAsString();
+  }
+
+  static StringRef getClassSymbolPrefix(ObjCKind Kind,
+                                        const ASTContext &Context) {
+    if (Context.getLangOpts().ObjCRuntime.isGNUFamily())
+      return Kind == ObjCMetaclass ? "_OBJC_METACLASS_" : "_OBJC_CLASS_";
+    return Kind == ObjCMetaclass ? "OBJC_METACLASS_$_" : "OBJC_CLASS_$_";
+  }
+
+  std::string getMangledStructor(const NamedDecl *ND, unsigned StructorType) {
+    std::string FrontendBuf;
+    llvm::raw_string_ostream FOS(FrontendBuf);
+
+    if (const auto *CD = dyn_cast_or_null<CXXConstructorDecl>(ND))
+      MC->mangleCXXCtor(CD, static_cast<CXXCtorType>(StructorType), FOS);
+    else if (const auto *DD = dyn_cast_or_null<CXXDestructorDecl>(ND))
+      MC->mangleCXXDtor(DD, static_cast<CXXDtorType>(StructorType), FOS);
+
+    std::string BackendBuf;
+    llvm::raw_string_ostream BOS(BackendBuf);
+
+    llvm::Mangler::getNameWithPrefix(BOS, FOS.str(), DL);
 
-CodegenNameGenerator::~CodegenNameGenerator() {
+    return BOS.str();
   }
 
+  std::string getMangledThunk(const CXXMethodDecl *MD, const ThunkInfo &T) {
+    std::string FrontendBuf;
+    llvm::raw_string_ostream FOS(FrontendBuf);
+
+    MC->mangleThunk(MD, T, FOS);
+
+    std::string BackendBuf;
+    llvm::raw_string_ostream BOS(BackendBuf);
+
+    llvm::Mangler::getNameWithPrefix(BOS, FOS.str(), DL);
+
+    return BOS.str();
+  }
+};
+
+CodegenNameGenerator::CodegenNameGenerator(ASTContext &Ctx)
+    : Impl(new Implementation(Ctx)) {}
+
+CodegenNameGenerator::~CodegenNameGenerator() { delete Impl; }
+
 bool CodegenNameGenerator::writeName(const Decl *D, raw_ostream &OS) {
   return Impl->writeName(D, OS);
 }
 
 std::string CodegenNameGenerator::getName(const Decl *D) {
   return Impl->getName(D);
 }
 
 std::vector<std::string> CodegenNameGenerator::getAllManglings(const Decl *D) {
   return Impl->getAllManglings(D);
 }

clang/lib/Index/FileIndexRecord.h

-//===--- FileIndexRecord.h - Index data per file ----------------*- C++ -*-===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-
-#ifndef LLVM_CLANG_LIB_INDEX_FILEINDEXRECORD_H
-#define LLVM_CLANG_LIB_INDEX_FILEINDEXRECORD_H
-
-#include "clang/Basic/SourceLocation.h"
-#include "clang/Index/DeclOccurrence.h"
-#include "clang/Index/IndexSymbol.h"
-#include "llvm/ADT/ArrayRef.h"
-#include "llvm/ADT/SmallVector.h"
-#include <vector>
-
-namespace clang {
-class IdentifierInfo;
-
-namespace index {
-
-/// Stores the declaration occurrences seen in a particular source or header
-/// file of a translation unit
-class FileIndexRecord {
-private:
-  FileID FID;
-  bool IsSystem;
-  std::vector<DeclOccurrence> Decls;
-
-public:
-  FileIndexRecord(FileID FID, bool IsSystem) : FID(FID), IsSystem(IsSystem) {}
-
-  ArrayRef<DeclOccurrence> getDeclOccurrencesSortedByOffset() const {
-    return Decls;
-  }
-
-  FileID getFileID() const { return FID; }
-  bool isSystem() const { return IsSystem; }
-
-  /// Adds an occurrence of the canonical declaration \c D at the supplied
-  /// \c Offset
-  ///
-  /// \param Roles the roles the occurrence fulfills in this position.
-  /// \param Offset the offset in the file of this occurrence.
-  /// \param D the canonical declaration this is an occurrence of.
-  /// \param Relations the set of symbols related to this occurrence.
-  void addDeclOccurence(SymbolRoleSet Roles, unsigned Offset, const Decl *D,
-                        ArrayRef<SymbolRelation> Relations);
-  void print(llvm::raw_ostream &OS) const;
-};
-
-} // end namespace index
-} // end namespace clang
-
-#endif // LLVM_CLANG_LIB_INDEX_FILEINDEXRECORD_H

clang/lib/Index/FileIndexRecord.cpp

-//===--- FileIndexRecord.cpp - Index data per file --------------*- C++ -*-===//
-//
-//                     The LLVM Compiler Infrastructure
-//
-// This file is distributed under the University of Illinois Open Source
-// License. See LICENSE.TXT for details.
-//
-//===----------------------------------------------------------------------===//
-
-#include "FileIndexRecord.h"
-#include "clang/AST/ASTContext.h"
-#include "clang/AST/DeclTemplate.h"
-#include "llvm/ADT/SmallString.h"
-#include "llvm/Support/Path.h"
-
-using namespace clang;
-using namespace clang::index;
-
-void FileIndexRecord::addDeclOccurence(SymbolRoleSet Roles, unsigned Offset,
-                                       const Decl *D,
-                                       ArrayRef<SymbolRelation> Relations) {
-  assert(D->isCanonicalDecl() &&
-         "Occurrences should be associated with their canonical decl");
-
-  auto IsNextOccurence = [&]() -> bool {
-    if (Decls.empty())
-      return true;
-    auto &Last = Decls.back();
-    return Last.Offset < Offset;
-  };
-
-  if (IsNextOccurence()) {
-    Decls.emplace_back(Roles, Offset, D, Relations);
-    return;
-  }
-
-  DeclOccurrence NewInfo(Roles, Offset, D, Relations);
-  // We keep Decls in order as we need to access them in this order in all cases.
-  auto It = llvm::upper_bound(Decls, NewInfo);
-  Decls.insert(It, std::move(NewInfo));
-}
-
-void FileIndexRecord::print(llvm::raw_ostream &OS) const {
-  OS << "DECLS BEGIN ---\n";
-  for (auto &DclInfo : Decls) {
-    const Decl *D = DclInfo.Dcl;
-    SourceManager &SM = D->getASTContext().getSourceManager();
-    SourceLocation Loc = SM.getFileLoc(D->getLocation());
-    PresumedLoc PLoc = SM.getPresumedLoc(Loc);
-    OS << llvm::sys::path::filename(PLoc.getFilename()) << ':' << PLoc.getLine()
-       << ':' << PLoc.getColumn();
-
-    if (auto ND = dyn_cast<NamedDecl>(D)) {
-      OS << ' ' << ND->getNameAsString();
-    }
-
-    OS << '\n';
-  }
-  OS << "DECLS END ---\n";
-}

clang/lib/Index/GenerateIndexAction.cpp

+//===- GenerateIndexAction.cpp --------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Index/GenerateIndexAction.h"
+
+#include "clang/Basic/Version.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Frontend/MultiplexConsumer.h"
+#include "clang/Index/IndexPPCallbacks.h"
+#include "clang/Lex/HeaderSearch.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Serialization/ASTReader.h"
+#include "llvm/ADT/ScopeExit.h"
+#include "llvm/Support/Errc.h"
+
+namespace clang {
+namespace index {
+
+static bool topLevelDeclVisitor(void *context, const Decl *D) {
+  IndexingContext &IndexCtx = *static_cast<IndexingContext *>(context);
+  return IndexCtx.indexTopLevelDecl(D);
+}
+
+static void indexTranslationUnit(ASTUnit &Unit, IndexingContext &IndexCtx) {
+  Unit.visitLocalTopLevelDecls(&IndexCtx, topLevelDeclVisitor);
+}
+
+static void indexPreprocessorMacros(const Preprocessor &PP,
+                                    IndexDataConsumer &DataConsumer) {
+  for (const auto &M : PP.macros())
+    if (MacroDirective *MD = M.second.getLatest())
+      DataConsumer.handleMacroOccurence(
+          M.first, MD->getMacroInfo(),
+          static_cast<unsigned>(SymbolRole::Definition), MD->getLocation());
+}
+
+void indexASTUnit(ASTUnit &Unit, IndexDataConsumer &DataConsumer,
+                  const IndexingOptions &Opts) {
+  IndexingContext IndexCtx(Opts, DataConsumer);
+  IndexCtx.setASTContext(Unit.getASTContext());
+  DataConsumer.initialize(Unit.getASTContext());
+  DataConsumer.setPreprocessor(Unit.getPreprocessorPtr());
+
+  if (Opts.IndexMacrosInPreprocessor)
+    indexPreprocessorMacros(Unit.getPreprocessor(), DataConsumer);
+  indexTranslationUnit(Unit, IndexCtx);
+  DataConsumer.finish();
+}
+
+void indexTopLevelDecls(ASTContext &Ctx, Preprocessor &PP,
+                        ArrayRef<const Decl *> Decls,
+                        IndexDataConsumer &DataConsumer,
+                        const IndexingOptions &Opts) {
+  IndexingContext IndexCtx(Opts, DataConsumer);
+  IndexCtx.setASTContext(Ctx);
+
+  DataConsumer.initialize(Ctx);
+
+  if (Opts.IndexMacrosInPreprocessor)
+    indexPreprocessorMacros(PP, DataConsumer);
+
+  for (const Decl *D : Decls)
+    IndexCtx.indexTopLevelDecl(D);
+  DataConsumer.finish();
+}
+
+std::unique_ptr<PPCallbacks> indexMacrosCallback(IndexDataConsumer &Consumer,
+                                                 const IndexingOptions &Opts) {
+  return llvm::make_unique<IndexPPCallbacks>(
+      std::make_shared<IndexingContext>(Opts, Consumer));
+}
+
+void indexModuleFile(serialization::ModuleFile &Mod, ASTReader &Reader,
+                     IndexDataConsumer &DataConsumer,
+                     const IndexingOptions &Opts) {
+  ASTContext &Ctx = Reader.getContext();
+  IndexingContext IndexCtx(Opts, DataConsumer);
+  IndexCtx.setASTContext(Ctx);
+  DataConsumer.initialize(Ctx);
+
+  if (Opts.IndexMacrosInPreprocessor)
+    indexPreprocessorMacros(Reader.getPreprocessor(), DataConsumer);
+
+  for (const Decl *D : Reader.getModuleFileLevelDecls(Mod)) {
+    IndexCtx.indexTopLevelDecl(D);
+  }
+  DataConsumer.finish();
+}
+
+class IndexASTConsumer : public ASTConsumer {
+  std::shared_ptr<IndexingContext> Ctx;
+
+public:
+  IndexASTConsumer(std::shared_ptr<IndexingContext> Ctx, CompilerInstance &CI)
+      : Ctx(Ctx) {
+    CI.getPreprocessor().addPPCallbacks(
+        llvm::make_unique<IndexPPCallbacks>(Ctx));
+  }
+
+protected:
+  void Initialize(ASTContext &Context) override {
+    Ctx->setASTContext(Context);
+    Ctx->getDataConsumer().initialize(Context);
+  }
+
+  bool HandleTopLevelDecl(DeclGroupRef DG) override {
+    return Ctx->indexDeclGroupRef(DG);
+  }
+
+  void HandleInterestingDecl(DeclGroupRef DG) override {
+    // Ignore deserialized decls.
+  }
+
+  void HandleTopLevelDeclInObjCContainer(DeclGroupRef DG) override {
+    Ctx->indexDeclGroupRef(DG);
+  }
+
+  void HandleTranslationUnit(ASTContext &Ctx) override {}
+};
+
+GenerateIndexAction::GenerateIndexAction(
+    std::shared_ptr<IndexDataConsumer> DataConsumer,
+    const IndexingOptions &Opts)
+    : DataConsumer(DataConsumer),
+      IndexCtx(new IndexingContext(Opts, *DataConsumer)) {}
+
+std::unique_ptr<ASTConsumer>
+GenerateIndexAction::CreateASTConsumer(CompilerInstance &CI, StringRef InFile) {
+
+  IndexCtx->getDataConsumer().setPreprocessor(CI.getPreprocessorPtr());
+  return llvm::make_unique<IndexASTConsumer>(IndexCtx, CI);
+}
+
+void GenerateIndexAction::EndSourceFileAction() {
+  if (DataConsumer)
+    DataConsumer->finish();
+}
+
+} // namespace index
+} // namespace clang
 No newline at end of file

clang/lib/Index/IndexAction.cpp

+//===- IndexAction.cpp - Interface for index actions ----------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Index/IndexAction.h"
+
+#include "clang/Frontend/MultiplexConsumer.h"
+
+#include <vector>
+
+namespace clang {
+namespace index {
+
+// Using distinct names for parameters in order to avoid confusion with name
+// resolution in assert.
+IndexActionWrapper::IndexActionWrapper(
+    std::unique_ptr<IndexAction> IndexingActionParam,
+    std::unique_ptr<FrontendAction> WrappedActionParam)
+    : WrapperFrontendAction(std::move(WrappedActionParam)),
+      IndexingAction(std::move(IndexingActionParam)) {
+  assert(IndexingAction);
+}
+
+std::unique_ptr<ASTConsumer>
+IndexActionWrapper::CreateASTConsumer(CompilerInstance &CI, StringRef InFile) {
+  std::vector<std::unique_ptr<ASTConsumer>> Consumers;
+  auto OtherConsumer = WrapperFrontendAction::CreateASTConsumer(CI, InFile);
+  if (OtherConsumer)
+    Consumers.push_back(std::move(OtherConsumer));
+  Consumers.push_back(IndexingAction->CreateASTConsumer(CI, InFile));
+
+  return llvm::make_unique<MultiplexConsumer>(std::move(Consumers));
+}
+
+void IndexActionWrapper::EndSourceFileAction() {
+  WrapperFrontendAction::EndSourceFileAction();
+  IndexingAction->EndSourceFileAction();
+}
+
+} // namespace index
+} // namespace clang

clang/lib/Index/IndexBody.cpp

 //===- IndexBody.cpp - Indexing statements --------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
-#include "IndexingContext.h"
-#include "clang/AST/RecursiveASTVisitor.h"
 #include "clang/AST/ASTLambda.h"
+#include "clang/AST/RecursiveASTVisitor.h"
+#include "clang/Index/IndexingContext.h"
 
 using namespace clang;
 using namespace clang::index;
 
 namespace {
 
 class BodyIndexer : public RecursiveASTVisitor<BodyIndexer> {
   IndexingContext &IndexCtx;

clang/lib/Index/IndexDecl.cpp

 //===- IndexDecl.cpp - Indexing declarations ------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
-#include "IndexingContext.h"
-#include "clang/Index/IndexDataConsumer.h"
 #include "clang/AST/DeclVisitor.h"
+#include "clang/Index/IndexDataConsumer.h"
+#include "clang/Index/IndexingContext.h"
 
 using namespace clang;
 using namespace index;
 
 #define TRY_DECL(D,CALL_EXPR)                                                  \
   do {                                                                         \
     if (!IndexCtx.shouldIndex(D)) return true;                                 \
     if (!CALL_EXPR)                                                            \

clang/lib/Index/IndexOptions.cpp

+//===--- IndexOptions.cpp - Options for indexing ----------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Index/IndexOptions.h"
+#include "clang/Index/IndexWhileBuilding/IndexDataFormat.h"
+#include "llvm/Support/Path.h"
+
+namespace clang {
+namespace index {
+
+IndexingOptions parseIndexGenerationOptions(const FrontendOptions &FEOpts) {
+  IndexingOptions IndexOpts;
+  if (FEOpts.IndexIgnoreSystemSymbols) {
+    IndexOpts.SystemSymbolFilter =
+        index::IndexingOptions::SystemSymbolFilterKind::None;
+  }
+  return IndexOpts;
+}
+
+} // namespace index
+} // namespace clang

clang/lib/Index/IndexTypeSourceInfo.cpp

 //===- IndexTypeSourceInfo.cpp - Indexing types ---------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
-#include "IndexingContext.h"
 #include "clang/AST/RecursiveASTVisitor.h"
+#include "clang/Index/IndexingContext.h"
 
 using namespace clang;
 using namespace index;
 
 namespace {
 
 class TypeIndexer : public RecursiveASTVisitor<TypeIndexer> {
   IndexingContext &IndexCtx;

clang/lib/Index/IndexWhileBuilding/BitstreamUtil.h

+//===--- BitstreamUtil.h - Bitstream manipulation utils -------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_LIB_INDEX_BITSTREAMUTIL_H
+#define LLVM_CLANG_LIB_INDEX_BITSTREAMUTIL_H
+
+#include "llvm/ADT/SmallVector.h"
+
+namespace llvm {
+class BitstreamWriter;
+}
+
+namespace clang {
+namespace index {
+namespace store {
+
+void emitBlockID(unsigned ID, const char *Name, llvm::BitstreamWriter &Stream,
+                 llvm::SmallVectorImpl<uint64_t> &Record);
+
+void emitRecordID(unsigned ID, const char *Name, llvm::BitstreamWriter &Stream,
+                  llvm::SmallVectorImpl<uint64_t> &Record);
+
+} // end namespace store
+} // end namespace index
+} // end namespace clang
+
+#endif

clang/lib/Index/IndexWhileBuilding/BitstreamUtil.cpp

+//===--- BitstreamUtil.cpp - Bitstream manipulation utils -----------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "BitstreamUtil.h"
+
+#include "llvm/Bitstream/BitCodes.h"
+#include "llvm/Bitstream/BitstreamWriter.h"
+
+using namespace clang;
+using namespace llvm;
+
+namespace clang {
+namespace index {
+namespace store {
+
+void emitBlockID(unsigned ID, const char *Name, llvm::BitstreamWriter &Stream,
+                 llvm::SmallVectorImpl<uint64_t> &Record) {
+  Record.clear();
+  Record.push_back(ID);
+  Stream.EmitRecord(bitc::BLOCKINFO_CODE_SETBID, Record);
+
+  // Emit the block name if present.
+  if (!Name || Name[0] == 0)
+    return;
+  Record.clear();
+  while (*Name)
+    Record.push_back(*Name++);
+  Stream.EmitRecord(bitc::BLOCKINFO_CODE_BLOCKNAME, Record);
+}
+
+void emitRecordID(unsigned ID, const char *Name, llvm::BitstreamWriter &Stream,
+                  llvm::SmallVectorImpl<uint64_t> &Record) {
+  Record.clear();
+  Record.push_back(ID);
+  while (*Name)
+    Record.push_back(*Name++);
+  Stream.EmitRecord(bitc::BLOCKINFO_CODE_SETRECORDNAME, Record);
+}
+
+} // end namespace store
+} // end namespace index
+} // end namespace clang

clang/lib/Index/IndexWhileBuilding/BitstreamVisitor.h

+//===--- BitstreamVisitor.h - Helper for reading a bitstream --------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_LIB_INDEX_BITSTREAMVISITOR_H
+#define LLVM_CLANG_LIB_INDEX_BITSTREAMVISITOR_H
+
+#include "clang/Basic/LLVM.h"
+#include "clang/Serialization/ASTReader.h"
+#include "llvm/Bitstream/BitstreamReader.h"
+#include <string>
+
+namespace clang {
+namespace index {
+namespace store {
+
+/// Helper class that saves the current stream position and
+/// then restores it when destroyed.
+struct SavedStreamPosition {
+  explicit SavedStreamPosition(llvm::BitstreamCursor &Cursor)
+      : Cursor(Cursor), Offset(Cursor.GetCurrentBitNo()) {}
+
+  ~SavedStreamPosition() {
+    if (llvm::Error Err = Cursor.JumpToBit(Offset))
+      llvm::report_fatal_error("SavedStreamPosition failed jumping: " +
+                               toString(std::move(Err)));
+  }
+
+private:
+  llvm::BitstreamCursor &Cursor;
+  uint64_t Offset;
+};
+
+enum class StreamVisit { Continue, Skip, Abort };
+
+template <typename ImplClass> class BitstreamVisitor {
+  SmallVector<unsigned, 4> BlockStack;
+
+protected:
+  llvm::BitstreamCursor &Stream;
+  Optional<llvm::BitstreamBlockInfo> BlockInfo;
+  std::string *Error;
+
+public:
+  BitstreamVisitor(llvm::BitstreamCursor &Stream) : Stream(Stream) {}
+
+  StreamVisit visitBlock(unsigned ID) { return StreamVisit::Continue; }
+
+  bool visit(std::string &Error) {
+    this->Error = &Error;
+
+    ASTReader::RecordData Record;
+    while (1) {
+      Expected<llvm::BitstreamEntry> MaybeEntry =
+          Stream.advance(llvm::BitstreamCursor::AF_DontPopBlockAtEnd);
+      if (!MaybeEntry) {
+        Error = toString(MaybeEntry.takeError());
+        return false;
+      }
+      llvm::BitstreamEntry Entry = MaybeEntry.get();
+
+      switch (Entry.Kind) {
+      case llvm::BitstreamEntry::Error:
+        Error = "malformed serialization";
+        return false;
+
+      case llvm::BitstreamEntry::EndBlock:
+        if (BlockStack.empty())
+          return true;
+        BlockStack.pop_back();
+        if (Stream.ReadBlockEnd()) {
+          Error = "malformed serialization";
+          return false;
+        }
+        if (Stream.AtEndOfStream())
+          return true;
+        break;
+
+      case llvm::BitstreamEntry::SubBlock: {
+        if (Entry.ID == llvm::bitc::BLOCKINFO_BLOCK_ID) {
+          Expected<Optional<llvm::BitstreamBlockInfo>> MaybeBlockInfo =
+              Stream.ReadBlockInfoBlock();
+          if (!MaybeBlockInfo) {
+            Error = toString(MaybeBlockInfo.takeError());
+            return false;
+          }
+          BlockInfo = MaybeBlockInfo.get();
+          if (!BlockInfo) {
+            Error = "malformed BlockInfoBlock";
+            return false;
+          }
+          Stream.setBlockInfo(&*BlockInfo);
+          break;
+        }
+
+        StreamVisit Ret = static_cast<ImplClass *>(this)->visitBlock(Entry.ID);
+        switch (Ret) {
+        case StreamVisit::Continue:
+          if (Stream.EnterSubBlock(Entry.ID)) {
+            Error = "malformed block record";
+            return false;
+          }
+          if (llvm::Error Err = readBlockAbbrevs(Stream)) {
+            Error = toString(std::move(Err));
+            return false;
+          }
+          BlockStack.push_back(Entry.ID);
+          break;
+
+        case StreamVisit::Skip:
+          if (Stream.SkipBlock()) {
+            Error = "malformed serialization";
+            return false;
+          }
+          if (Stream.AtEndOfStream())
+            return true;
+          break;
+
+        case StreamVisit::Abort:
+          return false;
+        }
+        break;
+      }
+
+      case llvm::BitstreamEntry::Record: {
+        Record.clear();
+        StringRef Blob;
+        Expected<unsigned> MaybeRecID =
+            Stream.readRecord(Entry.ID, Record, &Blob);
+        if (!MaybeRecID) {
+          Error = toString(MaybeRecID.takeError());
+          return false;
+        }
+        unsigned RecID = MaybeRecID.get();
+        unsigned BlockID = BlockStack.empty() ? 0 : BlockStack.back();
+        StreamVisit Ret = static_cast<ImplClass *>(this)->visitRecord(
+            BlockID, RecID, Record, Blob);
+        switch (Ret) {
+        case StreamVisit::Continue:
+          break;
+
+        case StreamVisit::Skip:
+          if (Expected<unsigned> Skipped = Stream.skipRecord(Entry.ID)) {
+            Error = toString(Skipped.takeError());
+            return false;
+          }
+          break;
+
+        case StreamVisit::Abort:
+          return false;
+        }
+        break;
+      }
+      }
+    }
+  }
+
+  static llvm::Error readBlockAbbrevs(llvm::BitstreamCursor &Cursor) {
+    while (true) {
+      uint64_t Offset = Cursor.GetCurrentBitNo();
+      Expected<unsigned> MaybeCode = Cursor.ReadCode();
+      if (!MaybeCode)
+        return MaybeCode.takeError();
+      unsigned Code = MaybeCode.get();
+
+      // We expect all abbrevs to be at the start of the block.
+      if (Code != llvm::bitc::DEFINE_ABBREV) {
+        if (llvm::Error Err = Cursor.JumpToBit(Offset))
+          return Err;
+        return llvm::Error::success();
+      }
+      if (llvm::Error Err = Cursor.ReadAbbrevRecord())
+        return Err;
+    }
+  }
+};
+
+} // end namespace store
+} // end namespace index
+} // end namespace clang
+
+#endif

clang/lib/Index/IndexWhileBuilding/CMakeLists.txt

+set(LLVM_LINK_COMPONENTS
+  BitReader
+  BitstreamReader
+  Core
+  Support
+  )
+
+add_clang_library(clangIndexWhileBuilding
+  BitstreamUtil.cpp
+  DeclOccurrenceCollector.cpp
+  EmitIndexAction.cpp
+  EmitIndexOptions.cpp
+  IndexDataFormat.cpp
+  IndexRecordReader.cpp
+  IndexRecordWriter.cpp
+  IndexUnitReader.cpp
+  IndexUnitWriter.cpp
+  PathStorage.cpp
+
+  LINK_LIBS
+  clangAST
+  clangBasic
+  clangFormat
+  clangFrontend
+  clangIndex
+  clangLex
+  clangRewrite
+  clangSerialization
+  clangToolingCore
+  )

clang/lib/Index/IndexWhileBuilding/DeclOccurrenceCollector.cpp

+//===- DeclOccurrenceCollector.cpp - Collecting data for index records ----===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Index/IndexWhileBuilding/DeclOccurrenceCollector.h"
+
+namespace clang {
+namespace index {
+
+bool DeclOccurrenceCollector::handleDeclOccurence(
+    const Decl *D, SymbolRoleSet Roles, ArrayRef<SymbolRelation> Relations,
+    SourceLocation Loc, ASTNodeInfo ASTNode) {
+  Loc = SM.getFileLoc(Loc);
+  if (Loc.isInvalid())
+    return true;
+
+  FileID FID;
+  unsigned Offset;
+  std::tie(FID, Offset) = SM.getDecomposedLoc(Loc);
+
+  if (FID.isInvalid())
+    return true;
+
+  const FileEntry *FE = SM.getFileEntryForID(FID);
+  if (!FE) // Ignore predefined buffers.
+    return true;
+
+  IndexRecordWriter &Rec =
+      RecordByFile.insert({FID, IndexRecordWriter(IndexCtx->isSystemFile(FID))})
+          .first->second;
+  Rec.addDeclOccurence(Roles, FID, Offset, D, Relations);
+  return true;
+}
+
+} // namespace index
+} // namespace clang
 No newline at end of file

clang/lib/Index/IndexWhileBuilding/EmitIndexAction.cpp

+//===- EmitIndexAction.cpp - Index emitting action
+//-------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Index/IndexWhileBuilding/EmitIndexAction.h"
+
+#include "clang/Basic/DiagnosticFrontend.h"
+#include "clang/Basic/Version.h"
+#include "clang/Frontend/MultiplexConsumer.h"
+#include "clang/Index/IndexPPCallbacks.h"
+#include "clang/Index/IndexWhileBuilding/IndexDataFormat.h"
+#include "clang/Lex/HeaderSearch.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Serialization/ASTReader.h"
+#include "llvm/ADT/ScopeExit.h"
+#include "llvm/Support/Errc.h"
+
+namespace clang {
+namespace index {
+
+using namespace llvm;
+
+static Module *getModBeingCompiled(const CompilerInstance &CI) {
+  return CI.getLangOpts().isCompilingModule()
+             ? CI.getPreprocessor().getHeaderSearchInfo().lookupModule(
+                   CI.getLangOpts().CurrentModule, /*AllowSearch=*/false)
+             : nullptr;
+}
+
+static std::string getOutputFile(const CompilerInstance &CI) {
+  std::string Result = CI.getFrontendOpts().OutputFile;
+  return !Result.empty()
+             ? Result
+             : CI.getFrontendOpts().Inputs[0].getFile().str() + ".o";
+}
+
+static const FileEntry *getMainFile(const CompilerInstance &CI) {
+  if (CI.getLangOpts().isCompilingModule() ||
+      CI.getFrontendOpts().ProgramAction == frontend::GeneratePCH)
+    return nullptr;
+
+  SourceManager &SM = CI.getSourceManager();
+  return SM.getFileEntryForID(SM.getMainFileID());
+}
+
+class IncludePPCallbacks : public PPCallbacks {
+  IndexingContext &IndexCtx;
+  EmitIndexOptions RecordOpts;
+  IndexUnitWriter &UnitWriter;
+  SourceManager &SourceMgr;
+
+public:
+  IncludePPCallbacks(IndexingContext &indexCtx, EmitIndexOptions recordOpts,
+                     IndexUnitWriter &UnitWriter, SourceManager &SourceMgr)
+      : IndexCtx(indexCtx), RecordOpts(recordOpts), UnitWriter(UnitWriter),
+        SourceMgr(SourceMgr) {}
+
+private:
+  virtual void
+  InclusionDirective(SourceLocation InclusionDirectiveLoc, const Token &,
+                     StringRef, bool, CharSourceRange,
+                     const FileEntry *IncludedFile, StringRef, StringRef,
+                     const Module *ImportedModule,
+                     SrcMgr::CharacteristicKind FileType) override {
+    // We don't want to treat imports as includes.
+    if (ImportedModule)
+      return;
+
+    if (RecordOpts.RecordIncludes ==
+        EmitIndexOptions::IncludesRecordingKind::None)
+      return;
+
+    if (!InclusionDirectiveLoc.isFileID() || !IncludedFile ||
+        !IncludedFile->isValid())
+      return;
+
+    std::pair<FileID, unsigned> InclusionDirectiveDecLoc =
+        SourceMgr.getDecomposedExpansionLoc(InclusionDirectiveLoc);
+
+    if (RecordOpts.RecordIncludes ==
+            EmitIndexOptions::IncludesRecordingKind::UserOnly &&
+        IndexCtx.isSystemFile(InclusionDirectiveDecLoc.first))
+      return; // Ignore includes of system headers.
+
+    auto *FE = SourceMgr.getFileEntryForID(InclusionDirectiveDecLoc.first);
+    if (!FE)
+      return;
+
+    UnitWriter.addIncludedFile(
+        FE,
+        SourceMgr.getLineNumber(InclusionDirectiveDecLoc.first,
+                                InclusionDirectiveDecLoc.second),
+        IncludedFile);
+  }
+};
+
+SourceFilesDependencyCollector::SourceFilesDependencyCollector(
+    const CompilerInstance &CI, IndexingContext &indexCtx,
+    EmitIndexOptions recordOpts)
+    : CI(CI), IndexCtx(indexCtx), RecordOpts(recordOpts),
+      UnitWriter(makeIndexUnitWriter(CI, RecordOpts, getOutputFile(CI),
+                                     getMainFile(CI),
+                                     getModBeingCompiled(CI))) {}
+
+void SourceFilesDependencyCollector::attachToPreprocessor(Preprocessor &PP) {
+  DependencyCollector::attachToPreprocessor(PP);
+  PP.addPPCallbacks(llvm::make_unique<IncludePPCallbacks>(
+      IndexCtx, RecordOpts, UnitWriter, PP.getSourceManager()));
+}
+
+/// \returns either a Module that is both a submodule of UnitModule and owns the
+/// header \p FE.
+static Module *findModuleForHeader(const CompilerInstance &CI,
+                                   Module &UnitModule, const FileEntry &FE) {
+  if (auto Mod = CI.getPreprocessor()
+                     .getHeaderSearchInfo()
+                     .findModuleForHeader(&FE)
+                     .getModule())
+    if (Mod->isSubModuleOf(&UnitModule))
+      return Mod;
+  return nullptr;
+}
+
+bool SourceFilesDependencyCollector::sawDependency(StringRef Filename,
+                                                   bool FromModule,
+                                                   bool IsSystem,
+                                                   bool IsModuleFile,
+                                                   bool IsMissing) {
+  bool sawIt = DependencyCollector::sawDependency(
+      Filename, FromModule, IsSystem, IsModuleFile, IsMissing);
+  if (sawIt) {
+    if (auto *FE = CI.getSourceManager().getFileManager().getFile(Filename)) {
+      assert(FE);
+
+      Module *ModForHead = nullptr;
+      {
+        Module *Mod = getModBeingCompiled(CI);
+        if (Mod) {
+          ModForHead = findModuleForHeader(CI, *Mod, *FE);
+          UnitWriter.addModule(ModForHead);
+        }
+      }
+
+      UnitWriter.addSourceFile(FE, IsSystem || isInSysroot(Filename),
+                               ModForHead);
+    }
+  }
+  return sawIt;
+}
+
+bool SourceFilesDependencyCollector::isInSysroot(StringRef Filename) {
+  const auto &Sysroot = CI.getHeaderSearchOpts().Sysroot;
+  // FIXME: Make this portable.
+  return !Sysroot.empty() && (Sysroot != "/") && Filename.startswith(Sysroot);
+}
+
+ActionContext::ActionContext(const IndexingOptions &IndexOpts,
+                             const CompilerInstance &CI,
+                             const EmitIndexOptions &RecordOpts)
+    : CI(CI), DeclsDataCollector(CI.getSourceManager()),
+      Ctx(IndexOpts, DeclsDataCollector), DepCollector(CI, Ctx, RecordOpts),
+      RecordOpts(RecordOpts), IndexOpts(IndexOpts) {
+  DepCollector.attachToPreprocessor(CI.getPreprocessor());
+  DeclsDataCollector.setIndexingContext(Ctx);
+}
+
+// FIXME: Some better implementation.
+template <unsigned N>
+static void reportError(const CompilerInstance &CI, const char (&Format)[N],
+                        const std::string &Param1 = "",
+                        const std::string &Param2 = "") {
+  DiagnosticsEngine &Diag = CI.getDiagnostics();
+  unsigned DiagID = Diag.getCustomDiagID(DiagnosticsEngine::Error, Format);
+  if (!Param2.empty())
+    Diag.Report(DiagID) << Param1 << Param2;
+  else
+    Diag.Report(DiagID) << Param1;
+};
+
+/// Writes records in \p RecordByFile to disk and adds them as dependencies to
+/// \p UnitModule.
+void writeIndexRecordsToDisk(
+    const llvm::DenseMap<FileID, IndexRecordWriter> &RecordByFile,
+    IndexUnitWriter &UnitWriter, const CompilerInstance &CI,
+    const EmitIndexOptions &RecordOpts, const Module *UnitModule) {
+  auto &ASTCtx = CI.getASTContext();
+  PrintingPolicy Policy(ASTCtx.getLangOpts());
+  Policy.SuppressTemplateArgsInCXXConstructors = true;
+
+  std::shared_ptr<CodegenNameGenerator> CGNameGen;
+  if (RecordOpts.RecordSymbolCodeGenName)
+    CGNameGen = std::make_shared<CodegenNameGenerator>(ASTCtx);
+
+  for (const auto &It : RecordByFile) {
+    const FileID FID = It.first;
+    const IndexRecordWriter &Rec = It.second;
+    const FileEntry *FE = CI.getSourceManager().getFileEntryForID(FID);
+    assert(FE);
+
+    if (Rec.isRecordEmpty())
+      continue;
+
+    llvm::Optional<std::string> RecordFilename = Rec.writeToDisk(
+        FE->getName(), RecordOpts, Policy, CGNameGen, CI.getDiagnostics());
+    if (!RecordFilename) {
+      reportError(CI,
+                  "failed to write record file for source file '%0' to disk",
+                  FE->getName(), "");
+      return;
+    }
+
+    Module *Mod = nullptr;
+    if (UnitModule) {
+      if (auto ModCandidate = CI.getPreprocessor()
+                                  .getHeaderSearchInfo()
+                                  .findModuleForHeader(FE)
+                                  .getModule()) {
+        if (ModCandidate->isSubModuleOf(UnitModule)) {
+          Mod = ModCandidate;
+          UnitWriter.addModule(Mod);
+        }
+      }
+    }
+
+    UnitWriter.addIndexRecord(RecordFilename.getValue(), FE, Rec.isSystem(),
+                              Mod);
+  }
+}
+
+void ActionContext::emitCollectedData() {
+  // FIXME: FrontendAction::EndSourceFile() should probably not call
+  // CI.getDiagnosticClient().EndSourceFile()' until after it has called
+  // 'EndSourceFileAction()', so that code executing during
+  // EndSourceFileAction() can emit diagnostics. If this is fixed,
+  // DiagClientBeginEndRAII can go away.
+  CI.getDiagnosticClient().BeginSourceFile(CI.getLangOpts());
+  auto DiagnosticClientEndSourceFile = llvm::make_scope_exit(
+      [this]() { CI.getDiagnosticClient().EndSourceFile(); });
+
+  std::string Error;
+  if (initIndexDirectory(RecordOpts, Error)) {
+    reportError(CI, "failed creating index directory %0", Error);
+    return;
+  }
+
+  IndexUnitWriter &UnitWriter = DepCollector.UnitWriter;
+
+  const Module *UnitModule = getModBeingCompiled(CI);
+  std::string ModuleName =
+      UnitModule ? UnitModule->getFullModuleName() : std::string();
+
+  if (auto Reader = CI.getModuleManager()) {
+    HeaderSearch &HS = CI.getPreprocessor().getHeaderSearchInfo();
+    Reader->getModuleManager().visit(
+        [&](serialization::ModuleFile &Mod) -> bool {
+          Module *UnitMod =
+              HS.lookupModule(Mod.ModuleName, /*AllowSearch=*/false);
+          const bool isSystemMod =
+              Mod.isModule() && UnitMod && UnitMod->IsSystem;
+
+          if (!isSystemMod || RecordOpts.RecordSystemDependencies) {
+            SmallString<256> AbsPath(Mod.File->getName());
+            CI.getFileManager().makeAbsolutePath(AbsPath);
+            UnitWriter.addIndexUnit(Mod.File, AbsPath, isSystemMod, UnitMod);
+            if (Mod.isModule()) {
+              llvm::Optional<bool> IndexUnitExist = doesIndexUnitForModuleExist(
+                  Mod.FileName, CI.getFileManager(), RecordOpts);
+
+              if (!IndexUnitExist.hasValue())
+                reportError(CI, "failed to stat index unit for module %0",
+                            AbsPath.str());
+
+              if (!IndexUnitExist.getValue())
+                emitIndexDataForModuleFile(Mod, CI, IndexOpts, RecordOpts);
+            }
+          }
+          return true; // skip module dependencies.
+        });
+  }
+
+  if (std::error_code EC = sys::fs::create_directory(RecordOpts.RecordDir)) {
+    reportError(CI, "failed to create directory '%0': %1", RecordOpts.RecordDir,
+                EC.message());
+    return;
+  }
+  writeIndexRecordsToDisk(DeclsDataCollector.getRecordByFile(), UnitWriter, CI,
+                          RecordOpts, UnitModule);
+
+  {
+    std::string Error;
+    if (UnitWriter.writeToDisk(Error)) {
+      reportError(CI, "failed writing unit data: %0", Error);
+      return;
+    }
+  }
+}
+
+class IndexRecordingASTConsumer : public ASTConsumer {
+  std::shared_ptr<ActionContext> Context;
+
+public:
+  IndexRecordingASTConsumer(std::shared_ptr<ActionContext> Context)
+      : Context(Context) {}
+
+protected:
+  void Initialize(ASTContext &ASTContextInst) override {
+    Context->Ctx.setASTContext(ASTContextInst);
+    Context->DeclsDataCollector.initialize(ASTContextInst);
+  }
+
+  bool HandleTopLevelDecl(DeclGroupRef DG) override {
+    return Context->Ctx.indexDeclGroupRef(DG);
+  }
+
+  void HandleInterestingDecl(DeclGroupRef DG) override {
+    // Ignore deserialized decls.
+  }
+
+  void HandleTopLevelDeclInObjCContainer(DeclGroupRef DG) override {
+    Context->Ctx.indexDeclGroupRef(DG);
+  }
+
+  void HandleTranslationUnit(ASTContext &Ctx) override {}
+};
+
+EmitIndexAction::EmitIndexAction(const IndexingOptions &IndexOpts,
+                                 const EmitIndexOptions &RecordOpts)
+    : RecordOpts(RecordOpts), IndexOpts(IndexOpts) {}
+
+std::unique_ptr<ASTConsumer>
+EmitIndexAction::CreateASTConsumer(CompilerInstance &CI, StringRef) {
+  Context = std::make_shared<ActionContext>(IndexOpts, CI, RecordOpts);
+  return llvm::make_unique<IndexRecordingASTConsumer>(Context);
+}
+
+void EmitIndexAction::EndSourceFileAction() {
+  if (Context)
+    Context->emitCollectedData();
+}
+
+void emitIndexDataForModuleFile(serialization::ModuleFile &Mod,
+                                const CompilerInstance &CI,
+                                const IndexingOptions &IndexOpts,
+                                const EmitIndexOptions &RecordOpts) {
+  // FIXME: FrontendAction::EndSourceFile() should probably not call
+  // CI.getDiagnosticClient().EndSourceFile()' until after it has called
+  // 'EndSourceFileAction()', so that code executing during
+  // EndSourceFileAction() can emit diagnostics. If this is fixed,
+  // DiagClientBeginEndRAII can go away.
+  CI.getDiagnosticClient().BeginSourceFile(CI.getLangOpts());
+  auto DiagnosticClientEndSourceFile = llvm::make_scope_exit(
+      [&CI]() { CI.getDiagnosticClient().EndSourceFile(); });
+
+  DiagnosticsEngine &Diag = CI.getDiagnostics();
+  Diag.Report(Mod.ImportLoc, diag::remark_index_producing_module_file_data)
+      << Mod.FileName;
+
+  Module *UnitModule = CI.getPreprocessor().getHeaderSearchInfo().lookupModule(
+      Mod.ModuleName, /*AllowSearch=*/false);
+
+  DeclOccurrenceCollector DeclsDataCollector(CI.getSourceManager());
+  {
+    IndexingContext IndexCtx(IndexOpts, DeclsDataCollector);
+    DeclsDataCollector.setIndexingContext(IndexCtx);
+
+    IndexCtx.setASTContext(CI.getASTContext());
+
+    for (const Decl *D : CI.getModuleManager()->getModuleFileLevelDecls(Mod)) {
+      IndexCtx.indexTopLevelDecl(D);
+    }
+    DeclsDataCollector.finish();
+  }
+
+  {
+    std::string Error;
+    if (initIndexDirectory(RecordOpts, Error)) {
+      reportError(CI, "failed creating index directory %0", Error);
+      return;
+    }
+  }
+
+  HeaderSearch &HS = CI.getPreprocessor().getHeaderSearchInfo();
+
+  SmallString<128> WorkDir;
+  if (!CI.getFileManager().getFileSystemOpts().WorkingDir.empty()) {
+    WorkDir = CI.getFileManager().getFileSystemOpts().WorkingDir;
+    if (!llvm::sys::path::is_absolute(WorkDir)) {
+      llvm::sys::fs::make_absolute(WorkDir);
+    }
+  } else {
+    std::error_code EC = llvm::sys::fs::current_path(WorkDir);
+    if (EC) {
+      reportError(CI, "failed to determine current working directory");
+    }
+  }
+
+  IndexUnitWriter UnitWriter =
+      makeIndexUnitWriter(CI, RecordOpts, Mod.FileName, nullptr, UnitModule);
+
+  if (UnitModule) {
+    serialization::ModuleFile *ModFile =
+        CI.getModuleManager()->getModuleManager().lookup(
+            UnitModule->getASTFile());
+    assert(ModFile && "no module file loaded for module ?");
+
+    for (auto *Mod : ModFile->Imports) {
+      auto *UnitMod = HS.lookupModule(Mod->ModuleName, /*AllowSearch=*/false);
+      if (!UnitMod->IsSystem || RecordOpts.RecordSystemDependencies) {
+        SmallString<256> AbsPath(Mod->File->getName());
+        CI.getFileManager().makeAbsolutePath(AbsPath);
+        UnitWriter.addIndexUnit(Mod->File, AbsPath, UnitMod->IsSystem, UnitMod);
+      }
+    }
+
+    CI.getModuleManager()->visitInputFiles(
+        *ModFile, RecordOpts.RecordSystemDependencies,
+        /*Complain=*/false,
+        [&](const serialization::InputFile &IF, bool isSystemFile) {
+          auto *FE = IF.getFile();
+          if (!FE)
+            return;
+          // Ignore module map files, they are not as important to track as
+          // source files and they may be auto-generated which would create an
+          // undesirable dependency on an intermediate build byproduct.
+          if (FE->getName().endswith("module.modulemap"))
+            return;
+
+          // Try to find a Module that is both a submodule of UnitModule and
+          // owns the header.
+          Module *ModForHead = nullptr;
+          if (auto Mod = CI.getPreprocessor()
+                             .getHeaderSearchInfo()
+                             .findModuleForHeader(FE)
+                             .getModule())
+            if (Mod->isSubModuleOf(UnitModule))
+              ModForHead = Mod;
+
+          UnitWriter.addModule(ModForHead);
+
+          UnitWriter.addSourceFile(FE, isSystemFile, ModForHead);
+        });
+  }
+
+  if (std::error_code EC = sys::fs::create_directory(RecordOpts.RecordDir)) {
+    reportError(CI, "failed to create directory '%0': %1", RecordOpts.RecordDir,
+                EC.message());
+    return;
+  }
+  writeIndexRecordsToDisk(DeclsDataCollector.getRecordByFile(), UnitWriter, CI,
+                          RecordOpts, UnitModule);
+
+  std::string Error;
+  if (UnitWriter.writeToDisk(Error)) {
+    reportError(CI, "failed writing unit data: %0", Error);
+    return;
+  }
+
+  if (UnitModule) {
+    serialization::ModuleFile *ModFile =
+        CI.getModuleManager()->getModuleManager().lookup(
+            UnitModule->getASTFile());
+    for (auto *Mod : ModFile->Imports) {
+      auto *UnitMod = HS.lookupModule(Mod->ModuleName, /*AllowSearch=*/false);
+
+      if (!UnitMod->IsSystem || RecordOpts.RecordSystemDependencies) {
+        if (Mod->isModule()) {
+          llvm::Optional<bool> IndexUnitExist = doesIndexUnitForModuleExist(
+              Mod->FileName, CI.getFileManager(), RecordOpts);
+          if (IndexUnitExist.hasValue() && !IndexUnitExist.getValue())
+            emitIndexDataForModuleFile(*Mod, CI, IndexOpts, RecordOpts);
+        }
+      }
+    }
+  }
+}
+
+bool emitIndexDataForModule(const Module *Mod, const CompilerInstance &CI) {
+  assert(Mod);
+  auto astReader = CI.getModuleManager();
+  serialization::ModuleFile *ModFile =
+      astReader->getModuleManager().lookup(Mod->getASTFile());
+  assert(ModFile && "no module file loaded for module ?");
+
+  const EmitIndexOptions RecOpts = parseEmitIndexOptions(CI.getFrontendOpts());
+
+  llvm::Optional<bool> IndexUnitExist = doesIndexUnitForModuleExist(
+      ModFile->FileName, CI.getFileManager(), RecOpts);
+  if (IndexUnitExist.hasValue() && IndexUnitExist.getValue())
+    return false;
+
+  emitIndexDataForModuleFile(
+      *ModFile, CI, parseIndexGenerationOptions(CI.getFrontendOpts()), RecOpts);
+  return true;
+}
+
+} // namespace index
+} // namespace clang

clang/lib/Index/IndexWhileBuilding/EmitIndexOptions.cpp

+//===--- IndexOptions.cpp - Options for indexing ----------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Index/IndexWhileBuilding/EmitIndexOptions.h"
+#include "clang/Index/IndexWhileBuilding/IndexDataFormat.h"
+#include "llvm/Support/Path.h"
+
+namespace clang {
+namespace index {
+
+EmitIndexOptions::EmitIndexOptions(const StringRef RootDir,
+                                   const bool RecordSymbolCodeGenName,
+                                   const bool RecordSystemDependencies,
+                                   const IncludesRecordingKind RecordIncludes)
+    : RootDir(RootDir), IndexDir(llvm::Twine(RootDir)
+                                     .concat(llvm::sys::path::get_separator())
+                                     .concat(store::getIndexDir())
+                                     .str()),
+      UnitDir(llvm::Twine(IndexDir)
+                  .concat(llvm::sys::path::get_separator())
+                  .concat(store::getIndexUnitSubdir())
+                  .str()),
+      RecordDir(llvm::Twine(IndexDir)
+                    .concat(llvm::sys::path::get_separator())
+                    .concat(store::getIndexRecordSubdir())
+                    .str()),
+      RecordSymbolCodeGenName(RecordSymbolCodeGenName),
+      RecordSystemDependencies(RecordSystemDependencies),
+      RecordIncludes(RecordIncludes) {}
+
+EmitIndexOptions parseEmitIndexOptions(const FrontendOptions &FEOpts) {
+  return EmitIndexOptions(FEOpts.IndexStorePath, FEOpts.IndexRecordCodegenName);
+}
+
+} // namespace index
+} // namespace clang

clang/lib/Index/IndexWhileBuilding/FSUtil.h

+//===--- FSUtil.h - File system utilities ---------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEX_FSUTIL_H
+#define LLVM_CLANG_INDEX_FSUTIL_H
+
+#include "clang/Index/IndexWhileBuilding/IndexRecordWriter.h"
+#include "llvm/Support/Errc.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/raw_ostream.h"
+
+namespace clang {
+namespace index {
+
+using namespace llvm;
+
+/// Creates a unique file with name according to the given \p TempPathModel,
+/// writes content of \p Buffer to the file and renames it to \p FinalPath.
+/// \returns false when the file was successfully created, true otherwise.
+inline bool atomicFileWrite(StringRef TempPathModel, StringRef FinalPath,
+                            const SmallString<512> &Buffer,
+                            std::string &Error) {
+  SmallString<128> GeneratedUniqPath;
+  int TempFD;
+  if (sys::fs::createUniqueFile(TempPathModel.str(), TempFD,
+                                GeneratedUniqPath)) {
+    raw_string_ostream Err(Error);
+    Err << "failed to create temporary file: " << TempPathModel;
+    return true;
+  }
+
+  raw_fd_ostream OS(TempFD, /*shouldClose=*/true);
+  OS.write(Buffer.data(), Buffer.size());
+  OS.close();
+  TempFD = -1;
+
+  if (OS.has_error()) {
+    raw_string_ostream Err(Error);
+    Err << "failed to write '" << GeneratedUniqPath
+        << "': " << OS.error().message();
+    OS.clear_error();
+    return true;
+  }
+
+  std::error_code EC = sys::fs::rename(/*from=*/GeneratedUniqPath.c_str(),
+                                       /*to=*/FinalPath.str().c_str());
+  if (EC) {
+    raw_string_ostream Err(Error);
+    Err << "failed to rename '" << GeneratedUniqPath << "' to '" << FinalPath
+        << "': " << EC.message();
+    return true;
+  }
+
+  return false;
+}
+
+} // end namespace index
+} // end namespace clang
+
+#endif
 No newline at end of file

clang/lib/Index/IndexWhileBuilding/IndexDataFormat.cpp

+//===--- IndexDataFormat.cpp - Layout of index on a filesystem ------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Index/IndexWhileBuilding/IndexDataFormat.h"
+
+#include "llvm/ADT/APInt.h"
+#include "llvm/ADT/Hashing.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/raw_ostream.h"
+
+using namespace llvm;
+
+namespace {
+
+#define STR_HELPER(x) #x
+#define STR(x) STR_HELPER(x)
+
+#define INDEX_FORMAT_VERSION_VALUE 5
+
+static const unsigned indexFormatVersion = INDEX_FORMAT_VERSION_VALUE;
+
+const char *const indexSubdir = "v" STR(INDEX_FORMAT_VERSION_VALUE);
+
+#undef INDEX_FORMAT_VERSION_VALUE
+#undef STR
+#undef STR_HELPER
+
+const char *const indexUnitSubdir = "units";
+
+const char *const indexRecordSubdir = "records";
+
+} // namespace
+
+namespace clang {
+namespace index {
+namespace store {
+
+unsigned getIndexFormatVersion() { return indexFormatVersion; }
+
+const char *getIndexDir() { return indexSubdir; }
+const char *getIndexUnitSubdir() { return indexUnitSubdir; }
+const char *getIndexRecordSubdir() { return indexRecordSubdir; }
+
+static void appendIndexSubDir(StringRef subdir, SmallVectorImpl<char> &Buffer) {
+  sys::path::append(Buffer, getIndexDir());
+  sys::path::append(Buffer, subdir);
+}
+
+void appendUnitBucketDir(StringRef UnitName, SmallVectorImpl<char> &Buffer) {
+  sys::path::append(Buffer, UnitName);
+}
+
+void appendUnitSubDir(SmallVectorImpl<char> &Buffer) {
+  return appendIndexSubDir(getIndexUnitSubdir(), Buffer);
+}
+
+void appendRecordSubDir(SmallVectorImpl<char> &Buffer) {
+  return appendIndexSubDir(getIndexRecordSubdir(), Buffer);
+}
+
+void appendRecordBucketDir(StringRef RecordName,
+                           SmallVectorImpl<char> &Buffer) {
+  // To avoid putting a huge number of files into the records directory, create
+  // subdirectories based on the last 2 characters from the hash.
+  sys::path::append(Buffer, RecordName.substr(RecordName.size() - 2));
+}
+
+void appendRecordFilename(StringRef RecordName, SmallVectorImpl<char> &Buffer) {
+  sys::path::append(Buffer, RecordName);
+}
+
+void getUnitNameForOutputFile(StringRef FilePath, SmallVectorImpl<char> &Str) {
+  StringRef Fname = sys::path::filename(FilePath);
+  Str.append(Fname.begin(), Fname.end());
+  Str.push_back('-');
+  llvm::hash_code PathHashVal = llvm::hash_value(FilePath);
+  llvm::APInt(64, PathHashVal).toString(Str, 36, /*Signed=*/false);
+}
+
+std::string generateFilenameForIndexRecord(
+    StringRef SourceFilename, const llvm::SmallString<512> &IndexRecordBuffer) {
+  const llvm::hash_code RecordHash = llvm::hash_combine_range(
+      IndexRecordBuffer.begin(), IndexRecordBuffer.end());
+
+  std::string RecordFilename;
+  llvm::raw_string_ostream RN(RecordFilename);
+  RN << llvm::sys::path::filename(SourceFilename);
+  RN << "-" << llvm::APInt(64, RecordHash).toString(36, /*Signed=*/false);
+  return RecordFilename;
+}
+
+} // end namespace store
+} // end namespace index
+} // end namespace clang
 No newline at end of file

clang/lib/Index/IndexWhileBuilding/IndexRecordReader.cpp

+//===--- IndexRecordReader.cpp - Index record deserialization -------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Index/IndexWhileBuilding/IndexRecordReader.h"
+#include "BitstreamVisitor.h"
+#include "indexstore/Conversions.h"
+#include "clang/Index/IndexWhileBuilding/IndexDataFormat.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/Bitstream/BitstreamReader.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/raw_ostream.h"
+
+namespace clang {
+namespace index {
+
+using namespace clang::index::store;
+using namespace llvm;
+
+class IndexBitstreamVisitor : public BitstreamVisitor<IndexBitstreamVisitor> {
+  IndexRecordReader &Reader;
+
+public:
+  IndexBitstreamVisitor(llvm::BitstreamCursor &Stream,
+                        IndexRecordReader &Reader)
+      : BitstreamVisitor(Stream), Reader(Reader) {}
+
+  StreamVisit visitBlock(unsigned ID) {
+    switch ((RecordBitBlock)ID) {
+    case REC_VERSION_BLOCK_ID:
+    case REC_DECLOFFSETS_BLOCK_ID:
+      return StreamVisit::Continue;
+
+    case REC_DECLS_BLOCK_ID:
+      Reader.DeclCursor = Stream;
+      if (Reader.DeclCursor.EnterSubBlock(ID)) {
+        *Error = "malformed block record";
+        return StreamVisit::Abort;
+      }
+      if (llvm::Error Err = readBlockAbbrevs(Reader.DeclCursor)) {
+        *Error = toString(std::move(Err));
+        return StreamVisit::Abort;
+      }
+      return StreamVisit::Skip;
+
+    case REC_DECLOCCURRENCES_BLOCK_ID:
+      Reader.OccurCursor = Stream;
+      if (Reader.OccurCursor.EnterSubBlock(ID)) {
+        *Error = "malformed block record";
+        return StreamVisit::Abort;
+      }
+      if (llvm::Error Err = readBlockAbbrevs(Reader.OccurCursor)) {
+        *Error = toString(std::move(Err));
+        return StreamVisit::Abort;
+      }
+      return StreamVisit::Skip;
+    }
+
+    // Some newly introduced block in a minor version update that we cannot
+    // handle.
+    return StreamVisit::Skip;
+  }
+
+  StreamVisit visitRecord(unsigned BlockID, unsigned RecID,
+                          SmallVectorImpl<uint64_t> &Record, StringRef Blob) {
+    switch (BlockID) {
+    case REC_VERSION_BLOCK_ID: {
+      unsigned StoreFormatVersion = Record[0];
+      if (StoreFormatVersion != getIndexFormatVersion()) {
+        llvm::raw_string_ostream OS(*Error);
+        OS << "Store format version mismatch: " << StoreFormatVersion;
+        OS << " , expected: " << getIndexFormatVersion();
+        return StreamVisit::Abort;
+      }
+      break;
+    }
+    case REC_DECLOFFSETS_BLOCK_ID:
+      assert(RecID == REC_DECLOFFSETS);
+      Reader.setDeclOffsets(
+          makeArrayRef((const uint32_t *)Blob.data(), Record[0]));
+      break;
+
+    case REC_DECLS_BLOCK_ID:
+    case REC_DECLOCCURRENCES_BLOCK_ID:
+      llvm_unreachable("shouldn't visit this block'");
+    }
+    return StreamVisit::Continue;
+  }
+};
+
+std::unique_ptr<IndexRecordReader> IndexRecordReader::createWithRecordFilename(
+    StringRef RecordFilename, StringRef StorePath, std::string &Error) {
+  SmallString<128> PathBuf = StorePath;
+  appendRecordSubDir(PathBuf);
+  appendRecordBucketDir(RecordFilename, PathBuf);
+  appendRecordFilename(RecordFilename, PathBuf);
+  return createWithFilePath(PathBuf.str(), Error);
+}
+
+std::unique_ptr<IndexRecordReader>
+IndexRecordReader::createWithFilePath(StringRef FilePath, std::string &Error) {
+  auto ErrOrBuf = MemoryBuffer::getFile(FilePath, /*FileSize=*/-1,
+                                        /*RequiresNullTerminator=*/false);
+  if (!ErrOrBuf) {
+    raw_string_ostream(Error) << "failed opening index record '" << FilePath
+                              << "': " << ErrOrBuf.getError().message();
+    return nullptr;
+  }
+  return createWithBuffer(std::move(*ErrOrBuf), Error);
+}
+
+std::unique_ptr<IndexRecordReader>
+IndexRecordReader::createWithBuffer(std::unique_ptr<llvm::MemoryBuffer> Buffer,
+                                    std::string &Error) {
+  llvm::BitstreamCursor Stream(*Buffer);
+
+  if (Stream.AtEndOfStream()) {
+    Error = "empty file";
+    return nullptr;
+  }
+
+  // Sniff for the signature.
+  for (unsigned char C : {'I', 'D', 'X', 'R'}) {
+    if (Expected<llvm::SimpleBitstreamCursor::word_t> Res = Stream.Read(8)) {
+      if (Res.get() == C)
+        continue;
+    } else {
+      Error = toString(Res.takeError());
+      return nullptr;
+    }
+    Error = "not a serialized index record file";
+    return nullptr;
+  }
+
+  std::unique_ptr<IndexRecordReader> Reader =
+      llvm::make_unique<IndexRecordReader>();
+  Reader->Buffer = std::move(Buffer);
+
+  IndexBitstreamVisitor BitVisitor(Stream, *Reader);
+  if (!BitVisitor.visit(Error))
+    return nullptr;
+
+  return Reader;
+}
+
+bool IndexRecordReader::searchDecls(
+    llvm::function_ref<DeclSearchCheck> Checker,
+    llvm::function_ref<void(const IndexRecordDecl *)> Receiver) {
+  for (unsigned I = 0, E = getNumDecls(); I != E; ++I) {
+    if (const IndexRecordDecl *D = Decls[I]) {
+      DeclSearchReturn Ret = Checker(*D);
+      if (Ret.AcceptDecl)
+        Receiver(D);
+      if (!Ret.ContinueSearch)
+        return false;
+      continue;
+    }
+
+    IndexRecordDecl LocalD;
+    readDecl(I, LocalD);
+    DeclSearchReturn Ret = Checker(LocalD);
+    if (Ret.AcceptDecl) {
+      IndexRecordDecl *D = Allocator.Allocate<IndexRecordDecl>();
+      *D = LocalD;
+      Decls[I] = D;
+      Receiver(D);
+    }
+    if (!Ret.ContinueSearch)
+      return false;
+  }
+  return true;
+}
+
+bool IndexRecordReader::foreachDecl(
+    bool NoCache, function_ref<bool(const IndexRecordDecl *)> Receiver) {
+  for (unsigned I = 0, E = getNumDecls(); I != E; ++I) {
+    if (const IndexRecordDecl *D = Decls[I]) {
+      if (!Receiver(D))
+        return false;
+      continue;
+    }
+
+    if (NoCache) {
+      IndexRecordDecl LocalD;
+      readDecl(I, LocalD);
+      if (!Receiver(&LocalD))
+        return false;
+    } else {
+      if (!Receiver(getDecl(I)))
+        return false;
+    }
+  }
+  return true;
+}
+
+bool IndexRecordReader::foreachOccurrence(
+    function_ref<bool(const IndexRecordOccurrence &)> Receiver,
+    ArrayRef<const IndexRecordDecl *> WhitelistDecls,
+    ArrayRef<const IndexRecordDecl *> WhitelistRelated) {
+  class OccurBitVisitor : public BitstreamVisitor<OccurBitVisitor> {
+    IndexRecordReader &Reader;
+    ArrayRef<const IndexRecordDecl *> WhitelistDecls;
+    ArrayRef<const IndexRecordDecl *> WhitelistRelated;
+    function_ref<bool(const IndexRecordOccurrence &)> Receiver;
+
+  public:
+    OccurBitVisitor(llvm::BitstreamCursor &Stream, IndexRecordReader &Reader,
+                    ArrayRef<const IndexRecordDecl *> WhitelistDecls,
+                    ArrayRef<const IndexRecordDecl *> WhitelistRelated,
+                    function_ref<bool(const IndexRecordOccurrence &)> Receiver)
+        : BitstreamVisitor(Stream), Reader(Reader),
+          WhitelistDecls(WhitelistDecls), WhitelistRelated(WhitelistRelated),
+          Receiver(std::move(Receiver)) {}
+
+    StreamVisit visitRecord(unsigned BlockID, unsigned RecID,
+                            SmallVectorImpl<uint64_t> &Record, StringRef Blob) {
+      assert(RecID == REC_DECLOCCURRENCE);
+      IndexRecordOccurrence RecOccur;
+      if (Reader.readOccurrence(Record, Blob, WhitelistDecls, WhitelistRelated,
+                                RecOccur))
+        if (!Receiver(RecOccur))
+          return StreamVisit::Abort;
+      return StreamVisit::Continue;
+    }
+  };
+
+  SavedStreamPosition SavedPosition(OccurCursor);
+  OccurBitVisitor Visitor(OccurCursor, *this, WhitelistDecls, WhitelistRelated,
+                          Receiver);
+  std::string Error;
+  return Visitor.visit(Error);
+}
+
+bool IndexRecordReader::foreachOccurrenceInLineRange(
+    unsigned lineStart, unsigned lineCount,
+    llvm::function_ref<bool(const IndexRecordOccurrence &)> Receiver) {
+  // FIXME: Use binary search and make this more efficient.
+  unsigned lineEnd = lineStart + lineCount;
+  return foreachOccurrence(
+      [&](const IndexRecordOccurrence &occur) -> bool {
+        if (occur.Line > lineEnd)
+          return false; // we're done.
+        if (occur.Line >= lineStart) {
+          if (!Receiver(occur))
+            return false;
+        }
+        return true;
+      },
+      None, None);
+}
+
+void IndexRecordReader::setDeclOffsets(ArrayRef<uint32_t> Offs) {
+  DeclOffsets = Offs;
+  Decls = Allocator.Allocate<const IndexRecordDecl *>(Offs.size());
+  memset(Decls, 0, sizeof(IndexRecordDecl *) * Offs.size());
+}
+
+unsigned IndexRecordReader::getNumDecls() const { return DeclOffsets.size(); }
+
+const IndexRecordDecl *IndexRecordReader::getDeclByID(unsigned DeclID) {
+  if (DeclID == 0)
+    return nullptr;
+  return getDecl(DeclID - 1);
+}
+
+const IndexRecordDecl *IndexRecordReader::getDecl(unsigned Index) {
+  assert(Index < getNumDecls());
+  if (const IndexRecordDecl *D = Decls[Index])
+    return D;
+
+  IndexRecordDecl *D = Allocator.Allocate<IndexRecordDecl>();
+  readDecl(Index, *D);
+  Decls[Index] = D;
+  return D;
+}
+
+/// Reads occurrence data.
+/// \param WhitelistDecls if non-empty indicates the list of decls that we want
+/// to get occurrences for. If empty then indicates that we want occurrences
+/// for all decls.
+/// \param WhitelistRelated Same as \c WhitelistDecls but for related decls.
+/// \returns true if the occurrence info was filled out, false if occurrence
+/// was ignored.
+bool IndexRecordReader::readOccurrence(
+    SmallVectorImpl<uint64_t> &Record, StringRef Blob,
+    ArrayRef<const IndexRecordDecl *> WhitelistDecls,
+    ArrayRef<const IndexRecordDecl *> WhitelistRelated,
+    IndexRecordOccurrence &RecOccur) {
+  auto isDeclIDContained = [](unsigned DeclID,
+                              ArrayRef<const IndexRecordDecl *> Ds) -> bool {
+    if (Ds.empty())
+      return true; // empty means accept all.
+    auto pred = [DeclID](const IndexRecordDecl *D) {
+      return D->DeclID == DeclID;
+    };
+    return std::find_if(Ds.begin(), Ds.end(), pred) != Ds.end();
+  };
+
+  unsigned I = 0;
+  unsigned DeclID = Record[I++];
+  if (!isDeclIDContained(DeclID, WhitelistDecls))
+    return false;
+
+  if (!WhitelistRelated.empty()) {
+    unsigned RelI = I + 3;
+    unsigned NumRelated = Record[RelI++];
+    bool FoundRelated = false;
+    while (NumRelated--) {
+      ++RelI; // roles;
+      unsigned RelDID = Record[RelI++];
+      if (isDeclIDContained(RelDID, WhitelistRelated)) {
+        FoundRelated = true;
+        break;
+      }
+    }
+    if (!FoundRelated)
+      return false;
+  }
+
+  RecOccur.Dcl = getDeclByID(DeclID);
+  RecOccur.Roles = getSymbolRoles(Record[I++]);
+  RecOccur.Line = Record[I++];
+  RecOccur.Column = Record[I++];
+
+  unsigned NumRelated = Record[I++];
+  while (NumRelated--) {
+    SymbolRoleSet RelRoles = getSymbolRoles(Record[I++]);
+    const IndexRecordDecl *RelD = getDeclByID(Record[I++]);
+    RecOccur.Relations.emplace_back(RelRoles, RelD);
+  }
+
+  return true;
+}
+
+void IndexRecordReader::readDecl(unsigned Index, IndexRecordDecl &RecD) {
+  SmallVector<uint64_t, 64> Record;
+  StringRef Blob;
+  if (llvm::Error Err = DeclCursor.JumpToBit(DeclOffsets[Index])) {
+    // FIXME this drops errors on the floor.
+    consumeError(std::move(Err));
+    llvm_unreachable("Failure in JumpToBit()");
+    // FIXME error handling
+    return;
+  }
+  Expected<unsigned> MaybeCode = DeclCursor.ReadCode();
+  if (!MaybeCode) {
+    // FIXME this drops errors on the floor.
+    consumeError(MaybeCode.takeError());
+    llvm_unreachable("Failure in ReadCode()");
+    // FIXME error handling
+    return;
+  }
+
+  Expected<unsigned> MaybeRecID =
+      DeclCursor.readRecord(MaybeCode.get(), Record, &Blob);
+  if (!MaybeRecID) {
+    // FIXME this drops errors on the floor.
+    consumeError(MaybeRecID.takeError());
+    llvm_unreachable("Failure in ReadRecord()");
+    // FIXME error handling
+    return;
+  }
+  assert(MaybeRecID.get() == REC_DECLINFO);
+  unsigned I = 0;
+  RecD.DeclID = Index + 1;
+  RecD.SymInfo.Kind = getSymbolKind((indexstore_symbol_kind_t)Record[I++]);
+  RecD.SymInfo.SubKind =
+      getSymbolSubKind((indexstore_symbol_subkind_t)Record[I++]);
+  RecD.SymInfo.Lang =
+      getSymbolLanguage((indexstore_symbol_language_t)Record[I++]);
+  RecD.SymInfo.Properties = getSymbolProperties(Record[I++]);
+  RecD.Roles = getSymbolRoles(Record[I++]);
+  RecD.RelatedRoles = getSymbolRoles(Record[I++]);
+  size_t NameLen = Record[I++];
+  size_t USRLen = Record[I++];
+  RecD.Name = Blob.substr(0, NameLen);
+  RecD.USR = Blob.substr(NameLen, USRLen);
+  RecD.CodeGenName = Blob.substr(NameLen + USRLen);
+}
+
+} // namespace index
+} // namespace clang
 No newline at end of file

clang/lib/Index/IndexWhileBuilding/IndexRecordWriter.cpp

+//===--- IndexRecordWriter.cpp - Index record serialization -----*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Index/IndexWhileBuilding/IndexRecordWriter.h"
+#include "BitstreamUtil.h"
+#include "FSUtil.h"
+#include "indexstore/Conversions.h"
+#include "indexstore/indexstore.h"
+#include "clang/AST/ASTContext.h"
+#include "clang/AST/DeclTemplate.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Index/IndexWhileBuilding/IndexDataFormat.h"
+#include "clang/Index/USRGeneration.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/Support/Errc.h"
+#include "llvm/Support/Path.h"
+
+namespace clang {
+namespace index {
+
+using namespace llvm;
+using namespace store;
+
+void IndexRecordWriter::addDeclOccurence(SymbolRoleSet Roles, FileID FID,
+                                         unsigned Offset, const Decl *D,
+                                         ArrayRef<SymbolRelation> Relations) {
+  assert(D->isCanonicalDecl() &&
+         "Occurrences should be associated with their canonical decl");
+
+  auto getIdxForDecl = [this](const Decl *D) -> unsigned {
+    const auto MaybeNewDecl = IndexForDecl.insert({D, Decls.size()});
+    if (MaybeNewDecl.second)
+      Decls.push_back(DeclInfo{D, SymbolRoleSet(), SymbolRoleSet()});
+    return MaybeNewDecl.first->second;
+  };
+
+  const unsigned DeclIdx = getIdxForDecl(D);
+  Decls[DeclIdx].Roles |= Roles;
+
+  const uint64_t REC_DECLOCCURRENCE = 3;
+
+  SourceManager &SM = D->getASTContext().getSourceManager();
+  const unsigned Line = SM.getLineNumber(FID, Offset);
+  const unsigned Column = SM.getColumnNumber(FID, Offset);
+
+  SmallVector<uint64_t, 10> OccurrenceData{
+      REC_DECLOCCURRENCE, // FIXME: This is constant value. We don't need to
+                          // store it.
+      DeclIdx + 1,        getIndexStoreRoles(Roles), Line, Column,
+      Relations.size()};
+
+  OccurrenceData.reserve(OccurrenceData.size() + 2 * Relations.size());
+  for (auto &Rel : Relations) {
+    OccurrenceData.push_back(getIndexStoreRoles(Rel.Roles));
+
+    const unsigned RelDeclIdx = getIdxForDecl(Rel.RelatedSymbol);
+    Decls[RelDeclIdx].RelatedRoles |= Rel.Roles;
+
+    OccurrenceData.push_back(RelDeclIdx + 1);
+  }
+
+  Occurrences.emplace(Offset, OccurrenceData);
+}
+
+static void writeHeader(BitstreamWriter &Stream) {
+  Stream.Emit('I', 8);
+  Stream.Emit('D', 8);
+  Stream.Emit('X', 8);
+  Stream.Emit('R', 8);
+}
+
+static void writeBlockInfo(BitstreamWriter &Stream) {
+  SmallVector<uint64_t, 64> Record;
+
+  Stream.EnterBlockInfoBlock();
+#define BLOCK(X) emitBlockID(X##_ID, #X, Stream, Record)
+#define RECORD(X) emitRecordID(X, #X, Stream, Record)
+
+  BLOCK(REC_VERSION_BLOCK);
+  RECORD(REC_VERSION);
+
+  BLOCK(REC_DECLS_BLOCK);
+  RECORD(REC_DECLINFO);
+
+  BLOCK(REC_DECLOFFSETS_BLOCK);
+  RECORD(REC_DECLOFFSETS);
+
+  BLOCK(REC_DECLOCCURRENCES_BLOCK);
+  RECORD(REC_DECLOCCURRENCE);
+
+#undef RECORD
+#undef BLOCK
+  Stream.ExitBlock();
+}
+
+static std::shared_ptr<BitCodeAbbrev> REC_VERSION_BLOCK_ID_Abbrev() {
+  static const std::shared_ptr<BitCodeAbbrev> CachedResult(new BitCodeAbbrev({
+      BitCodeAbbrevOp(REC_VERSION),
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 6) // Store format version
+  }));
+
+  return CachedResult;
+}
+
+static void writeVersionInfo(BitstreamWriter &Stream) {
+  using namespace llvm::sys;
+
+  Stream.EnterSubblock(REC_VERSION_BLOCK_ID, 3);
+  unsigned AbbrevCode = Stream.EmitAbbrev(REC_VERSION_BLOCK_ID_Abbrev());
+
+  const uint64_t Record[] = {REC_VERSION, getIndexFormatVersion()};
+  Stream.EmitRecordWithAbbrev(AbbrevCode, Record);
+
+  Stream.ExitBlock();
+}
+
+static std::shared_ptr<BitCodeAbbrev> DECLS_BLOCK_ID_Abbrev() {
+  static const std::shared_ptr<BitCodeAbbrev> CachedResult(new BitCodeAbbrev({
+      BitCodeAbbrevOp(REC_DECLINFO),
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 5), // Kind
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 5), // SubKind
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 5), // Language
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed,
+                      SymbolPropertyBitNum),                     // Properties
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, SymbolRoleBitNum), // Roles
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed,
+                      SymbolRoleBitNum),        // Related Roles
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 6), // Length of name in block
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 6), // Length of USR in block
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Blob) // Name + USR + CodeGen symbol name
+  }));
+
+  return CachedResult;
+}
+
+static std::shared_ptr<BitCodeAbbrev> DECLOFFSETS_BLOCK_ID_Abbrev() {
+  static const std::shared_ptr<BitCodeAbbrev> CachedResult(new BitCodeAbbrev({
+      BitCodeAbbrevOp(REC_DECLOFFSETS),
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 32), // Number of Decls
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Blob)       // Offsets array
+  }));
+
+  return CachedResult;
+}
+
+static std::shared_ptr<BitCodeAbbrev> DECLOCCURRENCES_BLOCK_ID_Abbrev() {
+  static const std::shared_ptr<BitCodeAbbrev> CachedResult(new BitCodeAbbrev({
+      BitCodeAbbrevOp(REC_DECLOCCURRENCE),
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 8),                  // Decl ID
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, SymbolRoleBitNum), // Roles
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 12),                 // Line
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 8),                  // Column
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 4),                  // Num related
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Array),  // Related Roles/IDs
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 16) // Roles or ID
+  }));
+
+  return CachedResult;
+}
+
+static SmallString<256> getUSR(const Decl *D) {
+  SmallString<256> Buf;
+  bool Ignore = generateUSRForDecl(D, Buf);
+  if (Ignore)
+    return {};
+  return Buf;
+}
+
+void IndexRecordWriter::getBitcodeRepresentation(
+    SmallString<512> &Buffer, const PrintingPolicy &Policy,
+    std::shared_ptr<CodegenNameGenerator> CGNameGen) const {
+  BitstreamWriter Stream(Buffer);
+  writeHeader(Stream);
+  writeBlockInfo(Stream);
+  writeVersionInfo(Stream);
+
+  SmallVector<uint32_t, 32> DeclOffsets;
+
+  {
+    Stream.EnterSubblock(REC_DECLS_BLOCK_ID, 3);
+    const unsigned AbbrevCode = Stream.EmitAbbrev(DECLS_BLOCK_ID_Abbrev());
+
+#ifndef NDEBUG
+    StringSet<> USRSet;
+    bool enableValidation = getenv("CLANG_INDEX_VALIDATION_CHECKS") != nullptr;
+#endif
+
+    DeclOffsets.reserve(Decls.size());
+
+    llvm::SmallString<256> Scratch;
+    for (auto &Declaration : Decls) {
+      DeclOffsets.push_back(Stream.GetCurrentBitNo());
+      Scratch.clear();
+
+      const Decl *D = static_cast<const Decl *>(Declaration.D);
+      auto Info = getSymbolInfo(D);
+      assert(Info.Kind != SymbolKind::Unknown);
+
+      std::string Name;
+      std::string CGName;
+      if (auto *ND = dyn_cast<NamedDecl>(D)) {
+        DeclarationName DeclName = ND->getDeclName();
+        if (!DeclName.isEmpty()) {
+          llvm::raw_string_ostream OS(Name);
+          DeclName.print(OS, Policy);
+        }
+        if (CGNameGen) {
+          llvm::raw_string_ostream OS(CGName);
+          CGNameGen->writeName(ND, OS);
+        }
+      }
+
+      const SmallString<256> USR = getUSR(D);
+      assert(!USR.empty() && "Recorded decl without USR!");
+
+#ifndef NDEBUG
+      if (enableValidation) {
+        bool IsNew = USRSet.insert(USR).second;
+        if (!IsNew) {
+          llvm::errs() << "Index: Duplicate USR! " << USR << "\n";
+          // FIXME: print more information so it's easier to find the
+          // declaration.
+        }
+      }
+#endif
+
+      const uint64_t Record[] = {REC_DECLINFO,
+                                 getIndexStoreKind(Info.Kind),
+                                 getIndexStoreSubKind(Info.SubKind),
+                                 getIndexStoreLang(Info.Lang),
+                                 getIndexStoreProperties(Info.Properties),
+                                 getIndexStoreRoles(Declaration.Roles),
+                                 getIndexStoreRoles(Declaration.RelatedRoles),
+                                 Name.size(),
+                                 USR.size()};
+
+      Stream.EmitRecordWithBlob(
+          AbbrevCode, Record,
+          llvm::Twine(Name).concat(USR).concat(CGName).str());
+    }
+
+    Stream.ExitBlock();
+  }
+
+  {
+    Stream.EnterSubblock(REC_DECLOFFSETS_BLOCK_ID, 3);
+    const unsigned AbbrevCode =
+        Stream.EmitAbbrev(DECLOFFSETS_BLOCK_ID_Abbrev());
+    const uint64_t Record[] = {REC_DECLOFFSETS, DeclOffsets.size()};
+    StringRef DeclOffsetsData(
+        reinterpret_cast<const char *>(DeclOffsets.data()),
+        sizeof(uint32_t) * DeclOffsets.size());
+
+    Stream.EmitRecordWithBlob(AbbrevCode, Record, DeclOffsetsData);
+    Stream.ExitBlock();
+  }
+
+  Stream.EnterSubblock(REC_DECLOCCURRENCES_BLOCK_ID, 3);
+  const unsigned AbbrevCode =
+      Stream.EmitAbbrev(DECLOCCURRENCES_BLOCK_ID_Abbrev());
+
+  for (auto &OffsetAndOccurrence : Occurrences) {
+    Stream.EmitRecordWithAbbrev(AbbrevCode, OffsetAndOccurrence.second);
+  }
+
+  Stream.ExitBlock();
+}
+
+template <unsigned N>
+static void reportError(DiagnosticsEngine &Diag, const char (&Format)[N],
+                        const std::string &Filename,
+                        const std::string &Message) {
+  unsigned DiagID = Diag.getCustomDiagID(DiagnosticsEngine::Error, Format);
+  Diag.Report(DiagID) << Filename << Message;
+}
+
+llvm::Optional<std::string>
+IndexRecordWriter::writeToDisk(StringRef SourceFileName,
+                               const EmitIndexOptions &RecordOpts,
+                               const PrintingPolicy &Policy,
+                               std::shared_ptr<CodegenNameGenerator> CGNameGen,
+                               DiagnosticsEngine &Diag) const {
+
+  llvm::SmallString<512> IndexRecordBuffer;
+  getBitcodeRepresentation(IndexRecordBuffer, Policy, CGNameGen);
+
+  const std::string RecordFilename =
+      generateFilenameForIndexRecord(SourceFileName, IndexRecordBuffer);
+
+  llvm::SmallString<256> RecordBucketSubdir(RecordOpts.RecordDir);
+  store::appendRecordBucketDir(RecordFilename, RecordBucketSubdir);
+
+  if (std::error_code EC = sys::fs::create_directory(RecordBucketSubdir)) {
+    reportError(Diag, "failed to create directory '%0': %1",
+                RecordBucketSubdir.str(), EC.message());
+    return llvm::None;
+  }
+
+  llvm::SmallString<256> RecordPath(RecordBucketSubdir);
+  store::appendRecordFilename(RecordFilename, RecordPath);
+
+  if (std::error_code EC = llvm::sys::fs::access(
+          RecordPath.str().str().c_str(), llvm::sys::fs::AccessMode::Exist)) {
+    if (EC == llvm::errc::no_such_file_or_directory) {
+      // Create a unique file to write to so that we can move the result into
+      // place atomically. If this process crashes we don't want to interfere
+      // with any other concurrent processes.
+      // FIXME: We should also make sure we don't eat up all the disk in case
+      // we're crashing repeatedly.
+      const SmallString<128> TempPath(
+          llvm::Twine(RecordPath).concat("-temp-%%%%%%%%").str());
+      std::string Error;
+      if (atomicFileWrite(TempPath, RecordPath, IndexRecordBuffer, Error)) {
+        // FIXME:  This assumes getDiagnosticClient().BeginSourceFile() has been
+        // called.
+        reportError(Diag, "failed writing record '%0': %1", RecordFilename,
+                    Error);
+        return llvm::None;
+      }
+    } else {
+      reportError(Diag, "failed stating record file '%0': %1", RecordFilename,
+                  "");
+      return llvm::None;
+    }
+  }
+
+  return RecordFilename;
+}
+
+} // end namespace index
+} // end namespace clang
 No newline at end of file

clang/lib/Index/IndexWhileBuilding/IndexUnitReader.cpp

+//===--- IndexUnitReader.cpp - Index unit deserialization -----------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Index/IndexWhileBuilding/IndexUnitReader.h"
+#include "BitstreamVisitor.h"
+#include "clang/Index/IndexWhileBuilding/IndexDataFormat.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/Optional.h"
+#include "llvm/ADT/ScopeExit.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Bitstream/BitstreamReader.h"
+#include "llvm/Support/Chrono.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/Process.h"
+#include "llvm/Support/raw_ostream.h"
+
+#include "clang/Index/IndexWhileBuilding/PathStorage.h"
+
+using namespace clang;
+using namespace clang::index;
+using namespace clang::index::store;
+using namespace llvm;
+
+namespace {
+
+class IndexUnitBitstreamVisitor
+    : public BitstreamVisitor<IndexUnitBitstreamVisitor> {
+public:
+  llvm::BitstreamCursor DependenciesBlockCursor;
+  llvm::BitstreamCursor IncludedFilesCursor;
+  StringRef ProviderIdentifier;
+  StringRef ProviderVersion;
+  bool IsSystemUnit;
+  bool IsModuleUnit;
+  bool IsDebugCompilation;
+  StringRef ModuleName;
+  StringRef Target;
+
+  int MainPathIndex;
+
+  std::vector<FileBitPath> Paths;
+  StringRef PathsBuffer;
+  size_t WorkDirOffset;
+  size_t WorkDirSize;
+  size_t OutputFileOffset;
+  size_t OutputFileSize;
+  size_t SysrootOffset;
+  size_t SysrootSize;
+
+  std::vector<ModuleInfo> Modules;
+  StringRef ModuleNamesBuffer;
+
+  IndexUnitBitstreamVisitor(llvm::BitstreamCursor &Stream)
+      : BitstreamVisitor(Stream) {}
+
+  StreamVisit visitBlock(unsigned ID) {
+    switch ((UnitBitBlock)ID) {
+    case UNIT_VERSION_BLOCK_ID:
+    case UNIT_INFO_BLOCK_ID:
+    case UNIT_PATHS_BLOCK_ID:
+    case UNIT_MODULES_BLOCK_ID:
+      return StreamVisit::Continue;
+
+    case UNIT_DEPENDENCIES_BLOCK_ID:
+      DependenciesBlockCursor = Stream;
+      if (DependenciesBlockCursor.EnterSubBlock(ID)) {
+        *Error = "malformed unit dependencies block record";
+        return StreamVisit::Abort;
+      }
+      if (llvm::Error Err = readBlockAbbrevs(DependenciesBlockCursor)) {
+        *Error = toString(std::move(Err));
+        return StreamVisit::Abort;
+      }
+      return StreamVisit::Skip;
+    case UNIT_INCLUDES_BLOCK_ID:
+      IncludedFilesCursor = Stream;
+      if (IncludedFilesCursor.EnterSubBlock(ID)) {
+        *Error = "malformed unit includes block record";
+        return StreamVisit::Abort;
+      }
+      if (llvm::Error Err = readBlockAbbrevs(IncludedFilesCursor)) {
+        *Error = toString(std::move(Err));
+        return StreamVisit::Abort;
+      }
+      return StreamVisit::Skip;
+    }
+
+    // Some newly introduced block in a minor version update that we cannot
+    // handle.
+    return StreamVisit::Skip;
+  }
+
+  StreamVisit visitRecord(unsigned BlockID, unsigned RecID,
+                          SmallVectorImpl<uint64_t> &Record, StringRef Blob) {
+    switch (BlockID) {
+    case UNIT_VERSION_BLOCK_ID: {
+      unsigned StoreFormatVersion = Record[0];
+      if (StoreFormatVersion != getIndexFormatVersion()) {
+        llvm::raw_string_ostream OS(*Error);
+        OS << "Store format version mismatch: " << StoreFormatVersion;
+        OS << " , expected: " << getIndexFormatVersion();
+        return StreamVisit::Abort;
+      }
+      break;
+    }
+
+    case UNIT_INFO_BLOCK_ID: {
+      assert(RecID == UNIT_INFO);
+      unsigned I = 0;
+      IsSystemUnit = Record[I++];
+
+      // Save these to lookup them up after we get the paths buffer.
+      WorkDirOffset = Record[I++];
+      WorkDirSize = Record[I++];
+      OutputFileOffset = Record[I++];
+      OutputFileSize = Record[I++];
+      SysrootOffset = Record[I++];
+      SysrootSize = Record[I++];
+      MainPathIndex = (int)Record[I++] - 1;
+      IsDebugCompilation = Record[I++];
+      IsModuleUnit = Record[I++];
+
+      const size_t moduleNameSize = Record[I++];
+      const size_t providerIdentifierSize = Record[I++];
+      const size_t providerVersionSize = Record[I++];
+      I++; // Reserved for ProviderDataVersion.
+      ModuleName = Blob.substr(0, moduleNameSize);
+      Blob = Blob.drop_front(moduleNameSize);
+      ProviderIdentifier = Blob.substr(0, providerIdentifierSize);
+      Blob = Blob.drop_front(providerIdentifierSize);
+      ProviderVersion = Blob.substr(0, providerVersionSize);
+      Target = Blob.drop_front(providerVersionSize);
+      break;
+    }
+
+    case UNIT_PATHS_BLOCK_ID:
+      switch (RecID) {
+      case UNIT_PATH: {
+        unsigned I = 0;
+        UnitFilePathPrefixKind Kind = (UnitFilePathPrefixKind)Record[I++];
+        size_t DirOffset = Record[I++];
+        size_t DirSize = Record[I++];
+        size_t FilenameOffset = Record[I++];
+        size_t FilenameSize = Record[I++];
+
+        Paths.emplace_back(Kind, BitPathComponent(DirOffset, DirSize),
+                           BitPathComponent(FilenameOffset, FilenameSize));
+      } break;
+      case UNIT_PATH_BUFFER:
+        PathsBuffer = Blob;
+        break;
+      default:
+        llvm_unreachable("shouldn't visit this record");
+      }
+      break;
+
+    case UNIT_MODULES_BLOCK_ID:
+      switch (RecID) {
+      case UNIT_MODULE: {
+        unsigned I = 0;
+        unsigned NameOffset = Record[I++];
+        unsigned NameSize = Record[I++];
+        Modules.push_back({NameOffset, NameSize});
+      } break;
+      case UNIT_MODULE_BUFFER:
+        ModuleNamesBuffer = Blob;
+        break;
+      default:
+        llvm_unreachable("shouldn't visit this record");
+      }
+      break;
+
+    case UNIT_DEPENDENCIES_BLOCK_ID:
+    case UNIT_INCLUDES_BLOCK_ID:
+      llvm_unreachable("shouldn't visit this block'");
+    }
+    return StreamVisit::Continue;
+  }
+};
+
+typedef std::function<bool(SmallVectorImpl<uint64_t> &Record, StringRef Blob)>
+    BlockVisitorCallback;
+
+class IndexUnitBlockBitstreamVisitor
+    : public BitstreamVisitor<IndexUnitBlockBitstreamVisitor> {
+  unsigned RecID;
+  BlockVisitorCallback Callback;
+
+public:
+  IndexUnitBlockBitstreamVisitor(unsigned RecID,
+                                 llvm::BitstreamCursor &BlockStream,
+                                 BlockVisitorCallback Callback)
+      : BitstreamVisitor(BlockStream), RecID(RecID), Callback(Callback) {}
+
+  StreamVisit visitRecord(unsigned BlockID, unsigned RecID,
+                          SmallVectorImpl<uint64_t> &Record, StringRef Blob) {
+    if (RecID != this->RecID)
+      llvm_unreachable("shouldn't be called with this RecID");
+
+    if (Callback(Record, Blob))
+      return StreamVisit::Continue;
+    return StreamVisit::Abort;
+  }
+};
+
+} // anonymous namespace
+
+std::unique_ptr<IndexUnitReader>
+IndexUnitReader::create(std::unique_ptr<MemoryBuffer> Buf,
+                        sys::TimePoint<> ModTime, std::string &Error) {
+  if (!Buf) {
+    Error = "null buffer";
+    return nullptr;
+  }
+
+  llvm::BitstreamCursor Stream(*Buf);
+
+  if (Stream.AtEndOfStream()) {
+    Error = "empty file";
+    return nullptr;
+  }
+
+  // Sniff for the signature.
+  for (unsigned char C : {'I', 'D', 'X', 'U'}) {
+    if (Expected<llvm::SimpleBitstreamCursor::word_t> Res = Stream.Read(8)) {
+      if (Res.get() == C)
+        continue;
+    } else {
+      Error = toString(Res.takeError());
+      return nullptr;
+    }
+    Error = "not a serialized index unit file";
+    return nullptr;
+  }
+
+  IndexUnitBitstreamVisitor Visitor(Stream);
+  if (!Visitor.visit(Error)) {
+    // Error is set by Visitor
+    return nullptr;
+  }
+
+  return llvm::make_unique<IndexUnitReader>(
+      std::move(Buf), std::move(Visitor.DependenciesBlockCursor),
+      std::move(Visitor.IncludedFilesCursor), Visitor.ProviderIdentifier,
+      Visitor.ProviderVersion, Visitor.IsSystemUnit, Visitor.IsModuleUnit,
+      Visitor.IsDebugCompilation, Visitor.ModuleName, Visitor.Target,
+      Visitor.MainPathIndex, std::move(Visitor.Paths), Visitor.PathsBuffer,
+      Visitor.WorkDirOffset, Visitor.WorkDirSize, Visitor.OutputFileOffset,
+      Visitor.OutputFileSize, Visitor.SysrootOffset, Visitor.SysrootSize,
+      std::move(Visitor.Modules), Visitor.ModuleNamesBuffer, ModTime);
+}
+
+std::unique_ptr<IndexUnitReader> IndexUnitReader::createWithUnitFilename(
+    StringRef UnitFilename, StringRef StorePath, std::string &Error) {
+  SmallString<128> PathBuf = StorePath;
+  appendUnitSubDir(PathBuf);
+  sys::path::append(PathBuf, UnitFilename);
+  return createWithFilePath(PathBuf.str(), Error);
+}
+
+std::unique_ptr<IndexUnitReader>
+IndexUnitReader::createWithFilePath(StringRef FilePath, std::string &Error) {
+  int FD;
+  std::error_code EC = sys::fs::openFileForRead(FilePath, FD);
+  if (EC) {
+    raw_string_ostream(Error)
+        << "Failed opening '" << FilePath << "': " << EC.message();
+    return nullptr;
+  }
+  assert(FD != -1);
+  auto FDScopeGuard = llvm::make_scope_exit(
+      [FD]() { llvm::sys::Process::SafelyCloseFileDescriptor(FD); });
+
+  sys::fs::file_status FileStat;
+  EC = sys::fs::status(FD, FileStat);
+  if (EC) {
+    Error = EC.message();
+    return nullptr;
+  }
+
+  auto ErrOrBuf = MemoryBuffer::getOpenFile(sys::fs::convertFDToNativeFile(FD),
+                                            FilePath, /*FileSize=*/-1,
+                                            /*RequiresNullTerminator=*/false);
+  if (!ErrOrBuf) {
+    raw_string_ostream(Error) << "Failed opening '" << FilePath
+                              << "': " << ErrOrBuf.getError().message();
+    return nullptr;
+  }
+
+  return IndexUnitReader::create(std::move(*ErrOrBuf),
+                                 FileStat.getLastModificationTime(), Error);
+}
+
+Optional<sys::TimePoint<>> IndexUnitReader::getModificationTimeForUnit(
+    StringRef UnitFilename, StringRef StorePath, std::string &Error) {
+  SmallString<128> PathBuf = StorePath;
+  appendUnitSubDir(PathBuf);
+  sys::path::append(PathBuf, UnitFilename);
+
+  sys::fs::file_status FileStat;
+  std::error_code EC = sys::fs::status(PathBuf.str(), FileStat);
+  if (EC) {
+    Error = EC.message();
+    return None;
+  }
+  return FileStat.getLastModificationTime();
+}
+
+StringRef IndexUnitReader::getProviderIdentifier() const {
+  return ProviderIdentifier;
+}
+
+StringRef IndexUnitReader::getProviderVersion() const {
+  return ProviderVersion;
+}
+
+llvm::sys::TimePoint<> IndexUnitReader::getModificationTime() const {
+  return ModTime;
+}
+
+StringRef IndexUnitReader::getWorkingDirectory() const {
+  return PathsBuffer.substr(WorkDirOffset, WorkDirSize);
+}
+
+StringRef IndexUnitReader::getOutputFile() const {
+  return PathsBuffer.substr(OutputFileOffset, OutputFileSize);
+}
+
+StringRef IndexUnitReader::getSysrootPath() const {
+  return PathsBuffer.substr(SysrootOffset, SysrootSize);
+}
+
+StringRef IndexUnitReader::getMainFilePath() const { return MainFilePath; }
+
+StringRef IndexUnitReader::getModuleName() const { return ModuleName; }
+
+StringRef IndexUnitReader::getTarget() const { return Target; }
+
+bool IndexUnitReader::hasMainFile() const { return !MainFilePath.empty(); }
+
+bool IndexUnitReader::isSystemUnit() const { return IsSystemUnit; }
+
+bool IndexUnitReader::isModuleUnit() const { return IsModuleUnit; }
+
+bool IndexUnitReader::isDebugCompilation() const { return IsDebugCompilation; }
+
+void IndexUnitReader::constructFilePath(SmallVectorImpl<char> &PathBuf,
+                                        int PathIndex) const {
+
+  if (PathIndex < 0)
+    return;
+  assert(PathIndex < Paths.size());
+  const FileBitPath &Path = Paths[PathIndex];
+  StringRef Prefix;
+  switch (Path.PrefixKind) {
+  case UNIT_PATH_PREFIX_NONE:
+    break;
+  case UNIT_PATH_PREFIX_WORKDIR:
+    Prefix = getWorkingDirectory();
+    break;
+  case UNIT_PATH_PREFIX_SYSROOT:
+    Prefix = getSysrootPath();
+    break;
+  }
+  PathBuf.append(Prefix.begin(), Prefix.end());
+  sys::path::append(
+      PathBuf, PathsBuffer.substr(Path.Dir.Offset, Path.Dir.Size),
+      PathsBuffer.substr(Path.Filename.Offset, Path.Filename.Size));
+}
+
+/// \c Index is the index in the \c getDependencies array.
+/// Unit dependencies are provided ahead of record ones, record ones
+/// ahead of the file ones.
+bool IndexUnitReader::foreachDependency(
+    llvm::function_ref<bool(const DependencyInfo &Info)> Receiver) {
+  store::SavedStreamPosition SavedDepPosition(DependenciesBlockCursor);
+  IndexUnitBlockBitstreamVisitor Visitor(
+      UNIT_DEPENDENCY, DependenciesBlockCursor,
+      [&](SmallVectorImpl<uint64_t> &Record, StringRef Blob) {
+        unsigned I = 0;
+        UnitDependencyKind UnitDepKind = (UnitDependencyKind)Record[I++];
+        bool IsSystem = Record[I++];
+        int PathIndex = (int)Record[I++] - 1;
+        int ModuleIndex = (int)Record[I++] - 1;
+        I++; // Reserved field.
+        I++; // Reserved field.
+        StringRef Name = Blob;
+
+        IndexUnitReader::DependencyKind DepKind;
+        switch (UnitDepKind) {
+        case UNIT_DEPEND_KIND_UNIT:
+          DepKind = IndexUnitReader::DependencyKind::Unit;
+          break;
+        case UNIT_DEPEND_KIND_RECORD:
+          DepKind = IndexUnitReader::DependencyKind::Record;
+          break;
+        case UNIT_DEPEND_KIND_FILE:
+          DepKind = IndexUnitReader::DependencyKind::File;
+          break;
+        }
+
+        SmallString<512> PathBuf;
+        constructFilePath(PathBuf, PathIndex);
+
+        StringRef ModuleName;
+        if (ModuleIndex >= 0) {
+          auto &ModInfo = Modules[ModuleIndex];
+          ModuleName = StringRef(ModuleNamesBuffer.data() + ModInfo.NameOffset,
+                                 ModInfo.NameSize);
+        }
+
+        return Receiver(IndexUnitReader::DependencyInfo{
+            DepKind, IsSystem, Name, PathBuf.str(), ModuleName});
+      });
+
+  std::string Error; // FIXME
+  return Visitor.visit(Error);
+}
+
+bool IndexUnitReader::foreachInclude(
+    llvm::function_ref<bool(const IncludeInfo &Info)> Receiver) {
+  store::SavedStreamPosition SavedIncPosition(IncludedFilesCursor);
+  IndexUnitBlockBitstreamVisitor Visitor(
+      UNIT_INCLUDE, IncludedFilesCursor,
+      [&](SmallVectorImpl<uint64_t> &Record, StringRef Blob) {
+        unsigned I = 0;
+        int SourcePathIndex = (int)Record[I++] - 1;
+        unsigned Line = Record[I++];
+        int TargetPathIndex = (int)Record[I++] - 1;
+
+        SmallString<512> SourceBuf, TargetBuf;
+        this->constructFilePath(SourceBuf, SourcePathIndex);
+        this->constructFilePath(TargetBuf, TargetPathIndex);
+        return Receiver(IndexUnitReader::IncludeInfo{SourceBuf.str(), Line,
+                                                     TargetBuf.str()});
+      });
+
+  std::string Error;
+  return Visitor.visit(Error);
+}
+
+IndexUnitReader::IndexUnitReader(
+    std::unique_ptr<llvm::MemoryBuffer> Buf,
+    llvm::BitstreamCursor &&DependenciesBlockCursor,
+    llvm::BitstreamCursor &&IncludedFilesCursor, StringRef ProviderIdentifier,
+    StringRef ProviderVersion, bool IsSystemUnit, bool IsModuleUnit,
+    bool IsDebugCompilation, StringRef ModuleName, StringRef Target,
+    int MainPathIndex, std::vector<FileBitPath> &&Paths, StringRef PathsBuffer,
+    // Offset-size pairs to the PathsBuffer.
+    size_t WorkDirOffset, size_t WorkDirSize, size_t OutputFileOffset,
+    size_t OutputFileSize, size_t SysrootOffset, size_t SysrootSize,
+    std::vector<ModuleInfo> &&Modules, StringRef ModuleNamesBuffer,
+    llvm::sys::TimePoint<> ModTime)
+    : Buf(std::move(Buf)), DependenciesBlockCursor(DependenciesBlockCursor),
+      IncludedFilesCursor(IncludedFilesCursor),
+      ProviderIdentifier(ProviderIdentifier), ProviderVersion(ProviderVersion),
+      IsSystemUnit(IsSystemUnit), IsModuleUnit(IsModuleUnit),
+      IsDebugCompilation(IsDebugCompilation), ModuleName(ModuleName),
+      Target(Target), Paths(Paths), PathsBuffer(PathsBuffer),
+      WorkDirOffset(WorkDirOffset), WorkDirSize(WorkDirSize),
+      OutputFileOffset(OutputFileOffset), OutputFileSize(OutputFileSize),
+      SysrootOffset(SysrootOffset), SysrootSize(SysrootSize), Modules(Modules),
+      ModuleNamesBuffer(ModuleNamesBuffer), ModTime(ModTime) {
+  constructFilePath(MainFilePath, MainPathIndex);
+}
 No newline at end of file

clang/lib/Index/IndexWhileBuilding/IndexUnitWriter.cpp

+//===--- IndexUnitWriter.cpp - Index unit serialization -------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Index/IndexWhileBuilding/IndexUnitWriter.h"
+#include "BitstreamUtil.h"
+#include "FSUtil.h"
+#include "clang/Basic/FileManager.h"
+#include "clang/Basic/Module.h"
+#include "clang/Basic/TargetOptions.h"
+#include "clang/Basic/Version.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Index/IndexWhileBuilding/IndexDataFormat.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/StringMap.h"
+#include "llvm/Bitstream/BitstreamWriter.h"
+#include "llvm/Support/Allocator.h"
+#include "llvm/Support/Errc.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/raw_ostream.h"
+
+namespace {
+// FIXME: should this be in clang/Basic/Version?
+std::string getClangVersion() {
+  // Try picking the version from an Apple Clang tag.
+  const std::string RepositoryPath = clang::getClangRepositoryPath();
+  llvm::StringRef BuildNumber(RepositoryPath);
+  if (BuildNumber.startswith("clang")) {
+    const size_t DashOffset = BuildNumber.find('-');
+    if (DashOffset != llvm::StringRef::npos)
+      return BuildNumber.substr(DashOffset + 1);
+  }
+  // Fallback to the generic version.
+  return CLANG_VERSION_STRING;
+}
+} // namespace
+
+namespace clang {
+namespace index {
+
+using namespace clang::index::store;
+using namespace llvm;
+using namespace llvm::sys;
+
+unsigned IndexUnitWriter::addModule(writer::OpaqueModule Mod) {
+  if (!Mod)
+    return 0;
+  const auto Inserted = ModToIdx.insert({Mod, ModToIdx.size() + 1});
+  if (Inserted.second) {
+    SmallString<64> ModuleName;
+    StringRef name = GetInfoForModuleFn(Mod, ModuleName).Name;
+    const size_t ModuleNameOffset = ModuleNamesBuf.size();
+    ModuleNamesBuf += name;
+
+    Modules.push_back({UNIT_MODULE, ModuleNameOffset, name.size()});
+  }
+  return Inserted.first->second;
+}
+
+IndexUnitWriter::IndexUnitWriter(
+    FileManager &FileMgr, const EmitIndexOptions &RecOpts,
+    StringRef ProviderIdentifier, StringRef ProviderVersion,
+    StringRef OutputFile, StringRef AbsPathToOutputFile,
+    const FileEntry *MainFile,
+    const llvm::Optional<ModuleIndexData> &ModuleData, bool IsDebugCompilation,
+    StringRef TargetTriple, StringRef WorkDir, StringRef SysrootPath,
+    writer::ModuleInfoWriterCallback GetInfoForModule)
+    : RecOpts(RecOpts), ProviderIdentifier(ProviderIdentifier),
+      ProviderVersion(ProviderVersion), OutputFile(OutputFile),
+      AbsPathToOutputFile(AbsPathToOutputFile), MainFile(MainFile),
+      ModuleData(ModuleData), IsDebugCompilation(IsDebugCompilation),
+      TargetTriple(TargetTriple), WorkDir(WorkDir), SysrootPath(SysrootPath),
+      PathStore(WorkDir, SysrootPath), GetInfoForModuleFn(GetInfoForModule) {}
+
+void IndexUnitWriter::addSourceFile(const FileEntry *File, bool IsSystem,
+                                    writer::OpaqueModule Mod) {
+  assert(File);
+
+  SourceFiles[File->getUID()] = std::array<uint64_t, 7>{
+      UNIT_DEPENDENCY,
+      UNIT_DEPEND_KIND_FILE,
+      IsSystem,
+      static_cast<uint64_t>(PathStore.getPathIndex(File) + 1),
+      addModule(Mod),
+      0, // Reserved.
+      0  // Reserved.
+  };
+}
+
+void IndexUnitWriter::addIndexRecord(StringRef FileName,
+                                     const FileEntry *SourceFile, bool IsSystem,
+                                     writer::OpaqueModule Mod) {
+  assert(SourceFile);
+
+  SourceFileHasARecord.insert(SourceFile->getUID());
+
+  IndexRecordDependencies.push_back(
+      std::make_pair<std::string, std::array<uint64_t, 7>>(
+          FileName,
+          {
+              UNIT_DEPENDENCY, UNIT_DEPEND_KIND_RECORD, IsSystem,
+              static_cast<uint64_t>(PathStore.getPathIndex(SourceFile) + 1),
+              addModule(Mod),
+              0, // Reserved.
+              0  // Reserved.
+          }));
+}
+
+void IndexUnitWriter::addIndexUnit(const FileEntry *ASTFile,
+                                   StringRef AbsolutePath, bool IsSystem,
+                                   writer::OpaqueModule Mod,
+                                   bool withoutUnitName) {
+  assert(ASTFile);
+  if (!ASTFilesOfSeenIndexUnits.insert(ASTFile->getUID()).second)
+    return;
+
+  SmallString<64> UnitName;
+  if (!withoutUnitName) {
+    getUnitNameForOutputFile(AbsolutePath, UnitName);
+  }
+
+  IndexUnitDependencies.emplace_back(
+      UnitName, std::array<uint64_t, 7>{
+                    UNIT_DEPENDENCY, UNIT_DEPEND_KIND_UNIT, IsSystem,
+                    static_cast<uint64_t>(PathStore.getPathIndex(ASTFile) + 1),
+                    addModule(Mod),
+                    0, // Reserved.
+                    0  // Reserved.
+                });
+}
+
+void IndexUnitWriter::addIncludedFile(const FileEntry *IncludingFile,
+                                      unsigned LineNo,
+                                      const FileEntry *IncludedFile) {
+  IncludedFiles.push_back(
+      {UNIT_INCLUDE, // FIXME: This is static data which don't have to be stored
+                     // here.
+       static_cast<uint64_t>(PathStore.getPathIndex(IncludingFile) + 1), LineNo,
+       static_cast<uint64_t>(PathStore.getPathIndex(IncludedFile) + 1)});
+};
+
+static void writeBlockInfo(BitstreamWriter &Stream) {
+  SmallVector<uint64_t, 64> Record;
+
+  Stream.EnterBlockInfoBlock();
+#define BLOCK(X) emitBlockID(X##_ID, #X, Stream, Record)
+#define RECORD(X) emitRecordID(X, #X, Stream, Record)
+
+  BLOCK(UNIT_VERSION_BLOCK);
+  RECORD(UNIT_VERSION);
+
+  BLOCK(UNIT_INFO_BLOCK);
+  RECORD(UNIT_INFO);
+
+  BLOCK(UNIT_DEPENDENCIES_BLOCK);
+  RECORD(UNIT_DEPENDENCY);
+
+  BLOCK(UNIT_INCLUDES_BLOCK);
+  RECORD(UNIT_INCLUDE);
+
+  BLOCK(UNIT_PATHS_BLOCK);
+  RECORD(UNIT_PATH);
+  RECORD(UNIT_PATH_BUFFER);
+
+  BLOCK(UNIT_MODULES_BLOCK);
+  RECORD(UNIT_MODULE);
+  RECORD(UNIT_MODULE_BUFFER);
+
+#undef RECORD
+#undef BLOCK
+  Stream.ExitBlock();
+}
+
+static std::shared_ptr<BitCodeAbbrev> UNIT_VERSION_BLOCK_ID_Abbrev() {
+  static const std::shared_ptr<BitCodeAbbrev> CachedResult(new BitCodeAbbrev({
+      BitCodeAbbrevOp(UNIT_VERSION),
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 6) // Store format version
+  }));
+
+  return CachedResult;
+}
+
+static void writeVersionInfo(BitstreamWriter &Stream) {
+  using namespace llvm::sys;
+
+  Stream.EnterSubblock(UNIT_VERSION_BLOCK_ID, 3);
+  unsigned AbbrevCode = Stream.EmitAbbrev(UNIT_VERSION_BLOCK_ID_Abbrev());
+
+  const uint64_t Record[2]{UNIT_VERSION, getIndexFormatVersion()};
+  Stream.EmitRecordWithAbbrev(AbbrevCode, Record);
+
+  Stream.ExitBlock();
+}
+
+bool IndexUnitWriter::writeToDisk(std::string &Error) const {
+  SmallString<512> BitcodeRepresentation;
+  getBitcodeRepresentation(BitcodeRepresentation, Error);
+
+  SmallString<256> UnitPath =
+      StringRef(llvm::Twine(RecOpts.UnitDir)
+                    .concat(llvm::sys::path::get_separator())
+                    .str());
+
+  getUnitNameForOutputFile(AbsPathToOutputFile, UnitPath);
+
+  const SmallString<128> TempPath(
+      llvm::Twine(path::parent_path(RecOpts.UnitDir))
+          .concat(llvm::sys::path::get_separator().str())
+          .concat(path::filename(UnitPath))
+          .concat("-%%%%%%%%")
+          .str());
+
+  return atomicFileWrite(TempPath, UnitPath, BitcodeRepresentation, Error);
+}
+
+void IndexUnitWriter::getBitcodeRepresentation(SmallString<512> &Buffer,
+                                               std::string &Error) const {
+  using namespace llvm::sys;
+
+  BitstreamWriter Stream(Buffer);
+  Stream.Emit('I', 8);
+  Stream.Emit('D', 8);
+  Stream.Emit('X', 8);
+  Stream.Emit('U', 8);
+
+  writeBlockInfo(Stream);
+  writeVersionInfo(Stream);
+  writeUnitInfo(Stream);
+  writeDependencies(Stream);
+  writeIncludes(Stream);
+  writePaths(Stream);
+  writeModules(Stream);
+}
+
+static std::shared_ptr<BitCodeAbbrev> UNIT_INFO_BLOCK_ID_Abbrev() {
+  static const std::shared_ptr<BitCodeAbbrev> CachedResult(new BitCodeAbbrev({
+      BitCodeAbbrevOp(UNIT_INFO),
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 1), // IsSystemUnit
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 10),  // WorkDir offset
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 8),   // WorkDir size
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 10),  // OutputFile offset
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 8),   // OutputFile size
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 10),  // Sysroot offset
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 8),   // Sysroot size
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 10),  // Main path id
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 1), // IsDebugCompilation
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 1), // IsModuleUnit
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 5),   // Module name size
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 5),   // ProviderIdentifier size
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 5),   // ProviderVersion size
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 5),   // ProviderDataVersion
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Blob) // Module name + ProviderIdentifier
+                                             // + ProviderVersion + target
+                                             // triple
+  }));
+
+  return CachedResult;
+}
+
+void IndexUnitWriter::writeUnitInfo(llvm::BitstreamWriter &Stream) const {
+  Stream.EnterSubblock(UNIT_INFO_BLOCK_ID, 3);
+  unsigned AbbrevCode = Stream.EmitAbbrev(UNIT_INFO_BLOCK_ID_Abbrev());
+
+  const std::string ModuleName = ModuleData.hasValue() ? ModuleData->Name : "";
+
+  const uint64_t Record[15]{
+      UNIT_INFO,
+      ModuleData.hasValue() &&
+          ModuleData->IsSystem, // We don't expect compilation of any source
+                                // code that would be considered system unless
+                                // it's a dependency in form of module.
+      PathStore.getPathOffset(WorkDir), WorkDir.size(),
+      PathStore.getPathOffset(OutputFile), OutputFile.size(),
+      PathStore.getPathOffset(SysrootPath), SysrootPath.size(),
+      static_cast<uint64_t>(PathStore.getPathIndex(MainFile) +
+                            1), // Make 1-based with 0=invalid
+      IsDebugCompilation, ModuleData.hasValue(), ModuleName.size(),
+      ProviderIdentifier.size(), ProviderVersion.size(),
+      // ProviderDataVersion is reserved. Not sure it is a good to idea to have
+      // clients consider the specifics of a 'provider data version', but
+      // reserving to avoid store format version change in case there is a use
+      // case in the future.
+      0 // ProviderDataVersion
+  };
+  const std::string InfoStrings = llvm::Twine(ModuleName)
+                                      .concat(ProviderIdentifier)
+                                      .concat(ProviderVersion)
+                                      .concat(TargetTriple)
+                                      .str();
+
+  Stream.EmitRecordWithBlob(AbbrevCode, Record, InfoStrings);
+
+  Stream.ExitBlock();
+}
+
+static std::shared_ptr<BitCodeAbbrev> UNIT_DEPENDENCIES_BLOCK_ID_Abbrev() {
+  static const std::shared_ptr<BitCodeAbbrev> CachedResult(new BitCodeAbbrev({
+      BitCodeAbbrevOp(UNIT_DEPENDENCY),
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed,
+                      UnitDependencyKindBitNum),  // Dependency kind
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 1), // IsSystem
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR,
+                      10), // PathIndex (1-based, 0 = none)
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR,
+                      8), // ModuleIndex (1-based, 0 = none)
+      // Reserved. These used to be time_t & file size but we decided against
+      // writing these in order to get reproducible build products (index data
+      // output being the same with the same inputs). Keep these reserved for
+      // the future, for coming up with a better scheme to track state of
+      // dependencies without using modification time.
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 0), // Reserved
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 0), // Reserved
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Blob)      // Name
+  }));
+
+  return CachedResult;
+}
+
+void IndexUnitWriter::writeDependencies(llvm::BitstreamWriter &Stream) const {
+  Stream.EnterSubblock(UNIT_DEPENDENCIES_BLOCK_ID, 3);
+
+  unsigned AbbrevCode = Stream.EmitAbbrev(UNIT_DEPENDENCIES_BLOCK_ID_Abbrev());
+
+  for (auto &IdxUnit : IndexUnitDependencies) {
+    Stream.EmitRecordWithBlob(AbbrevCode, IdxUnit.second, IdxUnit.first);
+  }
+
+  for (auto &recordData : IndexRecordDependencies) {
+    Stream.EmitRecordWithBlob(AbbrevCode, recordData.second, recordData.first);
+  }
+
+  for (const auto &FileAndData : SourceFiles) {
+    if (SourceFileHasARecord.count(FileAndData.first) > 0)
+      continue;
+    Stream.EmitRecordWithBlob(AbbrevCode, FileAndData.second, StringRef());
+  }
+
+  Stream.ExitBlock();
+}
+
+static std::shared_ptr<BitCodeAbbrev> UNIT_INCLUDES_BLOCK_ID_Abbrev() {
+  static const std::shared_ptr<BitCodeAbbrev> CachedResult(new BitCodeAbbrev({
+      BitCodeAbbrevOp(UNIT_INCLUDE),
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR,
+                      10), // source path index (1-based, 0 = no path)
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 12), // source include line
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR,
+                      10) // target path index (1-based, 0 = no path)
+  }));
+
+  return CachedResult;
+}
+
+void IndexUnitWriter::writeIncludes(llvm::BitstreamWriter &Stream) const {
+  Stream.EnterSubblock(UNIT_INCLUDES_BLOCK_ID, 3);
+
+  unsigned AbbrevCode = Stream.EmitAbbrev(UNIT_INCLUDES_BLOCK_ID_Abbrev());
+
+  for (const auto &IncludeDirective : IncludedFiles) {
+    Stream.EmitRecordWithAbbrev(AbbrevCode, IncludeDirective);
+  }
+
+  Stream.ExitBlock();
+}
+
+static std::shared_ptr<BitCodeAbbrev> UNIT_PATHS_BLOCK_ID_Paths_Abbrev() {
+  static const std::shared_ptr<BitCodeAbbrev> CachedResult(new BitCodeAbbrev({
+      BitCodeAbbrevOp(UNIT_PATH),
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed,
+                      UnitFilePathPrefixKindBitNum), // Path prefix kind
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 10),     // DirPath offset
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 8),      // DirPath size
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 10),     // Filename offset
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 6)       // Filename size
+  }));
+
+  return CachedResult;
+}
+
+static std::shared_ptr<BitCodeAbbrev>
+UNIT_PATHS_BLOCK_ID_Path_Buffers_Abbrev() {
+  static const std::shared_ptr<BitCodeAbbrev> CachedResult(new BitCodeAbbrev({
+      BitCodeAbbrevOp(UNIT_PATH_BUFFER),
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Blob) // Paths buffer
+  }));
+
+  return CachedResult;
+}
+
+void IndexUnitWriter::writePaths(llvm::BitstreamWriter &Stream) const {
+  Stream.EnterSubblock(UNIT_PATHS_BLOCK_ID, 3);
+
+  const unsigned PathAbbrevCode =
+      Stream.EmitAbbrev(UNIT_PATHS_BLOCK_ID_Paths_Abbrev());
+  const unsigned PathBufferAbbrevCode =
+      Stream.EmitAbbrev(UNIT_PATHS_BLOCK_ID_Path_Buffers_Abbrev());
+
+  for (auto &BitPath : PathStore.getBitPaths()) {
+    const uint64_t Record[6]{
+        UNIT_PATH,        BitPath.PrefixKind,      BitPath.Dir.Offset,
+        BitPath.Dir.Size, BitPath.Filename.Offset, BitPath.Filename.Size};
+    Stream.EmitRecordWithAbbrev(PathAbbrevCode, Record);
+  }
+
+  Stream.EmitRecordWithBlob(PathBufferAbbrevCode, UNIT_PATH_BUFFER,
+                            PathStore.getPathsBuffer());
+
+  Stream.ExitBlock();
+}
+
+static std::shared_ptr<BitCodeAbbrev> UNIT_MODULES_BLOCK_ID_Module_Abbrev() {
+  static const std::shared_ptr<BitCodeAbbrev> CachedResult(new BitCodeAbbrev({
+      BitCodeAbbrevOp(UNIT_MODULE),
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 9), // Module name offset
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 6)  // Module name size
+  }));
+
+  return CachedResult;
+}
+
+static std::shared_ptr<BitCodeAbbrev>
+UNIT_MODULES_BLOCK_ID_Module_Buffer_Abbrev() {
+  static const std::shared_ptr<BitCodeAbbrev> CachedResult(new BitCodeAbbrev({
+      BitCodeAbbrevOp(UNIT_MODULE_BUFFER),
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Blob) // Module names buffer
+  }));
+
+  return CachedResult;
+}
+
+void IndexUnitWriter::writeModules(llvm::BitstreamWriter &Stream) const {
+  Stream.EnterSubblock(UNIT_MODULES_BLOCK_ID, 3);
+  unsigned AbbrevCode =
+      Stream.EmitAbbrev(UNIT_MODULES_BLOCK_ID_Module_Abbrev());
+  unsigned BufferAbbrevCode =
+      Stream.EmitAbbrev(UNIT_MODULES_BLOCK_ID_Module_Buffer_Abbrev());
+
+  for (auto &Module : Modules) {
+    Stream.EmitRecordWithAbbrev(AbbrevCode, Module);
+  }
+
+  Stream.EmitRecordWithBlob(BufferAbbrevCode, UNIT_MODULE_BUFFER,
+                            ModuleNamesBuf.str());
+  Stream.ExitBlock();
+}
+
+static writer::ModuleInfo getModuleInfo(writer::OpaqueModule mod,
+                                        SmallVectorImpl<char> &Scratch) {
+  assert(mod);
+  writer::ModuleInfo info;
+  const std::string fullName =
+      static_cast<const Module *>(mod)->getFullModuleName();
+  unsigned offset = Scratch.size();
+  Scratch.append(fullName.begin(), fullName.end());
+  info.Name = StringRef(Scratch.data() + offset, fullName.size());
+  return info;
+}
+
+IndexUnitWriter makeIndexUnitWriter(const CompilerInstance &CI,
+                                    const EmitIndexOptions &RecordOpts,
+                                    StringRef OutputFile,
+                                    const FileEntry *MainFile,
+                                    Module *UnitModule) {
+  assert(MainFile == nullptr || UnitModule == nullptr);
+
+  SmallString<256> AbsOutputFilePath(OutputFile);
+  CI.getFileManager().makeAbsolutePath(AbsOutputFilePath);
+
+  llvm::Optional<ModuleIndexData> ModuleData =
+      [UnitModule]() -> llvm::Optional<ModuleIndexData> {
+    if (UnitModule) {
+      return ModuleIndexData{UnitModule->getFullModuleName(),
+                             static_cast<bool>(UnitModule->IsSystem)};
+    }
+    return llvm::None;
+  }();
+
+  SmallString<128> WorkDir;
+  if (!CI.getFileManager().getFileSystemOpts().WorkingDir.empty()) {
+    WorkDir = CI.getFileManager().getFileSystemOpts().WorkingDir;
+    if (!llvm::sys::path::is_absolute(WorkDir)) {
+      llvm::sys::fs::make_absolute(WorkDir);
+    }
+  } else {
+    std::error_code EC = llvm::sys::fs::current_path(WorkDir);
+    if (EC) {
+      // FIXME: error handling
+      llvm_unreachable("failed to determine current working directory");
+    }
+  }
+
+  return IndexUnitWriter{CI.getFileManager(),
+                         RecordOpts,
+                         "clang",
+                         getClangVersion(),
+                         OutputFile,
+                         AbsOutputFilePath,
+                         MainFile,
+                         ModuleData,
+                         CI.getCodeGenOpts().OptimizationLevel == 0,
+                         CI.getTargetOpts().Triple,
+                         WorkDir.str(),
+                         CI.getHeaderSearchOpts().Sysroot,
+                         getModuleInfo};
+}
+
+bool initIndexDirectory(const EmitIndexOptions &RecordingOpts,
+                        std::string &Error /* <- kill */) {
+  using namespace llvm::sys;
+  auto CreateDir = [&](StringRef DirPath) {
+    std::error_code EC = fs::create_directories(DirPath);
+    if (EC) {
+      llvm::raw_string_ostream Err(Error);
+      Err << "failed to create directory '" << DirPath << "': " << EC.message();
+      return true;
+    }
+    return false;
+  };
+  if (CreateDir(RecordingOpts.IndexDir))
+    return true;
+  if (CreateDir(RecordingOpts.RecordDir))
+    return true;
+  if (CreateDir(RecordingOpts.UnitDir))
+    return true;
+
+  return false;
+}
+
+llvm::Optional<bool>
+doesIndexUnitForModuleExist(StringRef ModuleFilePath,
+                            const FileManager &FileMgr,
+                            const EmitIndexOptions &RecOpts) {
+  SmallString<256> UnitPath(
+      StringRef(llvm::Twine(RecOpts.UnitDir)
+                    .concat(llvm::sys::path::get_separator().str())
+                    .str()));
+
+  SmallString<256> AbsPath(ModuleFilePath);
+  FileMgr.makeAbsolutePath(AbsPath);
+
+  getUnitNameForOutputFile(AbsPath, UnitPath);
+
+  llvm::sys::fs::file_status UnitStat;
+  if (std::error_code EC = llvm::sys::fs::status(UnitPath.c_str(), UnitStat)) {
+    if (EC == llvm::errc::no_such_file_or_directory)
+      return false;
+
+    return None;
+  }
+  return true;
+}
+
+} // namespace index
+} // namespace clang
 No newline at end of file

clang/lib/Index/IndexWhileBuilding/PathStorage.cpp

+//===--- PathStorage.cpp -- Index of paths ----------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Index/IndexWhileBuilding/PathStorage.h"
+
+using namespace llvm;
+using namespace clang;
+
+DirBitPath::DirBitPath(UnitFilePathPrefixKind Kind, BitPathComponent Dir)
+    : PrefixKind(Kind), Dir(Dir) {}
+DirBitPath::DirBitPath() : PrefixKind(UNIT_PATH_PREFIX_NONE) {}
+
+PathStorage::PathStorage(StringRef workDir, StringRef sysrootPath) {
+  WorkDir = workDir;
+  if (sysrootPath == "/")
+    sysrootPath = StringRef();
+  SysrootPath = sysrootPath;
+}
+
+int PathStorage::getPathIndex(const FileEntry *FE) {
+  if (!FE)
+    return -1;
+  auto Pair = FileToIndex.insert(std::make_pair(FE, FileBitPaths.size()));
+  bool IsNew = Pair.second;
+  size_t Index = Pair.first->getSecond();
+
+  if (IsNew) {
+    StringRef Filename = sys::path::filename(FE->getName());
+    DirBitPath Dir = getDirBitPath(sys::path::parent_path(FE->getName()));
+    FileBitPaths.emplace_back(
+        Dir.PrefixKind, Dir.Dir,
+        BitPathComponent(getPathOffset(Filename), Filename.size()));
+  }
+  return Index;
+}
+
+size_t PathStorage::getPathOffset(StringRef Path) {
+  if (Path.empty())
+    return 0;
+  size_t offset = PathsBuf.size();
+  PathsBuf += Path;
+  return offset;
+}
+
+static bool isPathInDir(StringRef dir, StringRef path) {
+  if (dir.empty() || !path.startswith(dir))
+    return false;
+  StringRef rest = path.drop_front(dir.size());
+  return !rest.empty() && sys::path::is_separator(rest.front());
+}
+
+DirBitPath PathStorage::getDirBitPath(StringRef dirStr) {
+  auto pair = Dirs.insert(std::make_pair(dirStr, DirBitPath()));
+  bool isNew = pair.second;
+  auto &dirPath = pair.first->second;
+
+  if (isNew) {
+    if (isPathInDir(SysrootPath, dirStr)) {
+      dirPath.PrefixKind = UNIT_PATH_PREFIX_SYSROOT;
+      dirStr = dirStr.drop_front(SysrootPath.size());
+      while (!dirStr.empty() && dirStr[0] == '/')
+        dirStr = dirStr.drop_front();
+    } else if (isPathInDir(WorkDir, dirStr)) {
+      dirPath.PrefixKind = UNIT_PATH_PREFIX_WORKDIR;
+      dirStr = dirStr.drop_front(WorkDir.size());
+      while (!dirStr.empty() && dirStr[0] == '/')
+        dirStr = dirStr.drop_front();
+    }
+    dirPath.Dir.Offset = getPathOffset(dirStr);
+    dirPath.Dir.Size = dirStr.size();
+  }
+  return dirPath;
+}

clang/lib/Index/IndexingAction.cpp

-//===- IndexingAction.cpp - Frontend index action -------------------------===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-
-#include "clang/Index/IndexingAction.h"
-#include "IndexingContext.h"
-#include "clang/Frontend/CompilerInstance.h"
-#include "clang/Frontend/FrontendAction.h"
-#include "clang/Frontend/MultiplexConsumer.h"
-#include "clang/Index/IndexDataConsumer.h"
-#include "clang/Lex/PPCallbacks.h"
-#include "clang/Lex/Preprocessor.h"
-#include "clang/Serialization/ASTReader.h"
-#include "llvm/ADT/STLExtras.h"
-#include <memory>
-
-using namespace clang;
-using namespace clang::index;
-
-bool IndexDataConsumer::handleDeclOccurence(const Decl *D, SymbolRoleSet Roles,
-                                            ArrayRef<SymbolRelation> Relations,
-                                            SourceLocation Loc,
-                                            ASTNodeInfo ASTNode) {
-  return true;
-}
-
-bool IndexDataConsumer::handleMacroOccurence(const IdentifierInfo *Name,
-                                             const MacroInfo *MI,
-                                             SymbolRoleSet Roles,
-                                             SourceLocation Loc) {
-  return true;
-}
-
-bool IndexDataConsumer::handleModuleOccurence(const ImportDecl *ImportD,
-                                              const Module *Mod,
-                                              SymbolRoleSet Roles,
-                                              SourceLocation Loc) {
-  return true;
-}
-
-namespace {
-
-class IndexASTConsumer : public ASTConsumer {
-  std::shared_ptr<Preprocessor> PP;
-  std::shared_ptr<IndexingContext> IndexCtx;
-
-public:
-  IndexASTConsumer(std::shared_ptr<Preprocessor> PP,
-                   std::shared_ptr<IndexingContext> IndexCtx)
-      : PP(std::move(PP)), IndexCtx(std::move(IndexCtx)) {}
-
-protected:
-  void Initialize(ASTContext &Context) override {
-    IndexCtx->setASTContext(Context);
-    IndexCtx->getDataConsumer().initialize(Context);
-    IndexCtx->getDataConsumer().setPreprocessor(PP);
-  }
-
-  bool HandleTopLevelDecl(DeclGroupRef DG) override {
-    return IndexCtx->indexDeclGroupRef(DG);
-  }
-
-  void HandleInterestingDecl(DeclGroupRef DG) override {
-    // Ignore deserialized decls.
-  }
-
-  void HandleTopLevelDeclInObjCContainer(DeclGroupRef DG) override {
-    IndexCtx->indexDeclGroupRef(DG);
-  }
-
-  void HandleTranslationUnit(ASTContext &Ctx) override {
-  }
-};
-
-class IndexPPCallbacks : public PPCallbacks {
-  std::shared_ptr<IndexingContext> IndexCtx;
-
-public:
-  IndexPPCallbacks(std::shared_ptr<IndexingContext> IndexCtx)
-      : IndexCtx(std::move(IndexCtx)) {}
-
-  void MacroExpands(const Token &MacroNameTok, const MacroDefinition &MD,
-                    SourceRange Range, const MacroArgs *Args) override {
-    IndexCtx->handleMacroReference(*MacroNameTok.getIdentifierInfo(),
-                                   Range.getBegin(), *MD.getMacroInfo());
-  }
-
-  void MacroDefined(const Token &MacroNameTok,
-                    const MacroDirective *MD) override {
-    IndexCtx->handleMacroDefined(*MacroNameTok.getIdentifierInfo(),
-                                 MacroNameTok.getLocation(),
-                                 *MD->getMacroInfo());
-  }
-
-  void MacroUndefined(const Token &MacroNameTok, const MacroDefinition &MD,
-                      const MacroDirective *Undef) override {
-    if (!MD.getMacroInfo())  // Ignore noop #undef.
-      return;
-    IndexCtx->handleMacroUndefined(*MacroNameTok.getIdentifierInfo(),
-                                   MacroNameTok.getLocation(),
-                                   *MD.getMacroInfo());
-  }
-};
-
-class IndexActionBase {
-protected:
-  std::shared_ptr<IndexDataConsumer> DataConsumer;
-  std::shared_ptr<IndexingContext> IndexCtx;
-
-  IndexActionBase(std::shared_ptr<IndexDataConsumer> dataConsumer,
-                  IndexingOptions Opts)
-      : DataConsumer(std::move(dataConsumer)),
-        IndexCtx(new IndexingContext(Opts, *DataConsumer)) {}
-
-  std::unique_ptr<IndexASTConsumer>
-  createIndexASTConsumer(CompilerInstance &CI) {
-    return llvm::make_unique<IndexASTConsumer>(CI.getPreprocessorPtr(),
-                                               IndexCtx);
-  }
-
-  std::unique_ptr<PPCallbacks> createIndexPPCallbacks() {
-    return llvm::make_unique<IndexPPCallbacks>(IndexCtx);
-  }
-
-  void finish() {
-    DataConsumer->finish();
-  }
-};
-
-class IndexAction : public ASTFrontendAction, IndexActionBase {
-public:
-  IndexAction(std::shared_ptr<IndexDataConsumer> DataConsumer,
-              IndexingOptions Opts)
-    : IndexActionBase(std::move(DataConsumer), Opts) {}
-
-protected:
-  std::unique_ptr<ASTConsumer> CreateASTConsumer(CompilerInstance &CI,
-                                                 StringRef InFile) override {
-    return createIndexASTConsumer(CI);
-  }
-
-  bool BeginSourceFileAction(clang::CompilerInstance &CI) override {
-    CI.getPreprocessor().addPPCallbacks(createIndexPPCallbacks());
-    return true;
-  }
-
-  void EndSourceFileAction() override {
-    FrontendAction::EndSourceFileAction();
-    finish();
-  }
-};
-
-class WrappingIndexAction : public WrapperFrontendAction, IndexActionBase {
-  bool IndexActionFailed = false;
-
-public:
-  WrappingIndexAction(std::unique_ptr<FrontendAction> WrappedAction,
-                      std::shared_ptr<IndexDataConsumer> DataConsumer,
-                      IndexingOptions Opts)
-    : WrapperFrontendAction(std::move(WrappedAction)),
-      IndexActionBase(std::move(DataConsumer), Opts) {}
-
-protected:
-  std::unique_ptr<ASTConsumer> CreateASTConsumer(CompilerInstance &CI,
-                                                 StringRef InFile) override {
-    auto OtherConsumer = WrapperFrontendAction::CreateASTConsumer(CI, InFile);
-    if (!OtherConsumer) {
-      IndexActionFailed = true;
-      return nullptr;
-    }
-
-    std::vector<std::unique_ptr<ASTConsumer>> Consumers;
-    Consumers.push_back(std::move(OtherConsumer));
-    Consumers.push_back(createIndexASTConsumer(CI));
-    return llvm::make_unique<MultiplexConsumer>(std::move(Consumers));
-  }
-
-  bool BeginSourceFileAction(clang::CompilerInstance &CI) override {
-    WrapperFrontendAction::BeginSourceFileAction(CI);
-    CI.getPreprocessor().addPPCallbacks(createIndexPPCallbacks());
-    return true;
-  }
-
-  void EndSourceFileAction() override {
-    // Invoke wrapped action's method.
-    WrapperFrontendAction::EndSourceFileAction();
-    if (!IndexActionFailed)
-      finish();
-  }
-};
-
-} // anonymous namespace
-
-std::unique_ptr<FrontendAction>
-index::createIndexingAction(std::shared_ptr<IndexDataConsumer> DataConsumer,
-                            IndexingOptions Opts,
-                            std::unique_ptr<FrontendAction> WrappedAction) {
-  if (WrappedAction)
-    return llvm::make_unique<WrappingIndexAction>(std::move(WrappedAction),
-                                                  std::move(DataConsumer),
-                                                  Opts);
-  return llvm::make_unique<IndexAction>(std::move(DataConsumer), Opts);
-}
-
-static bool topLevelDeclVisitor(void *context, const Decl *D) {
-  IndexingContext &IndexCtx = *static_cast<IndexingContext*>(context);
-  return IndexCtx.indexTopLevelDecl(D);
-}
-
-static void indexTranslationUnit(ASTUnit &Unit, IndexingContext &IndexCtx) {
-  Unit.visitLocalTopLevelDecls(&IndexCtx, topLevelDeclVisitor);
-}
-
-static void indexPreprocessorMacros(const Preprocessor &PP,
-                                    IndexDataConsumer &DataConsumer) {
-  for (const auto &M : PP.macros())
-    if (MacroDirective *MD = M.second.getLatest())
-      DataConsumer.handleMacroOccurence(
-          M.first, MD->getMacroInfo(),
-          static_cast<unsigned>(index::SymbolRole::Definition),
-          MD->getLocation());
-}
-
-void index::indexASTUnit(ASTUnit &Unit, IndexDataConsumer &DataConsumer,
-                         IndexingOptions Opts) {
-  IndexingContext IndexCtx(Opts, DataConsumer);
-  IndexCtx.setASTContext(Unit.getASTContext());
-  DataConsumer.initialize(Unit.getASTContext());
-  DataConsumer.setPreprocessor(Unit.getPreprocessorPtr());
-
-  if (Opts.IndexMacrosInPreprocessor)
-    indexPreprocessorMacros(Unit.getPreprocessor(), DataConsumer);
-  indexTranslationUnit(Unit, IndexCtx);
-  DataConsumer.finish();
-}
-
-void index::indexTopLevelDecls(ASTContext &Ctx, Preprocessor &PP,
-                               ArrayRef<const Decl *> Decls,
-                               IndexDataConsumer &DataConsumer,
-                               IndexingOptions Opts) {
-  IndexingContext IndexCtx(Opts, DataConsumer);
-  IndexCtx.setASTContext(Ctx);
-
-  DataConsumer.initialize(Ctx);
-
-  if (Opts.IndexMacrosInPreprocessor)
-    indexPreprocessorMacros(PP, DataConsumer);
-
-  for (const Decl *D : Decls)
-    IndexCtx.indexTopLevelDecl(D);
-  DataConsumer.finish();
-}
-
-std::unique_ptr<PPCallbacks>
-index::indexMacrosCallback(IndexDataConsumer &Consumer, IndexingOptions Opts) {
-  return llvm::make_unique<IndexPPCallbacks>(
-      std::make_shared<IndexingContext>(Opts, Consumer));
-}
-
-void index::indexModuleFile(serialization::ModuleFile &Mod, ASTReader &Reader,
-                            IndexDataConsumer &DataConsumer,
-                            IndexingOptions Opts) {
-  ASTContext &Ctx = Reader.getContext();
-  IndexingContext IndexCtx(Opts, DataConsumer);
-  IndexCtx.setASTContext(Ctx);
-  DataConsumer.initialize(Ctx);
-
-  if (Opts.IndexMacrosInPreprocessor)
-    indexPreprocessorMacros(Reader.getPreprocessor(), DataConsumer);
-
-  for (const Decl *D : Reader.getModuleFileLevelDecls(Mod)) {
-    IndexCtx.indexTopLevelDecl(D);
-  }
-  DataConsumer.finish();
-}

clang/lib/Index/IndexingContext.cpp

 //===- IndexingContext.cpp - Indexing context data ------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
-#include "IndexingContext.h"
-#include "clang/Basic/SourceLocation.h"
-#include "clang/Index/IndexDataConsumer.h"
+#include "clang/Index/IndexingContext.h"
 #include "clang/AST/ASTContext.h"
-#include "clang/AST/DeclTemplate.h"
 #include "clang/AST/DeclObjC.h"
+#include "clang/AST/DeclTemplate.h"
+#include "clang/Basic/SourceLocation.h"
 #include "clang/Basic/SourceManager.h"
+#include "clang/Index/IndexDataConsumer.h"
 
 using namespace clang;
 using namespace index;
 
 static bool isGeneratedDecl(const Decl *D) {
   if (auto *attr = D->getAttr<ExternalSourceSymbolAttr>()) {
     return attr->getGeneratedDeclaration();
   }
bool IndexingContext::importedModule(const ImportDecl *ImportD) {
   else
     Loc = ImportD->getLocation();
 
   SourceManager &SM = Ctx->getSourceManager();
   FileID FID = SM.getFileID(SM.getFileLoc(Loc));
   if (FID.isInvalid())
     return true;
 
-  bool Invalid = false;
-  const SrcMgr::SLocEntry &SEntry = SM.getSLocEntry(FID, &Invalid);
-  if (Invalid || !SEntry.isFile())
-    return true;
-
-  if (SEntry.getFile().getFileCharacteristic() != SrcMgr::C_User) {
+  if (isSystemFile(FID)) {
     switch (IndexOpts.SystemSymbolFilter) {
     case IndexingOptions::SystemSymbolFilterKind::None:
       return true;
     case IndexingOptions::SystemSymbolFilterKind::DeclarationsOnly:
     case IndexingOptions::SystemSymbolFilterKind::All:
       break;
     }
   }
if (isa<ObjCIvarDecl>(D))
     return false;
   if (isa<ObjCMethodDecl>(D))
     return false;
   if (isa<ImportDecl>(D))
     return false;
   return true;
 }
 
+void IndexingContext::setSysrootPath(StringRef path) {
+  // Ignore sysroot path if it points to root, otherwise every header will be
+  // treated as system one.
+  // FIXMIE: make this portable
+  if (path == "/")
+    path = StringRef();
+  SysrootPath = path;
+}
+
+bool IndexingContext::isSystemFile(FileID FID) {
+  if (LastFileCheck.first == FID)
+    return LastFileCheck.second;
+
+  auto cacheResult = [&](bool res) -> bool {
+    LastFileCheck = {FID, res};
+    return res;
+  };
+
+  bool Invalid = false;
+  const SrcMgr::SLocEntry &SEntry =
+      Ctx->getSourceManager().getSLocEntry(FID, &Invalid);
+  if (Invalid || !SEntry.isFile())
+    return cacheResult(false);
+
+  const SrcMgr::FileInfo &FI = SEntry.getFile();
+  if (FI.getFileCharacteristic() != SrcMgr::C_User)
+    return cacheResult(true);
+
+  auto *CC = FI.getContentCache();
+  if (!CC)
+    return cacheResult(false);
+  auto *FE = CC->OrigEntry;
+  if (!FE)
+    return cacheResult(false);
+
+  if (SysrootPath.empty())
+    return cacheResult(false);
+
+  // Check if directory is in sysroot so that we can consider system headers
+  // even the headers found via a user framework search path, pointing inside
+  // sysroot.
+  auto dirEntry = FE->getDir();
+  auto pair = DirEntries.insert(std::make_pair(dirEntry, false));
+  bool &isSystemDir = pair.first->second;
+  bool wasInserted = pair.second;
+  if (wasInserted) {
+    isSystemDir = StringRef(dirEntry->getName()).startswith(SysrootPath);
+  }
+  return cacheResult(isSystemDir);
+}
+
 static const CXXRecordDecl *
 getDeclContextForTemplateInstationPattern(const Decl *D) {
   if (const auto *CTSD =
           dyn_cast<ClassTemplateSpecializationDecl>(D->getDeclContext()))
     return CTSD->getTemplateInstantiationPattern();
   else if (const auto *RD = dyn_cast<CXXRecordDecl>(D->getDeclContext()))
     return RD->getInstantiatedFromMemberClass();
   return nullptr;

 
 bool IndexingContext::handleDeclOccurrence(const Decl *D, SourceLocation Loc,
                                            bool IsRef, const Decl *Parent,
                                            SymbolRoleSet Roles,
                                            ArrayRef<SymbolRelation> Relations,
                                            const Expr *OrigE,
                                            const Decl *OrigD,
                                            const DeclContext *ContainerDC) {
-  if (D->isImplicit() && !isa<ObjCMethodDecl>(D))
+  if (D->isImplicit() && !(isa<ObjCMethodDecl>(D) || isa<ObjCIvarDecl>(D)))
     return true;
   if (!isa<NamedDecl>(D) || shouldSkipNamelessDecl(cast<NamedDecl>(D)))
     return true;
 
   SourceManager &SM = Ctx->getSourceManager();
   FileID FID = SM.getFileID(SM.getFileLoc(Loc));
   if (FID.isInvalid())
     return true;
 
-  bool Invalid = false;
-  const SrcMgr::SLocEntry &SEntry = SM.getSLocEntry(FID, &Invalid);
-  if (Invalid || !SEntry.isFile())
-    return true;
-
-  if (SEntry.getFile().getFileCharacteristic() != SrcMgr::C_User) {
+  if (isSystemFile(FID)) {
     switch (IndexOpts.SystemSymbolFilter) {
     case IndexingOptions::SystemSymbolFilterKind::None:
       return true;
     case IndexingOptions::SystemSymbolFilterKind::DeclarationsOnly:
       if (!shouldReportOccurrenceForSystemDeclOnlyMode(IsRef, Roles, Relations))
         return true;
       break;
     case IndexingOptions::SystemSymbolFilterKind::All:

clang/lib/IndexDataStore/CMakeLists.txt

+set(LLVM_LINK_COMPONENTS
+  Support
+  )
+
+add_clang_library(clangIndexDataStore
+  IndexDataStore.cpp
+
+  LINK_LIBS
+  clangDirectoryWatcher
+  clangIndexWhileBuilding
+  )

clang/lib/IndexDataStore/IndexDataStore.cpp

+//===--- IndexDataStore.cpp - Index data store info -----------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/IndexDataStore/IndexDataStore.h"
+#include "clang/DirectoryWatcher/DirectoryWatcher.h"
+#include "clang/Index/IndexWhileBuilding/IndexDataFormat.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/Mutex.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/raw_ostream.h"
+
+using namespace clang;
+using namespace clang::index;
+using namespace clang::index::store;
+using namespace llvm;
+
+namespace {
+
+class UnitEventHandlerData {
+  mutable sys::Mutex Mtx;
+  IndexDataStore::UnitEventHandler Handler;
+
+public:
+  void setHandler(IndexDataStore::UnitEventHandler handler) {
+    sys::ScopedLock L(Mtx);
+    Handler = std::move(handler);
+  }
+  IndexDataStore::UnitEventHandler getHandler() const {
+    sys::ScopedLock L(Mtx);
+    return Handler;
+  }
+};
+
+class IndexDataStoreImpl {
+  std::string FilePath;
+  std::shared_ptr<UnitEventHandlerData> TheUnitEventHandlerData;
+  std::unique_ptr<DirectoryWatcher> DirWatcher;
+
+public:
+  explicit IndexDataStoreImpl(StringRef indexStorePath)
+      : FilePath(indexStorePath) {
+    TheUnitEventHandlerData = std::make_shared<UnitEventHandlerData>();
+  }
+
+  StringRef getFilePath() const { return FilePath; }
+  bool foreachUnitName(bool sorted,
+                       llvm::function_ref<bool(StringRef unitName)> receiver);
+  void setUnitEventHandler(IndexDataStore::UnitEventHandler Handler);
+  bool startEventListening(bool waitInitialSync, std::string &Error);
+  void stopEventListening();
+  void discardUnit(StringRef UnitName);
+  void discardRecord(StringRef RecordName);
+  void purgeStaleData();
+};
+
+} // anonymous namespace
+
+bool IndexDataStoreImpl::foreachUnitName(
+    bool sorted, llvm::function_ref<bool(StringRef unitName)> receiver) {
+  SmallString<128> UnitPath;
+  UnitPath = FilePath;
+  appendUnitSubDir(UnitPath);
+
+  std::vector<std::string> filenames;
+
+  std::error_code EC;
+  for (auto It = sys::fs::directory_iterator(UnitPath, EC),
+            End = sys::fs::directory_iterator();
+       !EC && It != End; It.increment(EC)) {
+    StringRef unitName = sys::path::filename(It->path());
+    if (!sorted) {
+      if (!receiver(unitName))
+        return false;
+    } else {
+      filenames.push_back(unitName);
+    }
+  }
+
+  if (sorted) {
+    std::sort(filenames.begin(), filenames.end());
+    for (auto &fname : filenames)
+      if (!receiver(fname))
+        return false;
+  }
+  return true;
+}
+
+void IndexDataStoreImpl::setUnitEventHandler(
+    IndexDataStore::UnitEventHandler handler) {
+  TheUnitEventHandlerData->setHandler(std::move(handler));
+}
+
+bool IndexDataStoreImpl::startEventListening(bool waitInitialSync,
+                                             std::string &Error) {
+  if (DirWatcher) {
+    Error = "event listener already active";
+    return true;
+  }
+
+  SmallString<128> UnitPath;
+  UnitPath = FilePath;
+  appendUnitSubDir(UnitPath);
+
+  auto localUnitEventHandlerData = TheUnitEventHandlerData;
+  auto OnUnitsChange = [localUnitEventHandlerData](
+                           ArrayRef<DirectoryWatcher::Event> Events,
+                           bool isInitial) {
+    SmallVector<IndexDataStore::UnitEvent, 16> UnitEvents;
+    UnitEvents.reserve(Events.size());
+    for (const DirectoryWatcher::Event &evt : Events) {
+      IndexDataStore::UnitEventKind K;
+      StringRef UnitName = sys::path::filename(evt.Filename);
+      switch (evt.Kind) {
+      case DirectoryWatcher::Event::EventKind::Removed:
+        K = IndexDataStore::UnitEventKind::Removed;
+        break;
+      case DirectoryWatcher::Event::EventKind::Modified:
+        K = IndexDataStore::UnitEventKind::Modified;
+        break;
+      case DirectoryWatcher::Event::EventKind::WatchedDirRemoved:
+        K = IndexDataStore::UnitEventKind::DirectoryDeleted;
+        UnitName = StringRef();
+        break;
+      case DirectoryWatcher::Event::EventKind::WatcherGotInvalidated:
+        K = IndexDataStore::UnitEventKind::Failure;
+        UnitName = StringRef();
+        break;
+      }
+      UnitEvents.push_back(IndexDataStore::UnitEvent{K, UnitName});
+    }
+
+    if (auto handler = localUnitEventHandlerData->getHandler()) {
+      IndexDataStore::UnitEventNotification EventNote{isInitial, UnitEvents};
+      handler(EventNote);
+    }
+  };
+
+  DirWatcher =
+      DirectoryWatcher::create(UnitPath.str(), OnUnitsChange, waitInitialSync);
+  if (!DirWatcher) {
+    Error = "failed to create directory watcher";
+    return true;
+  }
+
+  return false;
+}
+
+void IndexDataStoreImpl::stopEventListening() { DirWatcher.reset(); }
+
+void IndexDataStoreImpl::discardUnit(StringRef UnitName) {
+  SmallString<128> UnitPath;
+  UnitPath = FilePath;
+  appendUnitSubDir(UnitPath);
+  appendUnitBucketDir(UnitName, UnitPath);
+  sys::fs::remove(UnitPath);
+}
+
+void IndexDataStoreImpl::discardRecord(StringRef RecordName) {
+  // TODO check - is this correct or is the order of appended elements opposite?
+  SmallString<128> RecordPath;
+  RecordPath = FilePath;
+  appendRecordSubDir(RecordPath);
+  appendRecordBucketDir(RecordName, RecordPath);
+  appendRecordFilename(RecordName, RecordPath);
+  sys::fs::remove(RecordPath);
+}
+
+void IndexDataStoreImpl::purgeStaleData() {
+  // FIXME: Implement.
+}
+
+std::unique_ptr<IndexDataStore> IndexDataStore::create(StringRef IndexStorePath,
+                                                       std::string &Error) {
+  if (!sys::fs::exists(IndexStorePath)) {
+    raw_string_ostream OS(Error);
+    OS << "index store path does not exist: " << IndexStorePath;
+    return nullptr;
+  }
+
+  return std::unique_ptr<IndexDataStore>(
+      new IndexDataStore(new IndexDataStoreImpl(IndexStorePath)));
+}
+
+#define IMPL static_cast<IndexDataStoreImpl *>(Impl)
+
+IndexDataStore::~IndexDataStore() { delete IMPL; }
+
+StringRef IndexDataStore::getFilePath() const { return IMPL->getFilePath(); }
+
+bool IndexDataStore::foreachUnitName(
+    bool sorted, llvm::function_ref<bool(StringRef unitName)> receiver) {
+  return IMPL->foreachUnitName(sorted, std::move(receiver));
+}
+
+unsigned IndexDataStore::getFormatVersion() {
+  return clang::index::store::getIndexFormatVersion();
+}
+
+void IndexDataStore::setUnitEventHandler(UnitEventHandler Handler) {
+  return IMPL->setUnitEventHandler(std::move(Handler));
+}
+
+bool IndexDataStore::startEventListening(bool waitInitialSync,
+                                         std::string &Error) {
+  return IMPL->startEventListening(waitInitialSync, Error);
+}
+
+void IndexDataStore::stopEventListening() { return IMPL->stopEventListening(); }
+
+void IndexDataStore::discardUnit(StringRef UnitName) {
+  IMPL->discardUnit(UnitName);
+}
+
+void IndexDataStore::discardRecord(StringRef RecordName) {
+  IMPL->discardRecord(RecordName);
+}
+
+void IndexDataStore::purgeStaleData() { IMPL->purgeStaleData(); }

clang/test/Index/Store/Inputs/head.h

+
+extern void test1_func(void);
+extern void test2_func(void);

clang/test/Index/Store/Inputs/json.c.json

+{
+  "files": [
+    "/test1.o",
+    "/Inputs/test1.c",
+    "/Inputs/head.h",
+    "/test2.o",
+    "/Inputs/test2.c",
+    "/test3.o",
+    "/Inputs/test3.cpp"
+  ],
+  "symbols": [
+    {
+      "kind": "function",
+      "lang": "C",
+      "usr": "c:@F@test1_func",
+      "name": "test1_func",
+      "codegen": "_test1_func",
+      "roles": "Decl,Def"
+    },
+    {
+      "kind": "function",
+      "lang": "C",
+      "usr": "c:@F@test2_func",
+      "name": "test2_func",
+      "codegen": "_test2_func",
+      "roles": "Decl,Def"
+    },
+    {
+      "kind": "class",
+      "lang": "C++",
+      "usr": "c:@S@Base",
+      "name": "Base",
+      "roles": "Def,Ref,RelBase,RelCont"
+    },
+    {
+      "kind": "class",
+      "lang": "C++",
+      "usr": "c:@S@Sub",
+      "name": "Sub",
+      "roles": "Def",
+      "rel-roles": "RelBase,RelCont"
+    }
+  ],
+  "records": [
+    {
+      "occurrences": [
+        {
+          "symbol": 0,
+          "line": 3,
+          "col": 6,
+          "roles": "Def"
+        }
+      ]
+    },
+    {
+      "occurrences": [
+        {
+          "symbol": 0,
+          "line": 2,
+          "col": 13,
+          "roles": "Decl"
+        },
+        {
+          "symbol": 1,
+          "line": 3,
+          "col": 13,
+          "roles": "Decl"
+        }
+      ]
+    },
+    {
+      "occurrences": [
+        {
+          "symbol": 1,
+          "line": 3,
+          "col": 6,
+          "roles": "Def"
+        }
+      ]
+    },
+    {
+      "occurrences": [
+        {
+          "symbol": 2,
+          "line": 1,
+          "col": 7,
+          "roles": "Def"
+        },
+        {
+          "symbol": 3,
+          "line": 2,
+          "col": 7,
+          "roles": "Def"
+        },
+        {
+          "symbol": 2,
+          "line": 2,
+          "col": 20,
+          "roles": "Ref,RelBase,RelCont",
+          "relations": [
+            {
+              "symbol": 3,
+              "rel-roles": "RelBase,RelCont"
+            }
+          ]
+
+        }
+      ]
+    }
+  ],
+  "units": [
+    {
+      "triple": "x86_64-apple-macosx10.7.0",
+      "out-file": 0,
+      "sources": [
+        {
+          "file": 1,
+          "records": [0]
+        },
+        {
+          "file": 2,
+          "records": [1]
+        }
+      ]
+    },
+    {
+      "triple": "x86_64-apple-macosx10.7.0",
+      "out-file": 3,
+      "sources": [
+        {
+          "file": 4,
+          "records": [2]
+        },
+        {
+          "file": 2,
+          "records": [1]
+        }
+      ]
+    },
+    {
+      "triple": "x86_64-apple-macosx10.7.0",
+      "out-file": 5,
+      "sources": [
+        {
+          "file": 6,
+          "records": [3]
+        }
+      ]
+    }
+  ]
+}

clang/test/Index/Store/Inputs/module/ModDep.h

+#include "ModTop.h"
+
+void ModDep_func(ModTopStruct s);

clang/test/Index/Store/Inputs/module/ModSystem.h

+
+typedef struct {} ModSystemStruct;
+
+void ModSystem_func(void);

clang/test/Index/Store/Inputs/module/ModTop.h

+
+typedef struct {} ModTopStruct;
+
+void ModTop_func(void);

clang/test/Index/Store/Inputs/module/ModTopSub1.h

+void ModTopSub1_func(void);

clang/test/Index/Store/Inputs/module/ModTopSub2.h

+// This header has no symbols, intended to show up as file dependency.

clang/test/Index/Store/Inputs/module/module.modulemap

+module ModTop {
+  header "ModTop.h"
+  export *
+  module Sub1 {
+    header "ModTopSub1.h"
+  }
+  module Sub2 {
+    header "ModTopSub2.h"
+  }
+}
+module ModDep { header "ModDep.h" export * }
+module ModSystem [system] { header "ModSystem.h" export * }

clang/test/Index/Store/Inputs/overlay.yaml

+{
+  'version': 0,
+  'roots': [{ 'type': 'file', 'name': 'OUT_DIR/using-overlay.h',
+             'external-contents': 'INPUT_DIR/using-overlay.h'
+           }]
+}

clang/test/Index/Store/Inputs/print-unit.h

+#include "head.h"
+#include "using-overlay.h"

clang/test/Index/Store/Inputs/sys/another.h

+
+extern void sys_another_func(void);

clang/test/Index/Store/Inputs/sys/syshead.h

+
+#include "another.h"
+
+extern void sys_test1_func(void);

clang/test/Index/Store/Inputs/test1.c

+#include "head.h"
+
+void test1_func(void) {}

clang/test/Index/Store/Inputs/test2.c

+#include "head.h"
+
+void test2_func(void) {}

clang/test/Index/Store/Inputs/test3.cpp

+class Base {};
+class Sub : public Base {};

clang/test/Index/Store/Inputs/using-overlay.h

+void using_overlay(void);

clang/test/Index/Store/assembly-invocation.c

+// Make sure it doesn't crash.
+// RUN: %clang -target x86_64-apple-macosx10.7 -S %s -o %t.s
+// RUN: env CLANG_PROJECT_INDEX_PATH=%t.idx %clang -target x86_64-apple-macosx10.7 -c %t.s -o %t.o

clang/test/Index/Store/empty-unit.c

+void foo(int i);
+
+// RUN: rm -rf %t/idx
+// RUN: %clang_cc1 -index-store-path %t/idx %s -o %t.o
+// RUN: touch %t.empty
+
+// RUN: cp %t.empty $(find %t/idx -name "empty-unit.c*o*")
+// RUN: not c-index-test core -print-unit %t/idx 2> %t.err
+// RUN: FileCheck %s -input-file %t.err -check-prefix ERR-UNIT
+// ERR-UNIT: error loading unit: empty file
+
+// Also check for empty record files.
+// RUN: rm -rf %t/idx2
+// RUN: %clang_cc1 -index-store-path %t/idx2 %s -o %t.o
+// RUN: cp %t.empty $(find %t/idx2 -name "empty-unit.c-*")
+// RUN: not c-index-test core -print-record %t/idx2 2> %t2.err
+// RUN: FileCheck %s -input-file %t2.err -check-prefix ERR-RECORD
+// ERR-RECORD: error loading record: empty file

clang/test/Index/Store/external-source-symbol-hash.m

+// RUN: rm -rf %t.idx
+// RUN: %clang_cc1 %s -index-store-path %t.idx -D USE_EXTERNAL
+// RUN: c-index-test core -print-record %t.idx | FileCheck %s
+// RUN: %clang_cc1 %s -index-store-path %t.idx
+// RUN: find %t.idx/*/records -name "external-source-symbol-hash*" | count 2
+
+#ifdef USE_EXTERNAL
+#  define EXT_DECL(mod_name) __attribute__((external_source_symbol(language="Swift", defined_in=mod_name)))
+#else
+#  define EXT_DECL(mod_name)
+#endif
+
+#define NS_ENUM(_name, _type) enum _name:_type _name; enum _name : _type
+
+// Forward declarations should pick up the attribute from later decls
+@protocol P1;
+// CHECK: [[@LINE-1]]:11 | protocol/Swift | c:@M@some_module@objc(pl)P1 | Ref | rel: 0
+@class I2;
+// CHECK: [[@LINE-1]]:8 | class/Swift | c:@M@other_module@objc(cs)I2 | Ref | rel: 0
+enum E3: int;
+// CHECK: [[@LINE-1]]:6 | enum/Swift | c:@M@third_module@E@E3 | Decl | rel: 0
+
+void test(id<P1> first, I2 *second, enum E3 third) {}
+// CHECK: [[@LINE-1]]:14 | protocol/Swift | c:@M@some_module@objc(pl)P1 | Ref,RelCont | rel: 1
+// CHECK: [[@LINE-2]]:25 | class/Swift | c:@M@other_module@objc(cs)I2 | Ref,RelCont | rel: 1
+// CHECK: [[@LINE-3]]:42 | enum/Swift | c:@M@third_module@E@E3 | Ref,RelCont | rel: 1
+
+EXT_DECL("some_module")
+@protocol P1
+// CHECK: [[@LINE-1]]:11 | protocol/Swift | c:@M@some_module@objc(pl)P1 | Decl | rel: 0
+-(void)method;
+// CHECK: [[@LINE-1]]:8 | instance-method(protocol)/Swift | c:@M@some_module@objc(pl)P1(im)method | Decl,Dyn,RelChild | rel: 1
+@end
+
+EXT_DECL("other_module")
+@interface I2
+// CHECK: [[@LINE-1]]:12 | class/Swift | c:@M@other_module@objc(cs)I2 | Decl | rel: 0
+-(void)method;
+// CHECK: [[@LINE-1]]:8 | instance-method/Swift | c:@M@other_module@objc(cs)I2(im)method | Decl,Dyn,RelChild | rel: 1
+@end
+
+
+typedef NS_ENUM(E3, int) {
+// CHECK: [[@LINE-1]]:17 | enum/Swift | c:@M@third_module@E@E3 | Def | rel: 0
+  firstCase = 1,
+  // CHECK: [[@LINE-1]]:3 | enumerator/Swift | c:@M@third_module@E@E3@firstCase | Def,RelChild | rel: 1
+} EXT_DECL("third_module");

clang/test/Index/Store/handle-prebuilt-module.m

+// RUN: rm -rf %t
+// RUN: mkdir %t
+// RUN: %clang -arch x86_64 -mmacosx-version-min=10.7 -fsyntax-only %s -o %t.o -index-store-path %t/idx1 -fmodules -fmodules-cache-path=%t/mcp -I %S/Inputs/module -Rindex-store 2> %t.err1
+// RUN: %clang -arch x86_64 -mmacosx-version-min=10.7 -fsyntax-only %s -o %t.o -index-store-path %t/idx2 -fmodules -fmodules-cache-path=%t/mcp -I %S/Inputs/module -Rindex-store 2> %t.err2
+// RUN: %clang -arch x86_64 -mmacosx-version-min=10.7 -fsyntax-only %s -o %t.o -index-store-path %t/idx2 -fmodules -fmodules-cache-path=%t/mcp -I %S/Inputs/module -Rindex-store 2> %t.err3
+// RUN: FileCheck -input-file=%t.err1 -check-prefix=CREATING_MODULES %s -allow-empty
+// RUN: FileCheck -input-file=%t.err2 -check-prefix=CREATING_INDEX_DATA_FROM_MODULE_FILES %s
+// RUN: FileCheck -input-file=%t.err3 -check-prefix=EXISTING_INDEX_DATA_FROM_MODULE_FILES %s -allow-empty
+// RUN: c-index-test core -print-unit %t/idx1 | sort > %t/all-units1.txt
+// RUN: c-index-test core -print-unit %t/idx2 | sort > %t/all-units2.txt
+// RUN: c-index-test core -print-record %t/idx1 | sort > %t/all-records1.txt
+// RUN: c-index-test core -print-record %t/idx2 | sort > %t/all-records2.txt
+// RUN: diff -u %t/all-units1.txt %t/all-units2.txt
+// RUN: diff -u %t/all-records1.txt %t/all-records2.txt
+
+@import ModDep;
+
+// CREATING_MODULES-NOT: remark:
+
+// CREATING_INDEX_DATA_FROM_MODULE_FILES: remark: producing index data for module file {{.*}}ModDep{{.*}}.pcm
+// CREATING_INDEX_DATA_FROM_MODULE_FILES: remark: producing index data for module file {{.*}}ModTop{{.*}}.pcm
+
+// EXISTING_INDEX_DATA_FROM_MODULE_FILES-NOT: remark:

clang/test/Index/Store/json-with-module.m

+// RUN: rm -rf %t.idx %t.mcp
+// RUN: %clang -target x86_64-apple-darwin -arch x86_64 -mmacosx-version-min=10.7 -c %s -o %t.o -index-store-path %t.idx -fmodules -fmodules-cache-path=%t.mcp -Xclang -fdisable-module-hash -I %S/Inputs/module
+// RUN: c-index-test core -aggregate-json %t.idx -o %t.json
+// RUN: sed -e "s:%S::g" -e "s:%T::g" %t.json > %t.final.json
+// RUN: diff -u %s.json %t.final.json
+
+@import ModDep;

clang/test/Index/Store/json-with-module.m.json

+{
+  "files": [
+    "/json-with-module.m.tmp.mcp/ModDep.pcm",
+    "/json-with-module.m.tmp.mcp/ModTop.pcm",
+    "/Inputs/module/ModDep.h",
+    "/Inputs/module/ModTop.h",
+    "/Inputs/module/ModTopSub1.h",
+    "/Inputs/module/ModTopSub2.h",
+    "/json-with-module.m.tmp.o",
+    "/json-with-module.m",
+    "/Inputs/module/module.modulemap"
+  ],
+  "symbols": [
+    {
+      "kind": "function",
+      "lang": "C",
+      "usr": "c:@F@ModDep_func",
+      "name": "ModDep_func",
+      "roles": "Decl",
+      "rel-roles": "RelCont"
+    },
+    {
+      "kind": "type-alias",
+      "lang": "C",
+      "usr": "c:@T@ModTopStruct",
+      "name": "ModTopStruct",
+      "roles": "Def,Ref,RelCont"
+    },
+    {
+      "kind": "struct",
+      "lang": "C",
+      "usr": "c:@SA@ModTopStruct",
+      "name": "",
+      "roles": "Def"
+    },
+    {
+      "kind": "function",
+      "lang": "C",
+      "usr": "c:@F@ModTop_func",
+      "name": "ModTop_func",
+      "roles": "Decl"
+    },
+    {
+      "kind": "function",
+      "lang": "C",
+      "usr": "c:@F@ModTopSub1_func",
+      "name": "ModTopSub1_func",
+      "roles": "Decl"
+    }
+  ],
+  "records": [
+    {
+      "occurrences": [
+        {
+          "symbol": 0,
+          "line": 3,
+          "col": 6,
+          "roles": "Decl"
+        },
+        {
+          "symbol": 1,
+          "line": 3,
+          "col": 18,
+          "roles": "Ref,RelCont",
+          "relations": [
+            {
+              "symbol": 0,
+              "rel-roles": "RelCont"
+            }
+          ]
+
+        }
+      ]
+    },
+    {
+      "occurrences": [
+        {
+          "symbol": 2,
+          "line": 2,
+          "col": 9,
+          "roles": "Def"
+        },
+        {
+          "symbol": 1,
+          "line": 2,
+          "col": 19,
+          "roles": "Def"
+        },
+        {
+          "symbol": 3,
+          "line": 4,
+          "col": 6,
+          "roles": "Decl"
+        }
+      ]
+    },
+    {
+      "occurrences": [
+        {
+          "symbol": 4,
+          "line": 1,
+          "col": 6,
+          "roles": "Decl"
+        }
+      ]
+    }
+  ],
+  "units": [
+    {
+      "triple": "x86_64-apple-macosx10.7.0",
+      "out-file": 0,
+      "unit-dependencies": [1],
+      "sources": [
+        {
+          "file": 2,
+          "records": [0]
+        }
+      ]
+    },
+    {
+      "triple": "x86_64-apple-macosx10.7.0",
+      "out-file": 1,
+      "sources": [
+        {
+          "file": 3,
+          "records": [1]
+        },
+        {
+          "file": 4,
+          "records": [2]
+        },
+        {
+          "file": 5
+        }
+      ]
+    },
+    {
+      "triple": "x86_64-apple-macosx10.7.0",
+      "out-file": 6,
+      "unit-dependencies": [0],
+      "sources": [
+        {
+          "file": 7
+        },
+        {
+          "file": 8
+        }
+      ]
+    }
+  ]
+}

clang/test/Index/Store/json-with-pch.c

+// RUN: rm -rf %t.idx
+// RUN: %clang -target x86_64-apple-darwin -arch x86_64 -mmacosx-version-min=10.7 -x c-header %S/Inputs/head.h -o %t.h.pch -index-store-path %t.idx
+// RUN: %clang -target x86_64-apple-darwin -arch x86_64 -mmacosx-version-min=10.7 -c %s -o %t.o -index-store-path %t.idx -include %t.h -Werror
+// RUN: c-index-test core -aggregate-json %t.idx -o %t.json
+// RUN: sed -e "s:%S::g" -e "s:%T::g" %t.json > %t.final.json
+// RUN: diff -u %s.json %t.final.json
+
+int main() {
+  test1_func();
+}

clang/test/Index/Store/json-with-pch.c.json

+{
+  "files": [
+    "/json-with-pch.c.tmp.h.pch",
+    "/Inputs/head.h",
+    "/json-with-pch.c.tmp.o",
+    "/json-with-pch.c"
+  ],
+  "symbols": [
+    {
+      "kind": "function",
+      "lang": "C",
+      "usr": "c:@F@test1_func",
+      "name": "test1_func",
+      "roles": "Decl,Ref,Call,RelCall,RelCont"
+    },
+    {
+      "kind": "function",
+      "lang": "C",
+      "usr": "c:@F@test2_func",
+      "name": "test2_func",
+      "roles": "Decl"
+    },
+    {
+      "kind": "function",
+      "lang": "C",
+      "usr": "c:@F@main",
+      "name": "main",
+      "roles": "Def",
+      "rel-roles": "RelCall,RelCont"
+    }
+  ],
+  "records": [
+    {
+      "occurrences": [
+        {
+          "symbol": 0,
+          "line": 2,
+          "col": 13,
+          "roles": "Decl"
+        },
+        {
+          "symbol": 1,
+          "line": 3,
+          "col": 13,
+          "roles": "Decl"
+        }
+      ]
+    },
+    {
+      "occurrences": [
+        {
+          "symbol": 2,
+          "line": 8,
+          "col": 5,
+          "roles": "Def"
+        },
+        {
+          "symbol": 0,
+          "line": 9,
+          "col": 3,
+          "roles": "Ref,Call,RelCall,RelCont",
+          "relations": [
+            {
+              "symbol": 2,
+              "rel-roles": "RelCall,RelCont"
+            }
+          ]
+
+        }
+      ]
+    }
+  ],
+  "units": [
+    {
+      "triple": "x86_64-apple-macosx10.7.0",
+      "out-file": 0,
+      "sources": [
+        {
+          "file": 1,
+          "records": [0]
+        }
+      ]
+    },
+    {
+      "triple": "x86_64-apple-macosx10.7.0",
+      "out-file": 2,
+      "unit-dependencies": [0],
+      "sources": [
+        {
+          "file": 3,
+          "records": [1]
+        }
+      ]
+    }
+  ]
+}

clang/test/Index/Store/json.c

+// RUN: rm -rf %t.idx
+// RUN: mkdir -p %t.o
+// RUN: env CLANG_PROJECT_INDEX_PATH=%t.idx %clang -target x86_64-apple-macosx10.7 -arch x86_64 -mmacosx-version-min=10.7 -c %S/Inputs/test1.c -o %t.o/test1.o
+// RUN: env CLANG_PROJECT_INDEX_PATH=%t.idx %clang -target x86_64-apple-macosx10.7 -arch x86_64 -mmacosx-version-min=10.7 -c %S/Inputs/test2.c -o %t.o/test2.o
+// RUN: env CLANG_PROJECT_INDEX_PATH=%t.idx %clang -target x86_64-apple-macosx10.7 -arch x86_64 -mmacosx-version-min=10.7 -c %S/Inputs/test3.cpp -o %t.o/test3.o
+// RUN: c-index-test core -aggregate-json %t.idx -o %t.json
+// RUN: sed -e "s:%S::g" -e "s:%t.o::g" %t.json > %t.final.json
+// RUN: diff -u %S/Inputs/json.c.json %t.final.json

clang/test/Index/Store/print-record.mm

+// RUN: rm -rf %t.idx
+// RUN: %clang_cc1 %s -index-store-path %t.idx
+// RUN: c-index-test core -print-record %t.idx | FileCheck %s
+
+
+
+@class MyCls;
+
+@interface MyCls
+@end
+
+// CHECK: [[@LINE+2]]:6 | function/C | c:@F@foo#*$objc(cs)MyCls# | Decl | rel: 0
+// CHECK: [[@LINE+1]]:10 | class/ObjC | c:objc(cs)MyCls | Ref,RelCont | rel: 1
+void foo(MyCls *p);
+
+
+// RANGE-NOT: before_range
+void before_range();
+
+// RANGE: [[@LINE+1]]:6 | function/C | c:@F@in_range1# | Decl
+void in_range1();
+// RANGE: [[@LINE+1]]:6 | function/C | c:@F@in_range2# | Decl
+void in_range2();
+
+// RANGE-NOT: after_range
+void after_range();
+
+// RUN: c-index-test core -print-record %t.idx -filepath %s:21:23 | FileCheck -check-prefix=RANGE %s

clang/test/Index/Store/print-unit.c

+
+
+#include "print-unit.h"
+#include "syshead.h"
+
+void foo(int i);
+
+// RUN: rm -rf %t
+// RUN: %clang_cc1 -I %S/Inputs -isystem %S/Inputs/sys -index-store-path %t/idx %s -triple x86_64-apple-macosx10.8
+// RUN: c-index-test core -print-unit %t/idx | FileCheck %s
+// RUN: %clang_cc1 -I %S/Inputs -isystem %S/Inputs/sys -index-store-path %t/idx_opt1 %s -triple x86_64-apple-macosx10.8 -O2
+// RUN: c-index-test core -print-unit %t/idx_opt1 | FileCheck %s -check-prefix=OPT
+// RUN: %clang_cc1 -I %S/Inputs -isystem %S/Inputs/sys -index-store-path %t/idx_opt2 %s -triple x86_64-apple-macosx10.8 -Os
+// RUN: c-index-test core -print-unit %t/idx_opt2 | FileCheck %s -check-prefix=OPT
+
+// CHECK: print-unit.c.o
+// CHECK: provider: clang-
+// CHECK: is-system: 0
+// CHECK: has-main: 1
+// CHECK: main-path: {{.*}}/print-unit.c
+// CHECK: out-file: {{.*}}/print-unit.c.o
+// CHECK: target: x86_64-apple-macosx10.8
+// CHECK: is-debug: 1
+// CHECK: DEPEND START
+// CHECK: Record | user | {{.*}}/print-unit.c | print-unit.c-
+// CHECK: Record | user | {{.*}}/Inputs/head.h | head.h-
+// CHECK: Record | user | {{.*}}/Inputs/using-overlay.h | using-overlay.h-
+// CHECK: Record | system | {{.*}}/Inputs/sys/syshead.h | syshead.h-
+// CHECK: Record | system | {{.*}}/Inputs/sys/another.h | another.h-
+// CHECK: File | user | {{.*}}/Inputs/print-unit.h{{$}}
+// CHECK: DEPEND END (6)
+// CHECK: INCLUDE START
+// CHECK-DAG: {{.*}}/print-unit.c:3 | {{.*}}/Inputs/print-unit.h
+// CHECK-DAG: {{.*}}/print-unit.c:4 | {{.*}}/Inputs/sys/syshead.h
+// CHECK-DAG: {{.*}}/Inputs/print-unit.h:1 | {{.*}}/Inputs/head.h
+// CHECK-DAG: {{.*}}/Inputs/print-unit.h:2 | {{.*}}/Inputs/using-overlay.h
+// CHECK: INCLUDE END (4)
+
+// OPT: is-debug: 0

clang/test/Index/Store/print-units-with-modules.m

+// RUN: rm -rf %t.idx %t.mcp
+// RUN: %clang -arch x86_64 -mmacosx-version-min=10.7 -c %s -o %t.o -index-store-path %t.idx -fmodules -fmodules-cache-path=%t.mcp -Xclang -fdisable-module-hash -I %S/Inputs/module
+// RUN: c-index-test core -print-unit %t.idx | FileCheck %s
+
+@import ModDep;
+@import ModSystem;
+
+// CHECK: ModDep.pcm
+// CHECK: provider: clang-
+// CHECK: is-system: 0
+// CHECK: is-module: 1
+// CHECK: module-name: ModDep
+// CHECK: has-main: 0
+// CHECK: main-path: {{$}}
+// CHECK: out-file: {{.*}}/ModDep.pcm
+// CHECK: DEPEND START
+// CHECK: Unit | user | ModTop | {{.*}}/ModTop.pcm | ModTop.pcm
+// CHECK: Record | user | ModDep | {{.*}}/Inputs/module/ModDep.h | ModDep.h
+// CHECK: DEPEND END (2)
+
+// CHECK: ModSystem.pcm
+// CHECK: is-system: 1
+// CHECK: is-module: 1
+// CHECK: module-name: ModSystem
+// CHECK: has-main: 0
+// CHECK: main-path: {{$}}
+// CHECK: out-file: {{.*}}/ModSystem.pcm
+// CHECK: DEPEND START
+// CHECK: Record | system | ModSystem | {{.*}}/Inputs/module/ModSystem.h | ModSystem.h
+// CHECK: DEPEND END (1)
+
+// CHECK: ModTop.pcm
+// CHECK: is-system: 0
+// CHECK: is-module: 1
+// CHECK: module-name: ModTop
+// CHECK: has-main: 0
+// CHECK: main-path: {{$}}
+// CHECK: out-file: {{.*}}/ModTop.pcm
+// CHECK: DEPEND START
+// CHECK: Record | user | ModTop | {{.*}}/Inputs/module/ModTop.h | ModTop.h
+// CHECK: Record | user | ModTop.Sub1 | {{.*}}/Inputs/module/ModTopSub1.h | ModTopSub1.h
+// CHECK: File | user | ModTop.Sub2 | {{.*}}/Inputs/module/ModTopSub2.h{{$}}
+// CHECK: DEPEND END (3)
+
+// CHECK: print-units-with-modules.m.tmp.o
+// CHECK: is-system: 0
+// CHECK: is-module: 0
+// CHECK: module-name: <none>
+// CHECK: has-main: 1
+// CHECK: main-path: {{.*}}/print-units-with-modules.m
+// CHECK: out-file: {{.*}}/print-units-with-modules.m.tmp.o
+// CHECK: DEPEND START
+// CHECK: Unit | user | ModDep | {{.*}}/ModDep.pcm | ModDep.pcm
+// CHECK: Unit | system | ModSystem | {{.*}}/ModSystem.pcm | ModSystem.pcm
+// CHECK: File | user | {{.*}}/print-units-with-modules.m{{$}}
+// CHECK: File | user | {{.*}}/Inputs/module/module.modulemap{{$}}
+// CHECK: DEPEND END (4)

clang/test/Index/Store/print-units-with-pch.c

+// RUN: rm -rf %t.idx
+// RUN: %clang -target x86_64-apple-darwin -arch x86_64 -mmacosx-version-min=10.7 -x c-header %S/Inputs/head.h -o %t.h.pch -index-store-path %t.idx
+// RUN: %clang -target x86_64-apple-darwin -arch x86_64 -mmacosx-version-min=10.7 -c %s -o %t.o -index-store-path %t.idx -include %t.h -Werror
+// RUN: c-index-test core -print-unit %t.idx | FileCheck %s
+
+int main() {
+  test1_func();
+}
+
+// CHECK: print-units-with-pch.c.tmp.h.pch
+// CHECK: is-system: 0
+// CHECK: has-main: 0
+// CHECK: main-path: {{$}}
+// CHECK: out-file: {{.*}}/print-units-with-pch.c.tmp.h.pch
+// CHECK: DEPEND START
+// CHECK: Record | user | {{.*}}/Inputs/head.h | head.h
+// CHECK: DEPEND END (1)
+
+// CHECK: print-units-with-pch.c.tmp.o
+// CHECK: is-system: 0
+// CHECK: has-main: 1
+// CHECK: main-path: {{.*}}/print-units-with-pch.c
+// CHECK: out-file: {{.*}}/print-units-with-pch.c.tmp.o
+// CHECK: DEPEND START
+// CHECK: Unit | user | {{.*}}/print-units-with-pch.c.tmp.h.pch | print-units-with-pch.c.tmp.h.pch
+// CHECK: Record | user | {{.*}}/print-units-with-pch.c | print-units-with-pch.c
+// CHECK: DEPEND END (2)

clang/test/Index/Store/record-hash-crash-invalid-name.cpp

+// Makes sure it doesn't crash.
+
+// RUN: rm -rf %t
+// RUN: not %clang_cc1 %s -index-store-path %t/idx -std=c++14
+// RUN: c-index-test core -print-record %t/idx | FileCheck %s
+
+namespace rdar32474406 {
+void foo();
+typedef void (*Func_t)();
+// CHECK: [[@LINE+4]]:1 | type-alias/C | c:record-hash-crash-invalid-name.cpp@N@rdar32474406@T@Func_t | Ref,RelCont | rel: 1
+// CHECK-NEXT: RelCont | c:@N@rdar32474406
+// CHECK: [[@LINE+2]]:14 | function/C | c:@N@rdar32474406@F@foo# | Ref,RelCont | rel: 1
+// CHECK-NEXT: RelCont | c:@N@rdar32474406
+Func_t[] = { foo }; // invalid decomposition
+}

clang/test/Index/Store/record-hash-crash.cpp

+// Makes sure it doesn't crash.
+
+// RUN: rm -rf %t
+// RUN: %clang_cc1 %s -index-store-path %t/idx -std=c++14
+// RUN: c-index-test core -print-record %t/idx | FileCheck %s
+
+namespace crash1 {
+// CHECK: [[@LINE+1]]:6 | function/C
+auto getit() { return []() {}; }
+}
+
+namespace crash2 {
+// CHECK: [[@LINE+2]]:7 | class(Gen)/C++ | c:@N@crash2@ST>1#T@Foo | Decl,RelChild | rel: 1
+template <typename T>
+class Foo; // canonical decl
+
+// CHECK: [[@LINE+2]]:7 | class(Gen)/C++ | c:@N@crash2@ST>1#T@Foo | Def,RelChild | rel: 1
+template <typename T>
+class Foo {};
+
+// CHECK: [[@LINE+2]]:8 | struct(Gen)/C++ | c:@N@crash2@ST>1#t>1#pT@Wrapper | Def,RelChild | rel: 1
+template <template <typename... ARGS> class TYPE>
+struct Wrapper {};
+
+// CHECK: [[@LINE+3]]:8 | struct(Gen,TS)/C++ | c:@N@crash2@S@Wrapper>#@N@crash2@ST>1#T@Foo | Def,RelChild,RelSpecialization | rel: 2
+// CHECK: RelSpecialization | c:@N@crash2@ST>1#t>1#pT@Wrapper
+template <>
+struct Wrapper<Foo> {};
+}

clang/test/Index/Store/record-hash-using.cpp

+// RUN: rm -rf %t
+// RUN: %clang_cc1 %s -std=c++11 -index-store-path %t/idx -DTYPE1=A -DTYPE2=A -DTYPE3=T -DTYPE4=T
+// RUN: %clang_cc1 %s -std=c++11 -index-store-path %t/idx -DTYPE1=B -DTYPE2=A -DTYPE3=T -DTYPE4=T
+// RUN: %clang_cc1 %s -std=c++11 -index-store-path %t/idx -DTYPE1=A -DTYPE2=B -DTYPE3=T -DTYPE4=T
+// RUN: %clang_cc1 %s -std=c++11 -index-store-path %t/idx -DTYPE1=B -DTYPE2=B -DTYPE3=T -DTYPE4=T
+// RUN: find %t/idx/*/records -name "record-hash*" | count 4
+//
+// RUN: rm -rf %t
+// RUN: %clang_cc1 %s -std=c++11 -index-store-path %t/idx -DTYPE1=A -DTYPE2=A -DTYPE3=T -DTYPE4=T
+// RUN: %clang_cc1 %s -std=c++11 -index-store-path %t/idx -DTYPE1=A -DTYPE2=A -DTYPE3=U -DTYPE4=T
+// RUN: %clang_cc1 %s -std=c++11 -index-store-path %t/idx -DTYPE1=A -DTYPE2=A -DTYPE3=T -DTYPE4=U
+// RUN: %clang_cc1 %s -std=c++11 -index-store-path %t/idx -DTYPE1=A -DTYPE2=A -DTYPE3=U -DTYPE4=U
+// RUN: find %t/idx/*/records -name "record-hash*" | count 4
+
+template<typename T>
+struct A {
+  typedef int X;
+  void foo();
+};
+
+template<typename T>
+struct B : public A<T> {
+  typedef float X;
+  void foo(int);
+};
+
+template<typename T>
+struct C : public B<T> {
+// This should result in different records, due to the different types.
+  using TYPE1<T>::X;
+  using TYPE2<T>::foo;
+};
+
+template <typename T>
+struct D {
+  typedef T X;
+  void foo(T);
+};
+template <typename T, typename U>
+struct E : public D<T>, public D<U> {
+// This should result in different records, due to the different template parameter.
+  using D<TYPE3>::X;
+  using D<TYPE4>::foo;
+};

clang/test/Index/Store/record-hash.cpp

+// RUN: rm -rf %t
+// RUN: %clang_cc1 %s -index-store-path %t/idx -D THE_TYPE=long
+// RUN: %clang_cc1 %s -index-store-path %t/idx -D THE_TYPE=char
+// RUN: find %t/idx/*/records -name "record-hash*" | count 2
+
+template<typename T>
+class TC {};
+
+// This should result in different records, due to the different template parameter type.
+void some_func(TC<THE_TYPE>);

clang/test/Index/Store/relative-out-path.c

+// Needs 'find'.
+// REQUIRES: shell
+
+// RUN: rm -rf %t
+// RUN: mkdir -p %t
+// RUN: %clang %s -index-store-path %t/idx1 -c -o %t/outfile.o
+// RUN: cd %t
+// RUN: %clang %s -index-store-path %t/idx2 -c -o outfile.o
+// RUN: cd ..
+// RUN: %clang %s -index-store-path %t/idx3 -fsyntax-only -o outfile.o -working-directory=%t
+// RUN: diff -r -u %t/idx2 %t/idx3
+
+// RUN: find %t/idx1 -name '*outfile.o*' > %t/hashes.txt
+// RUN: find %t/idx3 -name '*outfile.o*' >> %t/hashes.txt
+// RUN: FileCheck %s --input-file=%t/hashes.txt
+// CHECK:      outfile.o[[OUT_HASH:.*$]]
+// CHECK-NEXT: outfile.o[[OUT_HASH]]
+
+void foo();

clang/test/Index/Store/syntax-only.c

+// RUN: rm -rf %t.idx
+// RUN: %clang -fsyntax-only %s -index-store-path %t.idx -o %T/syntax-only.c.myoutfile
+// RUN: c-index-test core -print-unit %t.idx | FileCheck %s -check-prefix=CHECK-UNIT
+// RUN: c-index-test core -print-record %t.idx | FileCheck %s -check-prefix=CHECK-RECORD
+
+// CHECK-UNIT: out-file: {{.*}}/syntax-only.c.myoutfile
+// CHECK-RECORD: function/C | foo | c:@F@foo
+
+void foo();

clang/test/Index/Store/unit-with-vfs.c

+// RUN: sed -e "s:INPUT_DIR:%S/Inputs:g" -e "s:OUT_DIR:%t:g" %S/Inputs/overlay.yaml > %t.yaml
+// REQUIRES: shell
+
+#include "using-overlay.h"
+
+// RUN: rm -rf %t.idx
+// RUN: %clang_cc1 %s -index-store-path %t.idx -I %t -ivfsoverlay %t.yaml
+// RUN: c-index-test core -print-unit %t.idx | FileCheck %s
+
+// CHECK: Record | user | {{.*}}test/Index/Store/Inputs/using-overlay.h

clang/test/Index/Store/unit-workdir-prefix.c

+#include "header.h"
+
+void foo(void) {
+  bar();
+}
+
+// RUN: rm -rf %t
+// RUN: mkdir -p %t/Directory
+// RUN: mkdir -p %t/Directory.surprise
+// RUN: mkdir -p %t/sdk
+// RUN: mkdir -p %t/sdk_other
+// RUN: echo "void bar(void);" > %t/sdk_other/header.h
+// RUN: cp %s %t/Directory.surprise/main.c
+//
+// RUN: %clang_cc1 -isystem %t/sdk_other -isysroot %t/sdk -index-store-path %t/idx %t/Directory.surprise/main.c -triple x86_64-apple-macosx10.8 -working-directory %t/Directory
+// RUN: c-index-test core -print-unit %t/idx | FileCheck %s
+
+// CHECK: main.c.o
+// CHECK: provider: clang-
+// CHECK: is-system: 0
+// CHECK: has-main: 1
+// CHECK: main-path: {{.*}}Directory.surprise{{/|\\}}main.c
+// CHECK: out-file: {{.*}}Directory.surprise{{/|\\}}main.c.o
+// CHECK: target: x86_64-apple-macosx10.8
+// CHECK: is-debug: 1
+// CHECK: DEPEND START
+// CHECK: Record | user | {{.*}}Directory.surprise{{/|\\}}main.c | main.c-
+// CHECK: Record | system | {{.*}}sdk_other{{/|\\}}header.h | header.h-

clang/test/Index/Store/using-libstdcpp-arc.mm

+// Test to make sure we don't crash, rdar://30816887&36162712.
+
+// RUN: mkdir -p %t/include
+// RUN: echo 'module Foo { header "test.h" }' > %t/include/module.modulemap
+// RUN: echo '' > %t/include/test.h
+
+// RUN: rm -rf %t.idx %t.mcp
+// RUN: %clang_cc1 %s -index-store-path %t.idx -I %t/include -fobjc-arc -fobjc-arc-cxxlib=libstdc++ -fmodules -fmodules-cache-path=%t.mcp -fimplicit-module-maps
+// RUN: c-index-test core -print-record %t.idx | FileCheck %s
+
+// RUN: rm -rf %t.idx2
+// RUN: %clang_cc1 %s -index-store-path %t.idx2 -I %t/include -fobjc-arc -fobjc-arc-cxxlib=libstdc++ -fmodules -fmodules-cache-path=%t.mcp -fimplicit-module-maps
+// RUN: c-index-test core -print-record %t.idx2 | FileCheck %s
+
+@import Foo;
+
+// CHECK: [[@LINE+1]]:6 | function/C
+void test1(void);

clang/tools/CMakeLists.txt

 create_subdirectory_options(CLANG TOOL)
 
 add_clang_subdirectory(diagtool)
 add_clang_subdirectory(driver)
 add_clang_subdirectory(clang-diff)
 add_clang_subdirectory(clang-format)
 add_clang_subdirectory(clang-format-vs)
 add_clang_subdirectory(clang-fuzzer)
 add_clang_subdirectory(clang-import-test)
 add_clang_subdirectory(clang-offload-bundler)
 add_clang_subdirectory(clang-scan-deps)
 
 add_clang_subdirectory(c-index-test)
+add_clang_subdirectory(IndexStore)
 
 add_clang_subdirectory(clang-rename)
 add_clang_subdirectory(clang-refactor)
 if(UNIX)
   add_clang_subdirectory(clang-shlib)
 endif()
 
 if(CLANG_ENABLE_ARCMT)

clang/tools/IndexStore/CMakeLists.txt

+include(CheckIncludeFiles)
+
+set(SOURCES
+  IndexStore.cpp
+
+  ADDITIONAL_HEADERS
+  ../../include/indexstore/indexstore.h
+  ../../include/indexstore/IndexStoreCXX.h
+  )
+
+set(LIBS
+  clangIndex
+  clangIndexWhileBuilding
+  clangIndexDataStore
+)
+
+if(NOT CMAKE_SYSTEM_NAME STREQUAL Windows)
+  set(LLVM_EXPORTED_SYMBOL_FILE ${CMAKE_CURRENT_SOURCE_DIR}/IndexStore.exports)
+endif()
+
+if(WIN32)
+  set(output_name "libIndexStore")
+else()
+  set(output_name "IndexStore")
+endif()
+
+add_clang_library(IndexStore SHARED
+  OUTPUT_NAME ${output_name}
+  ${SOURCES}
+
+  LINK_LIBS
+  ${LIBS}
+
+  LINK_COMPONENTS
+  ${LLVM_TARGETS_TO_BUILD}
+  Core
+  Support
+  )
+
+if(APPLE)
+  set(INDEXSTORE_LIBRARY_VERSION "${LLVM_VERSION_MAJOR}.${LLVM_VERSION_MINOR}.${LLVM_VERSION_PATCH}")
+
+  set(INDEXSTORE_LINK_FLAGS " -Wl,-compatibility_version -Wl,1")
+  set(INDEXSTORE_LINK_FLAGS "${INDEXSTORE_LINK_FLAGS} -Wl,-current_version -Wl,${INDEXSTORE_LIBRARY_VERSION}")
+
+  check_include_files("CoreServices/CoreServices.h" HAVE_CORESERVICES_H)
+  if(HAVE_CORESERVICES_H)
+    set(INDEXSTORE_LINK_FLAGS "${INDEXSTORE_LINK_FLAGS} -framework CoreServices")
+  endif()
+
+  set_property(TARGET IndexStore APPEND_STRING PROPERTY
+               LINK_FLAGS ${INDEXSTORE_LINK_FLAGS})
+endif()
+
+if (LLVM_INSTALL_TOOLCHAIN_ONLY)
+  install(TARGETS IndexStore
+    COMPONENT IndexStore
+    LIBRARY DESTINATION lib${LLVM_LIBDIR_SUFFIX}
+    ARCHIVE DESTINATION lib${LLVM_LIBDIR_SUFFIX}
+    RUNTIME DESTINATION bin)
+
+  if (NOT CMAKE_CONFIGURATION_TYPES)
+    add_llvm_install_targets(install-IndexStore
+                             DEPENDS IndexStore
+                             COMPONENT IndexStore)
+  endif()
+endif()
+
+set(INDEXSTORE_HEADERS_INSTALL_DESTINATION "local/include")
+
+install(DIRECTORY ../../include/indexstore
+  COMPONENT IndexStore
+  DESTINATION "${INDEXSTORE_HEADERS_INSTALL_DESTINATION}"
+  FILES_MATCHING
+  PATTERN "*.h"
+  PATTERN ".svn" EXCLUDE
+  )

clang/tools/IndexStore/IndexStore.cpp

+//===- IndexStore.cpp - Index store API -----------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements the API for the index store.
+//
+//===----------------------------------------------------------------------===//
+
+#include "indexstore/indexstore.h"
+#include "indexstore/Conversions.h"
+#include "clang/DirectoryWatcher/DirectoryWatcher.h"
+#include "clang/Index/IndexWhileBuilding/IndexDataFormat.h"
+#include "clang/Index/IndexWhileBuilding/IndexRecordReader.h"
+#include "clang/Index/IndexWhileBuilding/IndexUnitReader.h"
+#include "clang/Index/IndexWhileBuilding/IndexUnitWriter.h"
+#include "clang/IndexDataStore/IndexDataStore.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/Optional.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/Support/Chrono.h"
+#include "llvm/Support/ManagedStatic.h"
+
+#if INDEXSTORE_HAS_BLOCKS
+#include <Block.h>
+#endif
+
+using namespace clang;
+using namespace clang::index;
+using namespace llvm;
+
+static indexstore_string_ref_t toIndexStoreString(StringRef str) {
+  return indexstore_string_ref_t{str.data(), str.size()};
+}
+
+static timespec toTimeSpec(sys::TimePoint<> tp) {
+  std::chrono::seconds sec =
+      std::chrono::time_point_cast<std::chrono::seconds>(tp).time_since_epoch();
+  std::chrono::nanoseconds nsec =
+      std::chrono::time_point_cast<std::chrono::nanoseconds>(tp - sec)
+          .time_since_epoch();
+  timespec ts;
+  ts.tv_sec = sec.count();
+  ts.tv_nsec = nsec.count();
+  return ts;
+}
+
+//===----------------------------------------------------------------------===//
+// Fatal error handling
+//===----------------------------------------------------------------------===//
+
+static void fatal_error_handler(void *user_data, const std::string &reason,
+                                bool gen_crash_diag) {
+  // Write the result out to stderr avoiding errs() because raw_ostreams can
+  // call report_fatal_error.
+  fprintf(stderr, "INDEXSTORE FATAL ERROR: %s\n", reason.c_str());
+  ::abort();
+}
+
+namespace {
+struct RegisterFatalErrorHandler {
+  RegisterFatalErrorHandler() {
+    llvm::install_fatal_error_handler(fatal_error_handler, nullptr);
+  }
+};
+} // namespace
+
+static llvm::ManagedStatic<RegisterFatalErrorHandler>
+    RegisterFatalErrorHandlerOnce;
+
+//===----------------------------------------------------------------------===//
+// C API
+//===----------------------------------------------------------------------===//
+
+namespace {
+
+struct IndexStoreError {
+  std::string Error;
+};
+
+} // anonymous namespace
+
+const char *indexstore_error_get_description(indexstore_error_t err) {
+  return static_cast<IndexStoreError *>(err)->Error.c_str();
+}
+
+void indexstore_error_dispose(indexstore_error_t err) {
+  delete static_cast<IndexStoreError *>(err);
+}
+
+unsigned indexstore_format_version(void) {
+  return IndexDataStore::getFormatVersion();
+}
+
+indexstore_t indexstore_store_create(const char *store_path,
+                                     indexstore_error_t *c_error) {
+  // Look through the managed static to trigger construction of the managed
+  // static which registers our fatal error handler. This ensures it is only
+  // registered once.
+  (void)*RegisterFatalErrorHandlerOnce;
+
+  std::unique_ptr<IndexDataStore> store;
+  std::string error;
+  store = IndexDataStore::create(store_path, error);
+  if (!store) {
+    if (c_error)
+      *c_error = new IndexStoreError{error};
+    return nullptr;
+  }
+  return store.release();
+}
+
+void indexstore_store_dispose(indexstore_t store) {
+  delete static_cast<IndexDataStore *>(store);
+}
+
+#if INDEXSTORE_HAS_BLOCKS
+bool indexstore_store_units_apply(
+    indexstore_t c_store, unsigned sorted,
+    INDEXSTORE_NOESCAPE bool (^applier)(indexstore_string_ref_t unit_name)) {
+  IndexDataStore *store = static_cast<IndexDataStore *>(c_store);
+  return store->foreachUnitName(sorted, [&](StringRef unitName) -> bool {
+    return applier(toIndexStoreString(unitName));
+  });
+}
+#endif
+
+bool indexstore_store_units_apply_f(
+    indexstore_t c_store, unsigned sorted, void *context,
+    INDEXSTORE_NOESCAPE bool (*applier)(void *context,
+                                        indexstore_string_ref_t unit_name)) {
+  IndexDataStore *store = static_cast<IndexDataStore *>(c_store);
+  return store->foreachUnitName(sorted, [&](StringRef unitName) -> bool {
+    return applier(context, toIndexStoreString(unitName));
+  });
+}
+
+size_t indexstore_unit_event_notification_get_events_count(
+    indexstore_unit_event_notification_t c_evtnote) {
+  auto *evtnote =
+      static_cast<IndexDataStore::UnitEventNotification *>(c_evtnote);
+  return evtnote->Events.size();
+}
+
+indexstore_unit_event_t indexstore_unit_event_notification_get_event(
+    indexstore_unit_event_notification_t c_evtnote, size_t index) {
+  auto *evtnote =
+      static_cast<IndexDataStore::UnitEventNotification *>(c_evtnote);
+  return (indexstore_unit_event_t)&evtnote->Events[index];
+}
+
+bool indexstore_unit_event_notification_is_initial(
+    indexstore_unit_event_notification_t c_evtnote) {
+  auto *evtnote =
+      static_cast<IndexDataStore::UnitEventNotification *>(c_evtnote);
+  return evtnote->IsInitial;
+}
+
+indexstore_unit_event_kind_t
+indexstore_unit_event_get_kind(indexstore_unit_event_t c_evt) {
+  auto *evt = static_cast<IndexDataStore::UnitEvent *>(c_evt);
+  indexstore_unit_event_kind_t k;
+  switch (evt->Kind) {
+  case IndexDataStore::UnitEventKind::Removed:
+    k = INDEXSTORE_UNIT_EVENT_REMOVED;
+    break;
+  case IndexDataStore::UnitEventKind::Modified:
+    k = INDEXSTORE_UNIT_EVENT_MODIFIED;
+    break;
+  case IndexDataStore::UnitEventKind::DirectoryDeleted:
+    k = INDEXSTORE_UNIT_EVENT_DIRECTORY_DELETED;
+    break;
+  case IndexDataStore::UnitEventKind::Failure:
+    k = INDEXSTORE_UNIT_EVENT_FAILURE;
+    break;
+  }
+  return k;
+}
+
+indexstore_string_ref_t
+indexstore_unit_event_get_unit_name(indexstore_unit_event_t c_evt) {
+  auto *evt = static_cast<IndexDataStore::UnitEvent *>(c_evt);
+  return toIndexStoreString(evt->UnitName);
+}
+
+#if INDEXSTORE_HAS_BLOCKS
+void indexstore_store_set_unit_event_handler(
+    indexstore_t c_store, indexstore_unit_event_handler_t blk_handler) {
+  IndexDataStore *store = static_cast<IndexDataStore *>(c_store);
+  if (!blk_handler) {
+    store->setUnitEventHandler(nullptr);
+    return;
+  }
+
+  class BlockWrapper {
+    indexstore_unit_event_handler_t blk_handler;
+
+  public:
+    BlockWrapper(indexstore_unit_event_handler_t handler) {
+      blk_handler = Block_copy(handler);
+    }
+    BlockWrapper(const BlockWrapper &other) {
+      blk_handler = Block_copy(other.blk_handler);
+    }
+    ~BlockWrapper() { Block_release(blk_handler); }
+
+    void operator()(indexstore_unit_event_notification_t evt_note) const {
+      blk_handler(evt_note);
+    }
+  };
+
+  BlockWrapper handler(blk_handler);
+
+  store->setUnitEventHandler(
+      [handler](IndexDataStore::UnitEventNotification evtNote) {
+        handler(&evtNote);
+      });
+}
+#endif
+
+void indexstore_store_set_unit_event_handler_f(
+    indexstore_t c_store, void *context,
+    void (*fn_handler)(void *context, indexstore_unit_event_notification_t),
+    void (*finalizer)(void *context)) {
+  IndexDataStore *store = static_cast<IndexDataStore *>(c_store);
+  if (!fn_handler) {
+    store->setUnitEventHandler(nullptr);
+    return;
+  }
+
+  class BlockWrapper {
+    void *context;
+    void (*fn_handler)(void *context, indexstore_unit_event_notification_t);
+    void (*finalizer)(void *context);
+
+  public:
+    BlockWrapper(void *_context,
+                 void (*_fn_handler)(void *context,
+                                     indexstore_unit_event_notification_t),
+                 void (*_finalizer)(void *context))
+        : context(_context), fn_handler(_fn_handler), finalizer(_finalizer) {}
+
+    ~BlockWrapper() {
+      if (finalizer) {
+        finalizer(context);
+      }
+    }
+
+    void operator()(indexstore_unit_event_notification_t evt_note) const {
+      fn_handler(context, evt_note);
+    }
+  };
+
+  auto handler = std::make_shared<BlockWrapper>(context, fn_handler, finalizer);
+
+  store->setUnitEventHandler(
+      [handler](IndexDataStore::UnitEventNotification evtNote) {
+        (*handler)(&evtNote);
+      });
+}
+
+bool indexstore_store_start_unit_event_listening(
+    indexstore_t c_store, indexstore_unit_event_listen_options_t *client_opts,
+    size_t listen_options_struct_size, indexstore_error_t *c_error) {
+  IndexDataStore *store = static_cast<IndexDataStore *>(c_store);
+  indexstore_unit_event_listen_options_t listen_opts;
+  memset(&listen_opts, 0, sizeof(listen_opts));
+  unsigned clientOptSize = listen_options_struct_size < sizeof(listen_opts)
+                               ? listen_options_struct_size
+                               : sizeof(listen_opts);
+  memcpy(&listen_opts, client_opts, clientOptSize);
+
+  std::string error;
+  bool err = store->startEventListening(listen_opts.wait_initial_sync, error);
+  if (err && c_error)
+    *c_error = new IndexStoreError{error};
+  return err;
+}
+
+void indexstore_store_stop_unit_event_listening(indexstore_t c_store) {
+  IndexDataStore *store = static_cast<IndexDataStore *>(c_store);
+  store->stopEventListening();
+}
+
+void indexstore_store_discard_unit(indexstore_t c_store,
+                                   const char *unit_name) {
+  IndexDataStore *store = static_cast<IndexDataStore *>(c_store);
+  store->discardUnit(unit_name);
+}
+
+void indexstore_store_discard_record(indexstore_t c_store,
+                                     const char *record_name) {
+  IndexDataStore *store = static_cast<IndexDataStore *>(c_store);
+  store->discardRecord(record_name);
+}
+
+void indexstore_store_purge_stale_data(indexstore_t c_store) {
+  IndexDataStore *store = static_cast<IndexDataStore *>(c_store);
+  store->purgeStaleData();
+}
+
+indexstore_symbol_kind_t indexstore_symbol_get_kind(indexstore_symbol_t sym) {
+  return getIndexStoreKind(static_cast<IndexRecordDecl *>(sym)->SymInfo.Kind);
+}
+
+indexstore_symbol_subkind_t
+indexstore_symbol_get_subkind(indexstore_symbol_t sym) {
+  return getIndexStoreSubKind(
+      static_cast<IndexRecordDecl *>(sym)->SymInfo.SubKind);
+}
+
+indexstore_symbol_language_t
+indexstore_symbol_get_language(indexstore_symbol_t sym) {
+  return getIndexStoreLang(static_cast<IndexRecordDecl *>(sym)->SymInfo.Lang);
+}
+
+uint64_t indexstore_symbol_get_properties(indexstore_symbol_t sym) {
+  return getIndexStoreProperties(
+      static_cast<IndexRecordDecl *>(sym)->SymInfo.Properties);
+}
+
+uint64_t indexstore_symbol_get_roles(indexstore_symbol_t sym) {
+  return getIndexStoreRoles(static_cast<IndexRecordDecl *>(sym)->Roles);
+}
+
+uint64_t indexstore_symbol_get_related_roles(indexstore_symbol_t sym) {
+  return getIndexStoreRoles(static_cast<IndexRecordDecl *>(sym)->RelatedRoles);
+}
+
+indexstore_string_ref_t indexstore_symbol_get_name(indexstore_symbol_t sym) {
+  auto *D = static_cast<IndexRecordDecl *>(sym);
+  return toIndexStoreString(D->Name);
+}
+
+indexstore_string_ref_t indexstore_symbol_get_usr(indexstore_symbol_t sym) {
+  auto *D = static_cast<IndexRecordDecl *>(sym);
+  return toIndexStoreString(D->USR);
+}
+
+indexstore_string_ref_t
+indexstore_symbol_get_codegen_name(indexstore_symbol_t sym) {
+  auto *D = static_cast<IndexRecordDecl *>(sym);
+  return toIndexStoreString(D->CodeGenName);
+}
+
+uint64_t
+indexstore_symbol_relation_get_roles(indexstore_symbol_relation_t sym_rel) {
+  return getIndexStoreRoles(static_cast<IndexRecordRelation *>(sym_rel)->Roles);
+}
+
+indexstore_symbol_t
+indexstore_symbol_relation_get_symbol(indexstore_symbol_relation_t sym_rel) {
+  return (indexstore_symbol_t) static_cast<IndexRecordRelation *>(sym_rel)->Dcl;
+}
+
+indexstore_symbol_t
+indexstore_occurrence_get_symbol(indexstore_occurrence_t occur) {
+  return (indexstore_symbol_t) static_cast<IndexRecordOccurrence *>(occur)->Dcl;
+}
+
+#if INDEXSTORE_HAS_BLOCKS
+bool indexstore_occurrence_relations_apply(
+    indexstore_occurrence_t occur,
+    INDEXSTORE_NOESCAPE bool (^applier)(
+        indexstore_symbol_relation_t symbol_rel)) {
+  auto *recOccur = static_cast<IndexRecordOccurrence *>(occur);
+  for (auto &rel : recOccur->Relations) {
+    if (!applier(&rel))
+      return false;
+  }
+  return true;
+}
+#endif
+
+bool indexstore_occurrence_relations_apply_f(
+    indexstore_occurrence_t occur, void *context,
+    INDEXSTORE_NOESCAPE bool (*applier)(
+        void *context, indexstore_symbol_relation_t symbol_rel)) {
+  auto *recOccur = static_cast<IndexRecordOccurrence *>(occur);
+  for (auto &rel : recOccur->Relations) {
+    if (!applier(context, &rel))
+      return false;
+  }
+  return true;
+}
+
+uint64_t indexstore_occurrence_get_roles(indexstore_occurrence_t occur) {
+  return getIndexStoreRoles(static_cast<IndexRecordOccurrence *>(occur)->Roles);
+}
+
+void indexstore_occurrence_get_line_col(indexstore_occurrence_t occur,
+                                        unsigned *line, unsigned *column) {
+  auto *recOccur = static_cast<IndexRecordOccurrence *>(occur);
+  if (line)
+    *line = recOccur->Line;
+  if (column)
+    *column = recOccur->Column;
+}
+
+typedef void *indexstore_record_reader_t;
+
+indexstore_record_reader_t
+indexstore_record_reader_create(indexstore_t c_store, const char *record_name,
+                                indexstore_error_t *c_error) {
+  IndexDataStore *store = static_cast<IndexDataStore *>(c_store);
+  std::unique_ptr<IndexRecordReader> reader;
+  std::string error;
+  reader = IndexRecordReader::createWithRecordFilename(
+      record_name, store->getFilePath(), error);
+  if (!reader) {
+    if (c_error)
+      *c_error = new IndexStoreError{error};
+    return nullptr;
+  }
+  return reader.release();
+}
+
+void indexstore_record_reader_dispose(indexstore_record_reader_t rdr) {
+  auto *reader = static_cast<IndexRecordReader *>(rdr);
+  delete reader;
+}
+
+#if INDEXSTORE_HAS_BLOCKS
+/// Goes through the symbol data and passes symbols to \c receiver, for the
+/// symbol data that \c filter returns true on.
+///
+/// This allows allocating memory only for the record symbols that the caller is
+/// interested in.
+bool indexstore_record_reader_search_symbols(
+    indexstore_record_reader_t rdr,
+    INDEXSTORE_NOESCAPE bool (^filter)(indexstore_symbol_t symbol, bool *stop),
+    INDEXSTORE_NOESCAPE void (^receiver)(indexstore_symbol_t symbol)) {
+  auto *reader = static_cast<IndexRecordReader *>(rdr);
+
+  auto filterFn =
+      [&](const IndexRecordDecl &D) -> IndexRecordReader::DeclSearchReturn {
+    bool stop = false;
+    bool accept = filter((indexstore_symbol_t)&D, &stop);
+    return {accept, !stop};
+  };
+  auto receiverFn = [&](const IndexRecordDecl *D) {
+    receiver((indexstore_symbol_t)D);
+  };
+
+  return reader->searchDecls(filterFn, receiverFn);
+}
+
+bool indexstore_record_reader_symbols_apply(
+    indexstore_record_reader_t rdr, bool nocache,
+    INDEXSTORE_NOESCAPE bool (^applier)(indexstore_symbol_t symbol)) {
+  auto *reader = static_cast<IndexRecordReader *>(rdr);
+  auto receiverFn = [&](const IndexRecordDecl *D) -> bool {
+    return applier((indexstore_symbol_t)D);
+  };
+  return reader->foreachDecl(nocache, receiverFn);
+}
+
+bool indexstore_record_reader_occurrences_apply(
+    indexstore_record_reader_t rdr,
+    INDEXSTORE_NOESCAPE bool (^applier)(indexstore_occurrence_t occur)) {
+  auto *reader = static_cast<IndexRecordReader *>(rdr);
+  auto receiverFn = [&](const IndexRecordOccurrence &RO) -> bool {
+    return applier((indexstore_occurrence_t)&RO);
+  };
+  return reader->foreachOccurrence(receiverFn);
+}
+
+bool indexstore_record_reader_occurrences_in_line_range_apply(
+    indexstore_record_reader_t rdr, unsigned line_start, unsigned line_count,
+    INDEXSTORE_NOESCAPE bool (^applier)(indexstore_occurrence_t occur)) {
+  auto *reader = static_cast<IndexRecordReader *>(rdr);
+  auto receiverFn = [&](const IndexRecordOccurrence &RO) -> bool {
+    return applier((indexstore_occurrence_t)&RO);
+  };
+  return reader->foreachOccurrenceInLineRange(line_start, line_count,
+                                              receiverFn);
+}
+
+/// \param symbols if non-zero \c symbols_count, indicates the list of symbols
+/// that we want to get occurrences for. An empty array indicates that we want
+/// occurrences for all symbols.
+/// \param related_symbols Same as \c symbols but for related symbols.
+bool indexstore_record_reader_occurrences_of_symbols_apply(
+    indexstore_record_reader_t rdr, indexstore_symbol_t *symbols,
+    size_t symbols_count, indexstore_symbol_t *related_symbols,
+    size_t related_symbols_count,
+    INDEXSTORE_NOESCAPE bool (^applier)(indexstore_occurrence_t occur)) {
+  auto *reader = static_cast<IndexRecordReader *>(rdr);
+  auto receiverFn = [&](const IndexRecordOccurrence &RO) -> bool {
+    return applier((indexstore_occurrence_t)&RO);
+  };
+  return reader->foreachOccurrence(
+      receiverFn, {(IndexRecordDecl **)symbols, symbols_count},
+      {(IndexRecordDecl **)related_symbols, related_symbols_count});
+}
+#endif
+
+bool indexstore_record_reader_search_symbols_f(
+    indexstore_record_reader_t rdr, void *filter_ctx,
+    INDEXSTORE_NOESCAPE bool (*filter)(void *filter_ctx,
+                                       indexstore_symbol_t symbol, bool *stop),
+    void *receiver_ctx,
+    INDEXSTORE_NOESCAPE void (*receiver)(void *receiver_ctx,
+                                         indexstore_symbol_t symbol)) {
+  auto *reader = static_cast<IndexRecordReader *>(rdr);
+
+  auto filterFn =
+      [&](const IndexRecordDecl &D) -> IndexRecordReader::DeclSearchReturn {
+    bool stop = false;
+    bool accept = filter(filter_ctx, (indexstore_symbol_t)&D, &stop);
+    return {accept, !stop};
+  };
+  auto receiverFn = [&](const IndexRecordDecl *D) {
+    receiver(receiver_ctx, (indexstore_symbol_t)D);
+  };
+
+  return reader->searchDecls(filterFn, receiverFn);
+}
+
+bool indexstore_record_reader_symbols_apply_f(
+    indexstore_record_reader_t rdr, bool nocache, void *context,
+    INDEXSTORE_NOESCAPE bool (*applier)(void *context,
+                                        indexstore_symbol_t symbol)) {
+  auto *reader = static_cast<IndexRecordReader *>(rdr);
+  auto receiverFn = [&](const IndexRecordDecl *D) -> bool {
+    return applier(context, (indexstore_symbol_t)D);
+  };
+  return reader->foreachDecl(nocache, receiverFn);
+}
+
+bool indexstore_record_reader_occurrences_apply_f(
+    indexstore_record_reader_t rdr, void *context,
+    INDEXSTORE_NOESCAPE bool (*applier)(void *context,
+                                        indexstore_occurrence_t occur)) {
+  auto *reader = static_cast<IndexRecordReader *>(rdr);
+  auto receiverFn = [&](const IndexRecordOccurrence &RO) -> bool {
+    return applier(context, (indexstore_occurrence_t)&RO);
+  };
+  return reader->foreachOccurrence(receiverFn);
+}
+
+bool indexstore_record_reader_occurrences_in_line_range_apply_f(
+    indexstore_record_reader_t rdr, unsigned line_start, unsigned line_count,
+    void *context,
+    INDEXSTORE_NOESCAPE bool (*applier)(void *context,
+                                        indexstore_occurrence_t occur)) {
+  auto *reader = static_cast<IndexRecordReader *>(rdr);
+  auto receiverFn = [&](const IndexRecordOccurrence &RO) -> bool {
+    return applier(context, (indexstore_occurrence_t)&RO);
+  };
+  return reader->foreachOccurrenceInLineRange(line_start, line_count,
+                                              receiverFn);
+}
+
+bool indexstore_record_reader_occurrences_of_symbols_apply_f(
+    indexstore_record_reader_t rdr, indexstore_symbol_t *symbols,
+    size_t symbols_count, indexstore_symbol_t *related_symbols,
+    size_t related_symbols_count, void *context,
+    INDEXSTORE_NOESCAPE bool (*applier)(void *context,
+                                        indexstore_occurrence_t occur)) {
+  auto *reader = static_cast<IndexRecordReader *>(rdr);
+  auto receiverFn = [&](const IndexRecordOccurrence &RO) -> bool {
+    return applier(context, (indexstore_occurrence_t)&RO);
+  };
+  return reader->foreachOccurrence(
+      receiverFn, {(IndexRecordDecl **)symbols, symbols_count},
+      {(IndexRecordDecl **)related_symbols, related_symbols_count});
+}
+
+size_t indexstore_store_get_unit_name_from_output_path(indexstore_t store,
+                                                       const char *output_path,
+                                                       char *name_buf,
+                                                       size_t buf_size) {
+  llvm::SmallString<256> unitName;
+  clang::index::store::getUnitNameForOutputFile(output_path, unitName);
+  size_t nameLen = unitName.size();
+  if (buf_size != 0) {
+    strncpy(name_buf, unitName.c_str(), buf_size - 1);
+    name_buf[buf_size - 1] = '\0';
+  }
+  return nameLen;
+}
+
+bool indexstore_store_get_unit_modification_time(indexstore_t c_store,
+                                                 const char *unit_name,
+                                                 int64_t *seconds,
+                                                 int64_t *nanoseconds,
+                                                 indexstore_error_t *c_error) {
+  IndexDataStore *store = static_cast<IndexDataStore *>(c_store);
+  std::string error;
+  // FIXME: This provides mod time with second-only accuracy.
+  auto optModTime = IndexUnitReader::getModificationTimeForUnit(
+      unit_name, store->getFilePath(), error);
+  if (!optModTime) {
+    if (c_error)
+      *c_error = new IndexStoreError{error};
+    return true;
+  }
+
+  timespec ts = toTimeSpec(*optModTime);
+  if (seconds)
+    *seconds = ts.tv_sec;
+  if (nanoseconds)
+    *nanoseconds = ts.tv_nsec;
+
+  return false;
+}
+
+indexstore_unit_reader_t
+indexstore_unit_reader_create(indexstore_t c_store, const char *unit_name,
+                              indexstore_error_t *c_error) {
+  IndexDataStore *store = static_cast<IndexDataStore *>(c_store);
+  std::unique_ptr<IndexUnitReader> reader;
+  std::string error;
+  reader = IndexUnitReader::createWithUnitFilename(unit_name,
+                                                   store->getFilePath(), error);
+  if (!reader) {
+    if (c_error)
+      *c_error = new IndexStoreError{error};
+    return nullptr;
+  }
+  return reader.release();
+}
+
+void indexstore_unit_reader_dispose(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  delete reader;
+}
+
+indexstore_string_ref_t
+indexstore_unit_reader_get_provider_identifier(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return toIndexStoreString(reader->getProviderIdentifier());
+}
+
+indexstore_string_ref_t
+indexstore_unit_reader_get_provider_version(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return toIndexStoreString(reader->getProviderVersion());
+}
+
+void indexstore_unit_reader_get_modification_time(indexstore_unit_reader_t rdr,
+                                                  int64_t *seconds,
+                                                  int64_t *nanoseconds) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  // FIXME: This provides mod time with second-only accuracy.
+  sys::TimePoint<> timeVal = reader->getModificationTime();
+  timespec ts = toTimeSpec(timeVal);
+  if (seconds)
+    *seconds = ts.tv_sec;
+  if (nanoseconds)
+    *nanoseconds = ts.tv_nsec;
+}
+
+bool indexstore_unit_reader_is_system_unit(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return reader->isSystemUnit();
+}
+
+bool indexstore_unit_reader_is_module_unit(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return reader->isModuleUnit();
+}
+
+bool indexstore_unit_reader_is_debug_compilation(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return reader->isDebugCompilation();
+}
+
+bool indexstore_unit_reader_has_main_file(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return reader->hasMainFile();
+}
+
+indexstore_string_ref_t
+indexstore_unit_reader_get_main_file(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return toIndexStoreString(reader->getMainFilePath());
+}
+
+indexstore_string_ref_t
+indexstore_unit_reader_get_module_name(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return toIndexStoreString(reader->getModuleName());
+}
+
+indexstore_string_ref_t
+indexstore_unit_reader_get_working_dir(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return toIndexStoreString(reader->getWorkingDirectory());
+}
+
+indexstore_string_ref_t
+indexstore_unit_reader_get_output_file(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return toIndexStoreString(reader->getOutputFile());
+}
+
+indexstore_string_ref_t
+indexstore_unit_reader_get_sysroot_path(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return toIndexStoreString(reader->getSysrootPath());
+}
+
+indexstore_string_ref_t
+indexstore_unit_reader_get_target(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return toIndexStoreString(reader->getTarget());
+}
+
+indexstore_unit_dependency_kind_t
+indexstore_unit_dependency_get_kind(indexstore_unit_dependency_t c_dep) {
+  auto dep = static_cast<const IndexUnitReader::DependencyInfo *>(c_dep);
+  switch (dep->Kind) {
+  case IndexUnitReader::DependencyKind::Unit:
+    return INDEXSTORE_UNIT_DEPENDENCY_UNIT;
+  case IndexUnitReader::DependencyKind::Record:
+    return INDEXSTORE_UNIT_DEPENDENCY_RECORD;
+  case IndexUnitReader::DependencyKind::File:
+    return INDEXSTORE_UNIT_DEPENDENCY_FILE;
+  }
+}
+
+bool indexstore_unit_dependency_is_system(indexstore_unit_dependency_t c_dep) {
+  auto dep = static_cast<const IndexUnitReader::DependencyInfo *>(c_dep);
+  return dep->IsSystem;
+}
+
+indexstore_string_ref_t
+indexstore_unit_dependency_get_filepath(indexstore_unit_dependency_t c_dep) {
+  auto dep = static_cast<const IndexUnitReader::DependencyInfo *>(c_dep);
+  return toIndexStoreString(dep->FilePath);
+}
+
+indexstore_string_ref_t
+indexstore_unit_dependency_get_modulename(indexstore_unit_dependency_t c_dep) {
+  auto dep = static_cast<const IndexUnitReader::DependencyInfo *>(c_dep);
+  return toIndexStoreString(dep->ModuleName);
+}
+
+indexstore_string_ref_t
+indexstore_unit_dependency_get_name(indexstore_unit_dependency_t c_dep) {
+  auto dep = static_cast<const IndexUnitReader::DependencyInfo *>(c_dep);
+  return toIndexStoreString(dep->UnitOrRecordName);
+}
+
+indexstore_string_ref_t
+indexstore_unit_include_get_source_path(indexstore_unit_include_t c_inc) {
+  auto inc = static_cast<const IndexUnitReader::IncludeInfo *>(c_inc);
+  return toIndexStoreString(inc->SourcePath);
+}
+
+indexstore_string_ref_t
+indexstore_unit_include_get_target_path(indexstore_unit_include_t c_inc) {
+  auto inc = static_cast<const IndexUnitReader::IncludeInfo *>(c_inc);
+  return toIndexStoreString(inc->TargetPath);
+}
+
+unsigned
+indexstore_unit_include_get_source_line(indexstore_unit_include_t c_inc) {
+  auto inc = static_cast<const IndexUnitReader::IncludeInfo *>(c_inc);
+  return inc->SourceLine;
+}
+
+#if INDEXSTORE_HAS_BLOCKS
+bool indexstore_unit_reader_dependencies_apply(
+    indexstore_unit_reader_t rdr,
+    INDEXSTORE_NOESCAPE bool (^applier)(indexstore_unit_dependency_t)) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return reader->foreachDependency(
+      [&](const IndexUnitReader::DependencyInfo &depInfo) -> bool {
+        return applier((void *)&depInfo);
+      });
+}
+
+bool indexstore_unit_reader_includes_apply(
+    indexstore_unit_reader_t rdr,
+    INDEXSTORE_NOESCAPE bool (^applier)(indexstore_unit_include_t)) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return reader->foreachInclude(
+      [&](const IndexUnitReader::IncludeInfo &incInfo) -> bool {
+        return applier((void *)&incInfo);
+      });
+}
+#endif
+
+bool indexstore_unit_reader_dependencies_apply_f(
+    indexstore_unit_reader_t rdr, void *context,
+    INDEXSTORE_NOESCAPE bool (*applier)(void *context,
+                                        indexstore_unit_dependency_t)) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return reader->foreachDependency(
+      [&](const IndexUnitReader::DependencyInfo &depInfo) -> bool {
+        return applier(context, (void *)&depInfo);
+      });
+}
+
+bool indexstore_unit_reader_includes_apply_f(
+    indexstore_unit_reader_t rdr, void *context,
+    INDEXSTORE_NOESCAPE bool (*applier)(void *context,
+                                        indexstore_unit_include_t)) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return reader->foreachInclude(
+      [&](const IndexUnitReader::IncludeInfo &incInfo) -> bool {
+        return applier(context, (void *)&incInfo);
+      });
+}

clang/tools/IndexStore/IndexStore.exports

+indexstore_error_get_description
+indexstore_error_dispose
+indexstore_format_version
+indexstore_store_create
+indexstore_store_dispose
+indexstore_store_get_unit_modification_time
+indexstore_store_get_unit_name_from_output_path
+indexstore_store_units_apply
+indexstore_store_units_apply_f
+indexstore_store_set_unit_event_handler
+indexstore_store_set_unit_event_handler_f
+indexstore_store_start_unit_event_listening
+indexstore_store_stop_unit_event_listening
+indexstore_store_discard_unit
+indexstore_store_discard_record
+indexstore_store_purge_stale_data
+indexstore_symbol_get_kind
+indexstore_symbol_get_language
+indexstore_symbol_get_properties
+indexstore_symbol_get_roles
+indexstore_symbol_get_related_roles
+indexstore_symbol_get_subkind
+indexstore_symbol_get_name
+indexstore_symbol_get_usr
+indexstore_symbol_get_codegen_name
+indexstore_symbol_relation_get_roles
+indexstore_symbol_relation_get_symbol
+indexstore_occurrence_get_symbol
+indexstore_occurrence_get_roles
+indexstore_occurrence_get_line_col
+indexstore_occurrence_relations_apply
+indexstore_occurrence_relations_apply_f
+indexstore_record_reader_create
+indexstore_record_reader_dispose
+indexstore_record_reader_search_symbols
+indexstore_record_reader_search_symbols_f
+indexstore_record_reader_symbols_apply
+indexstore_record_reader_symbols_apply_f
+indexstore_record_reader_occurrences_apply
+indexstore_record_reader_occurrences_apply_f
+indexstore_record_reader_occurrences_in_line_range_apply
+indexstore_record_reader_occurrences_in_line_range_apply_f
+indexstore_record_reader_occurrences_of_symbols_apply
+indexstore_record_reader_occurrences_of_symbols_apply_f
+indexstore_unit_dependency_get_kind
+indexstore_unit_dependency_get_filepath
+indexstore_unit_dependency_get_modulename
+indexstore_unit_dependency_get_name
+indexstore_unit_dependency_is_system
+indexstore_unit_event_get_kind
+indexstore_unit_event_get_unit_name
+indexstore_unit_event_notification_get_event
+indexstore_unit_event_notification_get_events_count
+indexstore_unit_event_notification_is_initial
+indexstore_unit_reader_create
+indexstore_unit_reader_dispose
+indexstore_unit_reader_get_main_file
+indexstore_unit_reader_get_modification_time
+indexstore_unit_reader_get_module_name
+indexstore_unit_reader_get_provider_identifier
+indexstore_unit_reader_get_provider_version
+indexstore_unit_reader_get_working_dir
+indexstore_unit_reader_get_output_file
+indexstore_unit_reader_get_sysroot_path
+indexstore_unit_reader_get_target
+indexstore_unit_reader_dependencies_apply
+indexstore_unit_reader_dependencies_apply_f
+indexstore_unit_reader_includes_apply
+indexstore_unit_reader_includes_apply_f
+indexstore_unit_reader_has_main_file
+indexstore_unit_reader_is_debug_compilation
+indexstore_unit_reader_is_module_unit
+indexstore_unit_reader_is_system_unit
+indexstore_unit_include_get_source_path
+indexstore_unit_include_get_target_path
+indexstore_unit_include_get_source_line

clang/tools/c-index-test/CMakeLists.txt

+include(CheckIncludeFiles)
+
 set(LLVM_LINK_COMPONENTS
   support
 )
 
 add_clang_executable(c-index-test
   c-index-test.c
   core_main.cpp
+  JSONAggregation.cpp
   )
 
+set(INDEXSTORE_LIB)
+set(CINDEXTEST_LIBS)
+set(INDEXSTORE_LIB IndexStore)
+
 if(NOT MSVC)
   set_property(
     SOURCE c-index-test.c
     PROPERTY COMPILE_FLAGS "-std=gnu89"
     )
 endif()
 
 if (LLVM_BUILD_STATIC)
   target_link_libraries(c-index-test
     PRIVATE
     libclang_static
     clangCodeGen
     clangIndex
+    clangIndexWhileBuilding
+    ${CINDEXTEST_LIBS}
   )
 else()
   target_link_libraries(c-index-test
     PRIVATE
     libclang
+    ${INDEXSTORE_LIB}
     clangAST
     clangBasic
     clangCodeGen
+    clangDirectoryWatcher
+    clangIndexWhileBuilding
     clangFrontend
     clangIndex
     clangSerialization
+    ${CINDEXTEST_LIBS}
   )
 endif()
 
 set_target_properties(c-index-test
   PROPERTIES
   LINKER_LANGUAGE CXX)
 
 # If libxml2 is available, make it available for c-index-test.
 if (CLANG_HAVE_LIBXML)
   if ((CMAKE_OSX_SYSROOT) AND (EXISTS ${CMAKE_OSX_SYSROOT}/${LIBXML2_INCLUDE_DIR}))
     include_directories(SYSTEM ${CMAKE_OSX_SYSROOT}/${LIBXML2_INCLUDE_DIR})
   else()
     include_directories(SYSTEM ${LIBXML2_INCLUDE_DIR})
   endif()
   target_link_libraries(c-index-test PRIVATE ${LIBXML2_LIBRARIES})
 endif()
 
+if(APPLE)
+  check_include_files("CoreServices/CoreServices.h" HAVE_CORESERVICES_H)
+  if(HAVE_CORESERVICES_H)
+    target_link_libraries(c-index-test PRIVATE "-framework CoreServices")
+  endif()
+endif()
+
 if (NOT LLVM_INSTALL_TOOLCHAIN_ONLY)
   if(INTERNAL_INSTALL_PREFIX)
     set(INSTALL_DESTINATION "${INTERNAL_INSTALL_PREFIX}/bin")
     set_property(TARGET c-index-test APPEND PROPERTY INSTALL_RPATH
        "@executable_path/../../lib")
   else()
     set(INSTALL_DESTINATION bin)
   endif()

clang/tools/c-index-test/JSONAggregation.h

+//===--- JSONAggregation.h - Index data aggregation in JSON format --------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_TOOLS_CINDEXTEST_JSONAGGREGATION_H
+#define LLVM_CLANG_TOOLS_CINDEXTEST_JSONAGGREGATION_H
+
+#include "clang/Basic/LLVM.h"
+
+namespace clang {
+namespace index {
+
+/// Returns true if an error occurred, false otherwise.
+bool aggregateDataAsJSON(StringRef StorePath, raw_ostream &OS);
+
+} // end namespace index
+} // end namespace clang
+
+#endif

clang/tools/c-index-test/JSONAggregation.cpp

+//===--- JSONAggregation.cpp - Index data aggregation in JSON format ------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "JSONAggregation.h"
+#include "indexstore/Conversions.h"
+#include "indexstore/IndexStoreCXX.h"
+#include "clang/Frontend/Utils.h"
+#include "llvm/ADT/StringMap.h"
+#include "llvm/Support/Allocator.h"
+#include "llvm/Support/BuryPointer.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/raw_ostream.h"
+
+using namespace clang;
+using namespace clang::index;
+using namespace indexstore;
+using namespace llvm;
+
+namespace {
+
+typedef size_t FilePathIndex;
+typedef size_t RecordIndex;
+typedef size_t SymbolIndex;
+
+struct UnitSourceInfo {
+  FilePathIndex FilePath;
+  SmallVector<RecordIndex, 2> AssociatedRecords;
+};
+
+struct UnitInfo {
+  std::string Name;
+  SmallVector<UnitSourceInfo, 8> Sources;
+  SmallVector<std::string, 3> UnitDepends;
+  FilePathIndex OutFile;
+  StringRef Triple;
+};
+
+struct SymbolInfo {
+  SymbolKind Kind;
+  SymbolLanguage Lang;
+  StringRef USR;
+  StringRef Name;
+  StringRef CodegenName;
+  SymbolRoleSet Roles = 0;
+  SymbolRoleSet RelatedRoles = 0;
+};
+
+struct SymbolRelationInfo {
+  SymbolIndex RelatedSymbol;
+  SymbolRoleSet Roles;
+  SymbolRelationInfo(SymbolIndex relSymbol, SymbolRoleSet roles)
+      : RelatedSymbol(relSymbol), Roles(roles) {}
+};
+
+struct SymbolOccurrenceInfo {
+  SymbolIndex Symbol;
+  SymbolRoleSet Roles = 0;
+  std::vector<SymbolRelationInfo> Relations;
+  unsigned Line;
+  unsigned Column;
+};
+
+struct RecordInfo {
+  SmallVector<SymbolOccurrenceInfo, 8> Occurrences;
+};
+
+class Aggregator {
+  IndexStore Store;
+
+  BumpPtrAllocator Allocator;
+
+  StringMap<FilePathIndex, BumpPtrAllocator &> FilePathIndices;
+  std::vector<StringRef> FilePaths;
+  StringMap<char, BumpPtrAllocator &> Triples;
+
+  std::vector<std::unique_ptr<UnitInfo>> Units;
+
+  StringMap<RecordIndex, BumpPtrAllocator &> RecordIndices;
+  std::vector<std::unique_ptr<RecordInfo>> Records;
+
+  StringMap<SymbolIndex, BumpPtrAllocator &> SymbolIndices;
+  std::vector<SymbolInfo> Symbols;
+
+public:
+  explicit Aggregator(IndexStore store)
+      : Store(std::move(store)), FilePathIndices(Allocator), Triples(Allocator),
+        RecordIndices(Allocator), SymbolIndices(Allocator) {}
+
+  bool process();
+  void processUnit(StringRef name, IndexUnitReader &UnitReader);
+  void dumpJSON(raw_ostream &OS);
+
+private:
+  StringRef copyStr(StringRef str) {
+    if (str.empty())
+      return StringRef();
+    char *buf = Allocator.Allocate<char>(str.size());
+    std::copy(str.begin(), str.end(), buf);
+    return StringRef(buf, str.size());
+  }
+
+  StringRef getTripleString(StringRef inputTriple) {
+    return Triples.insert(std::make_pair(inputTriple, 0)).first->first();
+  }
+
+  FilePathIndex getFilePathIndex(StringRef path, StringRef workingDir);
+  RecordIndex getRecordIndex(StringRef recordFile);
+  SymbolIndex getSymbolIndex(IndexRecordSymbol sym);
+  std::unique_ptr<RecordInfo> processRecord(StringRef recordFile);
+};
+
+} // anonymous namespace
+
+bool Aggregator::process() {
+  bool succ =
+      Store.foreachUnit(/*sorted=*/true, [&](StringRef unitName) -> bool {
+        std::string error;
+        auto unitReader = IndexUnitReader(Store, unitName, error);
+        if (!unitReader) {
+          errs() << "error opening unit file '" << unitName << "': " << error
+                 << '\n';
+          return false;
+        }
+
+        processUnit(unitName, unitReader);
+        return true;
+      });
+
+  return !succ;
+}
+
+void Aggregator::processUnit(StringRef name, IndexUnitReader &UnitReader) {
+  auto workDir = UnitReader.getWorkingDirectory();
+  auto unit = llvm::make_unique<UnitInfo>();
+  unit->Name = name;
+  unit->Triple = getTripleString(UnitReader.getTarget());
+  unit->OutFile = getFilePathIndex(UnitReader.getOutputFile(), workDir);
+
+  struct DepInfo {
+    UnitSourceInfo source;
+    std::string unitName;
+  };
+  SmallVector<DepInfo, 32> Deps;
+  UnitReader.foreachDependency([&](IndexUnitDependency dep) -> bool {
+    Deps.resize(Deps.size() + 1);
+    auto &depInfo = Deps.back();
+    switch (dep.getKind()) {
+    case IndexUnitDependency::DependencyKind::Unit: {
+      depInfo.unitName = dep.getName();
+      StringRef filePath = dep.getFilePath();
+      if (!filePath.empty())
+        depInfo.source.FilePath = getFilePathIndex(filePath, workDir);
+      break;
+    }
+    case IndexUnitDependency::DependencyKind::Record: {
+      depInfo.source.FilePath = getFilePathIndex(dep.getFilePath(), workDir);
+      RecordIndex recIndex = getRecordIndex(dep.getName());
+      depInfo.source.AssociatedRecords.push_back(recIndex);
+      break;
+    }
+    case IndexUnitDependency::DependencyKind::File:
+      depInfo.source.FilePath = getFilePathIndex(dep.getFilePath(), workDir);
+    }
+    return true;
+  });
+
+  unit->Sources.reserve(Deps.size());
+  for (auto &dep : Deps) {
+    if (!dep.unitName.empty()) {
+      unit->UnitDepends.emplace_back(std::move(dep.unitName));
+    } else {
+      unit->Sources.push_back(std::move(dep.source));
+    }
+  }
+
+  Units.push_back(std::move(unit));
+}
+
+FilePathIndex Aggregator::getFilePathIndex(StringRef path,
+                                           StringRef workingDir) {
+  StringRef absPath;
+  SmallString<128> absPathBuf;
+  if (sys::path::is_absolute(path) || workingDir.empty()) {
+    absPath = path;
+  } else {
+    absPathBuf = workingDir;
+    sys::path::append(absPathBuf, path);
+    absPath = absPathBuf.str();
+  }
+
+  auto pair = FilePathIndices.insert(std::make_pair(absPath, FilePaths.size()));
+  bool wasInserted = pair.second;
+  if (wasInserted) {
+    FilePaths.push_back(pair.first->first());
+  }
+  return pair.first->second;
+}
+
+RecordIndex Aggregator::getRecordIndex(StringRef recordFile) {
+  auto pair = RecordIndices.insert(std::make_pair(recordFile, Records.size()));
+  bool wasInserted = pair.second;
+  if (wasInserted) {
+    Records.push_back(processRecord(recordFile));
+  }
+  return pair.first->second;
+}
+
+std::unique_ptr<RecordInfo> Aggregator::processRecord(StringRef recordFile) {
+  std::string error;
+  auto recordReader = IndexRecordReader(Store, recordFile, error);
+  if (!recordReader) {
+    errs() << "failed reading record file: " << recordFile << '\n';
+    ::exit(1);
+  }
+  auto record = llvm::make_unique<RecordInfo>();
+  recordReader.foreachOccurrence([&](IndexRecordOccurrence idxOccur) -> bool {
+    SymbolIndex symIdx = getSymbolIndex(idxOccur.getSymbol());
+    SymbolInfo &symInfo = Symbols[symIdx];
+    symInfo.Roles |= getSymbolRoles(idxOccur.getRoles());
+    SymbolOccurrenceInfo occurInfo;
+    occurInfo.Symbol = symIdx;
+    idxOccur.foreachRelation([&](IndexSymbolRelation rel) -> bool {
+      SymbolIndex relsymIdx = getSymbolIndex(rel.getSymbol());
+      SymbolInfo &relsymInfo = Symbols[relsymIdx];
+      relsymInfo.RelatedRoles |= getSymbolRoles(rel.getRoles());
+      occurInfo.Relations.emplace_back(relsymIdx,
+                                       getSymbolRoles(rel.getRoles()));
+      return true;
+    });
+    occurInfo.Roles = getSymbolRoles(idxOccur.getRoles());
+    std::tie(occurInfo.Line, occurInfo.Column) = idxOccur.getLineCol();
+    record->Occurrences.push_back(std::move(occurInfo));
+    return true;
+  });
+  return record;
+}
+
+SymbolIndex Aggregator::getSymbolIndex(IndexRecordSymbol sym) {
+  auto pair =
+      SymbolIndices.insert(std::make_pair(sym.getUSR(), Symbols.size()));
+  bool wasInserted = pair.second;
+  if (wasInserted) {
+    SymbolInfo symInfo;
+    symInfo.Kind = getSymbolKind(sym.getKind());
+    symInfo.Lang = getSymbolLanguage(sym.getLanguage());
+    symInfo.USR = pair.first->first();
+    symInfo.Name = copyStr(sym.getName());
+    symInfo.CodegenName = copyStr(sym.getCodegenName());
+    Symbols.push_back(std::move(symInfo));
+  }
+  return pair.first->second;
+}
+
+void Aggregator::dumpJSON(raw_ostream &OS) {
+  OS << "{\n";
+  OS.indent(2) << "\"files\": [\n";
+  for (unsigned i = 0, e = FilePaths.size(); i != e; ++i) {
+    OS.indent(4) << '\"' << FilePaths[i] << '\"';
+    if (i < e - 1)
+      OS << ',';
+    OS << '\n';
+  }
+  OS.indent(2) << "],\n";
+
+  OS.indent(2) << "\"symbols\": [\n";
+  for (unsigned i = 0, e = Symbols.size(); i != e; ++i) {
+    OS.indent(4) << "{\n";
+    SymbolInfo &symInfo = Symbols[i];
+    OS.indent(6) << "\"kind\": \"" << getSymbolKindString(symInfo.Kind)
+                 << "\",\n";
+    OS.indent(6) << "\"lang\": \"" << getSymbolLanguageString(symInfo.Lang)
+                 << "\",\n";
+    OS.indent(6) << "\"usr\": \"" << symInfo.USR << "\",\n";
+    OS.indent(6) << "\"name\": \"" << symInfo.Name << "\",\n";
+    if (!symInfo.CodegenName.empty())
+      OS.indent(6) << "\"codegen\": \"" << symInfo.CodegenName << "\",\n";
+    OS.indent(6) << "\"roles\": \"";
+    printSymbolRoles(symInfo.Roles, OS);
+    OS << '\"';
+    if (symInfo.RelatedRoles != 0) {
+      OS << ",\n";
+      OS.indent(6) << "\"rel-roles\": \"";
+      printSymbolRoles(symInfo.RelatedRoles, OS);
+      OS << '\"';
+    }
+    OS << '\n';
+    OS.indent(4) << "}";
+    if (i < e - 1)
+      OS << ',';
+    OS << '\n';
+  }
+  OS.indent(2) << "],\n";
+
+  OS.indent(2) << "\"records\": [\n";
+  for (unsigned i = 0, e = Records.size(); i != e; ++i) {
+    OS.indent(4) << "{\n";
+    RecordInfo &recInfo = *Records[i];
+    OS.indent(6) << "\"occurrences\": [\n";
+    for (unsigned oi = 0, oe = recInfo.Occurrences.size(); oi != oe; ++oi) {
+      OS.indent(8) << "{\n";
+      SymbolOccurrenceInfo &occurInfo = recInfo.Occurrences[oi];
+      OS.indent(10) << "\"symbol\": " << occurInfo.Symbol << ",\n";
+      OS.indent(10) << "\"line\": " << occurInfo.Line << ",\n";
+      OS.indent(10) << "\"col\": " << occurInfo.Column << ",\n";
+      OS.indent(10) << "\"roles\": \"";
+      printSymbolRoles(occurInfo.Roles, OS);
+      OS << '\"';
+      if (!occurInfo.Relations.empty()) {
+        OS << ",\n";
+        OS.indent(10) << "\"relations\": [\n";
+        for (unsigned ri = 0, re = occurInfo.Relations.size(); ri != re; ++ri) {
+          OS.indent(12) << "{\n";
+          SymbolRelationInfo &relInfo = occurInfo.Relations[ri];
+          OS.indent(14) << "\"symbol\": " << relInfo.RelatedSymbol << ",\n";
+          OS.indent(14) << "\"rel-roles\": \"";
+          printSymbolRoles(relInfo.Roles, OS);
+          OS << "\"\n";
+          OS.indent(12) << "}";
+          if (ri < re - 1)
+            OS << ',';
+          OS << '\n';
+        }
+        OS.indent(10) << "]\n";
+      }
+      OS << '\n';
+      OS.indent(8) << "}";
+      if (oi < oe - 1)
+        OS << ',';
+      OS << '\n';
+    }
+    OS.indent(6) << "]\n";
+    OS.indent(4) << "}";
+    if (i < e - 1)
+      OS << ',';
+    OS << '\n';
+  }
+  OS.indent(2) << "],\n";
+
+  StringMap<size_t> UnitIndicesByName;
+  for (unsigned i = 0, e = Units.size(); i != e; ++i) {
+    UnitInfo &unit = *Units[i];
+    UnitIndicesByName[unit.Name] = i;
+  }
+
+  OS.indent(2) << "\"units\": [\n";
+  for (unsigned i = 0, e = Units.size(); i != e; ++i) {
+    OS.indent(4) << "{\n";
+    UnitInfo &unit = *Units[i];
+    OS.indent(6) << "\"triple\": \"" << unit.Triple << "\",\n";
+    OS.indent(6) << "\"out-file\": " << unit.OutFile << ",\n";
+    if (!unit.UnitDepends.empty()) {
+      OS.indent(6) << "\"unit-dependencies\": [";
+      for (unsigned ui = 0, ue = unit.UnitDepends.size(); ui != ue; ++ui) {
+        OS << UnitIndicesByName[unit.UnitDepends[ui]];
+        if (ui < ue - 1)
+          OS << ", ";
+      }
+      OS << "],\n";
+    }
+    OS.indent(6) << "\"sources\": [\n";
+    for (unsigned si = 0, se = unit.Sources.size(); si != se; ++si) {
+      OS.indent(8) << "{\n";
+      UnitSourceInfo &source = unit.Sources[si];
+      OS.indent(10) << "\"file\": " << source.FilePath;
+      if (!source.AssociatedRecords.empty()) {
+        OS << ",\n";
+        OS.indent(10) << "\"records\": [";
+        for (unsigned ri = 0, re = source.AssociatedRecords.size(); ri != re;
+             ++ri) {
+          OS << source.AssociatedRecords[ri];
+          if (ri < re - 1)
+            OS << ", ";
+        }
+        OS << ']';
+      }
+      OS << '\n';
+      OS.indent(8) << "}";
+      if (si < se - 1)
+        OS << ',';
+      OS << '\n';
+    }
+    OS.indent(6) << "]\n";
+    OS.indent(4) << "}";
+    if (i < e - 1)
+      OS << ',';
+    OS << '\n';
+  }
+  OS.indent(2) << "]\n";
+  OS << "}\n";
+}
+
+bool index::aggregateDataAsJSON(StringRef StorePath, raw_ostream &OS) {
+  std::string error;
+  auto dataStore = IndexStore(StorePath, error);
+  if (!dataStore) {
+    errs() << "error opening store path '" << StorePath << "': " << error
+           << '\n';
+    return true;
+  }
+
+  // Explicitely avoid doing any memory cleanup for aggregator since the process
+  // is going to exit when we are done.
+  Aggregator *aggregator = new Aggregator(std::move(dataStore));
+  bool err = aggregator->process();
+  if (err)
+    return true;
+  aggregator->dumpJSON(OS);
+  llvm::BuryPointer(aggregator);
+  return false;
+}

clang/tools/c-index-test/core_main.cpp

 //===-- core_main.cpp - Core Index Tool testbed ---------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
+#include "JSONAggregation.h"
+#include "indexstore/Conversions.h"
+#include "indexstore/IndexStoreCXX.h"
 #include "clang/Basic/LangOptions.h"
 #include "clang/CodeGen/ObjectFilePCHContainerOperations.h"
+#include "clang/DirectoryWatcher/DirectoryWatcher.h"
 #include "clang/Frontend/ASTUnit.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/CompilerInvocation.h"
 #include "clang/Frontend/FrontendAction.h"
-#include "clang/Index/IndexingAction.h"
+#include "clang/Index/CodegenNameGenerator.h"
+#include "clang/Index/GenerateIndexAction.h"
 #include "clang/Index/IndexDataConsumer.h"
+#include "clang/Index/IndexWhileBuilding/IndexRecordReader.h"
+#include "clang/Index/IndexWhileBuilding/IndexUnitReader.h"
 #include "clang/Index/USRGeneration.h"
-#include "clang/Index/CodegenNameGenerator.h"
 #include "clang/Lex/Preprocessor.h"
 #include "clang/Serialization/ASTReader.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/FileSystem.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/PrettyStackTrace.h"
 #include "llvm/Support/Signals.h"
 #include "llvm/Support/raw_ostream.h"
-#include "llvm/Support/PrettyStackTrace.h"
+#include <thread>
 
 using namespace clang;
 using namespace clang::index;
 using namespace llvm;
 
 extern "C" int indextest_core_main(int argc, const char **argv);
 
 namespace {
 
 enum class ActionType {
   None,
   PrintSourceSymbols,
+  PrintRecord,
+  PrintUnit,
+  PrintStoreFormatVersion,
+  AggregateAsJSON,
+  WatchDir,
 };
 
 namespace options {
 
 static cl::OptionCategory IndexTestCoreCategory("index-test-core options");
 
-static cl::opt<ActionType>
-Action(cl::desc("Action:"), cl::init(ActionType::None),
-       cl::values(
-          clEnumValN(ActionType::PrintSourceSymbols,
-                     "print-source-symbols", "Print symbols from source")),
-       cl::cat(IndexTestCoreCategory));
+static cl::opt<ActionType> Action(
+    cl::desc("Action:"), cl::init(ActionType::None),
+        clEnumValN(ActionType::PrintSourceSymbols, "print-source-symbols",
+                   "Print symbols from source"),
+        clEnumValN(ActionType::PrintRecord, "print-record",
+                   "Print record info"),
+        clEnumValN(ActionType::PrintUnit, "print-unit", "Print unit info"),
+        clEnumValN(ActionType::PrintStoreFormatVersion,
+                   "print-store-format-version", "Print store format version"),
+        clEnumValN(ActionType::AggregateAsJSON, "aggregate-json",
+                   "Aggregate index data in JSON format"),
+        clEnumValN(ActionType::WatchDir, "watch-dir",
+                   "Watch directory for file events")),
 
+static cl::opt<std::string> OutputFile("o", cl::desc("output file"),
+                                       cl::cat(IndexTestCoreCategory));
+
+static cl::list<std::string> InputFiles(cl::Positional,
+                                        cl::desc("<filename>..."));
+
 static cl::extrahelp MoreHelp(
   "\nAdd \"-- <compiler arguments>\" at the end to setup the compiler "
   "invocation\n"
 );
 
 static cl::opt<bool>
 DumpModuleImports("dump-imported-module-files",
                cl::desc("Print symbols and input files from imported modules"));
 
 static cl::opt<bool>
 IncludeLocals("include-locals", cl::desc("Print local symbols"));
 
 static cl::opt<std::string>
 ModuleFilePath("module-file",
                cl::desc("Path to module file to print symbols from"));
 static cl::opt<std::string>
   ModuleFormat("fmodule-format", cl::init("raw"),
         cl::desc("Container format for clang modules and PCH, 'raw' or 'obj'"));
 
+static cl::opt<std::string> FilePathAndRange(
+    "filepath", cl::desc("File path that can optionally include a line range"));
 }
 } // anonymous namespace
 
 static void printSymbolInfo(SymbolInfo SymInfo, raw_ostream &OS);
 static void printSymbolNameAndUSR(const Decl *D, ASTContext &Ctx,
                                   raw_ostream &OS);
 static void printSymbolNameAndUSR(const clang::Module *Mod, raw_ostream &OS);
 
static bool printSourceSymbols(const char *Executable,
   auto CInvok = createInvocationFromCommandLine(ArgsWithProgName, Diags);
   if (!CInvok)
     return true;
 
   raw_ostream &OS = outs();
   auto DataConsumer = std::make_shared<PrintIndexDataConsumer>(OS);
   IndexingOptions IndexOpts;
   IndexOpts.IndexFunctionLocals = indexLocals;
-  std::unique_ptr<FrontendAction> IndexAction;
-  IndexAction = createIndexingAction(DataConsumer, IndexOpts,
-                                     /*WrappedAction=*/nullptr);
+  std::unique_ptr<FrontendAction> IndexAction =
+      llvm::make_unique<GenerateIndexAction>(DataConsumer, IndexOpts);
 
   auto PCHContainerOps = std::make_shared<PCHContainerOperations>();
   std::unique_ptr<ASTUnit> Unit(ASTUnit::LoadFromCompilerInvocationAction(
       std::move(CInvok), PCHContainerOps, Diags, IndexAction.get()));
 
   if (!Unit)
     return true;
 
static bool printSourceSymbolsFromModule(StringRef modulePath,
   PrintIndexDataConsumer DataConsumer(outs());
   IndexingOptions IndexOpts;
   indexASTUnit(*AU, DataConsumer, IndexOpts);
 
   return false;
 }
 
 //===----------------------------------------------------------------------===//
+// Print Record
+//===----------------------------------------------------------------------===//
+
+static void printSymbol(const IndexRecordDecl &Rec, raw_ostream &OS);
+static void printSymbol(const IndexRecordOccurrence &Rec, raw_ostream &OS);
+
+static int printRecord(StringRef Filename, raw_ostream &OS) {
+  std::string Error;
+  auto Reader = IndexRecordReader::createWithFilePath(Filename, Error);
+  if (!Reader) {
+    errs() << Error << '\n';
+    return true;
+  }
+
+  Reader->foreachDecl(/*noCache=*/true,
+                      [&](const IndexRecordDecl *Rec) -> bool {
+                        printSymbol(*Rec, OS);
+                        return true;
+                      });
+  OS << "------------\n";
+  Reader->foreachOccurrence([&](const IndexRecordOccurrence &Rec) -> bool {
+    printSymbol(Rec, OS);
+    return true;
+  });
+
+  return false;
+};
+
+//===----------------------------------------------------------------------===//
+// Print Store Records
+//===----------------------------------------------------------------------===//
+
+static void printSymbol(indexstore::IndexRecordSymbol Sym, raw_ostream &OS);
+static void printSymbol(indexstore::IndexRecordOccurrence Occur,
+                        raw_ostream &OS);
+
+static bool printStoreRecord(indexstore::IndexStore &Store, StringRef RecName,
+                             StringRef FilePath, raw_ostream &OS) {
+  std::string Error;
+  indexstore::IndexRecordReader Reader(Store, RecName, Error);
+  if (!Reader) {
+    errs() << "error loading record: " << Error << "\n";
+    return true;
+  }
+
+  StringRef Filename = sys::path::filename(FilePath);
+  OS << Filename << '\n';
+  OS << "------------\n";
+  Reader.foreachSymbol(/*noCache=*/true,
+                       [&](indexstore::IndexRecordSymbol Sym) -> bool {
+                         printSymbol(Sym, OS);
+                         return true;
+                       });
+  OS << "------------\n";
+  Reader.foreachOccurrence(
+      [&](indexstore::IndexRecordOccurrence Occur) -> bool {
+        printSymbol(Occur, OS);
+        return true;
+      });
+
+  return false;
+}
+
+static int printStoreRecords(StringRef StorePath, raw_ostream &OS) {
+  std::string Error;
+  indexstore::IndexStore Store(StorePath, Error);
+  if (!Store) {
+    errs() << "error loading store: " << Error << "\n";
+    return 1;
+  }
+
+  bool Success =
+      Store.foreachUnit(/*sorted=*/true, [&](StringRef UnitName) -> bool {
+        indexstore::IndexUnitReader Reader(Store, UnitName, Error);
+        if (!Reader) {
+          errs() << "error loading unit: " << Error << "\n";
+          return false;
+        }
+        return Reader.foreachDependency(
+            [&](indexstore::IndexUnitDependency Dep) -> bool {
+              if (Dep.getKind() ==
+                  indexstore::IndexUnitDependency::DependencyKind::Record) {
+                bool Err = printStoreRecord(Store, Dep.getName(),
+                                            Dep.getFilePath(), OS);
+                OS << '\n';
+                return !Err;
+              }
+              return true;
+            });
+      });
+
+  return !Success;
+}
+
+static std::string findRecordNameForFile(indexstore::IndexStore &store,
+                                         StringRef filePath) {
+  std::string recName;
+  store.foreachUnit(/*sorted=*/false, [&](StringRef unitName) -> bool {
+    std::string error;
+    indexstore::IndexUnitReader Reader(store, unitName, error);
+    if (!Reader) {
+      errs() << "error loading unit: " << error << "\n";
+      return false;
+    }
+    Reader.foreachDependency([&](indexstore::IndexUnitDependency Dep) -> bool {
+      if (Dep.getKind() ==
+          indexstore::IndexUnitDependency::DependencyKind::Record) {
+        if (Dep.getFilePath() == filePath) {
+          recName = Dep.getName();
+          return false;
+        }
+        return true;
+      }
+      return true;
+    });
+    return true;
+  });
+  return recName;
+}
+
+static int printStoreFileRecord(StringRef storePath, StringRef filePath,
+                                Optional<unsigned> lineStart,
+                                unsigned lineCount, raw_ostream &OS) {
+  std::string error;
+  indexstore::IndexStore store(storePath, error);
+  if (!store) {
+    errs() << "error loading store: " << error << "\n";
+    return 1;
+  }
+
+  std::string recName = findRecordNameForFile(store, filePath);
+  if (recName.empty()) {
+    errs() << "could not find record for '" << filePath << "'\n";
+    return 1;
+  }
+
+  if (!lineStart.hasValue())
+    return printStoreRecord(store, recName, filePath, OS);
+
+  indexstore::IndexRecordReader Reader(store, recName, error);
+  if (!Reader) {
+    errs() << "error loading record: " << error << "\n";
+    return 1;
+  }
+
+  Reader.foreachOccurrenceInLineRange(
+      *lineStart, lineCount,
+      [&](indexstore::IndexRecordOccurrence Occur) -> bool {
+        printSymbol(Occur, OS);
+        return true;
+      });
+
+  return 0;
+}
+
+//===----------------------------------------------------------------------===//
+// Print Unit
+//===----------------------------------------------------------------------===//
+
+static int printUnit(StringRef Filename, raw_ostream &OS) {
+  std::string Error;
+  auto Reader = IndexUnitReader::createWithFilePath(Filename, Error);
+  if (!Reader) {
+    errs() << Error << '\n';
+    return true;
+  }
+
+  OS << "provider: " << Reader->getProviderIdentifier() << '-'
+     << Reader->getProviderVersion() << '\n';
+  OS << "is-system: " << Reader->isSystemUnit() << '\n';
+  OS << "is-module: " << Reader->isModuleUnit() << '\n';
+  OS << "module-name: "
+     << (Reader->getModuleName().empty() ? "<none>" : Reader->getModuleName())
+     << '\n';
+  OS << "has-main: " << Reader->hasMainFile() << '\n';
+  OS << "main-path: " << Reader->getMainFilePath() << '\n';
+  OS << "work-dir: " << Reader->getWorkingDirectory() << '\n';
+  OS << "out-file: " << Reader->getOutputFile() << '\n';
+  OS << "target: " << Reader->getTarget() << '\n';
+  OS << "is-debug: " << Reader->isDebugCompilation() << '\n';
+  OS << "DEPEND START\n";
+  unsigned NumDepends = 0;
+  Reader->foreachDependency(
+      [&](const IndexUnitReader::DependencyInfo &Dep) -> bool {
+        switch (Dep.Kind) {
+        case IndexUnitReader::DependencyKind::Unit:
+          OS << "Unit | ";
+          break;
+        case IndexUnitReader::DependencyKind::Record:
+          OS << "Record | ";
+          break;
+        case IndexUnitReader::DependencyKind::File:
+          OS << "File | ";
+          break;
+        }
+        OS << (Dep.IsSystem ? "system" : "user");
+        OS << " | ";
+        if (!Dep.ModuleName.empty())
+          OS << Dep.ModuleName << " | ";
+        OS << Dep.FilePath;
+        if (!Dep.UnitOrRecordName.empty())
+          OS << " | " << Dep.UnitOrRecordName;
+        OS << '\n';
+        ++NumDepends;
+        return true;
+      });
+  OS << "DEPEND END (" << NumDepends << ")\n";
+  OS << "INCLUDE START\n";
+  unsigned NumIncludes = 0;
+  Reader->foreachInclude([&](const IndexUnitReader::IncludeInfo &Inc) -> bool {
+    OS << Inc.SourcePath << ":" << Inc.SourceLine << " | ";
+    OS << Inc.TargetPath << '\n';
+    ++NumIncludes;
+    return true;
+  });
+  OS << "INCLUDE END (" << NumIncludes << ")\n";
+
+  return false;
+};
+
+//===----------------------------------------------------------------------===//
+// Print Store Units
+//===----------------------------------------------------------------------===//
+
+static bool printStoreUnit(indexstore::IndexStore &Store, StringRef UnitName,
+                           raw_ostream &OS) {
+  std::string Error;
+  indexstore::IndexUnitReader Reader(Store, UnitName, Error);
+  if (!Reader) {
+    errs() << "error loading unit: " << Error << "\n";
+    return true;
+  }
+
+  OS << "provider: " << Reader.getProviderIdentifier() << '-'
+     << Reader.getProviderVersion() << '\n';
+  OS << "is-system: " << Reader.isSystemUnit() << '\n';
+  OS << "is-module: " << Reader.isModuleUnit() << '\n';
+  OS << "module-name: "
+     << (Reader.getModuleName().empty() ? "<none>" : Reader.getModuleName())
+     << '\n';
+  OS << "has-main: " << Reader.hasMainFile() << '\n';
+  OS << "main-path: " << Reader.getMainFilePath() << '\n';
+  OS << "work-dir: " << Reader.getWorkingDirectory() << '\n';
+  OS << "out-file: " << Reader.getOutputFile() << '\n';
+  OS << "target: " << Reader.getTarget() << '\n';
+  OS << "is-debug: " << Reader.isDebugCompilation() << '\n';
+  OS << "DEPEND START\n";
+  unsigned NumDepends = 0;
+  Reader.foreachDependency([&](indexstore::IndexUnitDependency Dep) -> bool {
+    switch (Dep.getKind()) {
+    case indexstore::IndexUnitDependency::DependencyKind::Unit:
+      OS << "Unit | ";
+      break;
+    case indexstore::IndexUnitDependency::DependencyKind::Record:
+      OS << "Record | ";
+      break;
+    case indexstore::IndexUnitDependency::DependencyKind::File:
+      OS << "File | ";
+      break;
+    }
+    OS << (Dep.isSystem() ? "system" : "user");
+    OS << " | ";
+    if (!Dep.getModuleName().empty())
+      OS << Dep.getModuleName() << " | ";
+    OS << Dep.getFilePath();
+    if (!Dep.getName().empty())
+      OS << " | " << Dep.getName();
+    OS << '\n';
+    ++NumDepends;
+    return true;
+  });
+  OS << "DEPEND END (" << NumDepends << ")\n";
+  OS << "INCLUDE START\n";
+  unsigned NumIncludes = 0;
+  Reader.foreachInclude([&](indexstore::IndexUnitInclude Inc) -> bool {
+    OS << Inc.getSourcePath() << ":" << Inc.getSourceLine() << " | ";
+    OS << Inc.getTargetPath() << '\n';
+    ++NumIncludes;
+    return true;
+  });
+  OS << "INCLUDE END (" << NumIncludes << ")\n";
+
+  return false;
+}
+
+static int printStoreUnits(StringRef StorePath, raw_ostream &OS) {
+  std::string Error;
+  indexstore::IndexStore Store(StorePath, Error);
+  if (!Store) {
+    errs() << "error loading store: " << Error << "\n";
+    return 1;
+  }
+
+  bool Success =
+      Store.foreachUnit(/*sorted=*/true, [&](StringRef UnitName) -> bool {
+        OS << UnitName << '\n';
+        OS << "--------\n";
+        bool err = printStoreUnit(Store, UnitName, OS);
+        OS << '\n';
+        return !err;
+      });
+
+  return !Success;
+}
+
+//===----------------------------------------------------------------------===//
 // Helper Utils
 //===----------------------------------------------------------------------===//
 
 static void printSymbolInfo(SymbolInfo SymInfo, raw_ostream &OS) {
   OS << getSymbolKindString(SymInfo.Kind);
   if (SymInfo.SubKind != SymbolSubKind::None)
     OS << '/' << getSymbolSubKindString(SymInfo.SubKind);
   if (SymInfo.Properties) {

 }
 
 static void printSymbolNameAndUSR(const clang::Module *Mod, raw_ostream &OS) {
   assert(Mod);
   OS << Mod->getFullModuleName() << " | ";
   generateFullUSRForModule(Mod, OS);
 }
 
+static void printSymbol(const IndexRecordDecl &Rec, raw_ostream &OS) {
+  printSymbolInfo(Rec.SymInfo, OS);
+  OS << " | ";
+
+  if (Rec.Name.empty())
+    OS << "<no-name>";
+  else
+    OS << Rec.Name;
+  OS << " | ";
+
+  if (Rec.USR.empty())
+    OS << "<no-usr>";
+  else
+    OS << Rec.USR;
+  OS << " | ";
+
+  if (Rec.CodeGenName.empty())
+    OS << "<no-cgname>";
+  else
+    OS << Rec.CodeGenName;
+  OS << " | ";
+
+  printSymbolRoles(Rec.Roles, OS);
+  OS << " - ";
+  printSymbolRoles(Rec.RelatedRoles, OS);
+  OS << '\n';
+}
+
+static void printSymbol(const IndexRecordOccurrence &Rec, raw_ostream &OS) {
+  OS << Rec.Line << ':' << Rec.Column << " | ";
+  printSymbolInfo(Rec.Dcl->SymInfo, OS);
+  OS << " | ";
+
+  if (Rec.Dcl->USR.empty())
+    OS << "<no-usr>";
+  else
+    OS << Rec.Dcl->USR;
+  OS << " | ";
+
+  printSymbolRoles(Rec.Roles, OS);
+  OS << " | ";
+  OS << "rel: " << Rec.Relations.size() << '\n';
+  for (auto &Rel : Rec.Relations) {
+    OS << '\t';
+    printSymbolRoles(Rel.Roles, OS);
+    OS << " | ";
+    if (Rel.Dcl->USR.empty())
+      OS << "<no-usr>";
+    else
+      OS << Rel.Dcl->USR;
+    OS << '\n';
+  }
+}
+
+static void printSymbol(indexstore::IndexRecordSymbol Sym, raw_ostream &OS) {
+  SymbolInfo SymInfo{getSymbolKind(Sym.getKind()),
+                     getSymbolSubKind(Sym.getSubKind()),
+                     getSymbolLanguage(Sym.getLanguage()),
+                     SymbolPropertySet(Sym.getProperties())};
+
+  printSymbolInfo(SymInfo, OS);
+  OS << " | ";
+
+  if (Sym.getName().empty())
+    OS << "<no-name>";
+  else
+    OS << Sym.getName();
+  OS << " | ";
+
+  if (Sym.getUSR().empty())
+    OS << "<no-usr>";
+  else
+    OS << Sym.getUSR();
+  OS << " | ";
+
+  if (Sym.getCodegenName().empty())
+    OS << "<no-cgname>";
+  else
+    OS << Sym.getCodegenName();
+  OS << " | ";
+
+  printSymbolRoles(getSymbolRoles(Sym.getRoles()), OS);
+  OS << " - ";
+  printSymbolRoles(getSymbolRoles(Sym.getRelatedRoles()), OS);
+  OS << '\n';
+}
+
+static void printSymbol(indexstore::IndexRecordOccurrence Occur,
+                        raw_ostream &OS) {
+  OS << Occur.getLineCol().first << ':' << Occur.getLineCol().second << " | ";
+  auto Sym = Occur.getSymbol();
+  SymbolInfo SymInfo{getSymbolKind(Sym.getKind()),
+                     getSymbolSubKind(Sym.getSubKind()),
+                     getSymbolLanguage(Sym.getLanguage()),
+                     SymbolPropertySet(Sym.getProperties())};
+
+  printSymbolInfo(SymInfo, OS);
+  OS << " | ";
+
+  if (Sym.getUSR().empty())
+    OS << "<no-usr>";
+  else
+    OS << Sym.getUSR();
+  OS << " | ";
+
+  unsigned NumRelations = 0;
+  Occur.foreachRelation([&](indexstore::IndexSymbolRelation) {
+    ++NumRelations;
+    return true;
+  });
+
+  printSymbolRoles(getSymbolRoles(Occur.getRoles()), OS);
+  OS << " | ";
+  OS << "rel: " << NumRelations << '\n';
+  Occur.foreachRelation([&](indexstore::IndexSymbolRelation Rel) {
+    OS << '\t';
+    printSymbolRoles(getSymbolRoles(Rel.getRoles()), OS);
+    OS << " | ";
+    auto Sym = Rel.getSymbol();
+    if (Sym.getUSR().empty())
+      OS << "<no-usr>";
+    else
+      OS << Sym.getUSR();
+    OS << '\n';
+    return true;
+  });
+}
+
+static int watchDirectory(StringRef dirPath) {
+  raw_ostream &OS = outs();
+  auto receiver = [&](ArrayRef<DirectoryWatcher::Event> Events,
+                      bool isInitial) {
+    OS << "-- " << Events.size() << " :\n";
+    for (auto evt : Events) {
+      switch (evt.Kind) {
+      case DirectoryWatcher::Event::EventKind::Modified:
+        OS << "modified: ";
+        break;
+      case DirectoryWatcher::Event::EventKind::Removed:
+        OS << "removed: ";
+        break;
+      case DirectoryWatcher::Event::EventKind::WatchedDirRemoved:
+        OS << "dir deleted: ";
+        break;
+      case DirectoryWatcher::Event::EventKind::WatcherGotInvalidated:
+        OS << "watcher got invalidated: ";
+        break;
+      }
+      OS << evt.Filename << '\n';
+    }
+  };
+  auto watcher = DirectoryWatcher::create(dirPath, receiver,
+                                          /*waitInitialSync=*/true);
+  if (!watcher) {
+    errs() << "failed creating directory watcher" << '\n';
+    return 1;
+  }
+
+  while (1) {
+    std::this_thread::yield();
+  }
+}
+
 //===----------------------------------------------------------------------===//
 // Command line processing.
 //===----------------------------------------------------------------------===//
 
+bool deconstructPathAndRange(StringRef input, std::string &filepath,
+                             Optional<unsigned> &lineStart,
+                             unsigned &lineCount) {
+  StringRef path, range;
+  std::tie(path, range) = input.split(':');
+  StringRef start, end;
+  std::tie(start, end) = range.split(':');
+  filepath = path;
+  lineCount = 0;
+  if (start.empty())
+    return false;
+  unsigned num;
+  if (start.getAsInteger(10, num)) {
+    errs() << "couldn't convert to integer: " << start << '\n';
+    return true;
+  }
+  lineStart = num;
+  if (end.empty())
+    return false;
+  if (end.getAsInteger(10, num)) {
+    errs() << "couldn't convert to integer: " << end << '\n';
+    return true;
+  }
+  lineCount = num - lineStart.getValue();
+  return false;
+}
+
 int indextest_core_main(int argc, const char **argv) {
   sys::PrintStackTraceOnErrorSignal(argv[0]);
   PrettyStackTraceProgram X(argc, argv);
   void *MainAddr = (void*) (intptr_t) indextest_core_main;
   std::string Executable = llvm::sys::fs::getMainExecutable(argv[0], MainAddr);
 
   assert(argv[1] == StringRef("core"));
   ++argv;
if (CompArgs.empty()) {
       errs() << "error: missing compiler args; pass '-- <compiler arguments>'\n";
       return 1;
     }
     return printSourceSymbols(Executable.c_str(), CompArgs,
                               options::DumpModuleImports,
                               options::IncludeLocals);
   }
 
+  if (options::Action == ActionType::PrintRecord) {
+    if (!options::FilePathAndRange.empty()) {
+      std::string filepath;
+      Optional<unsigned> lineStart;
+      unsigned lineCount;
+      if (deconstructPathAndRange(options::FilePathAndRange, filepath,
+                                  lineStart, lineCount))
+        return 1;
+
+      if (options::InputFiles.empty()) {
+        errs() << "error: missing index store path\n";
+        return 1;
+      }
+      return printStoreFileRecord(options::InputFiles[0], filepath, lineStart,
+                                  lineCount, outs());
+    }
+
+    if (options::InputFiles.empty()) {
+      errs() << "error: missing input file or directory\n";
+      return 1;
+    }
+
+    if (sys::fs::is_directory(options::InputFiles[0]))
+      return printStoreRecords(options::InputFiles[0], outs());
+    else
+      return printRecord(options::InputFiles[0], outs());
+  }
+
+  if (options::Action == ActionType::PrintUnit) {
+    if (options::InputFiles.empty()) {
+      errs() << "error: missing input file or directory\n";
+      return 1;
+    }
+
+    if (sys::fs::is_directory(options::InputFiles[0]))
+      return printStoreUnits(options::InputFiles[0], outs());
+    else
+      return printUnit(options::InputFiles[0], outs());
+  }
+
+  if (options::Action == ActionType::PrintStoreFormatVersion) {
+    outs() << indexstore::IndexStore::formatVersion() << '\n';
+  }
+
+  if (options::Action == ActionType::AggregateAsJSON) {
+    if (options::InputFiles.empty()) {
+      errs() << "error: missing input data store directory\n";
+      return 1;
+    }
+    StringRef storePath = options::InputFiles[0];
+    if (options::OutputFile.empty())
+      return aggregateDataAsJSON(storePath, outs());
+    std::error_code EC;
+    raw_fd_ostream OS(options::OutputFile, EC, llvm::sys::fs::F_None);
+    if (EC) {
+      errs() << "failed to open output file: " << EC.message() << '\n';
+      return 1;
+    }
+    return aggregateDataAsJSON(storePath, OS);
+  }
+
+  if (options::Action == ActionType::WatchDir) {
+    if (options::InputFiles.empty()) {
+      errs() << "error: missing directory path\n";
+      return 1;
+    }
+    return watchDirectory(options::InputFiles[0]);
+  }
+
   return 0;
 }

clang/tools/libclang/CXIndexDataConsumer.h

private:
 
   CXCursor getCursor(const Decl *D) {
     return cxcursor::MakeCXCursor(D, CXTU);
   }
 
   CXCursor getRefCursor(const NamedDecl *D, SourceLocation Loc);
 
   static bool shouldIgnoreIfImplicit(const Decl *D);
+
+  void initialize(ASTContext &Ctx) override {}
+
+  bool handleMacroOccurence(const IdentifierInfo *Name, const MacroInfo *MI,
+                            index::SymbolRoleSet Roles,
+                            SourceLocation Loc) override {
+    return true;
+  }
 };
 
 inline ScratchAlloc::ScratchAlloc(CXIndexDataConsumer &idxCtx) : IdxCtx(idxCtx) {
   ++IdxCtx.StrAdapterCount;
 }
 inline ScratchAlloc::ScratchAlloc(const ScratchAlloc &SA) : IdxCtx(SA.IdxCtx) {
   ++IdxCtx.StrAdapterCount;
 }

clang/tools/libclang/Indexing.cpp

 #include "CXString.h"
 #include "CXTranslationUnit.h"
 #include "clang/AST/ASTConsumer.h"
 #include "clang/Frontend/ASTUnit.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/CompilerInvocation.h"
 #include "clang/Frontend/FrontendAction.h"
 #include "clang/Frontend/Utils.h"
-#include "clang/Index/IndexingAction.h"
+#include "clang/Index/GenerateIndexAction.h"
+#include "clang/Index/IndexAction.h"
 #include "clang/Lex/HeaderSearch.h"
 #include "clang/Lex/PPCallbacks.h"
 #include "clang/Lex/PPConditionalDirectiveRecord.h"
 #include "clang/Lex/Preprocessor.h"
 #include "clang/Lex/PreprocessorOptions.h"
 #include "llvm/Support/CrashRecoveryContext.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Mutex.h"
static CXErrorCode clang_indexSourceFile_Impl(
   bool SkipBodies = (index_options & CXIndexOpt_SkipParsedBodiesInSession) &&
       CInvok->getLangOpts()->CPlusPlus;
   if (SkipBodies)
     CInvok->getFrontendOpts().SkipFunctionBodies = true;
 
   auto DataConsumer =
     std::make_shared<CXIndexDataConsumer>(client_data, CB, index_options,
                                           CXTU->getTU());
-  auto InterAction = llvm::make_unique<IndexingFrontendAction>(DataConsumer,
-                         SkipBodies ? IdxSession->SkipBodyData.get() : nullptr);
-  std::unique_ptr<FrontendAction> IndexAction;
-  IndexAction = createIndexingAction(DataConsumer,
-                                getIndexingOptionsFromCXOptions(index_options),
-                                     std::move(InterAction));
+
+  std::unique_ptr<FrontendAction> IndexAction =
+      llvm::make_unique<IndexActionWrapper>(
+          llvm::make_unique<GenerateIndexAction>(
+              DataConsumer, getIndexingOptionsFromCXOptions(index_options)),
+          llvm::make_unique<IndexingFrontendAction>(
+              DataConsumer,
+              SkipBodies ? IdxSession->SkipBodyData.get() : nullptr));
 
   // Recover resources if we crash before exiting this method.
   llvm::CrashRecoveryContextCleanupRegistrar<FrontendAction>
     IndexActionCleanup(IndexAction.get());
 
   bool Persistent = requestedToGetTU;
   bool OnlyLocalDecls = false;
   bool PrecompilePreamble = false;

clang/unittests/Index/IndexTests.cpp

 //===--- IndexTests.cpp - Test indexing actions -----------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "clang/AST/ASTConsumer.h"
 #include "clang/AST/ASTContext.h"
 #include "clang/AST/Decl.h"
 #include "clang/Basic/SourceLocation.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/FrontendAction.h"
+#include "clang/Index/GenerateIndexAction.h"
 #include "clang/Index/IndexDataConsumer.h"
 #include "clang/Index/IndexSymbol.h"
-#include "clang/Index/IndexingAction.h"
 #include "clang/Lex/Preprocessor.h"
 #include "clang/Tooling/Tooling.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/VirtualFileSystem.h"
 #include "gmock/gmock.h"
 #include "gtest/gtest.h"
 #include <memory>
 
llvm::raw_ostream &operator<<(llvm::raw_ostream &OS, const TestSymbol &S) {
   return OS << S.QName << '[' << S.WrittenPos << ']' << '@' << S.DeclPos << '('
             << static_cast<unsigned>(S.SymInfo.Kind) << ')';
 }
 
 class Indexer : public IndexDataConsumer {
 public:
   void initialize(ASTContext &Ctx) override {
     AST = &Ctx;
-    IndexDataConsumer::initialize(Ctx);
   }
 
   bool handleDeclOccurence(const Decl *D, SymbolRoleSet Roles,
                            ArrayRef<SymbolRelation>, SourceLocation Loc,
                            ASTNodeInfo) override {
     const auto *ND = llvm::dyn_cast<NamedDecl>(D);
     if (!ND)
       return true;
bool handleMacroOccurence(const IdentifierInfo *Name, const MacroInfo *MI,
     S.WrittenPos = Position::fromSourceLocation(Loc, AST->getSourceManager());
     S.DeclPos = Position::fromSourceLocation(MI->getDefinitionLoc(),
                                              AST->getSourceManager());
     S.Roles = Roles;
     Symbols.push_back(std::move(S));
     return true;
   }
 
+  virtual void setPreprocessor(std::shared_ptr<Preprocessor> PP) override {}
+  virtual bool handleModuleOccurence(const ImportDecl *ImportD,
+                                     const Module *Mod, SymbolRoleSet Roles,
+                                     SourceLocation Loc) override {
+    return true;
+  }
+  virtual void finish() override {}
+
   std::vector<TestSymbol> Symbols;
   const ASTContext *AST = nullptr;
 };
 
 class IndexAction : public ASTFrontendAction {
 public:
   IndexAction(std::shared_ptr<Indexer> Index,
               IndexingOptions Opts = IndexingOptions())

llvm/include/llvm/ADT/ArrayRef.h

namespace llvm {
   }
 
   /// Construct an ArrayRef from a std::vector.
   template<typename T>
   ArrayRef<T> makeArrayRef(const std::vector<T> &Vec) {
     return Vec;
   }
 
+  /// Construct an ArrayRef from a std::array.
+  template <typename T, std::size_t N>
+  ArrayRef<T> makeArrayRef(const std::array<T, N> &Arr) {
+    return Arr;
+  }
+
   /// Construct an ArrayRef from an ArrayRef (no-op) (const)
   template <typename T> ArrayRef<T> makeArrayRef(const ArrayRef<T> &Vec) {
     return Vec;
   }
 
   /// Construct an ArrayRef from an ArrayRef (no-op)
   template <typename T> ArrayRef<T> &makeArrayRef(ArrayRef<T> &Vec) {
     return Vec;

llvm/include/llvm/Bitstream/BitCodes.h

 
 /// BitCodeAbbrev - This class represents an abbreviation record.  An
 /// abbreviation allows a complex record that has redundancy to be stored in a
 /// specialized format instead of the fully-general, fully-vbr, format.
 class BitCodeAbbrev {
   SmallVector<BitCodeAbbrevOp, 32> OperandList;
 
 public:
+  BitCodeAbbrev() = default;
+
+  explicit BitCodeAbbrev(std::initializer_list<BitCodeAbbrevOp> OperandList)
+      : OperandList(OperandList) {}
+
   unsigned getNumOperandInfos() const {
     return static_cast<unsigned>(OperandList.size());
   }
   const BitCodeAbbrevOp &getOperandInfo(unsigned N) const {
     return OperandList[N];
   }
 
   void Add(const BitCodeAbbrevOp &OpInfo) {
     OperandList.push_back(OpInfo);
   }
 };
 } // End llvm namespace
 
 #endif