[index-while-building] PathIndexer

[jkorous]

Summary

Part of i-w-b upstreaming https://reviews.llvm.org/D64384

This is actually a bit polished PathStorage.

[gribozavr]

I tried my best to provide meaningful comments, however, I can't find users of PathIndexer in https://reviews.llvm.org/D64384, so I don't think I understand this abstraction at all. I would appreciate some big-picture description.

clang/include/clang/IndexSerialization/PathIndexer.h

20 ↗

(On Diff #217551)

"namespace clang"?

36 ↗

(On Diff #217551)

s/Represents// (throughout the patch)

"Represents" is generally a filler word -- of course types represent some concept, they rarely *are* that concept itself, except for concepts that we invent ourselves.

39 ↗

(On Diff #217551)

Would be best to not refer to one concept with two terms -- root and prefix.

40 ↗

(On Diff #217551)

"Path relative to the root specified by PrefixKind."

41 ↗

(On Diff #217551)

Const members in structs are not great because they make structs not act like values -- the assignment operator becomes deleted. (Throughout the patch.)

56 ↗

(On Diff #217551)

Builds => stores, or maintains. It does not walk the filesystem itself.

66 ↗

(On Diff #217551)

In this context, "\param".

80 ↗

(On Diff #217551)

Add an assert for uniqueness?

Rename to storeNewPath?

clang/lib/IndexSerialization/PathIndexer.cpp

15 ↗

(On Diff #217551)

It is OK to shadow member variables with parameters. Shadowing does the right thing in the member initializer list.

[gribozavr]

Oh, it is called PathStorage there. Looking.

[gribozavr]

clang/include/clang/IndexSerialization/PathIndexer.h

34 ↗

(On Diff #217551)

I believe it would be helpful to build a separate class that just does string interning, and let BitPathComponent be a reference to a string in that string pool. That would simplify code in PathIndexer, and separate the concerns of path handling and string storage. PathIndexer would expose a reference to the interned storage through a getter.

54 ↗

(On Diff #217551)

WDYT about moving DirBitPath and FileBitPath to be nested types in PathIndexer? You should be able to also remove "Bit" from their names then.

57 ↗

(On Diff #217551)

SerializablePaths? SerializablePathCollection? OnDiskPathCollection? (like clang/lib/Serialization/MultiOnDiskHashTable.h)

I'm not a fan of using the word "Indexer" for a passive entity.

76 ↗

(On Diff #217551)

WDYT about using optional instead of a special marker value?

clang/lib/IndexSerialization/PathIndexer.cpp

22 ↗

(On Diff #217551)

s/FileBitPaths.size()/FileToIndex.size()/ would probably be more intuitive.

[jkorous]

clang/include/clang/IndexSerialization/PathIndexer.h

20 ↗

(On Diff #217551)

clang::index?

41 ↗

(On Diff #217551)

I don't have a particularly strong opinion but in general I like immutability as an invariant.

76 ↗

(On Diff #217551)

Good point! Originally the -1 convention for both return values and indices being passed as parameters was used throughout the i-w-b. It avoided explicit error handling which allowed code to look simpler but was more difficult to reason about. This seems to be the last surviving specimen.

[jkorous]

Addressed the comments and tried to simplify things by splitting into multiple smaller classes.

[jkorous]

@arphaman can you please take a look at this patch?

[arphaman]

Sure, will do.

[jkorous]

Rebased on recent master + https://reviews.llvm.org/D66854

[jkorous]

Restore the patch. Previous update wasn't meant for this review - I confused the review IDs.

[arphaman]

clang/include/clang/IndexSerialization/SerializablePathCollection.h

32 ↗

(On Diff #279310)

I feel like StringOffSz isn't very informative. Can this be renamed to something like StringPool::OffsetSize, or maybe StringPool::Index.

clang/lib/IndexSerialization/SerializablePathCollection.cpp

48 ↗

(On Diff #279310)

Do you think you can use llvm::sys::path::starts_with instead? why is it important to return false when dir is equivalent to rest. Would we be able to store paths a path like /path/to/xyz_ab, when the CWD/Sysroot is /path/to/xyz when we don't have this check?

55 ↗

(On Diff #279310)

I looked through the uses of tryStoreFilePath in the full patch, and it doesn't seem like the use of try in the name is justified, since it could imply that some form of error could be produced as a result of a call. Can this be named more precisely, perhaps something like getStoreIndexForFile()? Also, uses seem to imply that it's always converted to uint64_t, can we avoid the conversion and return uint64_t here instead?

68 ↗

(On Diff #279310)

Same name critique, could this be named something like storeDirPath, without the try?

83 ↗

(On Diff #279310)

Please use is_separator instead of comparing to /

[jkorous]

clang/lib/IndexSerialization/SerializablePathCollection.cpp

48 ↗

(On Diff #279310)

I'll try to get rid of this. It's only used in tryStoreDirPath(). The logic is probably sound but maybe it could be implemented in a more readable fashion.

55 ↗

(On Diff #279310)

The semantics of the name are the same as STL containers - say std::map::try_emplace. I'm open to renaming but would like to have a name that conveys the fact that this is actually the method used for gathering the data for serialization.

68 ↗

(On Diff #279310)

Maybe storeUniqueDirPath or storeNewDirPath? Because we aren't always storing the input.

83 ↗

(On Diff #279310)

That's a good point!

[jkorous]

Addressed comments.

[jkorous]

Friendly ping

clang/include/clang/Basic/DiagnosticFrontendKinds.td

 def remark_module_build_done : Remark<"finished building module '%0'">,
   InGroup<ModuleBuild>;
 def err_modules_embed_file_not_found :
   Error<"file '%0' specified by '-fmodules-embed-file=' not found">,
   DefaultFatal;
 def err_module_header_file_not_found :
   Error<"module header file '%0' not found">, DefaultFatal;
 
+def remark_index_producing_module_file_data : Remark<"producing index data for "
+  "module file '%0'">,
+  InGroup<IndexStore>;
+
 def err_test_module_file_extension_version : Error<
   "test module file extension '%0' has different version (%1.%2) than expected "
   "(%3.%4)">;
 
 def err_missing_vfs_overlay_file : Error<
   "virtual filesystem overlay file '%0' not found">, DefaultFatal;
 def err_invalid_vfs_overlay : Error<
   "invalid virtual filesystem overlay file '%0'">, DefaultFatal;

clang/include/clang/Basic/DiagnosticGroups.td

 def MismatchedParameterTypes : DiagGroup<"mismatched-parameter-types">;
 def MismatchedReturnTypes : DiagGroup<"mismatched-return-types">;
 def MismatchedTags : DiagGroup<"mismatched-tags">;
 def MissingFieldInitializers : DiagGroup<"missing-field-initializers">;
 def ModuleBuild : DiagGroup<"module-build">;
 def ModuleImport : DiagGroup<"module-import">;
 def ModuleConflict : DiagGroup<"module-conflict">;
 def ModuleFileExtension : DiagGroup<"module-file-extension">;
+def IndexStore : DiagGroup<"index-store">;
 def NewlineEOF : DiagGroup<"newline-eof">;
 def Nullability : DiagGroup<"nullability">;
 def NullabilityDeclSpec : DiagGroup<"nullability-declspec">;
 def NullabilityInferredOnNestedType : DiagGroup<"nullability-inferred-on-nested-type">;
 def NullableToNonNullConversion : DiagGroup<"nullable-to-nonnull-conversion">;
 def NullabilityCompletenessOnArrays : DiagGroup<"nullability-completeness-on-arrays">;
 def NullabilityCompleteness : DiagGroup<"nullability-completeness",
                                         [NullabilityCompletenessOnArrays]>;

clang/include/clang/Driver/Job.h

 
 class Action;
 class InputInfo;
 class Tool;
 
 struct CrashReportInfo {
   StringRef Filename;
   StringRef VFSPath;
+  StringRef IndexStorePath;
 
-  CrashReportInfo(StringRef Filename, StringRef VFSPath)
-      : Filename(Filename), VFSPath(VFSPath) {}
+  CrashReportInfo(StringRef Filename, StringRef VFSPath,
+                  StringRef IndexStorePath)
+      : Filename(Filename), VFSPath(VFSPath), IndexStorePath(IndexStorePath) {}
 };
 
 // Encodes the kind of response file supported for a command invocation.
 // Response files are necessary if the command line gets too large, requiring
 // the arguments to be transferred to a file.
 struct ResponseFileSupport {
   enum ResponseFileKind {
     // Provides full support for response files, which means we can transfer

clang/include/clang/Driver/Options.td

 def objcmt_migrate_designated_init : Flag<["-"], "objcmt-migrate-designated-init">, Flags<[CC1Option]>,
   HelpText<"Enable migration to infer NS_DESIGNATED_INITIALIZER for initializer methods">;
 def objcmt_whitelist_dir_path: Joined<["-"], "objcmt-whitelist-dir-path=">, Flags<[CC1Option]>,
   HelpText<"Only modify files with a filename contained in the provided directory path">;
 // The misspelt "white-list" [sic] alias is due for removal.
 def : Joined<["-"], "objcmt-white-list-dir-path=">, Flags<[CC1Option]>,
     Alias<objcmt_whitelist_dir_path>;
 
+def index_store_path : Separate<["-"], "index-store-path">, Flags<[CC1Option]>,
+  HelpText<"Enable indexing with the specified data store path">;
+def index_ignore_system_symbols : Flag<["-"], "index-ignore-system-symbols">, Flags<[CC1Option]>,
+  HelpText<"Ignore symbols from system headers">;
+def index_record_mangled_name : Flag<["-"], "index-record-mangled-name">, Flags<[CC1Option]>,
+  HelpText<"Record the mangled name for symbols">;
+
 // Make sure all other -ccc- options are rejected.
 def ccc_ : Joined<["-"], "ccc-">, Group<internal_Group>, Flags<[Unsupported]>;
 
 // Standard Options
 
 def _HASH_HASH_HASH : Flag<["-"], "###">, Flags<[DriverOption, CoreOption]>,
     HelpText<"Print (but do not run) the commands to run for this compilation">;
 def _DASH_DASH : Option<["--"], "", KIND_REMAINING_ARGS>,

clang/include/clang/Frontend/FrontendAction.h

public:
   bool usesPreprocessorOnly() const override;
   TranslationUnitKind getTranslationUnitKind() override;
   bool hasPCHSupport() const override;
   bool hasASTFileSupport() const override;
   bool hasIRSupport() const override;
   bool hasCodeCompletionSupport() const override;
 };
 
+/// Class for adding custom ASTConsumer to a FrontendAction.
+/// Wraps the base action and forwards all method call except CreateASTConsumer.
+class ASTConsumerInjector : public WrapperFrontendAction {
+  std::function<std::unique_ptr<ASTConsumer>(CompilerInstance &CI,
+                                             StringRef InFile)>
+      ASTConsumerFactory;
+
+public:
+  /// \p BaseActionInput is the base action to which all method calls are
+  /// forwarded. \p ASTConsumerFactory is factory to produce the custom
+  /// ASTConsumer in CreateASTConsumer method.
+  ASTConsumerInjector(std::unique_ptr<FrontendAction> BaseActionInput,
+                      std::function<std::unique_ptr<ASTConsumer>(
+                          CompilerInstance &CI, StringRef InFile)>
+                          ASTConsumerFactory);
+
+  /// Returns MultiplexConsumer consumer with first element produced by
+  /// ASTConsumerFactory, followed by the ASTConsumer from the base action.
+  std::unique_ptr<ASTConsumer> CreateASTConsumer(CompilerInstance &CI,
+                                                 StringRef InFile) override;
+};
+
 }  // end namespace clang
 
 #endif

clang/include/clang/Frontend/FrontendOptions.h

ObjCMT_MigrateAll = (ObjCMT_Literals | ObjCMT_Subscripting |
                          ObjCMT_MigrateDecls | ObjCMT_PropertyDotSyntax)
   };
   unsigned ObjCMTAction = ObjCMT_None;
   std::string ObjCMTWhiteListPath;
 
   std::string MTMigrateDir;
   std::string ARCMTMigrateReportOut;
 
+  std::string IndexStorePath;
+  unsigned IndexIgnoreSystemSymbols : 1;
+  unsigned IndexRecordMangledName : 1;
+
   /// The input files and their types.
   SmallVector<FrontendInputFile, 0> Inputs;
 
   /// When the input is a module map, the original module map file from which
   /// that map was inferred, if any (for umbrella modules).
   std::string OriginalModuleMap;
 
   /// The output file, if any.
FrontendOptions()
       : DisableFree(false), RelocatablePCH(false), ShowHelp(false),
         ShowStats(false), ShowTimers(false), TimeTrace(false),
         ShowVersion(false), FixWhatYouCan(false), FixOnlyWarnings(false),
         FixAndRecompile(false), FixToTemporaries(false),
         ARCMTMigrateEmitARCErrors(false), SkipFunctionBodies(false),
         UseGlobalModuleIndex(true), GenerateGlobalModuleIndex(true),
         ASTDumpDecls(false), ASTDumpLookups(false),
         BuildingImplicitModule(false), ModulesEmbedAllFiles(false),
-        IncludeTimestamps(true), UseTemporary(true), TimeTraceGranularity(500) {}
+        IncludeTimestamps(true), UseTemporary(true),
+        IndexIgnoreSystemSymbols(false), IndexRecordMangledName(false),
+        TimeTraceGranularity(500) {}
 
   /// getInputKindForExtension - Return the appropriate input kind for a file
   /// extension. For example, "c" would return Language::C.
   ///
   /// \return The input kind for the extension, or Language::Unknown if the
   /// extension is not recognized.
   static InputKind getInputKindForExtension(StringRef Extension);
 };
 
 } // namespace clang
 
 #endif // LLVM_CLANG_FRONTEND_FRONTENDOPTIONS_H

clang/include/clang/Index/DeclOccurrence.h

-//===- DeclOccurrence.h - An occurrence of a decl within a file -*- C++ -*-===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-
-#ifndef LLVM_CLANG_INDEX_DECLOCCURRENCE_H
-#define LLVM_CLANG_INDEX_DECLOCCURRENCE_H
-
-#include "clang/Basic/LLVM.h"
-#include "clang/Index/IndexSymbol.h"
-#include "llvm/ADT/ArrayRef.h"
-#include "llvm/ADT/SmallVector.h"
-
-namespace clang {
-class Decl;
-
-namespace index {
-
-struct DeclOccurrence {
-  SymbolRoleSet Roles;
-  unsigned Offset;
-  const Decl *Dcl;
-  SmallVector<SymbolRelation, 3> Relations;
-
-  DeclOccurrence(SymbolRoleSet R, unsigned Offset, const Decl *D,
-                 ArrayRef<SymbolRelation> Relations)
-      : Roles(R), Offset(Offset), Dcl(D),
-        Relations(Relations.begin(), Relations.end()) {}
-
-  friend bool operator<(const DeclOccurrence &LHS, const DeclOccurrence &RHS) {
-    return LHS.Offset < RHS.Offset;
-  }
-};
-
-} // namespace index
-} // namespace clang
-
-#endif // LLVM_CLANG_INDEX_DECLOCCURRENCE_H

clang/include/clang/Index/IndexPPCallbacks.h

+//===--- IndexPPCallbacks.h - Gathering index record data -----------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEX_INDEXPPCALLBACKS_H
+#define LLVM_CLANG_INDEX_INDEXPPCALLBACKS_H
+
+#include "clang/Index/IndexingContext.h"
+#include "clang/Lex/PPCallbacks.h"
+
+#include <memory>
+
+namespace llvm {
+class BitstreamWriter;
+}
+
+namespace clang {
+class FileEntry;
+class FileManager;
+
+namespace index {
+
+class IndexPPCallbacks : public PPCallbacks {
+  std::shared_ptr<IndexingContext> IndexCtx;
+
+public:
+  IndexPPCallbacks(std::shared_ptr<IndexingContext> IndexCtx)
+      : IndexCtx(IndexCtx) {}
+
+  void MacroExpands(const Token &MacroNameTok, const MacroDefinition &MD,
+                    SourceRange Range, const MacroArgs *Args) override {
+    IndexCtx->handleMacroReference(*MacroNameTok.getIdentifierInfo(),
+                                   Range.getBegin(), *MD.getMacroInfo());
+  }
+
+  void MacroDefined(const Token &MacroNameTok,
+                    const MacroDirective *MD) override {
+    IndexCtx->handleMacroDefined(*MacroNameTok.getIdentifierInfo(),
+                                 MacroNameTok.getLocation(),
+                                 *MD->getMacroInfo());
+  }
+
+  void MacroUndefined(const Token &MacroNameTok, const MacroDefinition &MD,
+                      const MacroDirective *Undef) override {
+    if (!MD.getMacroInfo()) // Ignore noop #undef.
+      return;
+    IndexCtx->handleMacroUndefined(*MacroNameTok.getIdentifierInfo(),
+                                   MacroNameTok.getLocation(),
+                                   *MD.getMacroInfo());
+  }
+};
+
+} // end namespace index
+} // end namespace clang
+
+#endif

clang/include/clang/Index/IndexingOptions.h

struct IndexingOptions {
 
   // If set, skip indexing inside some declarations for performance.
   // This prevents traversal, so skipping a struct means its declaration an
   // members won't be indexed, but references elsewhere to that struct will be.
   // Currently this is only checked for top-level declarations.
   std::function<bool(const Decl *)> ShouldTraverseDecl;
 };
 
+IndexingOptions parseIndexGenerationOptions(const FrontendOptions &FEOpts);
+
 } // namespace index
 } // namespace clang
 
 #endif // LLVM_CLANG_INDEX_INDEXINGOPTIONS_H

clang/include/clang/IndexDataStore/IndexDataStore.h

+//===--- IndexDataStore.h - Index data store info -------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEX_INDEXDATASTORE_H
+#define LLVM_CLANG_INDEX_INDEXDATASTORE_H
+
+#include "clang/Basic/LLVM.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/Chrono.h"
+#include <functional>
+#include <memory>
+#include <string>
+
+namespace clang {
+namespace index {
+
+class IndexDataStore {
+public:
+  ~IndexDataStore();
+
+  static std::unique_ptr<IndexDataStore> create(StringRef IndexStorePath,
+                                                std::string &Error);
+
+  StringRef getFilePath() const;
+  bool foreachUnitName(bool sorted,
+                       llvm::function_ref<bool(StringRef unitName)> receiver);
+
+  static unsigned getFormatVersion();
+
+  enum class UnitEventKind {
+    Removed,
+    Modified,
+    /// The directory got deleted. No more events will follow.
+    DirectoryDeleted,
+    Failure
+  };
+  struct UnitEvent {
+    UnitEventKind Kind;
+    StringRef UnitName;
+  };
+  struct UnitEventNotification {
+    bool IsInitial;
+    ArrayRef<UnitEvent> Events;
+  };
+  typedef std::function<void(UnitEventNotification)> UnitEventHandler;
+
+  void setUnitEventHandler(UnitEventHandler Handler);
+  /// \returns true if an error occurred.
+  bool startEventListening(bool waitInitialSync, std::string &Error);
+  void stopEventListening();
+
+  void discardUnit(StringRef UnitName);
+  void discardRecord(StringRef RecordName);
+
+  void purgeStaleData();
+
+private:
+  IndexDataStore(void *Impl) : Impl(Impl) {}
+
+  void *Impl; // An IndexDataStoreImpl.
+};
+
+} // namespace index
+} // namespace clang
+
+#endif

clang/include/clang/IndexSerialization/DeclOccurrenceCollector.h

+//===--- DeclOccurrenceCollector.h - Collecting data for index records --*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEX_DECLOCCURENCECOLLECTOR_H
+#define LLVM_CLANG_INDEX_DECLOCCURENCECOLLECTOR_H
+
+#include "clang/Index/IndexDataConsumer.h"
+#include "clang/Index/IndexingContext.h"
+#include "clang/IndexSerialization/IndexRecordWriter.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/StringRef.h"
+#include <memory>
+
+namespace clang {
+namespace index {
+
+/// Collects occurences of declarations.
+class DeclOccurrenceCollector : public IndexDataConsumer {
+  SourceManager &SM;
+  // This exists because there's cyclic dependency between IndexingContext and
+  // DeclOccurrenceCollector.
+  IndexingContext *IndexCtx = nullptr;
+
+  llvm::DenseMap<FileID, IndexRecordWriter> RecordByFile;
+
+public:
+  explicit DeclOccurrenceCollector(SourceManager &SM) : SM(SM) {}
+  void setIndexingContext(IndexingContext &IC) { IndexCtx = &IC; }
+  bool handleDeclOccurrence(const Decl *D, SymbolRoleSet Roles,
+                            ArrayRef<SymbolRelation> Relations,
+                            SourceLocation Loc, ASTNodeInfo ASTNode) override;
+  const llvm::DenseMap<FileID, IndexRecordWriter> &getRecordByFile() const {
+    return RecordByFile;
+  }
+};
+
+} // namespace index
+} // namespace clang
+
+#endif

clang/include/clang/IndexSerialization/EmitIndex.h

+//===--- EmitIndex.h - Index emitting AST Consumer & functions --*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEX_EMITINDEXASTCONSUMER_H
+#define LLVM_CLANG_INDEX_EMITINDEXASTCONSUMER_H
+
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Frontend/FrontendAction.h"
+#include "clang/Index/IndexDataConsumer.h"
+#include "clang/Index/IndexingAction.h"
+#include "clang/Index/IndexingContext.h"
+#include "clang/IndexSerialization/DeclOccurrenceCollector.h"
+#include "clang/IndexSerialization/IndexRecordWriter.h"
+#include "clang/IndexSerialization/IndexUnitWriter.h"
+#include "llvm/ADT/StringRef.h"
+#include <memory>
+
+namespace clang {
+namespace index {
+
+class SourceFilesDependencyCollector : public DependencyCollector {
+  const CompilerInstance &CI;
+  IndexingContext &IndexCtx;
+  EmitIndexOptions RecordOpts;
+
+public:
+  IndexUnitWriter UnitWriter;
+
+  SourceFilesDependencyCollector(const CompilerInstance &CI,
+                                 IndexingContext &indexCtx,
+                                 EmitIndexOptions recordOpts);
+
+  virtual void attachToPreprocessor(Preprocessor &PP) override;
+
+private:
+  bool sawDependency(StringRef Filename, bool FromModule, bool IsSystem,
+                     bool IsModuleFile, bool IsMissing) override;
+  bool isInSysroot(StringRef Filename);
+};
+
+// FIXME:
+// This isn't nice - just a glue.
+// The reasons for this to exist are:
+// - We need access to CompilerInstance to create collectors which we get only
+// in CreateASTConsumer method, not in FrontendAction implementation's
+// constructor.
+// - createIndexRecordingASTConsumer returns unique_ptr but we need to preserve
+// some data until EndSourceFileAction.
+struct ActionContext {
+  const CompilerInstance &CI;
+  DeclOccurrenceCollector DeclsDataCollector;
+  IndexingContext Ctx;
+  SourceFilesDependencyCollector DepCollector;
+  EmitIndexOptions RecordOpts;
+  IndexingOptions IndexOpts;
+
+  ActionContext(const IndexingOptions &IndexOpts, const CompilerInstance &CI,
+                const EmitIndexOptions &RecordOpts);
+
+  void emitCollectedData();
+};
+
+class IndexRecordingASTConsumer final : public ASTConsumer {
+  ActionContext Context;
+
+public:
+  IndexRecordingASTConsumer(const IndexingOptions &IndexOpts,
+                            CompilerInstance &CI,
+                            const EmitIndexOptions &RecordOpts);
+
+protected:
+  void Initialize(ASTContext &ASTContextInst) override;
+
+  bool HandleTopLevelDecl(DeclGroupRef DG) override;
+
+  void HandleInterestingDecl(DeclGroupRef DG) override;
+
+  void HandleTopLevelDeclInObjCContainer(DeclGroupRef DG) override;
+
+  void HandleTranslationUnit(ASTContext &Ctx) override;
+};
+
+/// Checks if the unit file exists for \p Mod and emits index data in case it
+/// doesn't. Handles both index units and index records. \returns true if the
+/// index data were generated, false otherwise.
+bool emitIndexDataForModule(const Module *Mod, const CompilerInstance &CI);
+
+/// Emits index data for \p Mod.
+/// Handles both index units and index records.
+void emitIndexDataForModuleFile(serialization::ModuleFile &Mod,
+                                const CompilerInstance &CI,
+                                const IndexingOptions &IndexOpts,
+                                const EmitIndexOptions &RecordOpts);
+
+} // namespace index
+} // namespace clang
+
+#endif

clang/include/clang/IndexSerialization/EmitIndexOptions.h

+//===--- EmitIndexOptions.h - Options for indexing --------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEX_EMITINDEXOPTIONS_H
+#define LLVM_CLANG_INDEX_EMITINDEXOPTIONS_H
+
+#include "clang/Frontend/FrontendOptions.h"
+#include <memory>
+#include <string>
+
+namespace clang {
+namespace index {
+
+struct EmitIndexOptions {
+  enum class IncludesRecordingKind {
+    None,
+    UserOnly, // only record includes inside non-system files.
+    All,
+  };
+
+  const std::string RootDir;
+  // Caching the full paths.
+  const std::string IndexDir;
+  const std::string UnitDir;
+  const std::string RecordDir;
+
+  const bool RecordSymbolCodeGenName;
+  const bool RecordSystemDependencies;
+  const IncludesRecordingKind RecordIncludes;
+
+  EmitIndexOptions(const StringRef RootDir,
+                   const bool RecordSymbolCodeGenName = false,
+                   const bool RecordSystemDependencies = true,
+                   const IncludesRecordingKind RecordIncludes =
+                       IncludesRecordingKind::UserOnly);
+};
+
+EmitIndexOptions parseEmitIndexOptions(const FrontendOptions &FEOpts);
+
+} // namespace index
+} // namespace clang
+
+#endif

clang/include/clang/IndexSerialization/IndexDataFormat.h

+//===--- IndexDataFormat.h - Layout of index on a filesystem --------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_LIB_INDEX_INDEXDATAFORMAT_H
+#define LLVM_CLANG_LIB_INDEX_INDEXDATAFORMAT_H
+
+#include "llvm/ADT/SmallString.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Bitstream/BitCodes.h"
+
+namespace clang {
+namespace index {
+namespace store {
+
+/// Index is organized like this:
+/// (root of index dir)
+///   /<IndexDir>
+///     /<IndexRecordSubdir>
+///       /AA
+///         /(records)
+///       /AB
+///         /(records)
+///       /AC
+///         /(records)
+///       ...
+///     /<IndexUnitSubdir>
+///       /(units)
+
+/// \returns version of index format
+unsigned getIndexFormatVersion();
+
+/// \returns name of directory where index data lives inside the root directory.
+const char *getIndexDir();
+
+/// \returns name of directory where unit data lives inside the index directory.
+const char *getIndexUnitSubdir();
+
+/// \returns name of directory where record data lives inside the index
+/// directory.
+const char *getIndexRecordSubdir();
+
+/// \returns name of index record related to \p SourceFilename.
+/// Index record filename is the original source filename with hash of the index
+/// data as a suffix.
+std::string
+generateFilenameForIndexRecord(llvm::StringRef SourceFilename,
+                               const llvm::SmallString<512> &IndexRecordBuffer);
+
+/// Appends name of index unit related to \p OutputFilePath to \p Str.
+/// \p OutputFilePath is absolute path to output file.
+void getUnitNameForOutputFile(llvm::StringRef OutputFilePath,
+                              llvm::SmallVectorImpl<char> &Str);
+
+/// Appends index directory and unit subdirectory to \p Buffer without erasing
+/// it.
+void appendUnitSubDir(llvm::SmallVectorImpl<char> &Buffer);
+
+/// Appends index directory and record subdirectory to \p Buffer without erasing
+/// it.
+void appendRecordSubDir(llvm::SmallVectorImpl<char> &Buffer);
+
+/// This is currently a placeholder - just appends \p UnitName to \p Buffer.
+void appendUnitBucketDir(llvm::StringRef UnitName,
+                         llvm::SmallVectorImpl<char> &Buffer);
+
+/// We're bucketing record files according to the last two characters in their
+/// names in order to keep number of files in a single directory reasonable.
+/// Appends bucket subdirectory and delimiter.
+void appendRecordBucketDir(llvm::StringRef RecordName,
+                           llvm::SmallVectorImpl<char> &Buffer);
+
+/// Appends \p RecordName to \p Buffer.
+void appendRecordFilename(llvm::StringRef RecordName,
+                          llvm::SmallVectorImpl<char> &Buffer);
+
+enum RecordBitRecord {
+  REC_VERSION = 0,
+  REC_DECLINFO = 1,
+  REC_DECLOFFSETS = 2,
+  REC_DECLOCCURRENCE = 3,
+};
+
+enum RecordBitBlock {
+  REC_VERSION_BLOCK_ID = llvm::bitc::FIRST_APPLICATION_BLOCKID,
+  REC_DECLS_BLOCK_ID,
+  REC_DECLOFFSETS_BLOCK_ID,
+  REC_DECLOCCURRENCES_BLOCK_ID,
+};
+
+enum UnitBitRecord {
+  UNIT_VERSION = 0,
+  UNIT_INFO = 1,
+  UNIT_DEPENDENCY = 2,
+  UNIT_INCLUDE = 3,
+  UNIT_PATH = 4,
+  UNIT_PATH_BUFFER = 5,
+  UNIT_MODULE = 6,
+  UNIT_MODULE_BUFFER = 7,
+};
+
+enum UnitBitBlock {
+  UNIT_VERSION_BLOCK_ID = llvm::bitc::FIRST_APPLICATION_BLOCKID,
+  UNIT_INFO_BLOCK_ID,
+  UNIT_DEPENDENCIES_BLOCK_ID,
+  UNIT_INCLUDES_BLOCK_ID,
+  UNIT_PATHS_BLOCK_ID,
+  UNIT_MODULES_BLOCK_ID,
+};
+
+enum UnitDependencyKind {
+  UNIT_DEPEND_KIND_FILE = 0,
+  UNIT_DEPEND_KIND_RECORD = 1,
+  UNIT_DEPEND_KIND_UNIT = 2,
+};
+static const unsigned UnitDependencyKindBitNum = 2;
+static const unsigned UnitFilePathPrefixKindBitNum = 2;
+
+} // end namespace store
+} // end namespace index
+} // end namespace clang
+
+#endif

clang/include/clang/IndexSerialization/IndexRecordReader.h

+//===--- IndexRecordReader.h - Index record deserialization ---------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEX_INDEXRECORDREADER_H
+#define LLVM_CLANG_INDEX_INDEXRECORDREADER_H
+
+#include "clang/Index/IndexSymbol.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Bitstream/BitstreamReader.h"
+#include <memory>
+
+namespace llvm {
+class MemoryBuffer;
+}
+
+namespace clang {
+namespace index {
+
+struct IndexRecordDecl {
+  unsigned DeclID;
+  SymbolInfo SymInfo;
+  SymbolRoleSet Roles;
+  SymbolRoleSet RelatedRoles;
+  StringRef Name;
+  StringRef USR;
+  StringRef CodeGenName;
+};
+
+struct IndexRecordRelation {
+  SymbolRoleSet Roles;
+  const IndexRecordDecl *Dcl = nullptr;
+
+  IndexRecordRelation() = default;
+  IndexRecordRelation(SymbolRoleSet Roles, const IndexRecordDecl *Dcl)
+      : Roles(Roles), Dcl(Dcl) {}
+};
+
+struct IndexRecordOccurrence {
+  const IndexRecordDecl *Dcl;
+  SmallVector<IndexRecordRelation, 4> Relations;
+  SymbolRoleSet Roles;
+  unsigned Line;
+  unsigned Column;
+};
+
+/// Interface for reading index record data backed by a memory buffer.
+class IndexRecordReader {
+public:
+  static std::unique_ptr<IndexRecordReader>
+  createWithRecordFilename(StringRef RecordFilename, StringRef StorePath,
+                           std::string &Error);
+  static std::unique_ptr<IndexRecordReader>
+  createWithFilePath(StringRef FilePath, std::string &Error);
+  static std::unique_ptr<IndexRecordReader>
+  createWithBuffer(std::unique_ptr<llvm::MemoryBuffer> Buffer,
+                   std::string &Error);
+
+  struct DeclSearchReturn {
+    bool AcceptDecl;
+    bool ContinueSearch;
+  };
+  typedef DeclSearchReturn(DeclSearchCheck)(const IndexRecordDecl &);
+  /// Iterates over declarations and calls first \p Checker and conditionally
+  /// also \p Receiver for each of them. Receiver is called for those
+  /// declarations for which \p Checker returns AcceptDecl == true. Iteration is
+  /// terminated early if \p Checker returns ContinueSearch == false. Resulting
+  /// decls can be used as filter for \c foreachOccurrence. This allows
+  /// allocating memory only for the record decls that the caller is interested
+  /// in.
+  bool searchDecls(llvm::function_ref<DeclSearchCheck> Checker,
+                   llvm::function_ref<void(const IndexRecordDecl *)> Receiver);
+
+  /// Iterates over declarations and calls \p Receiver for each of them.
+  /// Avoids caching the declaration data if \p NoCache is true.
+  bool foreachDecl(bool NoCache,
+                   llvm::function_ref<bool(const IndexRecordDecl *)> Receiver);
+
+  /// Iterates over occurences of declarations and conditionally calls \p
+  /// Receiver for occurences of declarations. If \p WhitelistDecls is non-empty
+  /// \p Receiver is called only for occurence of declarations that it contains.
+  /// If \p WhitelistRelated is non-empty \p Receiver is called only for
+  /// occurences that are related to any of the declarations it contains. In
+  /// case \p WhitelistDecls is non-empty and a particular declaration isn't
+  /// present, it is skipped and isn't further inspected in regards to \p
+  /// WhitelistRelated.
+  bool foreachOccurrence(
+      llvm::function_ref<bool(const IndexRecordOccurrence &)> Receiver,
+      ArrayRef<const IndexRecordDecl *> WhitelistDecls = None,
+      ArrayRef<const IndexRecordDecl *> WhitelistRelated = None);
+
+  /// Iterates over occurences of declarations and conditionally calls \p
+  /// Receiver for occurences of declarations. \p Receiver will be called only
+  /// for occurences between \p lineStart and \p lineStart + \p lineCount (both
+  /// ends are included).
+  bool foreachOccurrenceInLineRange(
+      unsigned lineStart, unsigned lineCount,
+      llvm::function_ref<bool(const IndexRecordOccurrence &)> Receiver);
+
+  // TODO order
+private:
+  friend class IndexBitstreamVisitor;
+  llvm::BumpPtrAllocator Allocator;
+  std::unique_ptr<llvm::MemoryBuffer> Buffer;
+  llvm::BitstreamCursor DeclCursor;
+  llvm::BitstreamCursor OccurCursor;
+  llvm::ArrayRef<uint32_t> DeclOffsets;
+  const IndexRecordDecl **Decls;
+
+  void setDeclOffsets(ArrayRef<uint32_t> Offs);
+
+  unsigned getNumDecls() const;
+
+  const IndexRecordDecl *getDeclByID(unsigned DeclID);
+
+  const IndexRecordDecl *getDecl(unsigned Index);
+
+  /// Reads occurrence data.
+  /// \param WhitelistDecls if non-empty indicates the list of decls that we
+  /// want to get occurrences for. If empty then indicates that we want
+  /// occurrences for all decls. \param WhitelistRelated Same as \c
+  /// WhitelistDecls but for related decls. \returns true if the occurrence info
+  /// was filled out, false if occurrence was ignored.
+  bool readOccurrence(SmallVectorImpl<uint64_t> &Record, StringRef Blob,
+                      ArrayRef<const IndexRecordDecl *> WhitelistDecls,
+                      ArrayRef<const IndexRecordDecl *> WhitelistRelated,
+                      IndexRecordOccurrence &RecOccur);
+
+  void readDecl(unsigned Index, IndexRecordDecl &RecD);
+};
+
+} // namespace index
+} // namespace clang
+
+#endif

clang/include/clang/IndexSerialization/IndexRecordWriter.h

+//===--- IndexRecordWriter.h - Index record serialization-------- ---------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEX_INDEXRECORDWRITER_H
+#define LLVM_CLANG_INDEX_INDEXRECORDWRITER_H
+
+#include "clang/AST/DeclBase.h"
+#include "clang/AST/Mangle.h"
+#include "clang/Basic/SourceLocation.h"
+#include "clang/Index/IndexSymbol.h"
+#include "clang/IndexSerialization/EmitIndexOptions.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/Bitstream/BitstreamWriter.h"
+#include <map>
+#include <vector>
+
+namespace clang {
+namespace index {
+
+struct DeclInfo {
+  const Decl *D;
+  SymbolRoleSet Roles;
+  SymbolRoleSet RelatedRoles;
+};
+
+/// Collects declaration occurences in a header or source file and serializes
+/// them.
+class IndexRecordWriter {
+private:
+  const bool IsSystem;
+
+  llvm::DenseMap<const Decl *, unsigned> IndexForDecl;
+  std::vector<DeclInfo> Decls;
+  std::multimap<unsigned, SmallVector<uint64_t, 10>> Occurrences;
+
+public:
+  IndexRecordWriter(bool IsSystem) : IsSystem(IsSystem) {}
+
+  bool isSystem() const { return IsSystem; }
+
+  /// Adds an occurrence of the canonical declaration \c D at the supplied
+  /// \c Offset
+  ///
+  /// \param Roles the roles the occurrence fulfills in this position.
+  /// \param Offset the offset in the file of this occurrence.
+  /// \param D the canonical declaration this is an occurrence of.
+  /// \param Relations the set of symbols related to this occurrence.
+  void addDeclOccurence(SymbolRoleSet Roles, FileID FID, unsigned Offset,
+                        const Decl *D, ArrayRef<SymbolRelation> Relations);
+
+  bool isRecordEmpty() const { return Decls.empty(); }
+
+  /// \returns filename of the index record created or None in case of failure.
+  llvm::Optional<std::string>
+  writeToDisk(StringRef SourceFileName, const EmitIndexOptions &RecOpts,
+              const PrintingPolicy &Policy,
+              std::shared_ptr<ASTNameGenerator> MangledNameGen,
+              DiagnosticsEngine &Diag) const;
+
+private:
+  void getBitcodeRepresentation(
+      llvm::SmallString<512> &Buffer, const clang::PrintingPolicy &Policy,
+      std::shared_ptr<ASTNameGenerator> MangledNameGen) const;
+};
+
+} // end namespace index
+} // end namespace clang
+
+#endif

clang/include/clang/IndexSerialization/IndexUnitReader.h

+//===--- IndexUnitReader.h - Index unit deserialization -------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEX_INDEXUNITREADER_H
+#define LLVM_CLANG_INDEX_INDEXUNITREADER_H
+
+#include "clang/Basic/LLVM.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Bitstream/BitstreamReader.h"
+#include "llvm/Support/Chrono.h"
+#include "llvm/Support/MemoryBuffer.h"
+
+#include "clang/IndexSerialization/SerializablePathCollection.h"
+#include <memory>
+#include <vector>
+
+namespace clang {
+namespace index {
+
+struct ModuleInfo {
+  unsigned NameOffset;
+  unsigned NameSize;
+};
+
+/// Interface for reading index unit data backed by a memory buffer.
+class IndexUnitReader {
+public:
+  enum class DependencyKind {
+    Unit,
+    Record,
+    File,
+  };
+
+  IndexUnitReader(std::unique_ptr<llvm::MemoryBuffer> Buf,
+                  llvm::BitstreamCursor &&DependenciesBlockCursor,
+                  llvm::BitstreamCursor &&IncludeCursor,
+                  StringRef ProviderIdentifier, // < reference to the Buf data
+                  StringRef ProviderVersion,    // < reference to the Buf data
+                  bool IsSystemUnit, bool IsModuleUnit, bool IsDebugCompilation,
+                  StringRef ModuleName, // < reference to the Buf data
+                  StringRef Target,     // < reference to the Buf data
+                  int MainPathIndex, std::vector<PathPool::FilePath> &&Paths,
+                  StringRef PathsBuffer,
+                  // Offset-size pairs to the PathsBuffer.
+                  size_t WorkDirOffset, size_t WorkDirSize,
+                  size_t OutputFileOffset, size_t OutputFileSize,
+                  size_t SysrootOffset, size_t SysrootSize,
+                  std::vector<ModuleInfo> &&Modules,
+                  StringRef ModuleNamesBuffer, // < reference to the Buf data
+                  llvm::sys::TimePoint<> ModTime);
+
+  /// Creates IndexUnitReader from \p Buf.
+  /// \returns nullptr in case of any failure.
+  static std::unique_ptr<IndexUnitReader>
+  create(std::unique_ptr<llvm::MemoryBuffer> Buf,
+         llvm::sys::TimePoint<> ModTime, std::string &Error);
+
+  /// Creates IndexUnitReader from a file.
+  /// File location is derived from \p IndexUnitFilename and \p IndexRootDir.
+  static std::unique_ptr<IndexUnitReader>
+  createWithUnitFilename(StringRef IndexUnitFilename, StringRef IndexRootDir,
+                         std::string &Error);
+
+  /// Creates IndexUnitReader from a file at \p IndexUnitPath.
+  static std::unique_ptr<IndexUnitReader>
+  createWithFilePath(StringRef IndexUnitPath, std::string &Error);
+
+  /// Get Modification time of index unit file by getting it's status on
+  /// filesystem. File location is derived from \p IndexUnitFilename and \p
+  /// IndexRootDir. \returns nullptr in case of failure to get status of the
+  /// file.
+  static Optional<llvm::sys::TimePoint<>>
+  getModificationTimeForUnit(StringRef IndexUnitFilename,
+                             StringRef IndexRootDir, std::string &Error);
+
+  StringRef getProviderIdentifier() const;
+  StringRef getProviderVersion() const;
+
+  llvm::sys::TimePoint<> getModificationTime() const;
+  StringRef getWorkingDirectory() const;
+  StringRef getOutputFile() const;
+  StringRef getSysrootPath() const;
+  StringRef getMainFilePath() const;
+  StringRef getModuleName() const;
+  StringRef getTarget() const;
+  bool hasMainFile() const;
+  bool isSystemUnit() const;
+  bool isModuleUnit() const;
+  bool isDebugCompilation() const;
+
+  struct DependencyInfo {
+    DependencyKind Kind;
+    bool IsSystem;
+    StringRef UnitOrRecordName;
+    StringRef FilePath;
+    StringRef ModuleName;
+  };
+  /// Iterates over dependencies of this index unit and calls \p Receiver for
+  /// each of them. Order of dependencies is: units, records, files.
+  bool foreachDependency(
+      llvm::function_ref<bool(const DependencyInfo &Info)> Receiver);
+
+  struct IncludeInfo {
+    StringRef SourcePath;
+    unsigned SourceLine;
+    StringRef TargetPath;
+  };
+  /// Iterates over files included into this index unit and calls \p Receiver
+  /// for each of them.
+  bool
+  foreachInclude(llvm::function_ref<bool(const IncludeInfo &Info)> Receiver);
+
+private:
+  const std::unique_ptr<llvm::MemoryBuffer> Buf;
+
+  llvm::BitstreamCursor DependenciesBlockCursor;
+  llvm::BitstreamCursor IncludedFilesCursor;
+  StringRef ProviderIdentifier;
+  StringRef ProviderVersion;
+  const bool IsSystemUnit;
+  const bool IsModuleUnit;
+  const bool IsDebugCompilation;
+  StringRef ModuleName;
+  StringRef Target;
+
+  SmallString<128> MainFilePath;
+  const std::vector<PathPool::FilePath> Paths;
+
+  StringRef PathsBuffer;
+  // Offset-size pairs to the PathsBuffer.
+  const size_t WorkDirOffset;
+  const size_t WorkDirSize;
+  const size_t OutputFileOffset;
+  const size_t OutputFileSize;
+  const size_t SysrootOffset;
+  const size_t SysrootSize;
+
+  const std::vector<ModuleInfo> Modules;
+  StringRef ModuleNamesBuffer;
+  const llvm::sys::TimePoint<> ModTime;
+
+private:
+  /// Appends the full path related to \p PathIndex to \p PathBuf.
+  void constructFilePath(SmallVectorImpl<char> &PathBuf, int PathIndex) const;
+};
+
+} // namespace index
+} // namespace clang
+
+#endif

clang/include/clang/IndexSerialization/IndexUnitWriter.h

+//===--- IndexUnitWriter.h - Index unit serialization ---------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEX_INDEXUNITWRITER_H
+#define LLVM_CLANG_INDEX_INDEXUNITWRITER_H
+
+#include "clang/Basic/LLVM.h"
+#include "clang/IndexSerialization/EmitIndexOptions.h"
+#include "clang/IndexSerialization/SerializablePathCollection.h"
+#include "llvm/ADT/DenseSet.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/ADT/StringMap.h"
+#include "llvm/Support/Allocator.h"
+#include <set>
+#include <string>
+#include <vector>
+
+namespace llvm {
+class BitstreamWriter;
+}
+
+namespace clang {
+class CompilerInstance;
+class FileEntry;
+class FileManager;
+class Module;
+
+namespace index {
+
+namespace writer {
+/// An opaque pointer to a module used by the IndexUnitWriter to associate
+/// record and file dependencies with a module, and as a token for getting
+/// information about the module from the caller.
+typedef const void *OpaqueModule;
+
+struct ModuleInfo {
+  /// Full dot-separated module name.
+  StringRef Name;
+};
+
+typedef llvm::function_ref<ModuleInfo(OpaqueModule,
+                                      SmallVectorImpl<char> &Scratch)>
+    ModuleInfoWriterCallback;
+} // end namespace writer
+
+struct ModuleIndexData {
+  const std::string Name;
+  const bool IsSystem;
+};
+
+class IndexUnitWriter {
+private:
+  const EmitIndexOptions RecOpts;
+
+  // index unit metadata
+  const std::string ProviderIdentifier;
+  const std::string ProviderVersion;
+  const std::string OutputFile;
+  const std::string AbsPathToOutputFile;
+  const FileEntry *MainFile;
+
+  // Absence of value means we're not indexing a module.
+  const llvm::Optional<ModuleIndexData> ModuleData;
+
+  const bool IsDebugCompilation;
+  const std::string TargetTriple;
+  const std::string WorkDir;
+  // FIXME
+  // IIUC we're serializing this as a source of truth of what is considered a
+  // system part of directory tree but we should probably be using values of all
+  // of -isysroot, -isystem and -F options.
+  const std::string SysrootPath;
+
+  mutable SerializablePathCollection PathStore;
+  // Module dependencies
+  SmallString<512> ModuleNamesBuf;
+  std::vector<std::array<uint64_t, 3>> Modules;
+  // Maps module to one-based index in the Modules container.
+  llvm::DenseMap<writer::OpaqueModule, unsigned> ModToIdx;
+
+  // Source file dependencies - mapping from UID to data.
+  llvm::DenseMap<unsigned, std::array<uint64_t, 7>> SourceFiles;
+  // Source files that has a corresponding IndexRecord aren't written into
+  // index.
+  llvm::DenseSet<unsigned> SourceFileHasARecord;
+  std::vector<std::array<uint64_t, 4>> IncludedFiles;
+
+  std::vector<std::pair<SmallString<64>, std::array<uint64_t, 7>>>
+      IndexUnitDependencies;
+  llvm::DenseSet<unsigned> ASTFilesOfSeenIndexUnits;
+
+  std::vector<std::pair<std::string, std::array<uint64_t, 7>>>
+      IndexRecordDependencies;
+
+  std::function<writer::ModuleInfo(writer::OpaqueModule,
+                                   SmallVectorImpl<char> &Scratch)>
+      GetInfoForModuleFn;
+
+public:
+  /// Creates IndexUnitWriter for index unit represented either by \p MainFile
+  /// or \p ModuleData. \p MainFile the main file for a compiled source file.
+  /// This should be null for PCH and module units.
+  IndexUnitWriter(FileManager &FileMgr, const EmitIndexOptions &RecOpts,
+                  StringRef ProviderIdentifier, StringRef ProviderVersion,
+                  StringRef OutputFile, StringRef AbsPathToOutputFile,
+                  const FileEntry *MainFile,
+                  const llvm::Optional<ModuleIndexData> &ModuleData,
+                  bool IsDebugCompilation, StringRef TargetTriple,
+                  StringRef WorkDir, StringRef SysrootPath,
+                  writer::ModuleInfoWriterCallback GetInfoForModule);
+
+  unsigned addModule(writer::OpaqueModule Mod);
+
+  /// Stores dependency of this index unit on source file \p File.
+  /// If \p Mod != nullptr it means \p File is compiled as part of module \p
+  /// Mod.
+  void addSourceFile(const FileEntry *File, bool IsSystem,
+                     writer::OpaqueModule Mod);
+
+  /// Stores dependency of this index unit on index record.
+  /// Index record \p RecordFile contains index data of \p SourceFile.
+  /// If \p Mod != nullptr it means \p SourceFile is compiled as part of module
+  /// \p Mod.
+  void addIndexRecord(StringRef RecordFile, const FileEntry *SourceFile,
+                      bool IsSystem, writer::OpaqueModule Mod);
+
+  /// Stores dependency of this index unit on another index unit - \p File.
+  /// If \p Mod != nullptr it means \p File contains index data of module \p
+  /// Mod.
+  void addIndexUnit(const FileEntry *File, StringRef AbsolutePath,
+                    bool IsSystem, writer::OpaqueModule Mod,
+                    bool withoutUnitName = false);
+
+  /// Stores dependency of this index unit on \p IncludedFile.
+  void addIncludedFile(const FileEntry *IncludingFile, unsigned Line,
+                       const FileEntry *IncludedFile);
+
+  /// \returns true in case writing failed.
+  bool writeToDisk(std::string &Error) const;
+
+private:
+  void writeUnitInfo(llvm::BitstreamWriter &Stream) const;
+  void writeDependencies(llvm::BitstreamWriter &Stream) const;
+  void writeIncludes(llvm::BitstreamWriter &Stream) const;
+  void writePaths(llvm::BitstreamWriter &Stream) const;
+  void writeModules(llvm::BitstreamWriter &Stream) const;
+
+  void getBitcodeRepresentation(llvm::SmallString<512> &Buffer,
+                                std::string &Error) const;
+};
+
+IndexUnitWriter makeIndexUnitWriter(const CompilerInstance &CI,
+                                    const EmitIndexOptions &RecordOpts,
+                                    StringRef OutputFile,
+                                    const FileEntry *MainFile,
+                                    Module *UnitModule);
+
+bool initIndexDirectory(const EmitIndexOptions &RecordingOpts,
+                        std::string &Error);
+
+// \returns true in case index unit for \p ModuleFilePath exists, false in case
+// it doesn't and llvm::None in case it couldn't be stat()-ed.
+llvm::Optional<bool>
+doesIndexUnitForModuleExist(StringRef ModuleFilePath,
+                            const FileManager &FileMgr,
+                            const EmitIndexOptions &RecOpts);
+
+} // end namespace index
+} // end namespace clang
+
+#endif

clang/include/indexstore/Conversions.h

+//===--- IndexDataStoreSymbolUtils.h - Utilities for indexstore symbols ---===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEXSTORE_CONVERSIONS_H
+#define LLVM_CLANG_INDEXSTORE_CONVERSIONS_H
+
+#include "indexstore/indexstore.h"
+#include "clang/Index/IndexSymbol.h"
+
+namespace clang {
+namespace index {
+
+inline SymbolKind getSymbolKind(indexstore_symbol_kind_t K) {
+  switch ((uint64_t)K) {
+  default:
+  case INDEXSTORE_SYMBOL_KIND_UNKNOWN:
+    return SymbolKind::Unknown;
+  case INDEXSTORE_SYMBOL_KIND_MODULE:
+    return SymbolKind::Module;
+  case INDEXSTORE_SYMBOL_KIND_NAMESPACE:
+    return SymbolKind::Namespace;
+  case INDEXSTORE_SYMBOL_KIND_NAMESPACEALIAS:
+    return SymbolKind::NamespaceAlias;
+  case INDEXSTORE_SYMBOL_KIND_MACRO:
+    return SymbolKind::Macro;
+  case INDEXSTORE_SYMBOL_KIND_ENUM:
+    return SymbolKind::Enum;
+  case INDEXSTORE_SYMBOL_KIND_STRUCT:
+    return SymbolKind::Struct;
+  case INDEXSTORE_SYMBOL_KIND_CLASS:
+    return SymbolKind::Class;
+  case INDEXSTORE_SYMBOL_KIND_PROTOCOL:
+    return SymbolKind::Protocol;
+  case INDEXSTORE_SYMBOL_KIND_EXTENSION:
+    return SymbolKind::Extension;
+  case INDEXSTORE_SYMBOL_KIND_UNION:
+    return SymbolKind::Union;
+  case INDEXSTORE_SYMBOL_KIND_TYPEALIAS:
+    return SymbolKind::TypeAlias;
+  case INDEXSTORE_SYMBOL_KIND_FUNCTION:
+    return SymbolKind::Function;
+  case INDEXSTORE_SYMBOL_KIND_VARIABLE:
+    return SymbolKind::Variable;
+  case INDEXSTORE_SYMBOL_KIND_FIELD:
+    return SymbolKind::Field;
+  case INDEXSTORE_SYMBOL_KIND_ENUMCONSTANT:
+    return SymbolKind::EnumConstant;
+  case INDEXSTORE_SYMBOL_KIND_INSTANCEMETHOD:
+    return SymbolKind::InstanceMethod;
+  case INDEXSTORE_SYMBOL_KIND_CLASSMETHOD:
+    return SymbolKind::ClassMethod;
+  case INDEXSTORE_SYMBOL_KIND_STATICMETHOD:
+    return SymbolKind::StaticMethod;
+  case INDEXSTORE_SYMBOL_KIND_INSTANCEPROPERTY:
+    return SymbolKind::InstanceProperty;
+  case INDEXSTORE_SYMBOL_KIND_CLASSPROPERTY:
+    return SymbolKind::ClassProperty;
+  case INDEXSTORE_SYMBOL_KIND_STATICPROPERTY:
+    return SymbolKind::StaticProperty;
+  case INDEXSTORE_SYMBOL_KIND_CONSTRUCTOR:
+    return SymbolKind::Constructor;
+  case INDEXSTORE_SYMBOL_KIND_DESTRUCTOR:
+    return SymbolKind::Destructor;
+  case INDEXSTORE_SYMBOL_KIND_CONVERSIONFUNCTION:
+    return SymbolKind::ConversionFunction;
+  case INDEXSTORE_SYMBOL_KIND_PARAMETER:
+    return SymbolKind::Parameter;
+  case INDEXSTORE_SYMBOL_KIND_USING:
+    return SymbolKind::Using;
+  }
+}
+
+inline SymbolSubKind getSymbolSubKind(indexstore_symbol_subkind_t K) {
+  switch ((uint64_t)K) {
+  default:
+  case INDEXSTORE_SYMBOL_SUBKIND_NONE:
+    return SymbolSubKind::None;
+  case INDEXSTORE_SYMBOL_SUBKIND_CXXCOPYCONSTRUCTOR:
+    return SymbolSubKind::CXXCopyConstructor;
+  case INDEXSTORE_SYMBOL_SUBKIND_CXXMOVECONSTRUCTOR:
+    return SymbolSubKind::CXXMoveConstructor;
+  case INDEXSTORE_SYMBOL_SUBKIND_ACCESSORGETTER:
+    return SymbolSubKind::AccessorGetter;
+  case INDEXSTORE_SYMBOL_SUBKIND_ACCESSORSETTER:
+    return SymbolSubKind::AccessorSetter;
+  case INDEXSTORE_SYMBOL_SUBKIND_USINGTYPENAME:
+    return SymbolSubKind::UsingTypename;
+  case INDEXSTORE_SYMBOL_SUBKIND_USINGVALUE:
+    return SymbolSubKind::UsingValue;
+  }
+}
+
+inline SymbolLanguage getSymbolLanguage(indexstore_symbol_language_t L) {
+  switch ((uint64_t)L) {
+  default: // FIXME: add an unknown language?
+  case INDEXSTORE_SYMBOL_LANG_C:
+    return SymbolLanguage::C;
+  case INDEXSTORE_SYMBOL_LANG_OBJC:
+    return SymbolLanguage::ObjC;
+  case INDEXSTORE_SYMBOL_LANG_CXX:
+    return SymbolLanguage::CXX;
+  case INDEXSTORE_SYMBOL_LANG_SWIFT:
+    return SymbolLanguage::Swift;
+  }
+}
+
+inline SymbolPropertySet getSymbolProperties(uint64_t Props) {
+  SymbolPropertySet SymbolProperties = 0;
+  if (Props & INDEXSTORE_SYMBOL_PROPERTY_GENERIC)
+    SymbolProperties |= (SymbolPropertySet)SymbolProperty::Generic;
+  if (Props & INDEXSTORE_SYMBOL_PROPERTY_TEMPLATE_PARTIAL_SPECIALIZATION)
+    SymbolProperties |=
+        (SymbolPropertySet)SymbolProperty::TemplatePartialSpecialization;
+  if (Props & INDEXSTORE_SYMBOL_PROPERTY_TEMPLATE_SPECIALIZATION)
+    SymbolProperties |=
+        (SymbolPropertySet)SymbolProperty::TemplateSpecialization;
+  if (Props & INDEXSTORE_SYMBOL_PROPERTY_UNITTEST)
+    SymbolProperties |= (SymbolPropertySet)SymbolProperty::UnitTest;
+  if (Props & INDEXSTORE_SYMBOL_PROPERTY_IBANNOTATED)
+    SymbolProperties |= (SymbolPropertySet)SymbolProperty::IBAnnotated;
+  if (Props & INDEXSTORE_SYMBOL_PROPERTY_IBOUTLETCOLLECTION)
+    SymbolProperties |= (SymbolPropertySet)SymbolProperty::IBOutletCollection;
+  if (Props & INDEXSTORE_SYMBOL_PROPERTY_GKINSPECTABLE)
+    SymbolProperties |= (SymbolPropertySet)SymbolProperty::GKInspectable;
+  if (Props & INDEXSTORE_SYMBOL_PROPERTY_LOCAL)
+    SymbolProperties |= (SymbolPropertySet)SymbolProperty::Local;
+  if (Props & INDEXSTORE_SYMBOL_PROPERTY_PROTOCOL_INTERFACE)
+    SymbolProperties |= (SymbolPropertySet)SymbolProperty::ProtocolInterface;
+
+  return SymbolProperties;
+}
+
+inline SymbolRoleSet getSymbolRoles(uint64_t Roles) {
+  SymbolRoleSet SymbolRoles = 0;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_DECLARATION)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::Declaration;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_DEFINITION)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::Definition;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_REFERENCE)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::Reference;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_READ)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::Read;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_WRITE)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::Write;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_CALL)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::Call;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_DYNAMIC)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::Dynamic;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_ADDRESSOF)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::AddressOf;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_IMPLICIT)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::Implicit;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_UNDEFINITION)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::Undefinition;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_REL_CHILDOF)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::RelationChildOf;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_REL_BASEOF)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::RelationBaseOf;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_REL_OVERRIDEOF)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::RelationOverrideOf;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_REL_RECEIVEDBY)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::RelationReceivedBy;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_REL_CALLEDBY)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::RelationCalledBy;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_REL_EXTENDEDBY)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::RelationExtendedBy;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_REL_ACCESSOROF)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::RelationAccessorOf;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_REL_CONTAINEDBY)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::RelationContainedBy;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_REL_IBTYPEOF)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::RelationIBTypeOf;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_REL_SPECIALIZATIONOF)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::RelationSpecializationOf;
+  if (Roles & INDEXSTORE_SYMBOL_ROLE_NAMEREFERENCE)
+    SymbolRoles |= (SymbolRoleSet)SymbolRole::NameReference;
+
+  return SymbolRoles;
+}
+
+inline indexstore_symbol_kind_t getIndexStoreKind(SymbolKind K) {
+  switch (K) {
+  case SymbolKind::Unknown:
+  // FIXME: Tweak SymbolKind/SymbolSubKind to separate
+  // generic-ish/language-specific-ish concepts.
+  //
+  // We use SymbolKind/SymbolSubKind
+  // downstream for Swift and this would help us avoid intrusions here or
+  // upstream.
+  //
+  // Separation of concepts for C++ and Objective-C could also be
+  // improved. Once that is done on SymbolKind/SymbolSubKind level we should
+  // update indexstore_symbol_kind_t too.
+  //
+  // Initial discussion here:
+  // https://github.com/apple/llvm-project/pull/1099
+  case SymbolKind::TemplateTypeParm:
+  case SymbolKind::TemplateTemplateParm:
+  case SymbolKind::NonTypeTemplateParm:
+    return INDEXSTORE_SYMBOL_KIND_UNKNOWN;
+  case SymbolKind::Module:
+    return INDEXSTORE_SYMBOL_KIND_MODULE;
+  case SymbolKind::Namespace:
+    return INDEXSTORE_SYMBOL_KIND_NAMESPACE;
+  case SymbolKind::NamespaceAlias:
+    return INDEXSTORE_SYMBOL_KIND_NAMESPACEALIAS;
+  case SymbolKind::Macro:
+    return INDEXSTORE_SYMBOL_KIND_MACRO;
+  case SymbolKind::Enum:
+    return INDEXSTORE_SYMBOL_KIND_ENUM;
+  case SymbolKind::Struct:
+    return INDEXSTORE_SYMBOL_KIND_STRUCT;
+  case SymbolKind::Class:
+    return INDEXSTORE_SYMBOL_KIND_CLASS;
+  case SymbolKind::Protocol:
+    return INDEXSTORE_SYMBOL_KIND_PROTOCOL;
+  case SymbolKind::Extension:
+    return INDEXSTORE_SYMBOL_KIND_EXTENSION;
+  case SymbolKind::Union:
+    return INDEXSTORE_SYMBOL_KIND_UNION;
+  case SymbolKind::TypeAlias:
+    return INDEXSTORE_SYMBOL_KIND_TYPEALIAS;
+  case SymbolKind::Function:
+    return INDEXSTORE_SYMBOL_KIND_FUNCTION;
+  case SymbolKind::Variable:
+    return INDEXSTORE_SYMBOL_KIND_VARIABLE;
+  case SymbolKind::Field:
+    return INDEXSTORE_SYMBOL_KIND_FIELD;
+  case SymbolKind::EnumConstant:
+    return INDEXSTORE_SYMBOL_KIND_ENUMCONSTANT;
+  case SymbolKind::InstanceMethod:
+    return INDEXSTORE_SYMBOL_KIND_INSTANCEMETHOD;
+  case SymbolKind::ClassMethod:
+    return INDEXSTORE_SYMBOL_KIND_CLASSMETHOD;
+  case SymbolKind::StaticMethod:
+    return INDEXSTORE_SYMBOL_KIND_STATICMETHOD;
+  case SymbolKind::InstanceProperty:
+    return INDEXSTORE_SYMBOL_KIND_INSTANCEPROPERTY;
+  case SymbolKind::ClassProperty:
+    return INDEXSTORE_SYMBOL_KIND_CLASSPROPERTY;
+  case SymbolKind::StaticProperty:
+    return INDEXSTORE_SYMBOL_KIND_STATICPROPERTY;
+  case SymbolKind::Constructor:
+    return INDEXSTORE_SYMBOL_KIND_CONSTRUCTOR;
+  case SymbolKind::Destructor:
+    return INDEXSTORE_SYMBOL_KIND_DESTRUCTOR;
+  case SymbolKind::ConversionFunction:
+    return INDEXSTORE_SYMBOL_KIND_CONVERSIONFUNCTION;
+  case SymbolKind::Parameter:
+    return INDEXSTORE_SYMBOL_KIND_PARAMETER;
+  case SymbolKind::Using:
+    return INDEXSTORE_SYMBOL_KIND_USING;
+  }
+  llvm_unreachable("unexpected symbol kind");
+}
+
+inline indexstore_symbol_subkind_t getIndexStoreSubKind(SymbolSubKind K) {
+  switch (K) {
+  case SymbolSubKind::None:
+    return INDEXSTORE_SYMBOL_SUBKIND_NONE;
+  case SymbolSubKind::CXXCopyConstructor:
+    return INDEXSTORE_SYMBOL_SUBKIND_CXXCOPYCONSTRUCTOR;
+  case SymbolSubKind::CXXMoveConstructor:
+    return INDEXSTORE_SYMBOL_SUBKIND_CXXMOVECONSTRUCTOR;
+  case SymbolSubKind::AccessorGetter:
+    return INDEXSTORE_SYMBOL_SUBKIND_ACCESSORGETTER;
+  case SymbolSubKind::AccessorSetter:
+    return INDEXSTORE_SYMBOL_SUBKIND_ACCESSORSETTER;
+  case SymbolSubKind::UsingTypename:
+    return INDEXSTORE_SYMBOL_SUBKIND_USINGTYPENAME;
+  case SymbolSubKind::UsingValue:
+    return INDEXSTORE_SYMBOL_SUBKIND_USINGVALUE;
+  }
+  llvm_unreachable("unexpected symbol subkind");
+}
+
+inline indexstore_symbol_language_t getIndexStoreLang(SymbolLanguage L) {
+  switch (L) {
+  case SymbolLanguage::C:
+    return INDEXSTORE_SYMBOL_LANG_C;
+  case SymbolLanguage::ObjC:
+    return INDEXSTORE_SYMBOL_LANG_OBJC;
+  case SymbolLanguage::CXX:
+    return INDEXSTORE_SYMBOL_LANG_CXX;
+  case SymbolLanguage::Swift:
+    return INDEXSTORE_SYMBOL_LANG_SWIFT;
+  }
+  llvm_unreachable("unexpected symbol language");
+}
+
+inline uint64_t getIndexStoreProperties(SymbolPropertySet Props) {
+  uint64_t storeProp = 0;
+  applyForEachSymbolProperty(Props, [&](SymbolProperty prop) {
+    switch (prop) {
+    case SymbolProperty::Generic:
+      storeProp |= INDEXSTORE_SYMBOL_PROPERTY_GENERIC;
+      break;
+    case SymbolProperty::TemplatePartialSpecialization:
+      storeProp |= INDEXSTORE_SYMBOL_PROPERTY_TEMPLATE_PARTIAL_SPECIALIZATION;
+      break;
+    case SymbolProperty::TemplateSpecialization:
+      storeProp |= INDEXSTORE_SYMBOL_PROPERTY_TEMPLATE_SPECIALIZATION;
+      break;
+    case SymbolProperty::UnitTest:
+      storeProp |= INDEXSTORE_SYMBOL_PROPERTY_UNITTEST;
+      break;
+    case SymbolProperty::IBAnnotated:
+      storeProp |= INDEXSTORE_SYMBOL_PROPERTY_IBANNOTATED;
+      break;
+    case SymbolProperty::IBOutletCollection:
+      storeProp |= INDEXSTORE_SYMBOL_PROPERTY_IBOUTLETCOLLECTION;
+      break;
+    case SymbolProperty::GKInspectable:
+      storeProp |= INDEXSTORE_SYMBOL_PROPERTY_GKINSPECTABLE;
+      break;
+    case SymbolProperty::Local:
+      storeProp |= INDEXSTORE_SYMBOL_PROPERTY_LOCAL;
+      break;
+    case SymbolProperty::ProtocolInterface:
+      storeProp |= INDEXSTORE_SYMBOL_PROPERTY_PROTOCOL_INTERFACE;
+      break;
+    }
+  });
+  return storeProp;
+}
+
+inline uint64_t getIndexStoreRoles(SymbolRoleSet Roles) {
+  uint64_t storeRoles = 0;
+  applyForEachSymbolRole(Roles, [&](SymbolRole role) {
+    switch (role) {
+    case SymbolRole::Declaration:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_DECLARATION;
+      break;
+    case SymbolRole::Definition:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_DEFINITION;
+      break;
+    case SymbolRole::Reference:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REFERENCE;
+      break;
+    case SymbolRole::Read:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_READ;
+      break;
+    case SymbolRole::Write:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_WRITE;
+      break;
+    case SymbolRole::Call:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_CALL;
+      break;
+    case SymbolRole::Dynamic:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_DYNAMIC;
+      break;
+    case SymbolRole::AddressOf:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_ADDRESSOF;
+      break;
+    case SymbolRole::Implicit:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_IMPLICIT;
+      break;
+    case SymbolRole::Undefinition:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_UNDEFINITION;
+      break;
+    case SymbolRole::RelationChildOf:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REL_CHILDOF;
+      break;
+    case SymbolRole::RelationBaseOf:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REL_BASEOF;
+      break;
+    case SymbolRole::RelationOverrideOf:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REL_OVERRIDEOF;
+      break;
+    case SymbolRole::RelationReceivedBy:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REL_RECEIVEDBY;
+      break;
+    case SymbolRole::RelationCalledBy:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REL_CALLEDBY;
+      break;
+    case SymbolRole::RelationExtendedBy:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REL_EXTENDEDBY;
+      break;
+    case SymbolRole::RelationAccessorOf:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REL_ACCESSOROF;
+      break;
+    case SymbolRole::RelationContainedBy:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REL_CONTAINEDBY;
+      break;
+    case SymbolRole::RelationIBTypeOf:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REL_IBTYPEOF;
+      break;
+    case SymbolRole::RelationSpecializationOf:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_REL_SPECIALIZATIONOF;
+      break;
+    case SymbolRole::NameReference:
+      storeRoles |= INDEXSTORE_SYMBOL_ROLE_NAMEREFERENCE;
+      break;
+    }
+  });
+  return storeRoles;
+}
+
+} // end namespace index
+} // namespace clang
+
+#endif

clang/include/indexstore/IndexStoreCXX.h

+//===--- IndexStoreCXX.h - C++ wrapper for the Index Store C API. ---------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// Header-only C++ wrapper for the Index Store C API.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INDEXSTORE_INDEXSTORECXX_H
+#define LLVM_CLANG_INDEXSTORE_INDEXSTORECXX_H
+
+#include "indexstore/indexstore.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/Optional.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/SmallString.h"
+
+namespace indexstore {
+  using llvm::ArrayRef;
+  using llvm::Optional;
+  using llvm::StringRef;
+
+static inline StringRef stringFromIndexStoreStringRef(indexstore_string_ref_t str) {
+  return StringRef(str.data, str.length);
+}
+
+template<typename Ret, typename ...Params>
+static inline Ret functionPtrFromFunctionRef(void *ctx, Params ...params) {
+  auto fn = (llvm::function_ref<Ret(Params...)> *)ctx;
+  return (*fn)(std::forward<Params>(params)...);
+}
+
+class IndexRecordSymbol {
+  indexstore_symbol_t obj;
+  friend class IndexRecordReader;
+
+public:
+  IndexRecordSymbol(indexstore_symbol_t obj) : obj(obj) {}
+
+  indexstore_symbol_language_t getLanguage() {
+    return indexstore_symbol_get_language(obj);
+  }
+  indexstore_symbol_kind_t getKind() { return indexstore_symbol_get_kind(obj); }
+  indexstore_symbol_subkind_t getSubKind() { return indexstore_symbol_get_subkind(obj); }
+  uint64_t getProperties() {
+    return indexstore_symbol_get_properties(obj);
+  }
+  uint64_t getRoles() { return indexstore_symbol_get_roles(obj); }
+  uint64_t getRelatedRoles() { return indexstore_symbol_get_related_roles(obj); }
+  StringRef getName() { return stringFromIndexStoreStringRef(indexstore_symbol_get_name(obj)); }
+  StringRef getUSR() { return stringFromIndexStoreStringRef(indexstore_symbol_get_usr(obj)); }
+  StringRef getCodegenName() { return stringFromIndexStoreStringRef(indexstore_symbol_get_codegen_name(obj)); }
+};
+
+class IndexSymbolRelation {
+  indexstore_symbol_relation_t obj;
+
+public:
+  IndexSymbolRelation(indexstore_symbol_relation_t obj) : obj(obj) {}
+
+  uint64_t getRoles() { return indexstore_symbol_relation_get_roles(obj); }
+  IndexRecordSymbol getSymbol() { return indexstore_symbol_relation_get_symbol(obj); }
+};
+
+class IndexRecordOccurrence {
+  indexstore_occurrence_t obj;
+
+public:
+  IndexRecordOccurrence(indexstore_occurrence_t obj) : obj(obj) {}
+
+  IndexRecordSymbol getSymbol() { return indexstore_occurrence_get_symbol(obj); }
+  uint64_t getRoles() { return indexstore_occurrence_get_roles(obj); }
+
+  bool foreachRelation(llvm::function_ref<bool(IndexSymbolRelation)> receiver) {
+#if INDEXSTORE_HAS_BLOCKS
+    return indexstore_occurrence_relations_apply(obj, ^bool(indexstore_symbol_relation_t sym_rel) {
+      return receiver(sym_rel);
+    });
+#else
+    return indexstore_occurrence_relations_apply_f(obj, &receiver, functionPtrFromFunctionRef);
+#endif
+  }
+
+  std::pair<unsigned, unsigned> getLineCol() {
+    unsigned line, col;
+    indexstore_occurrence_get_line_col(obj, &line, &col);
+    return std::make_pair(line, col);
+  }
+};
+
+class IndexStore;
+typedef std::shared_ptr<IndexStore> IndexStoreRef;
+
+class IndexStore {
+  indexstore_t obj;
+  friend class IndexRecordReader;
+  friend class IndexUnitReader;
+
+public:
+  IndexStore(StringRef path, std::string &error) {
+    llvm::SmallString<64> buf = path;
+    indexstore_error_t c_err = nullptr;
+    obj = indexstore_store_create(buf.c_str(), &c_err);
+    if (c_err) {
+      error = indexstore_error_get_description(c_err);
+      indexstore_error_dispose(c_err);
+    }
+  }
+
+  IndexStore(IndexStore &&other) : obj(other.obj) {
+    other.obj = nullptr;
+  }
+
+  ~IndexStore() {
+    indexstore_store_dispose(obj);
+  }
+
+  static IndexStoreRef create(StringRef path, std::string &error) {
+    auto storeRef = std::make_shared<IndexStore>(path, error);
+    if (storeRef->isInvalid())
+      return nullptr;
+    return storeRef;
+  }
+
+  static unsigned formatVersion() {
+    return indexstore_format_version();
+  }
+
+  bool isValid() const { return obj; }
+  bool isInvalid() const { return !isValid(); }
+  explicit operator bool() const { return isValid(); }
+
+  bool foreachUnit(bool sorted, llvm::function_ref<bool(StringRef unitName)> receiver) {
+#if INDEXSTORE_HAS_BLOCKS
+    return indexstore_store_units_apply(obj, sorted, ^bool(indexstore_string_ref_t unit_name) {
+      return receiver(stringFromIndexStoreStringRef(unit_name));
+    });
+#else
+    return indexstore_store_units_apply_f(obj, sorted, &receiver, functionPtrFromFunctionRef);
+#endif
+  }
+
+  class UnitEvent {
+    indexstore_unit_event_t obj;
+  public:
+    UnitEvent(indexstore_unit_event_t obj) : obj(obj) {}
+
+    enum class Kind {
+      Removed,
+      Modified,
+      DirectoryDeleted,
+      Failure
+    };
+    Kind getKind() const {
+      indexstore_unit_event_kind_t c_k = indexstore_unit_event_get_kind(obj);
+      Kind K;
+      switch (c_k) {
+      case INDEXSTORE_UNIT_EVENT_REMOVED: K = Kind::Removed; break;
+      case INDEXSTORE_UNIT_EVENT_MODIFIED: K = Kind::Modified; break;
+      case INDEXSTORE_UNIT_EVENT_DIRECTORY_DELETED: K = Kind::DirectoryDeleted; break;
+      case INDEXSTORE_UNIT_EVENT_FAILURE: K = Kind::Failure; break;
+      }
+      return K;
+    }
+
+    StringRef getUnitName() const {
+      return stringFromIndexStoreStringRef(indexstore_unit_event_get_unit_name(obj));
+    }
+  };
+
+  class UnitEventNotification {
+    indexstore_unit_event_notification_t obj;
+  public:
+    UnitEventNotification(indexstore_unit_event_notification_t obj) : obj(obj) {}
+
+    bool isInitial() const { return indexstore_unit_event_notification_is_initial(obj); }
+    size_t getEventsCount() const { return indexstore_unit_event_notification_get_events_count(obj); }
+    UnitEvent getEvent(size_t index) const { return indexstore_unit_event_notification_get_event(obj, index); }
+  };
+
+  typedef std::function<void(UnitEventNotification)> UnitEventHandler;
+
+  void setUnitEventHandler(UnitEventHandler handler) {
+#if INDEXSTORE_HAS_BLOCKS
+    if (!handler) {
+      indexstore_store_set_unit_event_handler(obj, nullptr);
+      return;
+    }
+
+    indexstore_store_set_unit_event_handler(obj, ^(indexstore_unit_event_notification_t evt_note) {
+      handler(UnitEventNotification(evt_note));
+    });
+#else
+    if (!handler) {
+      indexstore_store_set_unit_event_handler_f(obj, nullptr, nullptr, nullptr);
+      return;
+    }
+
+    auto fnPtr = new UnitEventHandler(handler);
+    indexstore_store_set_unit_event_handler_f(obj, fnPtr, event_handler, event_handler_finalizer);
+#endif
+  }
+
+private:
+  static void event_handler(void *ctx, indexstore_unit_event_notification_t evt) {
+    auto fnPtr = (UnitEventHandler*)ctx;
+    (*fnPtr)(evt);
+  }
+  static void event_handler_finalizer(void *ctx) {
+    auto fnPtr = (UnitEventHandler*)ctx;
+    delete fnPtr;
+  }
+
+public:
+  bool startEventListening(bool waitInitialSync, std::string &error) {
+    indexstore_unit_event_listen_options_t opts;
+    opts.wait_initial_sync = waitInitialSync;
+    indexstore_error_t c_err = nullptr;
+    bool ret = indexstore_store_start_unit_event_listening(obj, &opts, sizeof(opts), &c_err);
+    if (c_err) {
+      error = indexstore_error_get_description(c_err);
+      indexstore_error_dispose(c_err);
+    }
+    return ret;
+  }
+
+  void stopEventListening() {
+    return indexstore_store_stop_unit_event_listening(obj);
+  }
+
+  void discardUnit(StringRef UnitName) {
+    llvm::SmallString<64> buf = UnitName;
+    indexstore_store_discard_unit(obj, buf.c_str());
+  }
+
+  void discardRecord(StringRef RecordName) {
+    llvm::SmallString<64> buf = RecordName;
+    indexstore_store_discard_record(obj, buf.c_str());
+  }
+
+  void getUnitNameFromOutputPath(StringRef outputPath, llvm::SmallVectorImpl<char> &nameBuf) {
+    llvm::SmallString<256> buf = outputPath;
+    llvm::SmallString<64> unitName;
+    unitName.resize(64);
+    size_t nameLen = indexstore_store_get_unit_name_from_output_path(obj, buf.c_str(), unitName.data(), unitName.size());
+    if (nameLen+1 > unitName.size()) {
+      unitName.resize(nameLen+1);
+      indexstore_store_get_unit_name_from_output_path(obj, buf.c_str(), unitName.data(), unitName.size());
+    }
+    nameBuf.append(unitName.begin(), unitName.begin()+nameLen);
+  }
+
+  void purgeStaleData() {
+    indexstore_store_purge_stale_data(obj);
+  }
+};
+
+class IndexRecordReader {
+  indexstore_record_reader_t obj;
+
+public:
+  IndexRecordReader(IndexStore &store, StringRef recordName, std::string &error) {
+    llvm::SmallString<64> buf = recordName;
+    indexstore_error_t c_err = nullptr;
+    obj = indexstore_record_reader_create(store.obj, buf.c_str(), &c_err);
+    if (c_err) {
+      error = indexstore_error_get_description(c_err);
+      indexstore_error_dispose(c_err);
+    }
+  }
+
+  IndexRecordReader(IndexRecordReader &&other) : obj(other.obj) {
+    other.obj = nullptr;
+  }
+
+  ~IndexRecordReader() {
+    indexstore_record_reader_dispose(obj);
+  }
+
+  bool isValid() const { return obj; }
+  bool isInvalid() const { return !isValid(); }
+  explicit operator bool() const { return isValid(); }
+
+  /// Goes through and passes record decls, after filtering using a \c Checker
+  /// function.
+  ///
+  /// Resulting decls can be used as filter for \c foreachOccurrence. This
+  /// allows allocating memory only for the record decls that the caller is
+  /// interested in.
+  bool searchSymbols(llvm::function_ref<bool(IndexRecordSymbol, bool &stop)> filter,
+                     llvm::function_ref<void(IndexRecordSymbol)> receiver) {
+#if INDEXSTORE_HAS_BLOCKS
+    return indexstore_record_reader_search_symbols(obj, ^bool(indexstore_symbol_t symbol, bool *stop) {
+      return filter(symbol, *stop);
+    }, ^(indexstore_symbol_t symbol) {
+      receiver(symbol);
+    });
+#else
+    return indexstore_record_reader_search_symbols_f(obj, &filter, functionPtrFromFunctionRef,
+                                                     &receiver, functionPtrFromFunctionRef);
+#endif
+  }
+
+  bool foreachSymbol(bool noCache, llvm::function_ref<bool(IndexRecordSymbol)> receiver) {
+#if INDEXSTORE_HAS_BLOCKS
+    return indexstore_record_reader_symbols_apply(obj, noCache, ^bool(indexstore_symbol_t sym) {
+      return receiver(sym);
+    });
+#else
+    return indexstore_record_reader_symbols_apply_f(obj, noCache, &receiver, functionPtrFromFunctionRef);
+#endif
+  }
+
+  /// \param DeclsFilter if non-empty indicates the list of decls that we want
+  /// to get occurrences for. An empty array indicates that we want occurrences
+  /// for all decls.
+  /// \param RelatedDeclsFilter Same as \c DeclsFilter but for related decls.
+  bool foreachOccurrence(ArrayRef<IndexRecordSymbol> symbolsFilter,
+                         ArrayRef<IndexRecordSymbol> relatedSymbolsFilter,
+              llvm::function_ref<bool(IndexRecordOccurrence)> receiver) {
+    llvm::SmallVector<indexstore_symbol_t, 16> c_symbolsFilter;
+    c_symbolsFilter.reserve(symbolsFilter.size());
+    for (IndexRecordSymbol sym : symbolsFilter) {
+      c_symbolsFilter.push_back(sym.obj);
+    }
+    llvm::SmallVector<indexstore_symbol_t, 16> c_relatedSymbolsFilter;
+    c_relatedSymbolsFilter.reserve(relatedSymbolsFilter.size());
+    for (IndexRecordSymbol sym : relatedSymbolsFilter) {
+      c_relatedSymbolsFilter.push_back(sym.obj);
+    }
+#if INDEXSTORE_HAS_BLOCKS
+    return indexstore_record_reader_occurrences_of_symbols_apply(obj,
+                                c_symbolsFilter.data(), c_symbolsFilter.size(),
+                                c_relatedSymbolsFilter.data(),
+                                c_relatedSymbolsFilter.size(),
+                                ^bool(indexstore_occurrence_t occur) {
+                                  return receiver(occur);
+                                });
+#else
+    return indexstore_record_reader_occurrences_of_symbols_apply_f(obj,
+                                c_symbolsFilter.data(), c_symbolsFilter.size(),
+                                c_relatedSymbolsFilter.data(),
+                                c_relatedSymbolsFilter.size(),
+                                &receiver, functionPtrFromFunctionRef);
+#endif
+  }
+
+  bool foreachOccurrence(
+              llvm::function_ref<bool(IndexRecordOccurrence)> receiver) {
+#if INDEXSTORE_HAS_BLOCKS
+    return indexstore_record_reader_occurrences_apply(obj, ^bool(indexstore_occurrence_t occur) {
+      return receiver(occur);
+    });
+#else
+    return indexstore_record_reader_occurrences_apply_f(obj, &receiver, functionPtrFromFunctionRef);
+#endif
+  }
+
+  bool foreachOccurrenceInLineRange(unsigned lineStart, unsigned lineEnd,
+              llvm::function_ref<bool(IndexRecordOccurrence)> receiver) {
+#if INDEXSTORE_HAS_BLOCKS
+    return indexstore_record_reader_occurrences_in_line_range_apply(obj,
+                                                                    lineStart,
+                                                                    lineEnd,
+                                          ^bool(indexstore_occurrence_t occur) {
+      return receiver(occur);
+    });
+#else
+    return indexstore_record_reader_occurrences_in_line_range_apply_f(obj,
+                                                                      lineStart,
+                                                                      lineEnd,
+                                         &receiver, functionPtrFromFunctionRef);
+#endif
+  }
+};
+
+class IndexUnitDependency {
+  indexstore_unit_dependency_t obj;
+  friend class IndexUnitReader;
+
+public:
+  IndexUnitDependency(indexstore_unit_dependency_t obj) : obj(obj) {}
+
+  enum class DependencyKind {
+    Unit,
+    Record,
+    File,
+  };
+  DependencyKind getKind() {
+    switch (indexstore_unit_dependency_get_kind(obj)) {
+    case INDEXSTORE_UNIT_DEPENDENCY_UNIT: return DependencyKind::Unit;
+    case INDEXSTORE_UNIT_DEPENDENCY_RECORD: return DependencyKind::Record;
+    case INDEXSTORE_UNIT_DEPENDENCY_FILE: return DependencyKind::File;
+    }
+  }
+  bool isSystem() { return indexstore_unit_dependency_is_system(obj); }
+  StringRef getName() { return stringFromIndexStoreStringRef(indexstore_unit_dependency_get_name(obj)); }
+  StringRef getFilePath() { return stringFromIndexStoreStringRef(indexstore_unit_dependency_get_filepath(obj)); }
+  StringRef getModuleName() { return stringFromIndexStoreStringRef(indexstore_unit_dependency_get_modulename(obj)); }
+
+};
+
+class IndexUnitInclude {
+  indexstore_unit_include_t obj;
+  friend class IndexUnitReader;
+
+public:
+  IndexUnitInclude(indexstore_unit_include_t obj) : obj(obj) {}
+
+  StringRef getSourcePath() {
+    return stringFromIndexStoreStringRef(indexstore_unit_include_get_source_path(obj));
+  }
+  StringRef getTargetPath() {
+    return stringFromIndexStoreStringRef(indexstore_unit_include_get_target_path(obj));
+  }
+  unsigned getSourceLine() {
+    return indexstore_unit_include_get_source_line(obj);
+  }
+};
+
+class IndexUnitReader {
+  indexstore_unit_reader_t obj;
+
+public:
+  IndexUnitReader(IndexStore &store, StringRef unitName, std::string &error) {
+    llvm::SmallString<64> buf = unitName;
+    indexstore_error_t c_err = nullptr;
+    obj = indexstore_unit_reader_create(store.obj, buf.c_str(), &c_err);
+    if (c_err) {
+      error = indexstore_error_get_description(c_err);
+      indexstore_error_dispose(c_err);
+    }
+  }
+
+  IndexUnitReader(IndexUnitReader &&other) : obj(other.obj) {
+    other.obj = nullptr;
+  }
+
+  ~IndexUnitReader() {
+    indexstore_unit_reader_dispose(obj);
+  }
+
+  bool isValid() const { return obj; }
+  bool isInvalid() const { return !isValid(); }
+  explicit operator bool() const { return isValid(); }
+
+  StringRef getProviderIdentifier() {
+    return stringFromIndexStoreStringRef(indexstore_unit_reader_get_provider_identifier(obj));
+  }
+  StringRef getProviderVersion() {
+    return stringFromIndexStoreStringRef(indexstore_unit_reader_get_provider_version(obj));
+  }
+
+  timespec getModificationTime() {
+    int64_t seconds, nanoseconds;
+    indexstore_unit_reader_get_modification_time(obj, &seconds, &nanoseconds);
+    timespec ts;
+    ts.tv_sec = seconds;
+    ts.tv_nsec = nanoseconds;
+    return ts;
+  }
+
+  bool isSystemUnit() { return indexstore_unit_reader_is_system_unit(obj); }
+  bool isModuleUnit() { return indexstore_unit_reader_is_module_unit(obj); }
+  bool isDebugCompilation() { return indexstore_unit_reader_is_debug_compilation(obj); }
+  bool hasMainFile() { return indexstore_unit_reader_has_main_file(obj); }
+
+  StringRef getMainFilePath() {
+    return stringFromIndexStoreStringRef(indexstore_unit_reader_get_main_file(obj));
+  }
+  StringRef getModuleName() {
+    return stringFromIndexStoreStringRef(indexstore_unit_reader_get_module_name(obj));
+  }
+  StringRef getWorkingDirectory() {
+    return stringFromIndexStoreStringRef(indexstore_unit_reader_get_working_dir(obj));
+  }
+  StringRef getOutputFile() {
+    return stringFromIndexStoreStringRef(indexstore_unit_reader_get_output_file(obj));
+  }
+  StringRef getSysrootPath() {
+    return stringFromIndexStoreStringRef(indexstore_unit_reader_get_sysroot_path(obj));
+  }
+  StringRef getTarget() {
+    return stringFromIndexStoreStringRef(indexstore_unit_reader_get_target(obj));
+  }
+
+  bool foreachDependency(llvm::function_ref<bool(IndexUnitDependency)> receiver) {
+#if INDEXSTORE_HAS_BLOCKS
+    return indexstore_unit_reader_dependencies_apply(obj, ^bool(indexstore_unit_dependency_t dep) {
+      return receiver(dep);
+    });
+#else
+    return indexstore_unit_reader_dependencies_apply_f(obj, &receiver, functionPtrFromFunctionRef);
+#endif
+  }
+
+  bool foreachInclude(llvm::function_ref<bool(IndexUnitInclude)> receiver) {
+#if INDEXSTORE_HAS_BLOCKS
+    return indexstore_unit_reader_includes_apply(obj, ^bool(indexstore_unit_include_t inc) {
+      return receiver(inc);
+    });
+#else
+    return indexstore_unit_reader_includes_apply_f(obj, &receiver, functionPtrFromFunctionRef);
+#endif
+  }
+};
+
+} // namespace indexstore
+
+#endif

clang/include/indexstore/indexstore.h

+/*===-- indexstore/indexstore.h - Index Store C API ----------------- C -*-===*\
+|*                                                                            *|
+|*                     The LLVM Compiler Infrastructure                       *|
+|*                                                                            *|
+|* This file is distributed under the University of Illinois Open Source      *|
+|* License. See LICENSE.TXT for details.                                      *|
+|*                                                                            *|
+|*===----------------------------------------------------------------------===*|
+|*                                                                            *|
+|* This header provides a C API for the index store.                          *|
+|*                                                                            *|
+\*===----------------------------------------------------------------------===*/
+
+#ifndef LLVM_CLANG_C_INDEXSTORE_INDEXSTORE_H
+#define LLVM_CLANG_C_INDEXSTORE_INDEXSTORE_H
+
+#include <stdbool.h>
+#include <stddef.h>
+#include <stdint.h>
+#include <time.h>
+
+/**
+ * \brief The version constants for the Index Store C API.
+ * INDEXSTORE_VERSION_MINOR should increase when there are API additions.
+ * INDEXSTORE_VERSION_MAJOR is intended for "major" source/ABI breaking changes.
+ */
+#define INDEXSTORE_VERSION_MAJOR 0
+#define INDEXSTORE_VERSION_MINOR 11
+
+#define INDEXSTORE_VERSION_ENCODE(major, minor) ( \
+      ((major) * 10000)                           \
+    + ((minor) *     1))
+
+#define INDEXSTORE_VERSION INDEXSTORE_VERSION_ENCODE( \
+    INDEXSTORE_VERSION_MAJOR,                         \
+    INDEXSTORE_VERSION_MINOR )
+
+#define INDEXSTORE_VERSION_STRINGIZE_(major, minor)   \
+    #major"."#minor
+#define INDEXSTORE_VERSION_STRINGIZE(major, minor)    \
+    INDEXSTORE_VERSION_STRINGIZE_(major, minor)
+
+#define INDEXSTORE_VERSION_STRING INDEXSTORE_VERSION_STRINGIZE( \
+    INDEXSTORE_VERSION_MAJOR,                                   \
+    INDEXSTORE_VERSION_MINOR)
+
+#ifdef  __cplusplus
+# define INDEXSTORE_BEGIN_DECLS  extern "C" {
+# define INDEXSTORE_END_DECLS    }
+#else
+# define INDEXSTORE_BEGIN_DECLS
+# define INDEXSTORE_END_DECLS
+#endif
+
+#ifndef INDEXSTORE_PUBLIC
+# ifdef _WIN32
+#  ifdef IndexStore_EXPORTS
+#    define INDEXSTORE_PUBLIC __declspec(dllexport)
+#  else
+#    define INDEXSTORE_PUBLIC __declspec(dllimport)
+#  endif
+# else
+#  define INDEXSTORE_PUBLIC
+# endif
+#endif
+
+#ifndef __has_feature
+# define __has_feature(x) 0
+#endif
+
+#ifndef __has_attribute
+# define __has_attribute(x) 0
+#endif
+
+#if __has_feature(blocks)
+# define INDEXSTORE_HAS_BLOCKS 1
+#else
+# define INDEXSTORE_HAS_BLOCKS 0
+#endif
+
+#if __has_attribute(noescape)
+# define INDEXSTORE_NOESCAPE __attribute__((noescape))
+#else
+# define INDEXSTORE_NOESCAPE
+#endif
+
+#if __has_attribute(flag_enum)
+# define INDEXSTORE_FLAG_ENUM_ATTR __attribute__((flag_enum))
+#else
+# define INDEXSTORE_FLAG_ENUM_ATTR
+#endif
+
+#if __has_attribute(enum_extensibility)
+# define INDEXSTORE_OPEN_ENUM_ATTR __attribute__((enum_extensibility(open)))
+#else
+# define INDEXSTORE_OPEN_ENUM_ATTR
+#endif
+
+#define INDEXSTORE_OPTIONS_ATTRS INDEXSTORE_OPEN_ENUM_ATTR INDEXSTORE_FLAG_ENUM_ATTR
+
+#if __has_feature(objc_fixed_enum)
+#ifdef __cplusplus
+# define INDEXSTORE_OPTIONS(_type, _name) enum INDEXSTORE_OPTIONS_ATTRS _name : _type
+#else
+# define INDEXSTORE_OPTIONS(_type, _name) enum _name : _type; typedef enum _name _name; enum INDEXSTORE_OPTIONS_ATTRS _name : _type
+#endif
+#endif
+
+#ifndef INDEXSTORE_OPTIONS
+# define INDEXSTORE_OPTIONS(_type, _name) typedef _type _name; enum INDEXSTORE_OPTIONS_ATTRS
+#endif
+
+INDEXSTORE_BEGIN_DECLS
+
+typedef void *indexstore_error_t;
+
+INDEXSTORE_PUBLIC const char *
+indexstore_error_get_description(indexstore_error_t);
+
+INDEXSTORE_PUBLIC void
+indexstore_error_dispose(indexstore_error_t);
+
+typedef struct {
+  const char *data;
+  size_t length;
+} indexstore_string_ref_t;
+
+INDEXSTORE_PUBLIC unsigned
+indexstore_format_version(void);
+
+typedef void *indexstore_t;
+
+INDEXSTORE_PUBLIC indexstore_t
+indexstore_store_create(const char *store_path, indexstore_error_t *error);
+
+INDEXSTORE_PUBLIC void
+indexstore_store_dispose(indexstore_t);
+
+#if INDEXSTORE_HAS_BLOCKS
+INDEXSTORE_PUBLIC bool
+indexstore_store_units_apply(indexstore_t, unsigned sorted,
+                             INDEXSTORE_NOESCAPE bool(^applier)(indexstore_string_ref_t unit_name));
+#endif
+
+INDEXSTORE_PUBLIC bool
+indexstore_store_units_apply_f(indexstore_t, unsigned sorted,
+                               void *context,
+              INDEXSTORE_NOESCAPE bool(*applier)(void *context, indexstore_string_ref_t unit_name));
+
+typedef void *indexstore_unit_event_notification_t;
+typedef void *indexstore_unit_event_t;
+
+INDEXSTORE_PUBLIC size_t
+indexstore_unit_event_notification_get_events_count(indexstore_unit_event_notification_t);
+
+INDEXSTORE_PUBLIC indexstore_unit_event_t
+indexstore_unit_event_notification_get_event(indexstore_unit_event_notification_t, size_t index);
+
+INDEXSTORE_PUBLIC bool
+indexstore_unit_event_notification_is_initial(indexstore_unit_event_notification_t);
+
+typedef enum {
+  INDEXSTORE_UNIT_EVENT_REMOVED = 2,
+  INDEXSTORE_UNIT_EVENT_MODIFIED = 3,
+  INDEXSTORE_UNIT_EVENT_DIRECTORY_DELETED = 4,
+  INDEXSTORE_UNIT_EVENT_FAILURE = 5,
+} indexstore_unit_event_kind_t;
+
+INDEXSTORE_PUBLIC indexstore_unit_event_kind_t
+indexstore_unit_event_get_kind(indexstore_unit_event_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_unit_event_get_unit_name(indexstore_unit_event_t);
+
+#if INDEXSTORE_HAS_BLOCKS
+typedef void (^indexstore_unit_event_handler_t)(indexstore_unit_event_notification_t);
+
+INDEXSTORE_PUBLIC void
+indexstore_store_set_unit_event_handler(indexstore_t,
+                                        indexstore_unit_event_handler_t handler);
+#endif
+
+INDEXSTORE_PUBLIC void
+indexstore_store_set_unit_event_handler_f(indexstore_t, void *context,
+            void(*handler)(void *context, indexstore_unit_event_notification_t),
+                                          void(*finalizer)(void *context));
+
+typedef struct {
+  /// If true, \c indexstore_store_start_unit_event_listening will block until
+  /// the initial set of units is passed to the unit event handler, otherwise
+  /// the function will return and the initial set will be passed asynchronously.
+  bool wait_initial_sync;
+} indexstore_unit_event_listen_options_t;
+
+INDEXSTORE_PUBLIC bool
+indexstore_store_start_unit_event_listening(indexstore_t,
+                                            indexstore_unit_event_listen_options_t *,
+                                            size_t listen_options_struct_size,
+                                            indexstore_error_t *error);
+
+INDEXSTORE_PUBLIC void
+indexstore_store_stop_unit_event_listening(indexstore_t);
+
+INDEXSTORE_PUBLIC void
+indexstore_store_discard_unit(indexstore_t, const char *unit_name);
+
+INDEXSTORE_PUBLIC void
+indexstore_store_discard_record(indexstore_t, const char *record_name);
+
+INDEXSTORE_PUBLIC void
+indexstore_store_purge_stale_data(indexstore_t);
+
+/// Determines the unit name from the \c output_path and writes it out in the
+/// \c name_buf buffer. It doesn't write more than \c buf_size.
+/// \returns the length of the name. If this is larger than \c buf_size, the
+/// caller should call the function again with a buffer of the appropriate size.
+INDEXSTORE_PUBLIC size_t
+indexstore_store_get_unit_name_from_output_path(indexstore_t store,
+                                                const char *output_path,
+                                                char *name_buf,
+                                                size_t buf_size);
+
+/// \returns true if an error occurred, false otherwise.
+INDEXSTORE_PUBLIC bool
+indexstore_store_get_unit_modification_time(indexstore_t store,
+                                            const char *unit_name,
+                                            int64_t *seconds,
+                                            int64_t *nanoseconds,
+                                            indexstore_error_t *error);
+
+typedef void *indexstore_symbol_t;
+
+typedef enum {
+  INDEXSTORE_SYMBOL_KIND_UNKNOWN = 0,
+  INDEXSTORE_SYMBOL_KIND_MODULE = 1,
+  INDEXSTORE_SYMBOL_KIND_NAMESPACE = 2,
+  INDEXSTORE_SYMBOL_KIND_NAMESPACEALIAS = 3,
+  INDEXSTORE_SYMBOL_KIND_MACRO = 4,
+  INDEXSTORE_SYMBOL_KIND_ENUM = 5,
+  INDEXSTORE_SYMBOL_KIND_STRUCT = 6,
+  INDEXSTORE_SYMBOL_KIND_CLASS = 7,
+  INDEXSTORE_SYMBOL_KIND_PROTOCOL = 8,
+  INDEXSTORE_SYMBOL_KIND_EXTENSION = 9,
+  INDEXSTORE_SYMBOL_KIND_UNION = 10,
+  INDEXSTORE_SYMBOL_KIND_TYPEALIAS = 11,
+  INDEXSTORE_SYMBOL_KIND_FUNCTION = 12,
+  INDEXSTORE_SYMBOL_KIND_VARIABLE = 13,
+  INDEXSTORE_SYMBOL_KIND_FIELD = 14,
+  INDEXSTORE_SYMBOL_KIND_ENUMCONSTANT = 15,
+  INDEXSTORE_SYMBOL_KIND_INSTANCEMETHOD = 16,
+  INDEXSTORE_SYMBOL_KIND_CLASSMETHOD = 17,
+  INDEXSTORE_SYMBOL_KIND_STATICMETHOD = 18,
+  INDEXSTORE_SYMBOL_KIND_INSTANCEPROPERTY = 19,
+  INDEXSTORE_SYMBOL_KIND_CLASSPROPERTY = 20,
+  INDEXSTORE_SYMBOL_KIND_STATICPROPERTY = 21,
+  INDEXSTORE_SYMBOL_KIND_CONSTRUCTOR = 22,
+  INDEXSTORE_SYMBOL_KIND_DESTRUCTOR = 23,
+  INDEXSTORE_SYMBOL_KIND_CONVERSIONFUNCTION = 24,
+  INDEXSTORE_SYMBOL_KIND_PARAMETER = 25,
+  INDEXSTORE_SYMBOL_KIND_USING = 26,
+
+  INDEXSTORE_SYMBOL_KIND_COMMENTTAG = 1000,
+} indexstore_symbol_kind_t;
+
+typedef enum {
+  INDEXSTORE_SYMBOL_SUBKIND_NONE = 0,
+  INDEXSTORE_SYMBOL_SUBKIND_CXXCOPYCONSTRUCTOR = 1,
+  INDEXSTORE_SYMBOL_SUBKIND_CXXMOVECONSTRUCTOR = 2,
+  INDEXSTORE_SYMBOL_SUBKIND_ACCESSORGETTER = 3,
+  INDEXSTORE_SYMBOL_SUBKIND_ACCESSORSETTER = 4,
+  INDEXSTORE_SYMBOL_SUBKIND_USINGTYPENAME = 5,
+  INDEXSTORE_SYMBOL_SUBKIND_USINGVALUE = 6,
+
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTACCESSORWILLSET = 1000,
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTACCESSORDIDSET = 1001,
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTACCESSORADDRESSOR = 1002,
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTACCESSORMUTABLEADDRESSOR = 1003,
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTEXTENSIONOFSTRUCT = 1004,
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTEXTENSIONOFCLASS = 1005,
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTEXTENSIONOFENUM = 1006,
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTEXTENSIONOFPROTOCOL = 1007,
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTPREFIXOPERATOR = 1008,
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTPOSTFIXOPERATOR = 1009,
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTINFIXOPERATOR = 1010,
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTSUBSCRIPT = 1011,
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTASSOCIATEDTYPE = 1012,
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTGENERICTYPEPARAM = 1013,
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTACCESSORREAD = 1014,
+  INDEXSTORE_SYMBOL_SUBKIND_SWIFTACCESSORMODIFY = 1015,
+} indexstore_symbol_subkind_t;
+
+INDEXSTORE_OPTIONS(uint64_t, indexstore_symbol_property_t) {
+  INDEXSTORE_SYMBOL_PROPERTY_GENERIC                          = 1 << 0,
+  INDEXSTORE_SYMBOL_PROPERTY_TEMPLATE_PARTIAL_SPECIALIZATION  = 1 << 1,
+  INDEXSTORE_SYMBOL_PROPERTY_TEMPLATE_SPECIALIZATION          = 1 << 2,
+  INDEXSTORE_SYMBOL_PROPERTY_UNITTEST                         = 1 << 3,
+  INDEXSTORE_SYMBOL_PROPERTY_IBANNOTATED                      = 1 << 4,
+  INDEXSTORE_SYMBOL_PROPERTY_IBOUTLETCOLLECTION               = 1 << 5,
+  INDEXSTORE_SYMBOL_PROPERTY_GKINSPECTABLE                    = 1 << 6,
+  INDEXSTORE_SYMBOL_PROPERTY_LOCAL                            = 1 << 7,
+  INDEXSTORE_SYMBOL_PROPERTY_PROTOCOL_INTERFACE               = 1 << 8,
+};
+
+typedef enum {
+  INDEXSTORE_SYMBOL_LANG_C = 0,
+  INDEXSTORE_SYMBOL_LANG_OBJC = 1,
+  INDEXSTORE_SYMBOL_LANG_CXX = 2,
+
+  INDEXSTORE_SYMBOL_LANG_SWIFT = 100,
+} indexstore_symbol_language_t;
+
+INDEXSTORE_OPTIONS(uint64_t, indexstore_symbol_role_t) {
+  INDEXSTORE_SYMBOL_ROLE_DECLARATION  = 1 << 0,
+  INDEXSTORE_SYMBOL_ROLE_DEFINITION   = 1 << 1,
+  INDEXSTORE_SYMBOL_ROLE_REFERENCE    = 1 << 2,
+  INDEXSTORE_SYMBOL_ROLE_READ         = 1 << 3,
+  INDEXSTORE_SYMBOL_ROLE_WRITE        = 1 << 4,
+  INDEXSTORE_SYMBOL_ROLE_CALL         = 1 << 5,
+  INDEXSTORE_SYMBOL_ROLE_DYNAMIC      = 1 << 6,
+  INDEXSTORE_SYMBOL_ROLE_ADDRESSOF    = 1 << 7,
+  INDEXSTORE_SYMBOL_ROLE_IMPLICIT     = 1 << 8,
+  INDEXSTORE_SYMBOL_ROLE_UNDEFINITION = 1 << 19,
+  INDEXSTORE_SYMBOL_ROLE_NAMEREFERENCE = 1 << 20,
+
+  // Relation roles.
+  INDEXSTORE_SYMBOL_ROLE_REL_CHILDOF     = 1 << 9,
+  INDEXSTORE_SYMBOL_ROLE_REL_BASEOF      = 1 << 10,
+  INDEXSTORE_SYMBOL_ROLE_REL_OVERRIDEOF  = 1 << 11,
+  INDEXSTORE_SYMBOL_ROLE_REL_RECEIVEDBY  = 1 << 12,
+  INDEXSTORE_SYMBOL_ROLE_REL_CALLEDBY    = 1 << 13,
+  INDEXSTORE_SYMBOL_ROLE_REL_EXTENDEDBY  = 1 << 14,
+  INDEXSTORE_SYMBOL_ROLE_REL_ACCESSOROF  = 1 << 15,
+  INDEXSTORE_SYMBOL_ROLE_REL_CONTAINEDBY = 1 << 16,
+  INDEXSTORE_SYMBOL_ROLE_REL_IBTYPEOF    = 1 << 17,
+  INDEXSTORE_SYMBOL_ROLE_REL_SPECIALIZATIONOF = 1 << 18,
+};
+
+INDEXSTORE_PUBLIC indexstore_symbol_language_t
+indexstore_symbol_get_language(indexstore_symbol_t);
+
+INDEXSTORE_PUBLIC indexstore_symbol_kind_t
+indexstore_symbol_get_kind(indexstore_symbol_t);
+
+INDEXSTORE_PUBLIC indexstore_symbol_subkind_t
+indexstore_symbol_get_subkind(indexstore_symbol_t);
+
+INDEXSTORE_PUBLIC indexstore_symbol_property_t
+indexstore_symbol_get_properties(indexstore_symbol_t);
+
+INDEXSTORE_PUBLIC indexstore_symbol_role_t
+indexstore_symbol_get_roles(indexstore_symbol_t);
+
+INDEXSTORE_PUBLIC indexstore_symbol_role_t
+indexstore_symbol_get_related_roles(indexstore_symbol_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_symbol_get_name(indexstore_symbol_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_symbol_get_usr(indexstore_symbol_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_symbol_get_codegen_name(indexstore_symbol_t);
+
+typedef void *indexstore_symbol_relation_t;
+
+INDEXSTORE_PUBLIC indexstore_symbol_role_t
+indexstore_symbol_relation_get_roles(indexstore_symbol_relation_t);
+
+INDEXSTORE_PUBLIC indexstore_symbol_t
+indexstore_symbol_relation_get_symbol(indexstore_symbol_relation_t);
+
+typedef void *indexstore_occurrence_t;
+
+INDEXSTORE_PUBLIC indexstore_symbol_t
+indexstore_occurrence_get_symbol(indexstore_occurrence_t);
+
+#if INDEXSTORE_HAS_BLOCKS
+INDEXSTORE_PUBLIC bool
+indexstore_occurrence_relations_apply(indexstore_occurrence_t,
+                      INDEXSTORE_NOESCAPE bool(^applier)(indexstore_symbol_relation_t symbol_rel));
+#endif
+
+INDEXSTORE_PUBLIC bool
+indexstore_occurrence_relations_apply_f(indexstore_occurrence_t,
+                                        void *context,
+        INDEXSTORE_NOESCAPE bool(*applier)(void *context, indexstore_symbol_relation_t symbol_rel));
+
+INDEXSTORE_PUBLIC indexstore_symbol_role_t
+indexstore_occurrence_get_roles(indexstore_occurrence_t);
+
+INDEXSTORE_PUBLIC void
+indexstore_occurrence_get_line_col(indexstore_occurrence_t,
+                              unsigned *line, unsigned *column);
+
+typedef void *indexstore_record_reader_t;
+
+INDEXSTORE_PUBLIC indexstore_record_reader_t
+indexstore_record_reader_create(indexstore_t store, const char *record_name,
+                                indexstore_error_t *error);
+
+INDEXSTORE_PUBLIC void
+indexstore_record_reader_dispose(indexstore_record_reader_t);
+
+#if INDEXSTORE_HAS_BLOCKS
+/// Goes through the symbol data and passes symbols to \c receiver, for the
+/// symbol data that \c filter returns true on.
+///
+/// This allows allocating memory only for the record symbols that the caller is
+/// interested in.
+INDEXSTORE_PUBLIC bool
+indexstore_record_reader_search_symbols(indexstore_record_reader_t,
+    INDEXSTORE_NOESCAPE bool(^filter)(indexstore_symbol_t symbol, bool *stop),
+    INDEXSTORE_NOESCAPE void(^receiver)(indexstore_symbol_t symbol));
+
+/// \param nocache if true, avoids allocating memory for the symbols.
+/// Useful when the caller does not intend to keep \c indexstore_record_reader_t
+/// for more queries.
+INDEXSTORE_PUBLIC bool
+indexstore_record_reader_symbols_apply(indexstore_record_reader_t,
+                                       bool nocache,
+                                    INDEXSTORE_NOESCAPE bool(^applier)(indexstore_symbol_t symbol));
+
+INDEXSTORE_PUBLIC bool
+indexstore_record_reader_occurrences_apply(indexstore_record_reader_t,
+                                 INDEXSTORE_NOESCAPE bool(^applier)(indexstore_occurrence_t occur));
+
+INDEXSTORE_PUBLIC bool
+indexstore_record_reader_occurrences_in_line_range_apply(indexstore_record_reader_t,
+                                                         unsigned line_start,
+                                                         unsigned line_count,
+                                 INDEXSTORE_NOESCAPE bool(^applier)(indexstore_occurrence_t occur));
+
+/// \param symbols if non-zero \c symbols_count, indicates the list of symbols
+/// that we want to get occurrences for. An empty array indicates that we want
+/// occurrences for all symbols.
+/// \param related_symbols Same as \c symbols but for related symbols.
+INDEXSTORE_PUBLIC bool
+indexstore_record_reader_occurrences_of_symbols_apply(indexstore_record_reader_t,
+        indexstore_symbol_t *symbols, size_t symbols_count,
+        indexstore_symbol_t *related_symbols, size_t related_symbols_count,
+        INDEXSTORE_NOESCAPE bool(^applier)(indexstore_occurrence_t occur));
+#endif
+
+INDEXSTORE_PUBLIC bool
+indexstore_record_reader_search_symbols_f(indexstore_record_reader_t,
+                                          void *filter_ctx,
+    INDEXSTORE_NOESCAPE bool(*filter)(void *filter_ctx, indexstore_symbol_t symbol, bool *stop),
+                                          void *receiver_ctx,
+    INDEXSTORE_NOESCAPE void(*receiver)(void *receiver_ctx, indexstore_symbol_t symbol));
+
+INDEXSTORE_PUBLIC bool
+indexstore_record_reader_symbols_apply_f(indexstore_record_reader_t,
+                                         bool nocache,
+                                         void *context,
+                     INDEXSTORE_NOESCAPE bool(*applier)(void *context, indexstore_symbol_t symbol));
+
+INDEXSTORE_PUBLIC bool
+indexstore_record_reader_occurrences_apply_f(indexstore_record_reader_t,
+                                             void *context,
+                  INDEXSTORE_NOESCAPE bool(*applier)(void *context, indexstore_occurrence_t occur));
+
+INDEXSTORE_PUBLIC bool
+indexstore_record_reader_occurrences_in_line_range_apply_f(indexstore_record_reader_t,
+                                                           unsigned line_start,
+                                                           unsigned line_count,
+                                                           void *context,
+                  INDEXSTORE_NOESCAPE bool(*applier)(void *context, indexstore_occurrence_t occur));
+
+INDEXSTORE_PUBLIC bool
+indexstore_record_reader_occurrences_of_symbols_apply_f(indexstore_record_reader_t,
+        indexstore_symbol_t *symbols, size_t symbols_count,
+        indexstore_symbol_t *related_symbols, size_t related_symbols_count,
+        void *context,
+        INDEXSTORE_NOESCAPE bool(*applier)(void *context, indexstore_occurrence_t occur));
+
+typedef void *indexstore_unit_reader_t;
+
+INDEXSTORE_PUBLIC indexstore_unit_reader_t
+indexstore_unit_reader_create(indexstore_t store, const char *unit_name,
+                              indexstore_error_t *error);
+
+INDEXSTORE_PUBLIC void
+indexstore_unit_reader_dispose(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_unit_reader_get_provider_identifier(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_unit_reader_get_provider_version(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC void
+indexstore_unit_reader_get_modification_time(indexstore_unit_reader_t,
+                                             int64_t *seconds,
+                                             int64_t *nanoseconds);
+
+INDEXSTORE_PUBLIC bool
+indexstore_unit_reader_is_system_unit(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC bool
+indexstore_unit_reader_is_module_unit(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC bool
+indexstore_unit_reader_is_debug_compilation(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC bool
+indexstore_unit_reader_has_main_file(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_unit_reader_get_main_file(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_unit_reader_get_module_name(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_unit_reader_get_working_dir(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_unit_reader_get_output_file(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_unit_reader_get_sysroot_path(indexstore_unit_reader_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_unit_reader_get_target(indexstore_unit_reader_t);
+
+typedef void *indexstore_unit_dependency_t;
+typedef void *indexstore_unit_include_t;
+
+typedef enum {
+  INDEXSTORE_UNIT_DEPENDENCY_UNIT = 1,
+  INDEXSTORE_UNIT_DEPENDENCY_RECORD = 2,
+  INDEXSTORE_UNIT_DEPENDENCY_FILE = 3,
+} indexstore_unit_dependency_kind_t;
+
+INDEXSTORE_PUBLIC indexstore_unit_dependency_kind_t
+indexstore_unit_dependency_get_kind(indexstore_unit_dependency_t);
+
+INDEXSTORE_PUBLIC bool
+indexstore_unit_dependency_is_system(indexstore_unit_dependency_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_unit_dependency_get_filepath(indexstore_unit_dependency_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_unit_dependency_get_modulename(indexstore_unit_dependency_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_unit_dependency_get_name(indexstore_unit_dependency_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_unit_include_get_source_path(indexstore_unit_include_t);
+
+INDEXSTORE_PUBLIC indexstore_string_ref_t
+indexstore_unit_include_get_target_path(indexstore_unit_include_t);
+
+INDEXSTORE_PUBLIC unsigned
+indexstore_unit_include_get_source_line(indexstore_unit_include_t);
+
+#if INDEXSTORE_HAS_BLOCKS
+INDEXSTORE_PUBLIC bool
+indexstore_unit_reader_dependencies_apply(indexstore_unit_reader_t,
+                             INDEXSTORE_NOESCAPE bool(^applier)(indexstore_unit_dependency_t));
+
+INDEXSTORE_PUBLIC bool
+indexstore_unit_reader_includes_apply(indexstore_unit_reader_t,
+                             INDEXSTORE_NOESCAPE bool(^applier)(indexstore_unit_include_t));
+#endif
+
+INDEXSTORE_PUBLIC bool
+indexstore_unit_reader_dependencies_apply_f(indexstore_unit_reader_t,
+                                            void *context,
+                   INDEXSTORE_NOESCAPE bool(*applier)(void *context, indexstore_unit_dependency_t));
+
+INDEXSTORE_PUBLIC bool
+indexstore_unit_reader_includes_apply_f(indexstore_unit_reader_t,
+                                        void *context,
+                      INDEXSTORE_NOESCAPE bool(*applier)(void *context, indexstore_unit_include_t));
+
+INDEXSTORE_END_DECLS
+
+#endif

clang/lib/CMakeLists.txt

 endif()
 add_subdirectory(Driver)
 add_subdirectory(Serialization)
 add_subdirectory(Frontend)
 add_subdirectory(FrontendTool)
 add_subdirectory(Tooling)
 add_subdirectory(DirectoryWatcher)
 add_subdirectory(Index)
+add_subdirectory(IndexDataStore)
 add_subdirectory(IndexSerialization)
 if(CLANG_ENABLE_STATIC_ANALYZER)
   add_subdirectory(StaticAnalyzer)
 endif()
 add_subdirectory(Format)
 add_subdirectory(Testing)

clang/lib/Driver/Driver.cpp

if (StringRef(TempFile).endswith(".cache")) {
       // In some cases (modules) we'll dump extra data to help with reproducing
       // the crash into a directory next to the output.
       VFS = llvm::sys::path::filename(TempFile);
       llvm::sys::path::append(VFS, "vfs", "vfs.yaml");
     }
   }
 
   // Assume associated files are based off of the first temporary file.
-  CrashReportInfo CrashInfo(TempFiles[0], VFS);
+  CrashReportInfo CrashInfo(
+      TempFiles[0], VFS,
+      C.getArgs().getLastArgValue(options::OPT_index_store_path));
 
   llvm::SmallString<128> Script(CrashInfo.Filename);
   llvm::sys::path::replace_extension(Script, "sh");
   std::error_code EC;
   llvm::raw_fd_ostream ScriptOS(Script, EC, llvm::sys::fs::CD_CreateNew);
   if (EC) {
     Diag(clang::diag::note_drv_command_failed_diag_msg)
         << "Error generating run script: " << Script << " " << EC.message();

clang/lib/Driver/Job.cpp

IsInclude = llvm::StringSwitch<bool>(Flag)
     .Cases("-idirafter", "-internal-isystem", "-iwithprefix", true)
     .Cases("-internal-externc-isystem", "-iprefix", true)
     .Cases("-iwithprefixbefore", "-isystem", "-iquote", true)
     .Cases("-isysroot", "-I", "-F", "-resource-dir", true)
     .Cases("-iframework", "-include-pch", true)
     .Default(false);
   if (IsInclude)
     return !HaveCrashVFS;
+  if (StringRef(Flag).startswith("-index-store-path"))
+    return true;
 
   // The remaining flags are treated as a single argument.
 
   // These flags are all of the form -Flag and have no second argument.
   ShouldSkip = llvm::StringSwitch<bool>(Flag)
     .Cases("-M", "-MM", "-MG", "-MP", "-MD", true)
     .Case("-MMD", true)
     .Default(false);
void Command::Print(raw_ostream &OS, const char *Terminator, bool Quote,
   ArrayRef<const char *> Args = Arguments;
   SmallVector<const char *, 128> ArgsRespFile;
   if (ResponseFile != nullptr) {
     buildArgvForResponseFile(ArgsRespFile);
     Args = ArrayRef<const char *>(ArgsRespFile).slice(1); // no executable name
   }
 
   bool HaveCrashVFS = CrashInfo && !CrashInfo->VFSPath.empty();
+  bool HaveIndexStorePath = CrashInfo && !CrashInfo->IndexStorePath.empty();
   for (size_t i = 0, e = Args.size(); i < e; ++i) {
     const char *const Arg = Args[i];
 
     if (CrashInfo) {
       int NumArgs = 0;
       bool IsInclude = false;
       if (skipArgs(Arg, HaveCrashVFS, NumArgs, IsInclude)) {
         i += NumArgs - 1;
if (CrashInfo && HaveCrashVFS) {
 
     std::string ModCachePath = "-fmodules-cache-path=";
     ModCachePath.append(RelModCacheDir.c_str());
 
     OS << ' ';
     llvm::sys::printArg(OS, ModCachePath, Quote);
   }
 
+  if (CrashInfo && HaveIndexStorePath) {
+    SmallString<128> IndexStoreDir;
+
+    if (HaveCrashVFS) {
+      IndexStoreDir = llvm::sys::path::parent_path(
+          llvm::sys::path::parent_path(CrashInfo->VFSPath));
+      llvm::sys::path::append(IndexStoreDir, "index-store");
+    } else {
+      IndexStoreDir = "index-store";
+    }
+
+    OS << ' ';
+    llvm::sys::printArg(OS, "-index-store-path", Quote);
+    OS << ' ';
+    llvm::sys::printArg(OS, IndexStoreDir.c_str(), Quote);
+  }
+
   if (ResponseFile != nullptr) {
     OS << "\n Arguments passed via response file:\n";
     writeResponseFile(OS);
     // Avoiding duplicated newline terminator, since FileLists are
     // newline-separated.
     if (ResponseSupport.ResponseKind != ResponseFileSupport::RF_FileList)
       OS << "\n";
     OS << " (end of response file)";

clang/lib/Driver/ToolChains/Clang.cpp

#endif
   // Pass the path to compiler resource files.
   CmdArgs.push_back("-resource-dir");
   CmdArgs.push_back(D.ResourceDir.c_str());
 
   Args.AddLastArg(CmdArgs, options::OPT_working_directory);
 
   RenderARCMigrateToolOptions(D, Args, CmdArgs);
 
+  if (Args.hasArg(options::OPT_index_store_path)) {
+    Args.AddLastArg(CmdArgs, options::OPT_index_store_path);
+    Args.AddLastArg(CmdArgs, options::OPT_index_ignore_system_symbols);
+    Args.AddLastArg(CmdArgs, options::OPT_index_record_mangled_name);
+
+    // If '-o' is passed along with '-fsyntax-only' pass it along the cc1
+    // invocation so that the index action knows what the out file is.
+    if (isa<CompileJobAction>(JA) && JA.getType() == types::TY_Nothing) {
+      Args.AddLastArg(CmdArgs, options::OPT_o);
+    }
+  }
+
+  if (const char *IdxStorePath = ::getenv("CLANG_PROJECT_INDEX_PATH")) {
+    CmdArgs.push_back("-index-store-path");
+    CmdArgs.push_back(IdxStorePath);
+    CmdArgs.push_back("-index-ignore-system-symbols");
+    CmdArgs.push_back("-index-record-mangled-name");
+  }
+
   // Add preprocessing options like -I, -D, etc. if we are using the
   // preprocessor.
   //
   // FIXME: Support -fpreprocessed
   if (types::getPreprocessedType(InputType) != types::TY_INVALID)
     AddPreprocessingOptions(C, JA, D, Args, CmdArgs, Output, Inputs);
 
   // Don't warn about "clang -c -DPIC -fPIC test.i" because libtool.m4 assumes

clang/lib/Frontend/CMakeLists.txt

add_clang_library(clangFrontend
   ClangDriverOptions
   ${optional_deps}
 
   LINK_LIBS
   clangAST
   clangBasic
   clangDriver
   clangEdit
+  clangIndexSerialization
   clangLex
   clangParse
   clangSema
   clangSerialization
   )

clang/lib/Frontend/CompilerInstance.cpp

 #include "clang/Frontend/FrontendAction.h"
 #include "clang/Frontend/FrontendActions.h"
 #include "clang/Frontend/FrontendDiagnostic.h"
 #include "clang/Frontend/LogDiagnosticPrinter.h"
 #include "clang/Frontend/SerializedDiagnosticPrinter.h"
 #include "clang/Frontend/TextDiagnosticPrinter.h"
 #include "clang/Frontend/Utils.h"
 #include "clang/Frontend/VerifyDiagnosticConsumer.h"
+#include "clang/Index/IndexingOptions.h"
+#include "clang/IndexSerialization/EmitIndex.h"
 #include "clang/Lex/HeaderSearch.h"
 #include "clang/Lex/Preprocessor.h"
 #include "clang/Lex/PreprocessorOptions.h"
 #include "clang/Sema/CodeCompleteConsumer.h"
 #include "clang/Sema/Sema.h"
 #include "clang/Serialization/ASTReader.h"
 #include "clang/Serialization/GlobalModuleIndex.h"
 #include "clang/Serialization/InMemoryModuleCache.h"
compileModuleImpl(CompilerInstance &ImportingInstance, SourceLocation ImportLoc,
 
   PreBuildStep(Instance);
 
   // Execute the action to actually build the module in-place. Use a separate
   // thread so that we get a stack large enough.
   llvm::CrashRecoveryContext CRC;
   CRC.RunSafelyOnThread(
       [&]() {
-        GenerateModuleFromModuleMapAction Action;
-        Instance.ExecuteAction(Action);
+        std::unique_ptr<FrontendAction> Action(
+            new GenerateModuleFromModuleMapAction);
+
+        const FrontendOptions &FEOpts = ImportingInstance.getFrontendOpts();
+        if (!FEOpts.IndexStorePath.empty()) {
+          Action = std::make_unique<ASTConsumerInjector>(
+              std::move(Action), [](CompilerInstance &CI, StringRef) {
+                const FrontendOptions &FEOpts = CI.getFrontendOpts();
+                return std::make_unique<index::IndexRecordingASTConsumer>(
+                    index::parseIndexGenerationOptions(FEOpts), CI,
+                    index::parseEmitIndexOptions(FEOpts));
+              });
+        }
+        Instance.ExecuteAction(*Action);
       },
       DesiredStackSize);
 
   PostBuildStep(Instance);
 
   ImportingInstance.getDiagnostics().Report(ImportLoc,
                                             diag::remark_module_build_done)
     << ModuleName;

clang/lib/Frontend/CompilerInvocation.cpp

Opts.ObjCMTWhiteListPath =
       std::string(Args.getLastArgValue(OPT_objcmt_whitelist_dir_path));
 
   if (Opts.ARCMTAction != FrontendOptions::ARCMT_None &&
       Opts.ObjCMTAction != FrontendOptions::ObjCMT_None) {
     Diags.Report(diag::err_drv_argument_not_allowed_with)
       << "ARC migration" << "ObjC migration";
   }
 
+  Opts.IndexStorePath = Args.getLastArgValue(OPT_index_store_path).str();
+  Opts.IndexIgnoreSystemSymbols = Args.hasArg(OPT_index_ignore_system_symbols);
+  Opts.IndexRecordMangledName = Args.hasArg(OPT_index_record_mangled_name);
+
   InputKind DashX(Language::Unknown);
   if (const Arg *A = Args.getLastArg(OPT_x)) {
     StringRef XValue = A->getValue();
 
     // Parse suffixes: '<lang>(-header|[-module-map][-cpp-output])'.
     // FIXME: Supporting '<lang>-header-cpp-output' would be useful.
     bool Preprocessed = XValue.consume_back("-cpp-output");
     bool ModuleMap = XValue.consume_back("-module-map");

clang/lib/Frontend/FrontendAction.cpp

 bool WrapperFrontendAction::hasCodeCompletionSupport() const {
   return WrappedAction->hasCodeCompletionSupport();
 }
 
 WrapperFrontendAction::WrapperFrontendAction(
     std::unique_ptr<FrontendAction> WrappedAction)
   : WrappedAction(std::move(WrappedAction)) {}
 
+ASTConsumerInjector::ASTConsumerInjector(
+    std::unique_ptr<FrontendAction> BaseActionInput,
+    std::function<std::unique_ptr<ASTConsumer>(CompilerInstance &CI,
+                                               StringRef InFile)>
+        ASTConsumerFactory)
+    : WrapperFrontendAction(std::move(BaseActionInput)),
+      ASTConsumerFactory(ASTConsumerFactory) {}
+
+std::unique_ptr<ASTConsumer>
+ASTConsumerInjector::CreateASTConsumer(CompilerInstance &CI, StringRef InFile) {
+  std::vector<std::unique_ptr<ASTConsumer>> Consumers;
+  Consumers.emplace_back(ASTConsumerFactory(CI, InFile));
+  Consumers.emplace_back(WrapperFrontendAction::CreateASTConsumer(CI, InFile));
+  return std::make_unique<MultiplexConsumer>(std::move(Consumers));
+}

clang/lib/FrontendTool/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   Option
   Support
   )
 
 set(link_libs
   clangBasic
   clangCodeGen
   clangDriver
   clangFrontend
+  clangIndex
+  clangIndexSerialization
   clangRewriteFrontend
   )
 
 if(CLANG_ENABLE_ARCMT)
   list(APPEND link_libs
     clangARCMigrate
     )
 endif()

clang/lib/FrontendTool/ExecuteCompilerInvocation.cpp

 #include "clang/Driver/Options.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/CompilerInvocation.h"
 #include "clang/Frontend/FrontendActions.h"
 #include "clang/Frontend/FrontendDiagnostic.h"
 #include "clang/Frontend/FrontendPluginRegistry.h"
 #include "clang/Frontend/Utils.h"
 #include "clang/FrontendTool/Utils.h"
+#include "clang/Index/IndexingAction.h"
+#include "clang/Index/IndexingOptions.h"
+#include "clang/IndexSerialization/EmitIndex.h"
 #include "clang/Rewrite/Frontend/FrontendActions.h"
 #include "clang/StaticAnalyzer/Frontend/AnalyzerHelpFlags.h"
 #include "clang/StaticAnalyzer/Frontend/FrontendActions.h"
 #include "llvm/Option/OptTable.h"
 #include "llvm/Option/Option.h"
 #include "llvm/Support/BuryPointer.h"
 #include "llvm/Support/DynamicLibrary.h"
 #include "llvm/Support/ErrorHandling.h"
if (CI.getFrontendOpts().ProgramAction != frontend::MigrateSource &&
     if (FEOpts.ObjCMTAction != FrontendOptions::ObjCMT_None) {
       Act = std::make_unique<arcmt::ObjCMigrateAction>(std::move(Act),
                                                         FEOpts.MTMigrateDir,
                                                         FEOpts.ObjCMTAction);
     }
   }
 #endif
 
+  if (!FEOpts.IndexStorePath.empty()) {
+    Act = std::make_unique<ASTConsumerInjector>(
+        std::move(Act), [](CompilerInstance &CI, StringRef) {
+          const FrontendOptions &FEOpts = CI.getFrontendOpts();
+          return std::make_unique<index::IndexRecordingASTConsumer>(
+              index::parseIndexGenerationOptions(FEOpts), CI,
+              index::parseEmitIndexOptions(FEOpts));
+        });
+  }
+
   // If there are any AST files to merge, create a frontend action
   // adaptor to perform the merge.
   if (!FEOpts.ASTMergeFiles.empty())
     Act = std::make_unique<ASTMergeAction>(std::move(Act),
                                             FEOpts.ASTMergeFiles);
 
   return Act;
 }

clang/lib/Index/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
+  BitReader
   Core
   Support
   )
 
 add_clang_library(clangIndex
   CommentToXML.cpp
-  FileIndexRecord.cpp
   IndexBody.cpp
   IndexDecl.cpp
   IndexingAction.cpp
   IndexingContext.cpp
+  IndexOptions.cpp
   IndexSymbol.cpp
   IndexTypeSourceInfo.cpp
   USRGeneration.cpp
 
   ADDITIONAL_HEADERS
-  IndexingContext.h
   SimpleFormatContext.h
 
   LINK_LIBS
   clangAST
   clangBasic
   clangFormat
   clangFrontend
   clangLex
   clangRewrite
   clangSerialization
   clangToolingCore
 
   DEPENDS
   omp_gen
   )

clang/lib/Index/FileIndexRecord.h

-//===--- FileIndexRecord.h - Index data per file ----------------*- C++ -*-===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-
-#ifndef LLVM_CLANG_LIB_INDEX_FILEINDEXRECORD_H
-#define LLVM_CLANG_LIB_INDEX_FILEINDEXRECORD_H
-
-#include "clang/Basic/SourceLocation.h"
-#include "clang/Index/DeclOccurrence.h"
-#include "clang/Index/IndexSymbol.h"
-#include "llvm/ADT/ArrayRef.h"
-#include "llvm/ADT/SmallVector.h"
-#include <vector>
-
-namespace clang {
-class IdentifierInfo;
-
-namespace index {
-
-/// Stores the declaration occurrences seen in a particular source or header
-/// file of a translation unit
-class FileIndexRecord {
-private:
-  FileID FID;
-  bool IsSystem;
-  std::vector<DeclOccurrence> Decls;
-
-public:
-  FileIndexRecord(FileID FID, bool IsSystem) : FID(FID), IsSystem(IsSystem) {}
-
-  ArrayRef<DeclOccurrence> getDeclOccurrencesSortedByOffset() const {
-    return Decls;
-  }
-
-  FileID getFileID() const { return FID; }
-  bool isSystem() const { return IsSystem; }
-
-  /// Adds an occurrence of the canonical declaration \c D at the supplied
-  /// \c Offset
-  ///
-  /// \param Roles the roles the occurrence fulfills in this position.
-  /// \param Offset the offset in the file of this occurrence.
-  /// \param D the canonical declaration this is an occurrence of.
-  /// \param Relations the set of symbols related to this occurrence.
-  void addDeclOccurence(SymbolRoleSet Roles, unsigned Offset, const Decl *D,
-                        ArrayRef<SymbolRelation> Relations);
-  void print(llvm::raw_ostream &OS) const;
-};
-
-} // end namespace index
-} // end namespace clang
-
-#endif // LLVM_CLANG_LIB_INDEX_FILEINDEXRECORD_H

clang/lib/Index/FileIndexRecord.cpp

-//===--- FileIndexRecord.cpp - Index data per file --------------*- C++ -*-===//
-//
-//                     The LLVM Compiler Infrastructure
-//
-// This file is distributed under the University of Illinois Open Source
-// License. See LICENSE.TXT for details.
-//
-//===----------------------------------------------------------------------===//
-
-#include "FileIndexRecord.h"
-#include "clang/AST/ASTContext.h"
-#include "clang/AST/DeclTemplate.h"
-#include "clang/Basic/SourceManager.h"
-#include "llvm/ADT/SmallString.h"
-#include "llvm/Support/Path.h"
-
-using namespace clang;
-using namespace clang::index;
-
-void FileIndexRecord::addDeclOccurence(SymbolRoleSet Roles, unsigned Offset,
-                                       const Decl *D,
-                                       ArrayRef<SymbolRelation> Relations) {
-  assert(D->isCanonicalDecl() &&
-         "Occurrences should be associated with their canonical decl");
-
-  auto IsNextOccurence = [&]() -> bool {
-    if (Decls.empty())
-      return true;
-    auto &Last = Decls.back();
-    return Last.Offset < Offset;
-  };
-
-  if (IsNextOccurence()) {
-    Decls.emplace_back(Roles, Offset, D, Relations);
-    return;
-  }
-
-  DeclOccurrence NewInfo(Roles, Offset, D, Relations);
-  // We keep Decls in order as we need to access them in this order in all cases.
-  auto It = llvm::upper_bound(Decls, NewInfo);
-  Decls.insert(It, std::move(NewInfo));
-}
-
-void FileIndexRecord::print(llvm::raw_ostream &OS) const {
-  OS << "DECLS BEGIN ---\n";
-  for (auto &DclInfo : Decls) {
-    const Decl *D = DclInfo.Dcl;
-    SourceManager &SM = D->getASTContext().getSourceManager();
-    SourceLocation Loc = SM.getFileLoc(D->getLocation());
-    PresumedLoc PLoc = SM.getPresumedLoc(Loc);
-    OS << llvm::sys::path::filename(PLoc.getFilename()) << ':' << PLoc.getLine()
-       << ':' << PLoc.getColumn();
-
-    if (auto ND = dyn_cast<NamedDecl>(D)) {
-      OS << ' ' << ND->getNameAsString();
-    }
-
-    OS << '\n';
-  }
-  OS << "DECLS END ---\n";
-}

clang/lib/Index/IndexBody.cpp

 //===- IndexBody.cpp - Indexing statements --------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
-#include "IndexingContext.h"
-#include "clang/AST/RecursiveASTVisitor.h"
 #include "clang/AST/ASTLambda.h"
+#include "clang/AST/RecursiveASTVisitor.h"
+#include "clang/Index/IndexingContext.h"
 
 using namespace clang;
 using namespace clang::index;
 
 namespace {
 
 class BodyIndexer : public RecursiveASTVisitor<BodyIndexer> {
   IndexingContext &IndexCtx;

clang/lib/Index/IndexDecl.cpp

 //===- IndexDecl.cpp - Indexing declarations ------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
-#include "IndexingContext.h"
 #include "clang/AST/Attr.h"
 #include "clang/AST/DeclVisitor.h"
 #include "clang/Index/IndexDataConsumer.h"
+#include "clang/Index/IndexingContext.h"
 
 using namespace clang;
 using namespace index;
 
 #define TRY_DECL(D,CALL_EXPR)                                                  \
   do {                                                                         \
     if (!IndexCtx.shouldIndex(D)) return true;                                 \
     if (!CALL_EXPR)                                                            \

clang/lib/Index/IndexOptions.cpp

+//===--- IndexOptions.cpp - Options for indexing ----------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Index/IndexingOptions.h"
+#include "llvm/Support/Path.h"
+
+namespace clang {
+namespace index {
+
+IndexingOptions parseIndexGenerationOptions(const FrontendOptions &FEOpts) {
+  IndexingOptions IndexOpts;
+  if (FEOpts.IndexIgnoreSystemSymbols) {
+    IndexOpts.SystemSymbolFilter =
+        index::IndexingOptions::SystemSymbolFilterKind::None;
+  }
+  return IndexOpts;
+}
+
+} // namespace index
+} // namespace clang

clang/lib/Index/IndexTypeSourceInfo.cpp

 //===- IndexTypeSourceInfo.cpp - Indexing types ---------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
-#include "IndexingContext.h"
 #include "clang/AST/RecursiveASTVisitor.h"
+#include "clang/Index/IndexingContext.h"
 
 using namespace clang;
 using namespace index;
 
 namespace {
 
 class TypeIndexer : public RecursiveASTVisitor<TypeIndexer> {
   IndexingContext &IndexCtx;

clang/lib/Index/IndexingAction.cpp

 //===- IndexingAction.cpp - Frontend index action -------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "clang/Index/IndexingAction.h"
-#include "IndexingContext.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/FrontendAction.h"
 #include "clang/Frontend/MultiplexConsumer.h"
 #include "clang/Index/IndexDataConsumer.h"
+#include "clang/Index/IndexingContext.h"
 #include "clang/Lex/PPCallbacks.h"
 #include "clang/Lex/Preprocessor.h"
 #include "clang/Serialization/ASTReader.h"
 #include "llvm/ADT/STLExtras.h"
 #include <memory>
 
 using namespace clang;
 using namespace clang::index;

clang/lib/Index/IndexingContext.cpp

 //===- IndexingContext.cpp - Indexing context data ------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
-#include "IndexingContext.h"
+#include "clang/Index/IndexingContext.h"
 #include "clang/AST/ASTContext.h"
 #include "clang/AST/Attr.h"
 #include "clang/AST/DeclObjC.h"
 #include "clang/AST/DeclTemplate.h"
 #include "clang/Basic/SourceLocation.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Index/IndexDataConsumer.h"
 
bool IndexingContext::importedModule(const ImportDecl *ImportD) {
   else
     Loc = ImportD->getLocation();
 
   SourceManager &SM = Ctx->getSourceManager();
   FileID FID = SM.getFileID(SM.getFileLoc(Loc));
   if (FID.isInvalid())
     return true;
 
-  bool Invalid = false;
-  const SrcMgr::SLocEntry &SEntry = SM.getSLocEntry(FID, &Invalid);
-  if (Invalid || !SEntry.isFile())
-    return true;
-
-  if (SEntry.getFile().getFileCharacteristic() != SrcMgr::C_User) {
+  if (SM.isInSystemHeader(Loc)) {
     switch (IndexOpts.SystemSymbolFilter) {
     case IndexingOptions::SystemSymbolFilterKind::None:
       return true;
     case IndexingOptions::SystemSymbolFilterKind::DeclarationsOnly:
     case IndexingOptions::SystemSymbolFilterKind::All:
       break;
     }
   }

 
 bool IndexingContext::handleDeclOccurrence(const Decl *D, SourceLocation Loc,
                                            bool IsRef, const Decl *Parent,
                                            SymbolRoleSet Roles,
                                            ArrayRef<SymbolRelation> Relations,
                                            const Expr *OrigE,
                                            const Decl *OrigD,
                                            const DeclContext *ContainerDC) {
-  if (D->isImplicit() && !isa<ObjCMethodDecl>(D))
+  if (D->isImplicit() && !(isa<ObjCMethodDecl>(D) || isa<ObjCIvarDecl>(D)))
     return true;
   if (!isa<NamedDecl>(D) || shouldSkipNamelessDecl(cast<NamedDecl>(D)))
     return true;
 
   SourceManager &SM = Ctx->getSourceManager();
   FileID FID = SM.getFileID(SM.getFileLoc(Loc));
   if (FID.isInvalid())
     return true;
 
-  bool Invalid = false;
-  const SrcMgr::SLocEntry &SEntry = SM.getSLocEntry(FID, &Invalid);
-  if (Invalid || !SEntry.isFile())
-    return true;
-
-  if (SEntry.getFile().getFileCharacteristic() != SrcMgr::C_User) {
+  if (SM.isInSystemHeader(Loc)) {
     switch (IndexOpts.SystemSymbolFilter) {
     case IndexingOptions::SystemSymbolFilterKind::None:
       return true;
     case IndexingOptions::SystemSymbolFilterKind::DeclarationsOnly:
       if (!shouldReportOccurrenceForSystemDeclOnlyMode(IsRef, Roles, Relations))
         return true;
       break;
     case IndexingOptions::SystemSymbolFilterKind::All:

clang/lib/IndexDataStore/CMakeLists.txt

+set(LLVM_LINK_COMPONENTS
+  Support
+  )
+
+add_clang_library(clangIndexDataStore
+  IndexDataStore.cpp
+
+  LINK_LIBS
+  clangDirectoryWatcher
+  clangIndexSerialization
+  )

clang/lib/IndexDataStore/IndexDataStore.cpp

+//===--- IndexDataStore.cpp - Index data store info -----------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/IndexDataStore/IndexDataStore.h"
+#include "clang/DirectoryWatcher/DirectoryWatcher.h"
+#include "clang/IndexSerialization/IndexDataFormat.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/Mutex.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/raw_ostream.h"
+
+using namespace clang;
+using namespace clang::index;
+using namespace clang::index::store;
+using namespace llvm;
+
+namespace {
+
+class UnitEventHandlerData {
+  mutable sys::Mutex Mtx;
+  IndexDataStore::UnitEventHandler Handler;
+
+public:
+  void setHandler(IndexDataStore::UnitEventHandler handler) {
+    sys::ScopedLock L(Mtx);
+    Handler = std::move(handler);
+  }
+  IndexDataStore::UnitEventHandler getHandler() const {
+    sys::ScopedLock L(Mtx);
+    return Handler;
+  }
+};
+
+class IndexDataStoreImpl {
+  std::string FilePath;
+  std::shared_ptr<UnitEventHandlerData> TheUnitEventHandlerData;
+  std::unique_ptr<DirectoryWatcher> DirWatcher;
+
+public:
+  explicit IndexDataStoreImpl(StringRef indexStorePath)
+      : FilePath(indexStorePath) {
+    TheUnitEventHandlerData = std::make_shared<UnitEventHandlerData>();
+  }
+
+  StringRef getFilePath() const { return FilePath; }
+  bool foreachUnitName(bool sorted,
+                       llvm::function_ref<bool(StringRef unitName)> receiver);
+  void setUnitEventHandler(IndexDataStore::UnitEventHandler Handler);
+  bool startEventListening(bool waitInitialSync, std::string &Error);
+  void stopEventListening();
+  void discardUnit(StringRef UnitName);
+  void discardRecord(StringRef RecordName);
+  void purgeStaleData();
+};
+
+} // anonymous namespace
+
+bool IndexDataStoreImpl::foreachUnitName(
+    bool sorted, llvm::function_ref<bool(StringRef unitName)> receiver) {
+  SmallString<128> UnitPath;
+  UnitPath = FilePath;
+  appendUnitSubDir(UnitPath);
+
+  std::vector<std::string> filenames;
+
+  std::error_code EC;
+  for (auto It = sys::fs::directory_iterator(UnitPath, EC),
+            End = sys::fs::directory_iterator();
+       !EC && It != End; It.increment(EC)) {
+    StringRef unitName = sys::path::filename(It->path());
+    if (!sorted) {
+      if (!receiver(unitName))
+        return false;
+    } else {
+      filenames.push_back(unitName.str());
+    }
+  }
+
+  if (sorted) {
+    std::sort(filenames.begin(), filenames.end());
+    for (auto &fname : filenames)
+      if (!receiver(fname))
+        return false;
+  }
+  return true;
+}
+
+void IndexDataStoreImpl::setUnitEventHandler(
+    IndexDataStore::UnitEventHandler handler) {
+  TheUnitEventHandlerData->setHandler(std::move(handler));
+}
+
+bool IndexDataStoreImpl::startEventListening(bool waitInitialSync,
+                                             std::string &Error) {
+  if (DirWatcher) {
+    Error = "event listener already active";
+    return true;
+  }
+
+  SmallString<128> UnitPath;
+  UnitPath = FilePath;
+  appendUnitSubDir(UnitPath);
+
+  auto localUnitEventHandlerData = TheUnitEventHandlerData;
+  auto OnUnitsChange = [localUnitEventHandlerData](
+                           ArrayRef<DirectoryWatcher::Event> Events,
+                           bool isInitial) {
+    SmallVector<IndexDataStore::UnitEvent, 16> UnitEvents;
+    UnitEvents.reserve(Events.size());
+    for (const DirectoryWatcher::Event &evt : Events) {
+      IndexDataStore::UnitEventKind K;
+      StringRef UnitName = sys::path::filename(evt.Filename);
+      switch (evt.Kind) {
+      case DirectoryWatcher::Event::EventKind::Removed:
+        K = IndexDataStore::UnitEventKind::Removed;
+        break;
+      case DirectoryWatcher::Event::EventKind::Modified:
+        K = IndexDataStore::UnitEventKind::Modified;
+        break;
+      case DirectoryWatcher::Event::EventKind::WatchedDirRemoved:
+        K = IndexDataStore::UnitEventKind::DirectoryDeleted;
+        UnitName = StringRef();
+        break;
+      case DirectoryWatcher::Event::EventKind::WatcherGotInvalidated:
+        K = IndexDataStore::UnitEventKind::Failure;
+        UnitName = StringRef();
+        break;
+      }
+      UnitEvents.push_back(IndexDataStore::UnitEvent{K, UnitName});
+    }
+
+    if (auto handler = localUnitEventHandlerData->getHandler()) {
+      IndexDataStore::UnitEventNotification EventNote{isInitial, UnitEvents};
+      handler(EventNote);
+    }
+  };
+
+  auto DirWatcherOrError =
+      DirectoryWatcher::create(UnitPath.str(), OnUnitsChange, waitInitialSync);
+  if (!DirWatcherOrError) {
+    Error = "failed to create directory watcher: " +
+            toString(DirWatcherOrError.takeError());
+    return true;
+  }
+  DirWatcher = std::move(*DirWatcherOrError);
+
+  return false;
+}
+
+void IndexDataStoreImpl::stopEventListening() { DirWatcher.reset(); }
+
+void IndexDataStoreImpl::discardUnit(StringRef UnitName) {
+  SmallString<128> UnitPath;
+  UnitPath = FilePath;
+  appendUnitSubDir(UnitPath);
+  appendUnitBucketDir(UnitName, UnitPath);
+  sys::fs::remove(UnitPath);
+}
+
+void IndexDataStoreImpl::discardRecord(StringRef RecordName) {
+  // TODO check - is this correct or is the order of appended elements opposite?
+  SmallString<128> RecordPath;
+  RecordPath = FilePath;
+  appendRecordSubDir(RecordPath);
+  appendRecordBucketDir(RecordName, RecordPath);
+  appendRecordFilename(RecordName, RecordPath);
+  sys::fs::remove(RecordPath);
+}
+
+void IndexDataStoreImpl::purgeStaleData() {
+  // FIXME: Implement.
+}
+
+std::unique_ptr<IndexDataStore> IndexDataStore::create(StringRef IndexStorePath,
+                                                       std::string &Error) {
+  if (!sys::fs::exists(IndexStorePath)) {
+    raw_string_ostream OS(Error);
+    OS << "index store path does not exist: " << IndexStorePath;
+    return nullptr;
+  }
+
+  return std::unique_ptr<IndexDataStore>(
+      new IndexDataStore(new IndexDataStoreImpl(IndexStorePath)));
+}
+
+#define IMPL static_cast<IndexDataStoreImpl *>(Impl)
+
+IndexDataStore::~IndexDataStore() { delete IMPL; }
+
+StringRef IndexDataStore::getFilePath() const { return IMPL->getFilePath(); }
+
+bool IndexDataStore::foreachUnitName(
+    bool sorted, llvm::function_ref<bool(StringRef unitName)> receiver) {
+  return IMPL->foreachUnitName(sorted, std::move(receiver));
+}
+
+unsigned IndexDataStore::getFormatVersion() {
+  return clang::index::store::getIndexFormatVersion();
+}
+
+void IndexDataStore::setUnitEventHandler(UnitEventHandler Handler) {
+  return IMPL->setUnitEventHandler(std::move(Handler));
+}
+
+bool IndexDataStore::startEventListening(bool waitInitialSync,
+                                         std::string &Error) {
+  return IMPL->startEventListening(waitInitialSync, Error);
+}
+
+void IndexDataStore::stopEventListening() { return IMPL->stopEventListening(); }
+
+void IndexDataStore::discardUnit(StringRef UnitName) {
+  IMPL->discardUnit(UnitName);
+}
+
+void IndexDataStore::discardRecord(StringRef RecordName) {
+  IMPL->discardRecord(RecordName);
+}
+
+void IndexDataStore::purgeStaleData() { IMPL->purgeStaleData(); }

clang/lib/IndexSerialization/BitstreamUtil.h

+//===--- BitstreamUtil.h - Bitstream manipulation utils -------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_LIB_INDEX_BITSTREAMUTIL_H
+#define LLVM_CLANG_LIB_INDEX_BITSTREAMUTIL_H
+
+#include "llvm/ADT/SmallVector.h"
+
+namespace llvm {
+class BitstreamWriter;
+}
+
+namespace clang {
+namespace index {
+namespace store {
+
+void emitBlockID(unsigned ID, const char *Name, llvm::BitstreamWriter &Stream,
+                 llvm::SmallVectorImpl<uint64_t> &Record);
+
+void emitRecordID(unsigned ID, const char *Name, llvm::BitstreamWriter &Stream,
+                  llvm::SmallVectorImpl<uint64_t> &Record);
+
+} // end namespace store
+} // end namespace index
+} // end namespace clang
+
+#endif

clang/lib/IndexSerialization/BitstreamUtil.cpp

+//===--- BitstreamUtil.cpp - Bitstream manipulation utils -----------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "BitstreamUtil.h"
+
+#include "llvm/Bitstream/BitCodes.h"
+#include "llvm/Bitstream/BitstreamWriter.h"
+
+using namespace clang;
+using namespace llvm;
+
+namespace clang {
+namespace index {
+namespace store {
+
+void emitBlockID(unsigned ID, const char *Name, llvm::BitstreamWriter &Stream,
+                 llvm::SmallVectorImpl<uint64_t> &Record) {
+  Record.clear();
+  Record.push_back(ID);
+  Stream.EmitRecord(bitc::BLOCKINFO_CODE_SETBID, Record);
+
+  // Emit the block name if present.
+  if (!Name || Name[0] == 0)
+    return;
+  Record.clear();
+  while (*Name)
+    Record.push_back(*Name++);
+  Stream.EmitRecord(bitc::BLOCKINFO_CODE_BLOCKNAME, Record);
+}
+
+void emitRecordID(unsigned ID, const char *Name, llvm::BitstreamWriter &Stream,
+                  llvm::SmallVectorImpl<uint64_t> &Record) {
+  Record.clear();
+  Record.push_back(ID);
+  while (*Name)
+    Record.push_back(*Name++);
+  Stream.EmitRecord(bitc::BLOCKINFO_CODE_SETRECORDNAME, Record);
+}
+
+} // end namespace store
+} // end namespace index
+} // end namespace clang

clang/lib/IndexSerialization/BitstreamVisitor.h

+//===--- BitstreamVisitor.h - Helper for reading a bitstream --------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_LIB_INDEX_BITSTREAMVISITOR_H
+#define LLVM_CLANG_LIB_INDEX_BITSTREAMVISITOR_H
+
+#include "clang/Basic/LLVM.h"
+#include "clang/Serialization/ASTReader.h"
+#include "llvm/Bitstream/BitstreamReader.h"
+#include <string>
+
+namespace clang {
+namespace index {
+namespace store {
+
+/// Helper class that saves the current stream position and
+/// then restores it when destroyed.
+struct SavedStreamPosition {
+  explicit SavedStreamPosition(llvm::BitstreamCursor &Cursor)
+      : Cursor(Cursor), Offset(Cursor.GetCurrentBitNo()) {}
+
+  ~SavedStreamPosition() {
+    if (llvm::Error Err = Cursor.JumpToBit(Offset))
+      llvm::report_fatal_error("SavedStreamPosition failed jumping: " +
+                               toString(std::move(Err)));
+  }
+
+private:
+  llvm::BitstreamCursor &Cursor;
+  uint64_t Offset;
+};
+
+enum class StreamVisit { Continue, Skip, Abort };
+
+template <typename ImplClass> class BitstreamVisitor {
+  SmallVector<unsigned, 4> BlockStack;
+
+protected:
+  llvm::BitstreamCursor &Stream;
+  Optional<llvm::BitstreamBlockInfo> BlockInfo;
+  std::string *Error;
+
+public:
+  BitstreamVisitor(llvm::BitstreamCursor &Stream) : Stream(Stream) {}
+
+  StreamVisit visitBlock(unsigned ID) { return StreamVisit::Continue; }
+
+  bool visit(std::string &Error) {
+    this->Error = &Error;
+
+    ASTReader::RecordData Record;
+    while (1) {
+      Expected<llvm::BitstreamEntry> MaybeEntry =
+          Stream.advance(llvm::BitstreamCursor::AF_DontPopBlockAtEnd);
+      if (!MaybeEntry) {
+        Error = toString(MaybeEntry.takeError());
+        return false;
+      }
+      llvm::BitstreamEntry Entry = MaybeEntry.get();
+
+      switch (Entry.Kind) {
+      case llvm::BitstreamEntry::Error:
+        Error = "malformed serialization";
+        return false;
+
+      case llvm::BitstreamEntry::EndBlock:
+        if (BlockStack.empty())
+          return true;
+        BlockStack.pop_back();
+        if (Stream.ReadBlockEnd()) {
+          Error = "malformed serialization";
+          return false;
+        }
+        if (Stream.AtEndOfStream())
+          return true;
+        break;
+
+      case llvm::BitstreamEntry::SubBlock: {
+        if (Entry.ID == llvm::bitc::BLOCKINFO_BLOCK_ID) {
+          Expected<Optional<llvm::BitstreamBlockInfo>> MaybeBlockInfo =
+              Stream.ReadBlockInfoBlock();
+          if (!MaybeBlockInfo) {
+            Error = toString(MaybeBlockInfo.takeError());
+            return false;
+          }
+          BlockInfo = MaybeBlockInfo.get();
+          if (!BlockInfo) {
+            Error = "malformed BlockInfoBlock";
+            return false;
+          }
+          Stream.setBlockInfo(&*BlockInfo);
+          break;
+        }
+
+        StreamVisit Ret = static_cast<ImplClass *>(this)->visitBlock(Entry.ID);
+        switch (Ret) {
+        case StreamVisit::Continue:
+          if (Stream.EnterSubBlock(Entry.ID)) {
+            Error = "malformed block record";
+            return false;
+          }
+          if (llvm::Error Err = readBlockAbbrevs(Stream)) {
+            Error = toString(std::move(Err));
+            return false;
+          }
+          BlockStack.push_back(Entry.ID);
+          break;
+
+        case StreamVisit::Skip:
+          if (Stream.SkipBlock()) {
+            Error = "malformed serialization";
+            return false;
+          }
+          if (Stream.AtEndOfStream())
+            return true;
+          break;
+
+        case StreamVisit::Abort:
+          return false;
+        }
+        break;
+      }
+
+      case llvm::BitstreamEntry::Record: {
+        Record.clear();
+        StringRef Blob;
+        Expected<unsigned> MaybeRecID =
+            Stream.readRecord(Entry.ID, Record, &Blob);
+        if (!MaybeRecID) {
+          Error = toString(MaybeRecID.takeError());
+          return false;
+        }
+        unsigned RecID = MaybeRecID.get();
+        unsigned BlockID = BlockStack.empty() ? 0 : BlockStack.back();
+        StreamVisit Ret = static_cast<ImplClass *>(this)->visitRecord(
+            BlockID, RecID, Record, Blob);
+        switch (Ret) {
+        case StreamVisit::Continue:
+          break;
+
+        case StreamVisit::Skip:
+          if (Expected<unsigned> Skipped = Stream.skipRecord(Entry.ID)) {
+            Error = toString(Skipped.takeError());
+            return false;
+          }
+          break;
+
+        case StreamVisit::Abort:
+          return false;
+        }
+        break;
+      }
+      }
+    }
+  }
+
+  static llvm::Error readBlockAbbrevs(llvm::BitstreamCursor &Cursor) {
+    while (true) {
+      uint64_t Offset = Cursor.GetCurrentBitNo();
+      Expected<unsigned> MaybeCode = Cursor.ReadCode();
+      if (!MaybeCode)
+        return MaybeCode.takeError();
+      unsigned Code = MaybeCode.get();
+
+      // We expect all abbrevs to be at the start of the block.
+      if (Code != llvm::bitc::DEFINE_ABBREV) {
+        if (llvm::Error Err = Cursor.JumpToBit(Offset))
+          return Err;
+        return llvm::Error::success();
+      }
+      if (llvm::Error Err = Cursor.ReadAbbrevRecord())
+        return Err;
+    }
+  }
+};
+
+} // end namespace store
+} // end namespace index
+} // end namespace clang
+
+#endif

clang/lib/IndexSerialization/CMakeLists.txt

+set(LLVM_LINK_COMPONENTS
+  BitReader
+  BitstreamReader
+  Core
+  Support
+  )
+
 add_clang_library(clangIndexSerialization
+  BitstreamUtil.cpp
+  DeclOccurrenceCollector.cpp
+  EmitIndex.cpp
+  EmitIndexOptions.cpp
+  IndexDataFormat.cpp
+  IndexRecordReader.cpp
+  IndexRecordWriter.cpp
+  IndexUnitReader.cpp
+  IndexUnitWriter.cpp
   SerializablePathCollection.cpp
 
   LINK_LIBS
+  clangAST
   clangBasic
+  clangFormat
+  clangFrontend
+  clangIndex
+  clangLex
+  clangRewrite
+  clangSerialization
+  clangToolingCore
   )

clang/lib/IndexSerialization/DeclOccurrenceCollector.cpp

+//===- DeclOccurrenceCollector.cpp - Collecting data for index records ----===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/IndexSerialization/DeclOccurrenceCollector.h"
+
+namespace clang {
+namespace index {
+
+bool DeclOccurrenceCollector::handleDeclOccurrence(
+    const Decl *D, SymbolRoleSet Roles, ArrayRef<SymbolRelation> Relations,
+    SourceLocation Loc, ASTNodeInfo ASTNode) {
+  Loc = SM.getFileLoc(Loc);
+  if (Loc.isInvalid())
+    return true;
+
+  FileID FID;
+  unsigned Offset;
+  std::tie(FID, Offset) = SM.getDecomposedLoc(Loc);
+
+  if (FID.isInvalid())
+    return true;
+
+  const FileEntry *FE = SM.getFileEntryForID(FID);
+  if (!FE) // Ignore predefined buffers.
+    return true;
+
+  IndexRecordWriter &Rec =
+      RecordByFile.insert({FID, IndexRecordWriter(SM.isInSystemHeader(Loc))})
+          .first->second;
+  Rec.addDeclOccurence(Roles, FID, Offset, D, Relations);
+  return true;
+}
+
+} // namespace index
+} // namespace clang
 No newline at end of file

clang/lib/IndexSerialization/EmitIndex.cpp

+//===- EmitIndex.cpp - Index emitting AST consumer & functions ------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/IndexSerialization/EmitIndex.h"
+
+#include "clang/Basic/DiagnosticFrontend.h"
+#include "clang/Basic/Version.h"
+#include "clang/Frontend/MultiplexConsumer.h"
+#include "clang/Index/IndexPPCallbacks.h"
+#include "clang/Index/IndexingOptions.h"
+#include "clang/IndexSerialization/IndexDataFormat.h"
+#include "clang/Lex/HeaderSearch.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Serialization/ASTReader.h"
+#include "llvm/ADT/ScopeExit.h"
+#include "llvm/Support/Errc.h"
+
+namespace clang {
+namespace index {
+
+using namespace llvm;
+
+static Module *getModBeingCompiled(const CompilerInstance &CI) {
+  return CI.getLangOpts().isCompilingModule()
+             ? CI.getPreprocessor().getHeaderSearchInfo().lookupModule(
+                   CI.getLangOpts().CurrentModule, /*AllowSearch=*/false)
+             : nullptr;
+}
+
+static std::string getOutputFile(const CompilerInstance &CI) {
+  std::string Result = CI.getFrontendOpts().OutputFile;
+  return !Result.empty()
+             ? Result
+             : CI.getFrontendOpts().Inputs[0].getFile().str() + ".o";
+}
+
+static const FileEntry *getMainFile(const CompilerInstance &CI) {
+  if (CI.getLangOpts().isCompilingModule() ||
+      CI.getFrontendOpts().ProgramAction == frontend::GeneratePCH)
+    return nullptr;
+
+  SourceManager &SM = CI.getSourceManager();
+  return SM.getFileEntryForID(SM.getMainFileID());
+}
+
+class IncludePPCallbacks : public PPCallbacks {
+  EmitIndexOptions RecordOpts;
+  IndexUnitWriter &UnitWriter;
+  SourceManager &SourceMgr;
+
+public:
+  IncludePPCallbacks(EmitIndexOptions recordOpts, IndexUnitWriter &UnitWriter,
+                     SourceManager &SourceMgr)
+      : RecordOpts(recordOpts), UnitWriter(UnitWriter), SourceMgr(SourceMgr) {}
+
+private:
+  virtual void
+  InclusionDirective(SourceLocation InclusionDirectiveLoc, const Token &,
+                     StringRef, bool, CharSourceRange,
+                     const FileEntry *IncludedFile, StringRef, StringRef,
+                     const Module *ImportedModule,
+                     SrcMgr::CharacteristicKind FileType) override {
+    // We don't want to treat imports as includes.
+    if (ImportedModule)
+      return;
+
+    if (RecordOpts.RecordIncludes ==
+        EmitIndexOptions::IncludesRecordingKind::None)
+      return;
+
+    if (!InclusionDirectiveLoc.isFileID() || !IncludedFile ||
+        !IncludedFile->isValid())
+      return;
+
+    std::pair<FileID, unsigned> InclusionDirectiveDecLoc =
+        SourceMgr.getDecomposedExpansionLoc(InclusionDirectiveLoc);
+
+    if (RecordOpts.RecordIncludes ==
+            EmitIndexOptions::IncludesRecordingKind::UserOnly &&
+        SourceMgr.isInSystemHeader(InclusionDirectiveLoc))
+      return; // Ignore includes of system headers.
+
+    auto *FE = SourceMgr.getFileEntryForID(InclusionDirectiveDecLoc.first);
+    if (!FE)
+      return;
+
+    UnitWriter.addIncludedFile(
+        FE,
+        SourceMgr.getLineNumber(InclusionDirectiveDecLoc.first,
+                                InclusionDirectiveDecLoc.second),
+        IncludedFile);
+  }
+};
+
+SourceFilesDependencyCollector::SourceFilesDependencyCollector(
+    const CompilerInstance &CI, IndexingContext &indexCtx,
+    EmitIndexOptions recordOpts)
+    : CI(CI), IndexCtx(indexCtx), RecordOpts(recordOpts),
+      UnitWriter(makeIndexUnitWriter(CI, RecordOpts, getOutputFile(CI),
+                                     getMainFile(CI),
+                                     getModBeingCompiled(CI))) {}
+
+void SourceFilesDependencyCollector::attachToPreprocessor(Preprocessor &PP) {
+  DependencyCollector::attachToPreprocessor(PP);
+  PP.addPPCallbacks(std::make_unique<IncludePPCallbacks>(
+      RecordOpts, UnitWriter, PP.getSourceManager()));
+}
+
+/// \returns either a Module that is both a submodule of UnitModule and owns the
+/// header \p FE.
+static Module *findModuleForHeader(const CompilerInstance &CI,
+                                   Module &UnitModule, const FileEntry &FE) {
+  if (auto Mod = CI.getPreprocessor()
+                     .getHeaderSearchInfo()
+                     .findModuleForHeader(&FE)
+                     .getModule())
+    if (Mod->isSubModuleOf(&UnitModule))
+      return Mod;
+  return nullptr;
+}
+
+bool SourceFilesDependencyCollector::sawDependency(StringRef Filename,
+                                                   bool FromModule,
+                                                   bool IsSystem,
+                                                   bool IsModuleFile,
+                                                   bool IsMissing) {
+  bool sawIt = DependencyCollector::sawDependency(
+      Filename, FromModule, IsSystem, IsModuleFile, IsMissing);
+  if (sawIt) {
+    auto FEOrError = CI.getSourceManager().getFileManager().getFile(Filename);
+    if (FEOrError) {
+      Module *ModForHead = nullptr;
+      {
+        Module *Mod = getModBeingCompiled(CI);
+        if (Mod) {
+          ModForHead = findModuleForHeader(CI, *Mod, **FEOrError);
+          UnitWriter.addModule(ModForHead);
+        }
+      }
+
+      UnitWriter.addSourceFile(*FEOrError, IsSystem || isInSysroot(Filename),
+                               ModForHead);
+    }
+  }
+  return sawIt;
+}
+
+bool SourceFilesDependencyCollector::isInSysroot(StringRef Filename) {
+  const auto &Sysroot = CI.getHeaderSearchOpts().Sysroot;
+  // FIXME: Make this portable.
+  return !Sysroot.empty() && (Sysroot != "/") && Filename.startswith(Sysroot);
+}
+
+ActionContext::ActionContext(const IndexingOptions &IndexOpts,
+                             const CompilerInstance &CI,
+                             const EmitIndexOptions &RecordOpts)
+    : CI(CI), DeclsDataCollector(CI.getSourceManager()),
+      Ctx(IndexOpts, DeclsDataCollector), DepCollector(CI, Ctx, RecordOpts),
+      RecordOpts(RecordOpts), IndexOpts(IndexOpts) {
+  DepCollector.attachToPreprocessor(CI.getPreprocessor());
+  DeclsDataCollector.setIndexingContext(Ctx);
+}
+
+// FIXME: Some better implementation.
+template <unsigned N>
+static void reportError(const CompilerInstance &CI, const char (&Format)[N],
+                        const std::string &Param1 = "",
+                        const std::string &Param2 = "") {
+  DiagnosticsEngine &Diag = CI.getDiagnostics();
+  unsigned DiagID = Diag.getCustomDiagID(DiagnosticsEngine::Error, Format);
+  if (!Param2.empty())
+    Diag.Report(DiagID) << Param1 << Param2;
+  else
+    Diag.Report(DiagID) << Param1;
+};
+
+/// Writes records in \p RecordByFile to disk and adds them as dependencies to
+/// \p UnitModule.
+void writeIndexRecordsToDisk(
+    const llvm::DenseMap<FileID, IndexRecordWriter> &RecordByFile,
+    IndexUnitWriter &UnitWriter, const CompilerInstance &CI,
+    const EmitIndexOptions &RecordOpts, const Module *UnitModule) {
+  auto &ASTCtx = CI.getASTContext();
+  PrintingPolicy Policy(ASTCtx.getLangOpts());
+  Policy.SuppressTemplateArgsInCXXConstructors = true;
+
+  std::shared_ptr<ASTNameGenerator> MangledNameGen;
+  if (RecordOpts.RecordSymbolCodeGenName)
+    MangledNameGen = std::make_shared<ASTNameGenerator>(ASTCtx);
+
+  for (const auto &It : RecordByFile) {
+    const FileID FID = It.first;
+    const IndexRecordWriter &Rec = It.second;
+    const FileEntry *FE = CI.getSourceManager().getFileEntryForID(FID);
+    assert(FE);
+
+    if (Rec.isRecordEmpty())
+      continue;
+
+    llvm::Optional<std::string> RecordFilename = Rec.writeToDisk(
+        FE->getName(), RecordOpts, Policy, MangledNameGen, CI.getDiagnostics());
+    if (!RecordFilename) {
+      reportError(CI,
+                  "failed to write record file for source file '%0' to disk",
+                  FE->getName().str(), "");
+      return;
+    }
+
+    Module *Mod = nullptr;
+    if (UnitModule) {
+      if (auto ModCandidate = CI.getPreprocessor()
+                                  .getHeaderSearchInfo()
+                                  .findModuleForHeader(FE)
+                                  .getModule()) {
+        if (ModCandidate->isSubModuleOf(UnitModule)) {
+          Mod = ModCandidate;
+          UnitWriter.addModule(Mod);
+        }
+      }
+    }
+
+    UnitWriter.addIndexRecord(RecordFilename.getValue(), FE, Rec.isSystem(),
+                              Mod);
+  }
+}
+
+void ActionContext::emitCollectedData() {
+  // FIXME: FrontendAction::EndSourceFile() should probably not call
+  // CI.getDiagnosticClient().EndSourceFile()' until after it has called
+  // 'EndSourceFileAction()', so that code executing during
+  // EndSourceFileAction() can emit diagnostics. If this is fixed,
+  // DiagClientBeginEndRAII can go away.
+  CI.getDiagnosticClient().BeginSourceFile(CI.getLangOpts());
+  auto DiagnosticClientEndSourceFile = llvm::make_scope_exit(
+      [this]() { CI.getDiagnosticClient().EndSourceFile(); });
+
+  std::string Error;
+  if (initIndexDirectory(RecordOpts, Error)) {
+    reportError(CI, "failed creating index directory %0", Error);
+    return;
+  }
+
+  IndexUnitWriter &UnitWriter = DepCollector.UnitWriter;
+
+  const Module *UnitModule = getModBeingCompiled(CI);
+  std::string ModuleName =
+      UnitModule ? UnitModule->getFullModuleName() : std::string();
+
+  if (auto Reader = CI.getASTReader()) {
+    HeaderSearch &HS = CI.getPreprocessor().getHeaderSearchInfo();
+    Reader->getModuleManager().visit(
+        [&](serialization::ModuleFile &Mod) -> bool {
+          Module *UnitMod =
+              HS.lookupModule(Mod.ModuleName, /*AllowSearch=*/false);
+          const bool isSystemMod =
+              Mod.isModule() && UnitMod && UnitMod->IsSystem;
+
+          if (!isSystemMod || RecordOpts.RecordSystemDependencies) {
+            SmallString<256> AbsPath(Mod.File->getName());
+            CI.getFileManager().makeAbsolutePath(AbsPath);
+            UnitWriter.addIndexUnit(Mod.File, AbsPath, isSystemMod, UnitMod);
+            if (Mod.isModule()) {
+              llvm::Optional<bool> IndexUnitExist = doesIndexUnitForModuleExist(
+                  Mod.FileName, CI.getFileManager(), RecordOpts);
+
+              if (!IndexUnitExist.hasValue())
+                reportError(CI, "failed to stat index unit for module %0",
+                            AbsPath.str().str());
+
+              if (!IndexUnitExist.getValue())
+                emitIndexDataForModuleFile(Mod, CI, IndexOpts, RecordOpts);
+            }
+          }
+          return true; // skip module dependencies.
+        });
+  }
+
+  if (std::error_code EC = sys::fs::create_directory(RecordOpts.RecordDir)) {
+    reportError(CI, "failed to create directory '%0': %1", RecordOpts.RecordDir,
+                EC.message());
+    return;
+  }
+  writeIndexRecordsToDisk(DeclsDataCollector.getRecordByFile(), UnitWriter, CI,
+                          RecordOpts, UnitModule);
+
+  {
+    std::string Error;
+    if (UnitWriter.writeToDisk(Error)) {
+      reportError(CI, "failed writing unit data: %0", Error);
+      return;
+    }
+  }
+}
+
+IndexRecordingASTConsumer::IndexRecordingASTConsumer(
+    const IndexingOptions &IndexOpts, CompilerInstance &CI,
+    const EmitIndexOptions &RecordOpts)
+    : Context(IndexOpts, CI, RecordOpts) {}
+
+void IndexRecordingASTConsumer::Initialize(ASTContext &ASTContextInst) {
+  Context.Ctx.setASTContext(ASTContextInst);
+  Context.DeclsDataCollector.initialize(ASTContextInst);
+}
+
+bool IndexRecordingASTConsumer::HandleTopLevelDecl(DeclGroupRef DG) {
+  return Context.Ctx.indexDeclGroupRef(DG);
+}
+
+void IndexRecordingASTConsumer::HandleInterestingDecl(DeclGroupRef DG) {
+  // Ignore deserialized decls.
+}
+
+void IndexRecordingASTConsumer::HandleTopLevelDeclInObjCContainer(
+    DeclGroupRef DG) {
+  Context.Ctx.indexDeclGroupRef(DG);
+}
+
+void IndexRecordingASTConsumer::HandleTranslationUnit(ASTContext &Ctx) {
+  Context.emitCollectedData();
+}
+
+void emitIndexDataForModuleFile(serialization::ModuleFile &Mod,
+                                const CompilerInstance &CI,
+                                const IndexingOptions &IndexOpts,
+                                const EmitIndexOptions &RecordOpts) {
+  // FIXME: FrontendAction::EndSourceFile() should probably not call
+  // CI.getDiagnosticClient().EndSourceFile()' until after it has called
+  // 'EndSourceFileAction()', so that code executing during
+  // EndSourceFileAction() can emit diagnostics. If this is fixed,
+  // DiagClientBeginEndRAII can go away.
+  CI.getDiagnosticClient().BeginSourceFile(CI.getLangOpts());
+  auto DiagnosticClientEndSourceFile = llvm::make_scope_exit(
+      [&CI]() { CI.getDiagnosticClient().EndSourceFile(); });
+
+  DiagnosticsEngine &Diag = CI.getDiagnostics();
+  Diag.Report(Mod.ImportLoc, diag::remark_index_producing_module_file_data)
+      << Mod.FileName;
+
+  Module *UnitModule = CI.getPreprocessor().getHeaderSearchInfo().lookupModule(
+      Mod.ModuleName, /*AllowSearch=*/false);
+
+  DeclOccurrenceCollector DeclsDataCollector(CI.getSourceManager());
+  {
+    IndexingContext IndexCtx(IndexOpts, DeclsDataCollector);
+    DeclsDataCollector.setIndexingContext(IndexCtx);
+
+    IndexCtx.setASTContext(CI.getASTContext());
+
+    for (const Decl *D : CI.getASTReader()->getModuleFileLevelDecls(Mod)) {
+      IndexCtx.indexTopLevelDecl(D);
+    }
+    DeclsDataCollector.finish();
+  }
+
+  {
+    std::string Error;
+    if (initIndexDirectory(RecordOpts, Error)) {
+      reportError(CI, "failed creating index directory %0", Error);
+      return;
+    }
+  }
+
+  HeaderSearch &HS = CI.getPreprocessor().getHeaderSearchInfo();
+
+  SmallString<128> WorkDir;
+  if (!CI.getFileManager().getFileSystemOpts().WorkingDir.empty()) {
+    WorkDir = CI.getFileManager().getFileSystemOpts().WorkingDir;
+    if (!llvm::sys::path::is_absolute(WorkDir)) {
+      llvm::sys::fs::make_absolute(WorkDir);
+    }
+  } else {
+    std::error_code EC = llvm::sys::fs::current_path(WorkDir);
+    if (EC) {
+      reportError(CI, "failed to determine current working directory");
+    }
+  }
+
+  IndexUnitWriter UnitWriter =
+      makeIndexUnitWriter(CI, RecordOpts, Mod.FileName, nullptr, UnitModule);
+
+  if (UnitModule) {
+    serialization::ModuleFile *ModFile =
+        CI.getASTReader()->getModuleManager().lookup(UnitModule->getASTFile());
+    assert(ModFile && "no module file loaded for module ?");
+
+    for (auto *Mod : ModFile->Imports) {
+      auto *UnitMod = HS.lookupModule(Mod->ModuleName, /*AllowSearch=*/false);
+      if (!UnitMod->IsSystem || RecordOpts.RecordSystemDependencies) {
+        SmallString<256> AbsPath(Mod->File->getName());
+        CI.getFileManager().makeAbsolutePath(AbsPath);
+        UnitWriter.addIndexUnit(Mod->File, AbsPath, UnitMod->IsSystem, UnitMod);
+      }
+    }
+
+    CI.getASTReader()->visitInputFiles(
+        *ModFile, RecordOpts.RecordSystemDependencies,
+        /*Complain=*/false,
+        [&](const serialization::InputFile &IF, bool isSystemFile) {
+          auto *FE = IF.getFile();
+          if (!FE)
+            return;
+          // Ignore module map files, they are not as important to track as
+          // source files and they may be auto-generated which would create an
+          // undesirable dependency on an intermediate build byproduct.
+          if (FE->getName().endswith("module.modulemap"))
+            return;
+
+          // Try to find a Module that is both a submodule of UnitModule and
+          // owns the header.
+          Module *ModForHead = nullptr;
+          if (auto Mod = CI.getPreprocessor()
+                             .getHeaderSearchInfo()
+                             .findModuleForHeader(FE)
+                             .getModule())
+            if (Mod->isSubModuleOf(UnitModule))
+              ModForHead = Mod;
+
+          UnitWriter.addModule(ModForHead);
+
+          UnitWriter.addSourceFile(FE, isSystemFile, ModForHead);
+        });
+  }
+
+  if (std::error_code EC = sys::fs::create_directory(RecordOpts.RecordDir)) {
+    reportError(CI, "failed to create directory '%0': %1", RecordOpts.RecordDir,
+                EC.message());
+    return;
+  }
+  writeIndexRecordsToDisk(DeclsDataCollector.getRecordByFile(), UnitWriter, CI,
+                          RecordOpts, UnitModule);
+
+  std::string Error;
+  if (UnitWriter.writeToDisk(Error)) {
+    reportError(CI, "failed writing unit data: %0", Error);
+    return;
+  }
+
+  if (UnitModule) {
+    serialization::ModuleFile *ModFile =
+        CI.getASTReader()->getModuleManager().lookup(UnitModule->getASTFile());
+    for (auto *Mod : ModFile->Imports) {
+      auto *UnitMod = HS.lookupModule(Mod->ModuleName, /*AllowSearch=*/false);
+
+      if (!UnitMod->IsSystem || RecordOpts.RecordSystemDependencies) {
+        if (Mod->isModule()) {
+          llvm::Optional<bool> IndexUnitExist = doesIndexUnitForModuleExist(
+              Mod->FileName, CI.getFileManager(), RecordOpts);
+          if (IndexUnitExist.hasValue() && !IndexUnitExist.getValue())
+            emitIndexDataForModuleFile(*Mod, CI, IndexOpts, RecordOpts);
+        }
+      }
+    }
+  }
+}
+
+bool emitIndexDataForModule(const Module *Mod, const CompilerInstance &CI) {
+  assert(Mod);
+  auto astReader = CI.getASTReader();
+  serialization::ModuleFile *ModFile =
+      astReader->getModuleManager().lookup(Mod->getASTFile());
+  assert(ModFile && "no module file loaded for module ?");
+
+  const EmitIndexOptions RecOpts = parseEmitIndexOptions(CI.getFrontendOpts());
+
+  llvm::Optional<bool> IndexUnitExist = doesIndexUnitForModuleExist(
+      ModFile->FileName, CI.getFileManager(), RecOpts);
+  if (IndexUnitExist.hasValue() && IndexUnitExist.getValue())
+    return false;
+
+  emitIndexDataForModuleFile(
+      *ModFile, CI, parseIndexGenerationOptions(CI.getFrontendOpts()), RecOpts);
+  return true;
+}
+
+} // namespace index
+} // namespace clang

clang/lib/IndexSerialization/EmitIndexOptions.cpp

+//===--- EmitIndexOptions.cpp - Options for indexing ------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/IndexSerialization/EmitIndexOptions.h"
+#include "clang/IndexSerialization/IndexDataFormat.h"
+#include "llvm/Support/Path.h"
+
+namespace clang {
+namespace index {
+
+EmitIndexOptions::EmitIndexOptions(const StringRef RootDir,
+                                   const bool RecordSymbolCodeGenName,
+                                   const bool RecordSystemDependencies,
+                                   const IncludesRecordingKind RecordIncludes)
+    : RootDir(RootDir), IndexDir(llvm::Twine(RootDir)
+                                     .concat(llvm::sys::path::get_separator())
+                                     .concat(store::getIndexDir())
+                                     .str()),
+      UnitDir(llvm::Twine(IndexDir)
+                  .concat(llvm::sys::path::get_separator())
+                  .concat(store::getIndexUnitSubdir())
+                  .str()),
+      RecordDir(llvm::Twine(IndexDir)
+                    .concat(llvm::sys::path::get_separator())
+                    .concat(store::getIndexRecordSubdir())
+                    .str()),
+      RecordSymbolCodeGenName(RecordSymbolCodeGenName),
+      RecordSystemDependencies(RecordSystemDependencies),
+      RecordIncludes(RecordIncludes) {}
+
+EmitIndexOptions parseEmitIndexOptions(const FrontendOptions &FEOpts) {
+  return EmitIndexOptions(FEOpts.IndexStorePath, FEOpts.IndexRecordMangledName);
+}
+
+} // namespace index
+} // namespace clang

clang/lib/IndexSerialization/IndexDataFormat.cpp

+//===--- IndexDataFormat.cpp - Layout of index on a filesystem ------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/IndexSerialization/IndexDataFormat.h"
+
+#include "llvm/ADT/APInt.h"
+#include "llvm/ADT/Hashing.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/raw_ostream.h"
+
+using namespace llvm;
+
+namespace {
+
+#define STR_HELPER(x) #x
+#define STR(x) STR_HELPER(x)
+
+#define INDEX_FORMAT_VERSION_VALUE 5
+
+static const unsigned indexFormatVersion = INDEX_FORMAT_VERSION_VALUE;
+
+const char *const indexSubdir = "v" STR(INDEX_FORMAT_VERSION_VALUE);
+
+#undef INDEX_FORMAT_VERSION_VALUE
+#undef STR
+#undef STR_HELPER
+
+const char *const indexUnitSubdir = "units";
+
+const char *const indexRecordSubdir = "records";
+
+} // namespace
+
+namespace clang {
+namespace index {
+namespace store {
+
+unsigned getIndexFormatVersion() { return indexFormatVersion; }
+
+const char *getIndexDir() { return indexSubdir; }
+const char *getIndexUnitSubdir() { return indexUnitSubdir; }
+const char *getIndexRecordSubdir() { return indexRecordSubdir; }
+
+static void appendIndexSubDir(StringRef subdir, SmallVectorImpl<char> &Buffer) {
+  sys::path::append(Buffer, getIndexDir());
+  sys::path::append(Buffer, subdir);
+}
+
+void appendUnitBucketDir(StringRef UnitName, SmallVectorImpl<char> &Buffer) {
+  sys::path::append(Buffer, UnitName);
+}
+
+void appendUnitSubDir(SmallVectorImpl<char> &Buffer) {
+  return appendIndexSubDir(getIndexUnitSubdir(), Buffer);
+}
+
+void appendRecordSubDir(SmallVectorImpl<char> &Buffer) {
+  return appendIndexSubDir(getIndexRecordSubdir(), Buffer);
+}
+
+void appendRecordBucketDir(StringRef RecordName,
+                           SmallVectorImpl<char> &Buffer) {
+  // To avoid putting a huge number of files into the records directory, create
+  // subdirectories based on the last 2 characters from the hash.
+  sys::path::append(Buffer, RecordName.substr(RecordName.size() - 2));
+}
+
+void appendRecordFilename(StringRef RecordName, SmallVectorImpl<char> &Buffer) {
+  sys::path::append(Buffer, RecordName);
+}
+
+void getUnitNameForOutputFile(StringRef FilePath, SmallVectorImpl<char> &Str) {
+  StringRef Fname = sys::path::filename(FilePath);
+  Str.append(Fname.begin(), Fname.end());
+  Str.push_back('-');
+  llvm::hash_code PathHashVal = llvm::hash_value(FilePath);
+  llvm::APInt(64, PathHashVal).toString(Str, 36, /*Signed=*/false);
+}
+
+std::string generateFilenameForIndexRecord(
+    StringRef SourceFilename, const llvm::SmallString<512> &IndexRecordBuffer) {
+  const llvm::hash_code RecordHash = llvm::hash_combine_range(
+      IndexRecordBuffer.begin(), IndexRecordBuffer.end());
+
+  std::string RecordFilename;
+  llvm::raw_string_ostream RN(RecordFilename);
+  RN << llvm::sys::path::filename(SourceFilename);
+  RN << "-" << llvm::APInt(64, RecordHash).toString(36, /*Signed=*/false);
+  return RecordFilename;
+}
+
+} // end namespace store
+} // end namespace index
+} // end namespace clang
 No newline at end of file

clang/lib/IndexSerialization/IndexRecordReader.cpp

+//===--- IndexRecordReader.cpp - Index record deserialization -------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/IndexSerialization/IndexRecordReader.h"
+#include "BitstreamVisitor.h"
+#include "indexstore/Conversions.h"
+#include "clang/IndexSerialization/IndexDataFormat.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/Bitstream/BitstreamReader.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/raw_ostream.h"
+
+namespace clang {
+namespace index {
+
+using namespace clang::index::store;
+using namespace llvm;
+
+class IndexBitstreamVisitor : public BitstreamVisitor<IndexBitstreamVisitor> {
+  IndexRecordReader &Reader;
+
+public:
+  IndexBitstreamVisitor(llvm::BitstreamCursor &Stream,
+                        IndexRecordReader &Reader)
+      : BitstreamVisitor(Stream), Reader(Reader) {}
+
+  StreamVisit visitBlock(unsigned ID) {
+    switch ((RecordBitBlock)ID) {
+    case REC_VERSION_BLOCK_ID:
+    case REC_DECLOFFSETS_BLOCK_ID:
+      return StreamVisit::Continue;
+
+    case REC_DECLS_BLOCK_ID:
+      Reader.DeclCursor = Stream;
+      if (Reader.DeclCursor.EnterSubBlock(ID)) {
+        *Error = "malformed block record";
+        return StreamVisit::Abort;
+      }
+      if (llvm::Error Err = readBlockAbbrevs(Reader.DeclCursor)) {
+        *Error = toString(std::move(Err));
+        return StreamVisit::Abort;
+      }
+      return StreamVisit::Skip;
+
+    case REC_DECLOCCURRENCES_BLOCK_ID:
+      Reader.OccurCursor = Stream;
+      if (Reader.OccurCursor.EnterSubBlock(ID)) {
+        *Error = "malformed block record";
+        return StreamVisit::Abort;
+      }
+      if (llvm::Error Err = readBlockAbbrevs(Reader.OccurCursor)) {
+        *Error = toString(std::move(Err));
+        return StreamVisit::Abort;
+      }
+      return StreamVisit::Skip;
+    }
+
+    // Some newly introduced block in a minor version update that we cannot
+    // handle.
+    return StreamVisit::Skip;
+  }
+
+  StreamVisit visitRecord(unsigned BlockID, unsigned RecID,
+                          SmallVectorImpl<uint64_t> &Record, StringRef Blob) {
+    switch (BlockID) {
+    case REC_VERSION_BLOCK_ID: {
+      unsigned StoreFormatVersion = Record[0];
+      if (StoreFormatVersion != getIndexFormatVersion()) {
+        llvm::raw_string_ostream OS(*Error);
+        OS << "Store format version mismatch: " << StoreFormatVersion;
+        OS << " , expected: " << getIndexFormatVersion();
+        return StreamVisit::Abort;
+      }
+      break;
+    }
+    case REC_DECLOFFSETS_BLOCK_ID:
+      assert(RecID == REC_DECLOFFSETS);
+      Reader.setDeclOffsets(
+          makeArrayRef((const uint32_t *)Blob.data(), Record[0]));
+      break;
+
+    case REC_DECLS_BLOCK_ID:
+    case REC_DECLOCCURRENCES_BLOCK_ID:
+      llvm_unreachable("shouldn't visit this block'");
+    }
+    return StreamVisit::Continue;
+  }
+};
+
+std::unique_ptr<IndexRecordReader> IndexRecordReader::createWithRecordFilename(
+    StringRef RecordFilename, StringRef StorePath, std::string &Error) {
+  SmallString<128> PathBuf = StorePath;
+  appendRecordSubDir(PathBuf);
+  appendRecordBucketDir(RecordFilename, PathBuf);
+  appendRecordFilename(RecordFilename, PathBuf);
+  return createWithFilePath(PathBuf.str(), Error);
+}
+
+std::unique_ptr<IndexRecordReader>
+IndexRecordReader::createWithFilePath(StringRef FilePath, std::string &Error) {
+  auto ErrOrBuf = MemoryBuffer::getFile(FilePath, /*FileSize=*/-1,
+                                        /*RequiresNullTerminator=*/false);
+  if (!ErrOrBuf) {
+    raw_string_ostream(Error) << "failed opening index record '" << FilePath
+                              << "': " << ErrOrBuf.getError().message();
+    return nullptr;
+  }
+  return createWithBuffer(std::move(*ErrOrBuf), Error);
+}
+
+std::unique_ptr<IndexRecordReader>
+IndexRecordReader::createWithBuffer(std::unique_ptr<llvm::MemoryBuffer> Buffer,
+                                    std::string &Error) {
+  llvm::BitstreamCursor Stream(*Buffer);
+
+  if (Stream.AtEndOfStream()) {
+    Error = "empty file";
+    return nullptr;
+  }
+
+  // Sniff for the signature.
+  for (unsigned char C : {'I', 'D', 'X', 'R'}) {
+    if (Expected<llvm::SimpleBitstreamCursor::word_t> Res = Stream.Read(8)) {
+      if (Res.get() == C)
+        continue;
+    } else {
+      Error = toString(Res.takeError());
+      return nullptr;
+    }
+    Error = "not a serialized index record file";
+    return nullptr;
+  }
+
+  std::unique_ptr<IndexRecordReader> Reader =
+      std::make_unique<IndexRecordReader>();
+  Reader->Buffer = std::move(Buffer);
+
+  IndexBitstreamVisitor BitVisitor(Stream, *Reader);
+  if (!BitVisitor.visit(Error))
+    return nullptr;
+
+  return Reader;
+}
+
+bool IndexRecordReader::searchDecls(
+    llvm::function_ref<DeclSearchCheck> Checker,
+    llvm::function_ref<void(const IndexRecordDecl *)> Receiver) {
+  for (unsigned I = 0, E = getNumDecls(); I != E; ++I) {
+    if (const IndexRecordDecl *D = Decls[I]) {
+      DeclSearchReturn Ret = Checker(*D);
+      if (Ret.AcceptDecl)
+        Receiver(D);
+      if (!Ret.ContinueSearch)
+        return false;
+      continue;
+    }
+
+    IndexRecordDecl LocalD;
+    readDecl(I, LocalD);
+    DeclSearchReturn Ret = Checker(LocalD);
+    if (Ret.AcceptDecl) {
+      IndexRecordDecl *D = Allocator.Allocate<IndexRecordDecl>();
+      *D = LocalD;
+      Decls[I] = D;
+      Receiver(D);
+    }
+    if (!Ret.ContinueSearch)
+      return false;
+  }
+  return true;
+}
+
+bool IndexRecordReader::foreachDecl(
+    bool NoCache, function_ref<bool(const IndexRecordDecl *)> Receiver) {
+  for (unsigned I = 0, E = getNumDecls(); I != E; ++I) {
+    if (const IndexRecordDecl *D = Decls[I]) {
+      if (!Receiver(D))
+        return false;
+      continue;
+    }
+
+    if (NoCache) {
+      IndexRecordDecl LocalD;
+      readDecl(I, LocalD);
+      if (!Receiver(&LocalD))
+        return false;
+    } else {
+      if (!Receiver(getDecl(I)))
+        return false;
+    }
+  }
+  return true;
+}
+
+bool IndexRecordReader::foreachOccurrence(
+    function_ref<bool(const IndexRecordOccurrence &)> Receiver,
+    ArrayRef<const IndexRecordDecl *> WhitelistDecls,
+    ArrayRef<const IndexRecordDecl *> WhitelistRelated) {
+  class OccurBitVisitor : public BitstreamVisitor<OccurBitVisitor> {
+    IndexRecordReader &Reader;
+    ArrayRef<const IndexRecordDecl *> WhitelistDecls;
+    ArrayRef<const IndexRecordDecl *> WhitelistRelated;
+    function_ref<bool(const IndexRecordOccurrence &)> Receiver;
+
+  public:
+    OccurBitVisitor(llvm::BitstreamCursor &Stream, IndexRecordReader &Reader,
+                    ArrayRef<const IndexRecordDecl *> WhitelistDecls,
+                    ArrayRef<const IndexRecordDecl *> WhitelistRelated,
+                    function_ref<bool(const IndexRecordOccurrence &)> Receiver)
+        : BitstreamVisitor(Stream), Reader(Reader),
+          WhitelistDecls(WhitelistDecls), WhitelistRelated(WhitelistRelated),
+          Receiver(std::move(Receiver)) {}
+
+    StreamVisit visitRecord(unsigned BlockID, unsigned RecID,
+                            SmallVectorImpl<uint64_t> &Record, StringRef Blob) {
+      assert(RecID == REC_DECLOCCURRENCE);
+      IndexRecordOccurrence RecOccur;
+      if (Reader.readOccurrence(Record, Blob, WhitelistDecls, WhitelistRelated,
+                                RecOccur))
+        if (!Receiver(RecOccur))
+          return StreamVisit::Abort;
+      return StreamVisit::Continue;
+    }
+  };
+
+  SavedStreamPosition SavedPosition(OccurCursor);
+  OccurBitVisitor Visitor(OccurCursor, *this, WhitelistDecls, WhitelistRelated,
+                          Receiver);
+  std::string Error;
+  return Visitor.visit(Error);
+}
+
+bool IndexRecordReader::foreachOccurrenceInLineRange(
+    unsigned lineStart, unsigned lineCount,
+    llvm::function_ref<bool(const IndexRecordOccurrence &)> Receiver) {
+  // FIXME: Use binary search and make this more efficient.
+  unsigned lineEnd = lineStart + lineCount;
+  return foreachOccurrence(
+      [&](const IndexRecordOccurrence &occur) -> bool {
+        if (occur.Line > lineEnd)
+          return false; // we're done.
+        if (occur.Line >= lineStart) {
+          if (!Receiver(occur))
+            return false;
+        }
+        return true;
+      },
+      None, None);
+}
+
+void IndexRecordReader::setDeclOffsets(ArrayRef<uint32_t> Offs) {
+  DeclOffsets = Offs;
+  Decls = Allocator.Allocate<const IndexRecordDecl *>(Offs.size());
+  memset(Decls, 0, sizeof(IndexRecordDecl *) * Offs.size());
+}
+
+unsigned IndexRecordReader::getNumDecls() const { return DeclOffsets.size(); }
+
+const IndexRecordDecl *IndexRecordReader::getDeclByID(unsigned DeclID) {
+  if (DeclID == 0)
+    return nullptr;
+  return getDecl(DeclID - 1);
+}
+
+const IndexRecordDecl *IndexRecordReader::getDecl(unsigned Index) {
+  assert(Index < getNumDecls());
+  if (const IndexRecordDecl *D = Decls[Index])
+    return D;
+
+  IndexRecordDecl *D = Allocator.Allocate<IndexRecordDecl>();
+  readDecl(Index, *D);
+  Decls[Index] = D;
+  return D;
+}
+
+/// Reads occurrence data.
+/// \param WhitelistDecls if non-empty indicates the list of decls that we want
+/// to get occurrences for. If empty then indicates that we want occurrences
+/// for all decls.
+/// \param WhitelistRelated Same as \c WhitelistDecls but for related decls.
+/// \returns true if the occurrence info was filled out, false if occurrence
+/// was ignored.
+bool IndexRecordReader::readOccurrence(
+    SmallVectorImpl<uint64_t> &Record, StringRef Blob,
+    ArrayRef<const IndexRecordDecl *> WhitelistDecls,
+    ArrayRef<const IndexRecordDecl *> WhitelistRelated,
+    IndexRecordOccurrence &RecOccur) {
+  auto isDeclIDContained = [](unsigned DeclID,
+                              ArrayRef<const IndexRecordDecl *> Ds) -> bool {
+    if (Ds.empty())
+      return true; // empty means accept all.
+    auto pred = [DeclID](const IndexRecordDecl *D) {
+      return D->DeclID == DeclID;
+    };
+    return std::find_if(Ds.begin(), Ds.end(), pred) != Ds.end();
+  };
+
+  unsigned I = 0;
+  unsigned DeclID = Record[I++];
+  if (!isDeclIDContained(DeclID, WhitelistDecls))
+    return false;
+
+  if (!WhitelistRelated.empty()) {
+    unsigned RelI = I + 3;
+    unsigned NumRelated = Record[RelI++];
+    bool FoundRelated = false;
+    while (NumRelated--) {
+      ++RelI; // roles;
+      unsigned RelDID = Record[RelI++];
+      if (isDeclIDContained(RelDID, WhitelistRelated)) {
+        FoundRelated = true;
+        break;
+      }
+    }
+    if (!FoundRelated)
+      return false;
+  }
+
+  RecOccur.Dcl = getDeclByID(DeclID);
+  RecOccur.Roles = getSymbolRoles(Record[I++]);
+  RecOccur.Line = Record[I++];
+  RecOccur.Column = Record[I++];
+
+  unsigned NumRelated = Record[I++];
+  while (NumRelated--) {
+    SymbolRoleSet RelRoles = getSymbolRoles(Record[I++]);
+    const IndexRecordDecl *RelD = getDeclByID(Record[I++]);
+    RecOccur.Relations.emplace_back(RelRoles, RelD);
+  }
+
+  return true;
+}
+
+void IndexRecordReader::readDecl(unsigned Index, IndexRecordDecl &RecD) {
+  SmallVector<uint64_t, 64> Record;
+  StringRef Blob;
+  if (llvm::Error Err = DeclCursor.JumpToBit(DeclOffsets[Index])) {
+    // FIXME this drops errors on the floor.
+    consumeError(std::move(Err));
+    llvm_unreachable("Failure in JumpToBit()");
+    // FIXME error handling
+    return;
+  }
+  Expected<unsigned> MaybeCode = DeclCursor.ReadCode();
+  if (!MaybeCode) {
+    // FIXME this drops errors on the floor.
+    consumeError(MaybeCode.takeError());
+    llvm_unreachable("Failure in ReadCode()");
+    // FIXME error handling
+    return;
+  }
+
+  Expected<unsigned> MaybeRecID =
+      DeclCursor.readRecord(MaybeCode.get(), Record, &Blob);
+  if (!MaybeRecID) {
+    // FIXME this drops errors on the floor.
+    consumeError(MaybeRecID.takeError());
+    llvm_unreachable("Failure in ReadRecord()");
+    // FIXME error handling
+    return;
+  }
+  assert(MaybeRecID.get() == REC_DECLINFO);
+  unsigned I = 0;
+  RecD.DeclID = Index + 1;
+  RecD.SymInfo.Kind = getSymbolKind((indexstore_symbol_kind_t)Record[I++]);
+  RecD.SymInfo.SubKind =
+      getSymbolSubKind((indexstore_symbol_subkind_t)Record[I++]);
+  RecD.SymInfo.Lang =
+      getSymbolLanguage((indexstore_symbol_language_t)Record[I++]);
+  RecD.SymInfo.Properties = getSymbolProperties(Record[I++]);
+  RecD.Roles = getSymbolRoles(Record[I++]);
+  RecD.RelatedRoles = getSymbolRoles(Record[I++]);
+  size_t NameLen = Record[I++];
+  size_t USRLen = Record[I++];
+  RecD.Name = Blob.substr(0, NameLen);
+  RecD.USR = Blob.substr(NameLen, USRLen);
+  RecD.CodeGenName = Blob.substr(NameLen + USRLen);
+}
+
+} // namespace index
+} // namespace clang
 No newline at end of file

clang/lib/IndexSerialization/IndexRecordWriter.cpp

+//===--- IndexRecordWriter.cpp - Index record serialization -----*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/IndexSerialization/IndexRecordWriter.h"
+#include "BitstreamUtil.h"
+#include "indexstore/Conversions.h"
+#include "indexstore/indexstore.h"
+#include "clang/AST/ASTContext.h"
+#include "clang/AST/DeclTemplate.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Index/USRGeneration.h"
+#include "clang/IndexSerialization/IndexDataFormat.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/Support/Errc.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/FileUtilities.h"
+#include "llvm/Support/Path.h"
+
+namespace clang {
+namespace index {
+
+using namespace llvm;
+using namespace store;
+
+void IndexRecordWriter::addDeclOccurence(SymbolRoleSet Roles, FileID FID,
+                                         unsigned Offset, const Decl *D,
+                                         ArrayRef<SymbolRelation> Relations) {
+  assert(D->isCanonicalDecl() &&
+         "Occurrences should be associated with their canonical decl");
+
+  auto getIdxForDecl = [this](const Decl *D) -> unsigned {
+    const auto MaybeNewDecl = IndexForDecl.insert({D, Decls.size()});
+    if (MaybeNewDecl.second)
+      Decls.push_back(DeclInfo{D, SymbolRoleSet(), SymbolRoleSet()});
+    return MaybeNewDecl.first->second;
+  };
+
+  const unsigned DeclIdx = getIdxForDecl(D);
+  Decls[DeclIdx].Roles |= Roles;
+
+  const uint64_t REC_DECLOCCURRENCE = 3;
+
+  SourceManager &SM = D->getASTContext().getSourceManager();
+  const unsigned Line = SM.getLineNumber(FID, Offset);
+  const unsigned Column = SM.getColumnNumber(FID, Offset);
+
+  SmallVector<uint64_t, 10> OccurrenceData{
+      REC_DECLOCCURRENCE, // FIXME: This is constant value. We don't need to
+                          // store it.
+      DeclIdx + 1, getIndexStoreRoles(Roles), Line, Column, Relations.size()};
+
+  OccurrenceData.reserve(OccurrenceData.size() + 2 * Relations.size());
+  for (auto &Rel : Relations) {
+    OccurrenceData.push_back(getIndexStoreRoles(Rel.Roles));
+
+    const unsigned RelDeclIdx = getIdxForDecl(Rel.RelatedSymbol);
+    Decls[RelDeclIdx].RelatedRoles |= Rel.Roles;
+
+    OccurrenceData.push_back(RelDeclIdx + 1);
+  }
+
+  Occurrences.emplace(Offset, OccurrenceData);
+}
+
+static void writeHeader(BitstreamWriter &Stream) {
+  Stream.Emit('I', 8);
+  Stream.Emit('D', 8);
+  Stream.Emit('X', 8);
+  Stream.Emit('R', 8);
+}
+
+static void writeBlockInfo(BitstreamWriter &Stream) {
+  SmallVector<uint64_t, 64> Record;
+
+  Stream.EnterBlockInfoBlock();
+#define BLOCK(X) emitBlockID(X##_ID, #X, Stream, Record)
+#define RECORD(X) emitRecordID(X, #X, Stream, Record)
+
+  BLOCK(REC_VERSION_BLOCK);
+  RECORD(REC_VERSION);
+
+  BLOCK(REC_DECLS_BLOCK);
+  RECORD(REC_DECLINFO);
+
+  BLOCK(REC_DECLOFFSETS_BLOCK);
+  RECORD(REC_DECLOFFSETS);
+
+  BLOCK(REC_DECLOCCURRENCES_BLOCK);
+  RECORD(REC_DECLOCCURRENCE);
+
+#undef RECORD
+#undef BLOCK
+  Stream.ExitBlock();
+}
+
+static std::shared_ptr<BitCodeAbbrev> REC_VERSION_BLOCK_ID_Abbrev() {
+  static const std::shared_ptr<BitCodeAbbrev> CachedResult(new BitCodeAbbrev({
+      BitCodeAbbrevOp(REC_VERSION),
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 6) // Store format version
+  }));
+
+  return CachedResult;
+}
+
+static void writeVersionInfo(BitstreamWriter &Stream) {
+  using namespace llvm::sys;
+
+  Stream.EnterSubblock(REC_VERSION_BLOCK_ID, 3);
+  unsigned AbbrevCode = Stream.EmitAbbrev(REC_VERSION_BLOCK_ID_Abbrev());
+
+  const uint64_t Record[] = {REC_VERSION, getIndexFormatVersion()};
+  Stream.EmitRecordWithAbbrev(AbbrevCode, Record);
+
+  Stream.ExitBlock();
+}
+
+static std::shared_ptr<BitCodeAbbrev> DECLS_BLOCK_ID_Abbrev() {
+  static const std::shared_ptr<BitCodeAbbrev> CachedResult(new BitCodeAbbrev({
+      BitCodeAbbrevOp(REC_DECLINFO),
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 5), // Kind
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 5), // SubKind
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 5), // Language
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed,
+                      SymbolPropertyBitNum),                     // Properties
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, SymbolRoleBitNum), // Roles
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed,
+                      SymbolRoleBitNum),        // Related Roles
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 6), // Length of name in block
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 6), // Length of USR in block
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Blob) // Name + USR + CodeGen symbol name
+  }));
+
+  return CachedResult;
+}
+
+static std::shared_ptr<BitCodeAbbrev> DECLOFFSETS_BLOCK_ID_Abbrev() {
+  static const std::shared_ptr<BitCodeAbbrev> CachedResult(new BitCodeAbbrev({
+      BitCodeAbbrevOp(REC_DECLOFFSETS),
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 32), // Number of Decls
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Blob)       // Offsets array
+  }));
+
+  return CachedResult;
+}
+
+static std::shared_ptr<BitCodeAbbrev> DECLOCCURRENCES_BLOCK_ID_Abbrev() {
+  static const std::shared_ptr<BitCodeAbbrev> CachedResult(new BitCodeAbbrev({
+      BitCodeAbbrevOp(REC_DECLOCCURRENCE),
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 8),                  // Decl ID
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, SymbolRoleBitNum), // Roles
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 12),                 // Line
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 8),                  // Column
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 4),                  // Num related
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Array),  // Related Roles/IDs
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 16) // Roles or ID
+  }));
+
+  return CachedResult;
+}
+
+static SmallString<256> getUSR(const Decl *D) {
+  SmallString<256> Buf;
+  bool Ignore = generateUSRForDecl(D, Buf);
+  if (Ignore)
+    return {};
+  return Buf;
+}
+
+void IndexRecordWriter::getBitcodeRepresentation(
+    SmallString<512> &Buffer, const PrintingPolicy &Policy,
+    std::shared_ptr<ASTNameGenerator> MangledNameGen) const {
+  BitstreamWriter Stream(Buffer);
+  writeHeader(Stream);
+  writeBlockInfo(Stream);
+  writeVersionInfo(Stream);
+
+  SmallVector<uint32_t, 32> DeclOffsets;
+
+  {
+    Stream.EnterSubblock(REC_DECLS_BLOCK_ID, 3);
+    const unsigned AbbrevCode = Stream.EmitAbbrev(DECLS_BLOCK_ID_Abbrev());
+
+#ifndef NDEBUG
+    StringSet<> USRSet;
+    bool enableValidation = getenv("CLANG_INDEX_VALIDATION_CHECKS") != nullptr;
+#endif
+
+    DeclOffsets.reserve(Decls.size());
+
+    llvm::SmallString<256> Scratch;
+    for (auto &Declaration : Decls) {
+      DeclOffsets.push_back(Stream.GetCurrentBitNo());
+      Scratch.clear();
+
+      const Decl *D = static_cast<const Decl *>(Declaration.D);
+      auto Info = getSymbolInfo(D);
+      assert(Info.Kind != SymbolKind::Unknown);
+
+      std::string Name;
+      std::string CGName;
+      if (auto *ND = dyn_cast<NamedDecl>(D)) {
+        DeclarationName DeclName = ND->getDeclName();
+        if (!DeclName.isEmpty()) {
+          llvm::raw_string_ostream OS(Name);
+          DeclName.print(OS, Policy);
+        }
+        if (MangledNameGen) {
+          llvm::raw_string_ostream OS(CGName);
+          MangledNameGen->writeName(ND, OS);
+        }
+      }
+
+      const SmallString<256> USR = getUSR(D);
+      assert(!USR.empty() && "Recorded decl without USR!");
+
+#ifndef NDEBUG
+      if (enableValidation) {
+        bool IsNew = USRSet.insert(USR).second;
+        if (!IsNew) {
+          llvm::errs() << "Index: Duplicate USR! " << USR << "\n";
+          // FIXME: print more information so it's easier to find the
+          // declaration.
+        }
+      }
+#endif
+
+      const uint64_t Record[] = {REC_DECLINFO,
+                                 getIndexStoreKind(Info.Kind),
+                                 getIndexStoreSubKind(Info.SubKind),
+                                 getIndexStoreLang(Info.Lang),
+                                 getIndexStoreProperties(Info.Properties),
+                                 getIndexStoreRoles(Declaration.Roles),
+                                 getIndexStoreRoles(Declaration.RelatedRoles),
+                                 Name.size(),
+                                 USR.size()};
+
+      Stream.EmitRecordWithBlob(
+          AbbrevCode, Record,
+          llvm::Twine(Name).concat(USR).concat(CGName).str());
+    }
+
+    Stream.ExitBlock();
+  }
+
+  {
+    Stream.EnterSubblock(REC_DECLOFFSETS_BLOCK_ID, 3);
+    const unsigned AbbrevCode =
+        Stream.EmitAbbrev(DECLOFFSETS_BLOCK_ID_Abbrev());
+    const uint64_t Record[] = {REC_DECLOFFSETS, DeclOffsets.size()};
+    StringRef DeclOffsetsData(
+        reinterpret_cast<const char *>(DeclOffsets.data()),
+        sizeof(uint32_t) * DeclOffsets.size());
+
+    Stream.EmitRecordWithBlob(AbbrevCode, Record, DeclOffsetsData);
+    Stream.ExitBlock();
+  }
+
+  Stream.EnterSubblock(REC_DECLOCCURRENCES_BLOCK_ID, 3);
+  const unsigned AbbrevCode =
+      Stream.EmitAbbrev(DECLOCCURRENCES_BLOCK_ID_Abbrev());
+
+  for (auto &OffsetAndOccurrence : Occurrences) {
+    Stream.EmitRecordWithAbbrev(AbbrevCode, OffsetAndOccurrence.second);
+  }
+
+  Stream.ExitBlock();
+}
+
+template <unsigned N>
+static void reportError(DiagnosticsEngine &Diag, const char (&Format)[N],
+                        const std::string &Filename,
+                        const std::string &Message) {
+  unsigned DiagID = Diag.getCustomDiagID(DiagnosticsEngine::Error, Format);
+  Diag.Report(DiagID) << Filename << Message;
+}
+
+llvm::Optional<std::string>
+IndexRecordWriter::writeToDisk(StringRef SourceFileName,
+                               const EmitIndexOptions &RecordOpts,
+                               const PrintingPolicy &Policy,
+                               std::shared_ptr<ASTNameGenerator> MangledNameGen,
+                               DiagnosticsEngine &Diag) const {
+
+  llvm::SmallString<512> IndexRecordBuffer;
+  getBitcodeRepresentation(IndexRecordBuffer, Policy, MangledNameGen);
+
+  const std::string RecordFilename =
+      generateFilenameForIndexRecord(SourceFileName, IndexRecordBuffer);
+
+  llvm::SmallString<256> RecordBucketSubdir(RecordOpts.RecordDir);
+  store::appendRecordBucketDir(RecordFilename, RecordBucketSubdir);
+
+  if (std::error_code EC = sys::fs::create_directory(RecordBucketSubdir)) {
+    reportError(Diag, "failed to create directory '%0': %1",
+                RecordBucketSubdir.str().str(), EC.message());
+    return llvm::None;
+  }
+
+  llvm::SmallString<256> RecordPath(RecordBucketSubdir);
+  store::appendRecordFilename(RecordFilename, RecordPath);
+
+  if (std::error_code EC = llvm::sys::fs::access(
+          RecordPath.str().str().c_str(), llvm::sys::fs::AccessMode::Exist)) {
+    if (EC == llvm::errc::no_such_file_or_directory) {
+      // Create a unique file to write to so that we can move the result into
+      // place atomically. If this process crashes we don't want to interfere
+      // with any other concurrent processes.
+      // FIXME: We should also make sure we don't eat up all the disk in case
+      // we're crashing repeatedly.
+      const SmallString<128> TempPath(
+          llvm::Twine(RecordPath).concat("-temp-%%%%%%%%").str());
+
+      llvm::Error Err =
+          llvm::writeFileAtomically(TempPath, RecordPath, IndexRecordBuffer);
+      if (Err) {
+        // FIXME:  This assumes getDiagnosticClient().BeginSourceFile() has been
+        // called.
+        reportError(Diag, "failed writing record '%0': %1", RecordFilename,
+                    llvm::toString(std::move(Err)));
+        return llvm::None;
+      }
+    } else {
+      reportError(Diag, "failed stating record file '%0': %1", RecordFilename,
+                  "");
+      return llvm::None;
+    }
+  }
+
+  return RecordFilename;
+}
+
+} // end namespace index
+} // end namespace clang
 No newline at end of file

clang/lib/IndexSerialization/IndexUnitReader.cpp

+//===--- IndexUnitReader.cpp - Index unit deserialization -----------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/IndexSerialization/IndexUnitReader.h"
+#include "BitstreamVisitor.h"
+#include "clang/IndexSerialization/IndexDataFormat.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/Optional.h"
+#include "llvm/ADT/ScopeExit.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Bitstream/BitstreamReader.h"
+#include "llvm/Support/Chrono.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/Process.h"
+#include "llvm/Support/raw_ostream.h"
+
+#include "clang/IndexSerialization/SerializablePathCollection.h"
+
+using namespace clang;
+using namespace clang::index;
+using namespace clang::index::store;
+using namespace llvm;
+
+namespace {
+
+class IndexUnitBitstreamVisitor
+    : public BitstreamVisitor<IndexUnitBitstreamVisitor> {
+public:
+  llvm::BitstreamCursor DependenciesBlockCursor;
+  llvm::BitstreamCursor IncludedFilesCursor;
+  StringRef ProviderIdentifier;
+  StringRef ProviderVersion;
+  bool IsSystemUnit;
+  bool IsModuleUnit;
+  bool IsDebugCompilation;
+  StringRef ModuleName;
+  StringRef Target;
+
+  int MainPathIndex;
+
+  std::vector<PathPool::FilePath> Paths;
+  StringRef PathsBuffer;
+  size_t WorkDirOffset;
+  size_t WorkDirSize;
+  size_t OutputFileOffset;
+  size_t OutputFileSize;
+  size_t SysrootOffset;
+  size_t SysrootSize;
+
+  std::vector<ModuleInfo> Modules;
+  StringRef ModuleNamesBuffer;
+
+  IndexUnitBitstreamVisitor(llvm::BitstreamCursor &Stream)
+      : BitstreamVisitor(Stream) {}
+
+  StreamVisit visitBlock(unsigned ID) {
+    switch ((UnitBitBlock)ID) {
+    case UNIT_VERSION_BLOCK_ID:
+    case UNIT_INFO_BLOCK_ID:
+    case UNIT_PATHS_BLOCK_ID:
+    case UNIT_MODULES_BLOCK_ID:
+      return StreamVisit::Continue;
+
+    case UNIT_DEPENDENCIES_BLOCK_ID:
+      DependenciesBlockCursor = Stream;
+      if (DependenciesBlockCursor.EnterSubBlock(ID)) {
+        *Error = "malformed unit dependencies block record";
+        return StreamVisit::Abort;
+      }
+      if (llvm::Error Err = readBlockAbbrevs(DependenciesBlockCursor)) {
+        *Error = toString(std::move(Err));
+        return StreamVisit::Abort;
+      }
+      return StreamVisit::Skip;
+    case UNIT_INCLUDES_BLOCK_ID:
+      IncludedFilesCursor = Stream;
+      if (IncludedFilesCursor.EnterSubBlock(ID)) {
+        *Error = "malformed unit includes block record";
+        return StreamVisit::Abort;
+      }
+      if (llvm::Error Err = readBlockAbbrevs(IncludedFilesCursor)) {
+        *Error = toString(std::move(Err));
+        return StreamVisit::Abort;
+      }
+      return StreamVisit::Skip;
+    }
+
+    // Some newly introduced block in a minor version update that we cannot
+    // handle.
+    return StreamVisit::Skip;
+  }
+
+  StreamVisit visitRecord(unsigned BlockID, unsigned RecID,
+                          SmallVectorImpl<uint64_t> &Record, StringRef Blob) {
+    switch (BlockID) {
+    case UNIT_VERSION_BLOCK_ID: {
+      unsigned StoreFormatVersion = Record[0];
+      if (StoreFormatVersion != getIndexFormatVersion()) {
+        llvm::raw_string_ostream OS(*Error);
+        OS << "Store format version mismatch: " << StoreFormatVersion;
+        OS << " , expected: " << getIndexFormatVersion();
+        return StreamVisit::Abort;
+      }
+      break;
+    }
+
+    case UNIT_INFO_BLOCK_ID: {
+      assert(RecID == UNIT_INFO);
+      unsigned I = 0;
+      IsSystemUnit = Record[I++];
+
+      // Save these to lookup them up after we get the paths buffer.
+      WorkDirOffset = Record[I++];
+      WorkDirSize = Record[I++];
+      OutputFileOffset = Record[I++];
+      OutputFileSize = Record[I++];
+      SysrootOffset = Record[I++];
+      SysrootSize = Record[I++];
+      MainPathIndex = (int)Record[I++] - 1;
+      IsDebugCompilation = Record[I++];
+      IsModuleUnit = Record[I++];
+
+      const size_t moduleNameSize = Record[I++];
+      const size_t providerIdentifierSize = Record[I++];
+      const size_t providerVersionSize = Record[I++];
+      I++; // Reserved for ProviderDataVersion.
+      ModuleName = Blob.substr(0, moduleNameSize);
+      Blob = Blob.drop_front(moduleNameSize);
+      ProviderIdentifier = Blob.substr(0, providerIdentifierSize);
+      Blob = Blob.drop_front(providerIdentifierSize);
+      ProviderVersion = Blob.substr(0, providerVersionSize);
+      Target = Blob.drop_front(providerVersionSize);
+      break;
+    }
+
+    case UNIT_PATHS_BLOCK_ID:
+      switch (RecID) {
+      case UNIT_PATH: {
+        unsigned I = 0;
+        PathPool::RootDirKind Kind = (PathPool::RootDirKind)Record[I++];
+        size_t DirOffset = Record[I++];
+        size_t DirSize = Record[I++];
+        size_t FilenameOffset = Record[I++];
+        size_t FilenameSize = Record[I++];
+
+        Paths.emplace_back(
+            PathPool::DirPath(Kind,
+                              StringPool::StringOffSz(DirOffset, DirSize)),
+            StringPool::StringOffSz(FilenameOffset, FilenameSize));
+      } break;
+      case UNIT_PATH_BUFFER:
+        PathsBuffer = Blob;
+        break;
+      default:
+        llvm_unreachable("shouldn't visit this record");
+      }
+      break;
+
+    case UNIT_MODULES_BLOCK_ID:
+      switch (RecID) {
+      case UNIT_MODULE: {
+        unsigned I = 0;
+        unsigned NameOffset = Record[I++];
+        unsigned NameSize = Record[I++];
+        Modules.push_back({NameOffset, NameSize});
+      } break;
+      case UNIT_MODULE_BUFFER:
+        ModuleNamesBuffer = Blob;
+        break;
+      default:
+        llvm_unreachable("shouldn't visit this record");
+      }
+      break;
+
+    case UNIT_DEPENDENCIES_BLOCK_ID:
+    case UNIT_INCLUDES_BLOCK_ID:
+      llvm_unreachable("shouldn't visit this block'");
+    }
+    return StreamVisit::Continue;
+  }
+};
+
+typedef std::function<bool(SmallVectorImpl<uint64_t> &Record, StringRef Blob)>
+    BlockVisitorCallback;
+
+class IndexUnitBlockBitstreamVisitor
+    : public BitstreamVisitor<IndexUnitBlockBitstreamVisitor> {
+  unsigned RecID;
+  BlockVisitorCallback Callback;
+
+public:
+  IndexUnitBlockBitstreamVisitor(unsigned RecID,
+                                 llvm::BitstreamCursor &BlockStream,
+                                 BlockVisitorCallback Callback)
+      : BitstreamVisitor(BlockStream), RecID(RecID), Callback(Callback) {}
+
+  StreamVisit visitRecord(unsigned BlockID, unsigned RecID,
+                          SmallVectorImpl<uint64_t> &Record, StringRef Blob) {
+    if (RecID != this->RecID)
+      llvm_unreachable("shouldn't be called with this RecID");
+
+    if (Callback(Record, Blob))
+      return StreamVisit::Continue;
+    return StreamVisit::Abort;
+  }
+};
+
+} // anonymous namespace
+
+std::unique_ptr<IndexUnitReader>
+IndexUnitReader::create(std::unique_ptr<MemoryBuffer> Buf,
+                        sys::TimePoint<> ModTime, std::string &Error) {
+  if (!Buf) {
+    Error = "null buffer";
+    return nullptr;
+  }
+
+  llvm::BitstreamCursor Stream(*Buf);
+
+  if (Stream.AtEndOfStream()) {
+    Error = "empty file";
+    return nullptr;
+  }
+
+  // Sniff for the signature.
+  for (unsigned char C : {'I', 'D', 'X', 'U'}) {
+    if (Expected<llvm::SimpleBitstreamCursor::word_t> Res = Stream.Read(8)) {
+      if (Res.get() == C)
+        continue;
+    } else {
+      Error = toString(Res.takeError());
+      return nullptr;
+    }
+    Error = "not a serialized index unit file";
+    return nullptr;
+  }
+
+  IndexUnitBitstreamVisitor Visitor(Stream);
+  if (!Visitor.visit(Error)) {
+    // Error is set by Visitor
+    return nullptr;
+  }
+
+  return std::make_unique<IndexUnitReader>(
+      std::move(Buf), std::move(Visitor.DependenciesBlockCursor),
+      std::move(Visitor.IncludedFilesCursor), Visitor.ProviderIdentifier,
+      Visitor.ProviderVersion, Visitor.IsSystemUnit, Visitor.IsModuleUnit,
+      Visitor.IsDebugCompilation, Visitor.ModuleName, Visitor.Target,
+      Visitor.MainPathIndex, std::move(Visitor.Paths), Visitor.PathsBuffer,
+      Visitor.WorkDirOffset, Visitor.WorkDirSize, Visitor.OutputFileOffset,
+      Visitor.OutputFileSize, Visitor.SysrootOffset, Visitor.SysrootSize,
+      std::move(Visitor.Modules), Visitor.ModuleNamesBuffer, ModTime);
+}
+
+std::unique_ptr<IndexUnitReader> IndexUnitReader::createWithUnitFilename(
+    StringRef UnitFilename, StringRef StorePath, std::string &Error) {
+  SmallString<128> PathBuf = StorePath;
+  appendUnitSubDir(PathBuf);
+  sys::path::append(PathBuf, UnitFilename);
+  return createWithFilePath(PathBuf.str(), Error);
+}
+
+std::unique_ptr<IndexUnitReader>
+IndexUnitReader::createWithFilePath(StringRef FilePath, std::string &Error) {
+  int FD;
+  std::error_code EC = sys::fs::openFileForRead(FilePath, FD);
+  if (EC) {
+    raw_string_ostream(Error)
+        << "Failed opening '" << FilePath << "': " << EC.message();
+    return nullptr;
+  }
+  assert(FD != -1);
+  auto FDScopeGuard = llvm::make_scope_exit(
+      [FD]() { llvm::sys::Process::SafelyCloseFileDescriptor(FD); });
+
+  sys::fs::file_status FileStat;
+  EC = sys::fs::status(FD, FileStat);
+  if (EC) {
+    Error = EC.message();
+    return nullptr;
+  }
+
+  auto ErrOrBuf = MemoryBuffer::getOpenFile(sys::fs::convertFDToNativeFile(FD),
+                                            FilePath, /*FileSize=*/-1,
+                                            /*RequiresNullTerminator=*/false);
+  if (!ErrOrBuf) {
+    raw_string_ostream(Error) << "Failed opening '" << FilePath
+                              << "': " << ErrOrBuf.getError().message();
+    return nullptr;
+  }
+
+  return IndexUnitReader::create(std::move(*ErrOrBuf),
+                                 FileStat.getLastModificationTime(), Error);
+}
+
+Optional<sys::TimePoint<>> IndexUnitReader::getModificationTimeForUnit(
+    StringRef UnitFilename, StringRef StorePath, std::string &Error) {
+  SmallString<128> PathBuf = StorePath;
+  appendUnitSubDir(PathBuf);
+  sys::path::append(PathBuf, UnitFilename);
+
+  sys::fs::file_status FileStat;
+  std::error_code EC = sys::fs::status(PathBuf.str(), FileStat);
+  if (EC) {
+    Error = EC.message();
+    return None;
+  }
+  return FileStat.getLastModificationTime();
+}
+
+StringRef IndexUnitReader::getProviderIdentifier() const {
+  return ProviderIdentifier;
+}
+
+StringRef IndexUnitReader::getProviderVersion() const {
+  return ProviderVersion;
+}
+
+llvm::sys::TimePoint<> IndexUnitReader::getModificationTime() const {
+  return ModTime;
+}
+
+StringRef IndexUnitReader::getWorkingDirectory() const {
+  return PathsBuffer.substr(WorkDirOffset, WorkDirSize);
+}
+
+StringRef IndexUnitReader::getOutputFile() const {
+  return PathsBuffer.substr(OutputFileOffset, OutputFileSize);
+}
+
+StringRef IndexUnitReader::getSysrootPath() const {
+  return PathsBuffer.substr(SysrootOffset, SysrootSize);
+}
+
+StringRef IndexUnitReader::getMainFilePath() const { return MainFilePath; }
+
+StringRef IndexUnitReader::getModuleName() const { return ModuleName; }
+
+StringRef IndexUnitReader::getTarget() const { return Target; }
+
+bool IndexUnitReader::hasMainFile() const { return !MainFilePath.empty(); }
+
+bool IndexUnitReader::isSystemUnit() const { return IsSystemUnit; }
+
+bool IndexUnitReader::isModuleUnit() const { return IsModuleUnit; }
+
+bool IndexUnitReader::isDebugCompilation() const { return IsDebugCompilation; }
+
+void IndexUnitReader::constructFilePath(SmallVectorImpl<char> &PathBuf,
+                                        int PathIndex) const {
+
+  if (PathIndex < 0)
+    return;
+  assert(static_cast<unsigned long>(PathIndex) < Paths.size());
+  const PathPool::FilePath &Path = Paths[PathIndex];
+  StringRef Prefix;
+  switch (Path.Dir.Root) {
+  case PathPool::RootDirKind::Regular:
+    break;
+  case PathPool::RootDirKind::CurrentWorkDir:
+    Prefix = getWorkingDirectory();
+    break;
+  case PathPool::RootDirKind::SysRoot:
+    Prefix = getSysrootPath();
+    break;
+  }
+  PathBuf.append(Prefix.begin(), Prefix.end());
+  sys::path::append(
+      PathBuf, PathsBuffer.substr(Path.Dir.Path.Offset, Path.Dir.Path.Size),
+      PathsBuffer.substr(Path.Filename.Offset, Path.Filename.Size));
+}
+
+/// \c Index is the index in the \c getDependencies array.
+/// Unit dependencies are provided ahead of record ones, record ones
+/// ahead of the file ones.
+bool IndexUnitReader::foreachDependency(
+    llvm::function_ref<bool(const DependencyInfo &Info)> Receiver) {
+  store::SavedStreamPosition SavedDepPosition(DependenciesBlockCursor);
+  IndexUnitBlockBitstreamVisitor Visitor(
+      UNIT_DEPENDENCY, DependenciesBlockCursor,
+      [&](SmallVectorImpl<uint64_t> &Record, StringRef Blob) {
+        unsigned I = 0;
+        UnitDependencyKind UnitDepKind = (UnitDependencyKind)Record[I++];
+        bool IsSystem = Record[I++];
+        int PathIndex = (int)Record[I++] - 1;
+        int ModuleIndex = (int)Record[I++] - 1;
+        I++; // Reserved field.
+        I++; // Reserved field.
+        StringRef Name = Blob;
+
+        IndexUnitReader::DependencyKind DepKind;
+        switch (UnitDepKind) {
+        case UNIT_DEPEND_KIND_UNIT:
+          DepKind = IndexUnitReader::DependencyKind::Unit;
+          break;
+        case UNIT_DEPEND_KIND_RECORD:
+          DepKind = IndexUnitReader::DependencyKind::Record;
+          break;
+        case UNIT_DEPEND_KIND_FILE:
+          DepKind = IndexUnitReader::DependencyKind::File;
+          break;
+        }
+
+        SmallString<512> PathBuf;
+        constructFilePath(PathBuf, PathIndex);
+
+        StringRef ModuleName;
+        if (ModuleIndex >= 0) {
+          auto &ModInfo = Modules[ModuleIndex];
+          ModuleName = StringRef(ModuleNamesBuffer.data() + ModInfo.NameOffset,
+                                 ModInfo.NameSize);
+        }
+
+        return Receiver(IndexUnitReader::DependencyInfo{
+            DepKind, IsSystem, Name, PathBuf.str(), ModuleName});
+      });
+
+  std::string Error; // FIXME
+  return Visitor.visit(Error);
+}
+
+bool IndexUnitReader::foreachInclude(
+    llvm::function_ref<bool(const IncludeInfo &Info)> Receiver) {
+  store::SavedStreamPosition SavedIncPosition(IncludedFilesCursor);
+  IndexUnitBlockBitstreamVisitor Visitor(
+      UNIT_INCLUDE, IncludedFilesCursor,
+      [&](SmallVectorImpl<uint64_t> &Record, StringRef Blob) {
+        unsigned I = 0;
+        int SourcePathIndex = (int)Record[I++] - 1;
+        unsigned Line = Record[I++];
+        int TargetPathIndex = (int)Record[I++] - 1;
+
+        SmallString<512> SourceBuf, TargetBuf;
+        this->constructFilePath(SourceBuf, SourcePathIndex);
+        this->constructFilePath(TargetBuf, TargetPathIndex);
+        return Receiver(IndexUnitReader::IncludeInfo{SourceBuf.str(), Line,
+                                                     TargetBuf.str()});
+      });
+
+  std::string Error;
+  return Visitor.visit(Error);
+}
+
+IndexUnitReader::IndexUnitReader(
+    std::unique_ptr<llvm::MemoryBuffer> Buf,
+    llvm::BitstreamCursor &&DependenciesBlockCursor,
+    llvm::BitstreamCursor &&IncludedFilesCursor, StringRef ProviderIdentifier,
+    StringRef ProviderVersion, bool IsSystemUnit, bool IsModuleUnit,
+    bool IsDebugCompilation, StringRef ModuleName, StringRef Target,
+    int MainPathIndex, std::vector<PathPool::FilePath> &&Paths,
+    StringRef PathsBuffer,
+    // Offset-size pairs to the PathsBuffer.
+    size_t WorkDirOffset, size_t WorkDirSize, size_t OutputFileOffset,
+    size_t OutputFileSize, size_t SysrootOffset, size_t SysrootSize,
+    std::vector<ModuleInfo> &&Modules, StringRef ModuleNamesBuffer,
+    llvm::sys::TimePoint<> ModTime)
+    : Buf(std::move(Buf)), DependenciesBlockCursor(DependenciesBlockCursor),
+      IncludedFilesCursor(IncludedFilesCursor),
+      ProviderIdentifier(ProviderIdentifier), ProviderVersion(ProviderVersion),
+      IsSystemUnit(IsSystemUnit), IsModuleUnit(IsModuleUnit),
+      IsDebugCompilation(IsDebugCompilation), ModuleName(ModuleName),
+      Target(Target), Paths(Paths), PathsBuffer(PathsBuffer),
+      WorkDirOffset(WorkDirOffset), WorkDirSize(WorkDirSize),
+      OutputFileOffset(OutputFileOffset), OutputFileSize(OutputFileSize),
+      SysrootOffset(SysrootOffset), SysrootSize(SysrootSize), Modules(Modules),
+      ModuleNamesBuffer(ModuleNamesBuffer), ModTime(ModTime) {
+  constructFilePath(MainFilePath, MainPathIndex);
+}
 No newline at end of file

clang/lib/IndexSerialization/IndexUnitWriter.cpp

+//===--- IndexUnitWriter.cpp - Index unit serialization -------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/IndexSerialization/IndexUnitWriter.h"
+#include "BitstreamUtil.h"
+#include "clang/Basic/FileManager.h"
+#include "clang/Basic/Module.h"
+#include "clang/Basic/TargetOptions.h"
+#include "clang/Basic/Version.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/IndexSerialization/IndexDataFormat.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/StringMap.h"
+#include "llvm/Bitstream/BitstreamWriter.h"
+#include "llvm/Support/Allocator.h"
+#include "llvm/Support/Errc.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/FileUtilities.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/raw_ostream.h"
+
+namespace {
+// FIXME: should this be in clang/Basic/Version?
+std::string getClangVersion() {
+  // Try picking the version from an Apple Clang tag.
+  const std::string RepositoryPath = clang::getClangRepositoryPath();
+  llvm::StringRef BuildNumber(RepositoryPath);
+  if (BuildNumber.startswith("clang")) {
+    const size_t DashOffset = BuildNumber.find('-');
+    if (DashOffset != llvm::StringRef::npos)
+      return BuildNumber.str().substr(DashOffset + 1);
+  }
+  // Fallback to the generic version.
+  return CLANG_VERSION_STRING;
+}
+} // namespace
+
+namespace clang {
+namespace index {
+
+using namespace clang::index::store;
+using namespace llvm;
+using namespace llvm::sys;
+
+unsigned IndexUnitWriter::addModule(writer::OpaqueModule Mod) {
+  if (!Mod)
+    return 0;
+  const auto Inserted = ModToIdx.insert({Mod, ModToIdx.size() + 1});
+  if (Inserted.second) {
+    SmallString<64> ModuleName;
+    StringRef name = GetInfoForModuleFn(Mod, ModuleName).Name;
+    const size_t ModuleNameOffset = ModuleNamesBuf.size();
+    ModuleNamesBuf += name;
+
+    Modules.push_back({UNIT_MODULE, ModuleNameOffset, name.size()});
+  }
+  return Inserted.first->second;
+}
+
+IndexUnitWriter::IndexUnitWriter(
+    FileManager &FileMgr, const EmitIndexOptions &RecOpts,
+    StringRef ProviderIdentifier, StringRef ProviderVersion,
+    StringRef OutputFile, StringRef AbsPathToOutputFile,
+    const FileEntry *MainFile,
+    const llvm::Optional<ModuleIndexData> &ModuleData, bool IsDebugCompilation,
+    StringRef TargetTriple, StringRef WorkDir, StringRef SysrootPath,
+    writer::ModuleInfoWriterCallback GetInfoForModule)
+    : RecOpts(RecOpts), ProviderIdentifier(ProviderIdentifier),
+      ProviderVersion(ProviderVersion), OutputFile(OutputFile),
+      AbsPathToOutputFile(AbsPathToOutputFile), MainFile(MainFile),
+      ModuleData(ModuleData), IsDebugCompilation(IsDebugCompilation),
+      TargetTriple(TargetTriple), WorkDir(WorkDir), SysrootPath(SysrootPath),
+      PathStore(WorkDir, SysrootPath, OutputFile),
+      GetInfoForModuleFn(GetInfoForModule) {}
+
+void IndexUnitWriter::addSourceFile(const FileEntry *File, bool IsSystem,
+                                    writer::OpaqueModule Mod) {
+  assert(File);
+
+  SourceFiles[File->getUID()] = std::array<uint64_t, 7>{
+      UNIT_DEPENDENCY,
+      UNIT_DEPEND_KIND_FILE,
+      IsSystem,
+      File ? static_cast<uint64_t>(PathStore.tryStoreFilePath(*File) + 1) : 0,
+      addModule(Mod),
+      0, // Reserved.
+      0  // Reserved.
+  };
+}
+
+void IndexUnitWriter::addIndexRecord(StringRef FileName,
+                                     const FileEntry *SourceFile, bool IsSystem,
+                                     writer::OpaqueModule Mod) {
+  assert(SourceFile);
+
+  SourceFileHasARecord.insert(SourceFile->getUID());
+
+  IndexRecordDependencies.push_back(
+      std::make_pair<std::string, std::array<uint64_t, 7>>(
+          FileName.str(),
+          {
+              UNIT_DEPENDENCY, UNIT_DEPEND_KIND_RECORD, IsSystem,
+              SourceFile ? static_cast<uint64_t>(
+                               PathStore.tryStoreFilePath(*SourceFile) + 1)
+                         : 0,
+              addModule(Mod),
+              0, // Reserved.
+              0  // Reserved.
+          }));
+}
+
+void IndexUnitWriter::addIndexUnit(const FileEntry *ASTFile,
+                                   StringRef AbsolutePath, bool IsSystem,
+                                   writer::OpaqueModule Mod,
+                                   bool withoutUnitName) {
+  assert(ASTFile);
+  if (!ASTFilesOfSeenIndexUnits.insert(ASTFile->getUID()).second)
+    return;
+
+  SmallString<64> UnitName;
+  if (!withoutUnitName) {
+    getUnitNameForOutputFile(AbsolutePath, UnitName);
+  }
+
+  IndexUnitDependencies.emplace_back(
+      UnitName, std::array<uint64_t, 7>{
+                    UNIT_DEPENDENCY, UNIT_DEPEND_KIND_UNIT, IsSystem,
+                    ASTFile ? static_cast<uint64_t>(
+                                  PathStore.tryStoreFilePath(*ASTFile) + 1)
+                            : 0,
+                    addModule(Mod),
+                    0, // Reserved.
+                    0  // Reserved.
+                });
+}
+
+void IndexUnitWriter::addIncludedFile(const FileEntry *IncludingFile,
+                                      unsigned LineNo,
+                                      const FileEntry *IncludedFile) {
+  IncludedFiles.push_back(
+      {UNIT_INCLUDE, // FIXME: This is static data which don't have to be stored
+                     // here.
+       IncludingFile ? static_cast<uint64_t>(
+                           PathStore.tryStoreFilePath(*IncludingFile) + 1)
+                     : 0,
+       LineNo,
+       IncludedFile ? static_cast<uint64_t>(
+                          PathStore.tryStoreFilePath(*IncludedFile) + 1)
+                    : 0});
+};
+
+static void writeBlockInfo(BitstreamWriter &Stream) {
+  SmallVector<uint64_t, 64> Record;
+
+  Stream.EnterBlockInfoBlock();
+#define BLOCK(X) emitBlockID(X##_ID, #X, Stream, Record)
+#define RECORD(X) emitRecordID(X, #X, Stream, Record)
+
+  BLOCK(UNIT_VERSION_BLOCK);
+  RECORD(UNIT_VERSION);
+
+  BLOCK(UNIT_INFO_BLOCK);
+  RECORD(UNIT_INFO);
+
+  BLOCK(UNIT_DEPENDENCIES_BLOCK);
+  RECORD(UNIT_DEPENDENCY);
+
+  BLOCK(UNIT_INCLUDES_BLOCK);
+  RECORD(UNIT_INCLUDE);
+
+  BLOCK(UNIT_PATHS_BLOCK);
+  RECORD(UNIT_PATH);
+  RECORD(UNIT_PATH_BUFFER);
+
+  BLOCK(UNIT_MODULES_BLOCK);
+  RECORD(UNIT_MODULE);
+  RECORD(UNIT_MODULE_BUFFER);
+
+#undef RECORD
+#undef BLOCK
+  Stream.ExitBlock();
+}
+
+static std::shared_ptr<BitCodeAbbrev> UNIT_VERSION_BLOCK_ID_Abbrev() {
+  static const std::shared_ptr<BitCodeAbbrev> CachedResult(new BitCodeAbbrev({
+      BitCodeAbbrevOp(UNIT_VERSION),
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 6) // Store format version
+  }));
+
+  return CachedResult;
+}
+
+static void writeVersionInfo(BitstreamWriter &Stream) {
+  using namespace llvm::sys;
+
+  Stream.EnterSubblock(UNIT_VERSION_BLOCK_ID, 3);
+  unsigned AbbrevCode = Stream.EmitAbbrev(UNIT_VERSION_BLOCK_ID_Abbrev());
+
+  const uint64_t Record[2]{UNIT_VERSION, getIndexFormatVersion()};
+  Stream.EmitRecordWithAbbrev(AbbrevCode, Record);
+
+  Stream.ExitBlock();
+}
+
+bool IndexUnitWriter::writeToDisk(std::string &Error) const {
+  SmallString<512> BitcodeRepresentation;
+  getBitcodeRepresentation(BitcodeRepresentation, Error);
+
+  SmallString<256> UnitPath =
+      StringRef(llvm::Twine(RecOpts.UnitDir)
+                    .concat(llvm::sys::path::get_separator())
+                    .str());
+
+  getUnitNameForOutputFile(AbsPathToOutputFile, UnitPath);
+
+  const SmallString<128> TempPath(
+      llvm::Twine(path::parent_path(RecOpts.UnitDir))
+          .concat(llvm::sys::path::get_separator().str())
+          .concat(path::filename(UnitPath))
+          .concat("-%%%%%%%%")
+          .str());
+
+  llvm::Error Err =
+      llvm::writeFileAtomically(TempPath, UnitPath, BitcodeRepresentation);
+  if (Err) {
+    Error = llvm::toString(std::move(Err));
+    return true;
+  }
+  return false;
+}
+
+void IndexUnitWriter::getBitcodeRepresentation(SmallString<512> &Buffer,
+                                               std::string &Error) const {
+  using namespace llvm::sys;
+
+  BitstreamWriter Stream(Buffer);
+  Stream.Emit('I', 8);
+  Stream.Emit('D', 8);
+  Stream.Emit('X', 8);
+  Stream.Emit('U', 8);
+
+  writeBlockInfo(Stream);
+  writeVersionInfo(Stream);
+  writeUnitInfo(Stream);
+  writeDependencies(Stream);
+  writeIncludes(Stream);
+  writePaths(Stream);
+  writeModules(Stream);
+}
+
+static std::shared_ptr<BitCodeAbbrev> UNIT_INFO_BLOCK_ID_Abbrev() {
+  static const std::shared_ptr<BitCodeAbbrev> CachedResult(new BitCodeAbbrev({
+      BitCodeAbbrevOp(UNIT_INFO),
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 1), // IsSystemUnit
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 10),  // WorkDir offset
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 8),   // WorkDir size
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 10),  // OutputFile offset
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 8),   // OutputFile size
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 10),  // Sysroot offset
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 8),   // Sysroot size
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 10),  // Main path id
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 1), // IsDebugCompilation
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 1), // IsModuleUnit
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 5),   // Module name size
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 5),   // ProviderIdentifier size
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 5),   // ProviderVersion size
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 5),   // ProviderDataVersion
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Blob) // Module name + ProviderIdentifier
+                                             // + ProviderVersion + target
+                                             // triple
+  }));
+
+  return CachedResult;
+}
+
+void IndexUnitWriter::writeUnitInfo(llvm::BitstreamWriter &Stream) const {
+  Stream.EnterSubblock(UNIT_INFO_BLOCK_ID, 3);
+  unsigned AbbrevCode = Stream.EmitAbbrev(UNIT_INFO_BLOCK_ID_Abbrev());
+
+  const std::string ModuleName = ModuleData.hasValue() ? ModuleData->Name : "";
+
+  const uint64_t Record[15]{
+      UNIT_INFO,
+      ModuleData.hasValue() &&
+          ModuleData->IsSystem, // We don't expect compilation of any source
+                                // code that would be considered system unless
+                                // it's a dependency in form of module.
+      PathStore.WorkDirPath.Offset, PathStore.WorkDirPath.Size,
+      PathStore.OutputFilePath.Offset, PathStore.OutputFilePath.Size,
+      PathStore.SysRootPath.Offset, PathStore.SysRootPath.Size,
+      MainFile
+          ? static_cast<uint64_t>(PathStore.tryStoreFilePath(*MainFile) + 1)
+          : 0,
+      IsDebugCompilation, ModuleData.hasValue(), ModuleName.size(),
+      ProviderIdentifier.size(), ProviderVersion.size(),
+      // ProviderDataVersion is reserved. Not sure it is a good to idea to have
+      // clients consider the specifics of a 'provider data version', but
+      // reserving to avoid store format version change in case there is a use
+      // case in the future.
+      0 // ProviderDataVersion
+  };
+  const std::string InfoStrings = llvm::Twine(ModuleName)
+                                      .concat(ProviderIdentifier)
+                                      .concat(ProviderVersion)
+                                      .concat(TargetTriple)
+                                      .str();
+
+  Stream.EmitRecordWithBlob(AbbrevCode, Record, InfoStrings);
+
+  Stream.ExitBlock();
+}
+
+static std::shared_ptr<BitCodeAbbrev> UNIT_DEPENDENCIES_BLOCK_ID_Abbrev() {
+  static const std::shared_ptr<BitCodeAbbrev> CachedResult(new BitCodeAbbrev({
+      BitCodeAbbrevOp(UNIT_DEPENDENCY),
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed,
+                      UnitDependencyKindBitNum),  // Dependency kind
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 1), // IsSystem
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR,
+                      10), // PathIndex (1-based, 0 = none)
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR,
+                      8), // ModuleIndex (1-based, 0 = none)
+      // Reserved. These used to be time_t & file size but we decided against
+      // writing these in order to get reproducible build products (index data
+      // output being the same with the same inputs). Keep these reserved for
+      // the future, for coming up with a better scheme to track state of
+      // dependencies without using modification time.
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 0), // Reserved
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed, 0), // Reserved
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Blob)      // Name
+  }));
+
+  return CachedResult;
+}
+
+void IndexUnitWriter::writeDependencies(llvm::BitstreamWriter &Stream) const {
+  Stream.EnterSubblock(UNIT_DEPENDENCIES_BLOCK_ID, 3);
+
+  unsigned AbbrevCode = Stream.EmitAbbrev(UNIT_DEPENDENCIES_BLOCK_ID_Abbrev());
+
+  for (auto &IdxUnit : IndexUnitDependencies) {
+    Stream.EmitRecordWithBlob(AbbrevCode, IdxUnit.second, IdxUnit.first);
+  }
+
+  for (auto &recordData : IndexRecordDependencies) {
+    Stream.EmitRecordWithBlob(AbbrevCode, recordData.second, recordData.first);
+  }
+
+  for (const auto &FileAndData : SourceFiles) {
+    if (SourceFileHasARecord.count(FileAndData.first) > 0)
+      continue;
+    Stream.EmitRecordWithBlob(AbbrevCode, FileAndData.second, StringRef());
+  }
+
+  Stream.ExitBlock();
+}
+
+static std::shared_ptr<BitCodeAbbrev> UNIT_INCLUDES_BLOCK_ID_Abbrev() {
+  static const std::shared_ptr<BitCodeAbbrev> CachedResult(new BitCodeAbbrev({
+      BitCodeAbbrevOp(UNIT_INCLUDE),
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR,
+                      10), // source path index (1-based, 0 = no path)
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 12), // source include line
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR,
+                      10) // target path index (1-based, 0 = no path)
+  }));
+
+  return CachedResult;
+}
+
+void IndexUnitWriter::writeIncludes(llvm::BitstreamWriter &Stream) const {
+  Stream.EnterSubblock(UNIT_INCLUDES_BLOCK_ID, 3);
+
+  unsigned AbbrevCode = Stream.EmitAbbrev(UNIT_INCLUDES_BLOCK_ID_Abbrev());
+
+  for (const auto &IncludeDirective : IncludedFiles) {
+    Stream.EmitRecordWithAbbrev(AbbrevCode, IncludeDirective);
+  }
+
+  Stream.ExitBlock();
+}
+
+static std::shared_ptr<BitCodeAbbrev> UNIT_PATHS_BLOCK_ID_Paths_Abbrev() {
+  static const std::shared_ptr<BitCodeAbbrev> CachedResult(new BitCodeAbbrev({
+      BitCodeAbbrevOp(UNIT_PATH),
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Fixed,
+                      UnitFilePathPrefixKindBitNum), // Path prefix kind
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 10),     // DirPath offset
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 8),      // DirPath size
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 10),     // Filename offset
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 6)       // Filename size
+  }));
+
+  return CachedResult;
+}
+
+static std::shared_ptr<BitCodeAbbrev>
+UNIT_PATHS_BLOCK_ID_Path_Buffers_Abbrev() {
+  static const std::shared_ptr<BitCodeAbbrev> CachedResult(new BitCodeAbbrev({
+      BitCodeAbbrevOp(UNIT_PATH_BUFFER),
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Blob) // Paths buffer
+  }));
+
+  return CachedResult;
+}
+
+void IndexUnitWriter::writePaths(llvm::BitstreamWriter &Stream) const {
+  Stream.EnterSubblock(UNIT_PATHS_BLOCK_ID, 3);
+
+  const unsigned PathAbbrevCode =
+      Stream.EmitAbbrev(UNIT_PATHS_BLOCK_ID_Paths_Abbrev());
+  const unsigned PathBufferAbbrevCode =
+      Stream.EmitAbbrev(UNIT_PATHS_BLOCK_ID_Path_Buffers_Abbrev());
+
+  for (auto &FilePath : PathStore.getFilePaths()) {
+    const uint64_t Record[6]{UNIT_PATH,
+                             static_cast<uint64_t>(FilePath.Dir.Root),
+                             FilePath.Dir.Path.Offset,
+                             FilePath.Dir.Path.Size,
+                             FilePath.Filename.Offset,
+                             FilePath.Filename.Size};
+    Stream.EmitRecordWithAbbrev(PathAbbrevCode, Record);
+  }
+
+  Stream.EmitRecordWithBlob(PathBufferAbbrevCode, UNIT_PATH_BUFFER,
+                            PathStore.getPathsBuffer());
+
+  Stream.ExitBlock();
+}
+
+static std::shared_ptr<BitCodeAbbrev> UNIT_MODULES_BLOCK_ID_Module_Abbrev() {
+  static const std::shared_ptr<BitCodeAbbrev> CachedResult(new BitCodeAbbrev({
+      BitCodeAbbrevOp(UNIT_MODULE),
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 9), // Module name offset
+      BitCodeAbbrevOp(BitCodeAbbrevOp::VBR, 6)  // Module name size
+  }));
+
+  return CachedResult;
+}
+
+static std::shared_ptr<BitCodeAbbrev>
+UNIT_MODULES_BLOCK_ID_Module_Buffer_Abbrev() {
+  static const std::shared_ptr<BitCodeAbbrev> CachedResult(new BitCodeAbbrev({
+      BitCodeAbbrevOp(UNIT_MODULE_BUFFER),
+      BitCodeAbbrevOp(BitCodeAbbrevOp::Blob) // Module names buffer
+  }));
+
+  return CachedResult;
+}
+
+void IndexUnitWriter::writeModules(llvm::BitstreamWriter &Stream) const {
+  Stream.EnterSubblock(UNIT_MODULES_BLOCK_ID, 3);
+  unsigned AbbrevCode =
+      Stream.EmitAbbrev(UNIT_MODULES_BLOCK_ID_Module_Abbrev());
+  unsigned BufferAbbrevCode =
+      Stream.EmitAbbrev(UNIT_MODULES_BLOCK_ID_Module_Buffer_Abbrev());
+
+  for (auto &Module : Modules) {
+    Stream.EmitRecordWithAbbrev(AbbrevCode, Module);
+  }
+
+  Stream.EmitRecordWithBlob(BufferAbbrevCode, UNIT_MODULE_BUFFER,
+                            ModuleNamesBuf.str());
+  Stream.ExitBlock();
+}
+
+static writer::ModuleInfo getModuleInfo(writer::OpaqueModule mod,
+                                        SmallVectorImpl<char> &Scratch) {
+  assert(mod);
+  writer::ModuleInfo info;
+  const std::string fullName =
+      static_cast<const Module *>(mod)->getFullModuleName();
+  unsigned offset = Scratch.size();
+  Scratch.append(fullName.begin(), fullName.end());
+  info.Name = StringRef(Scratch.data() + offset, fullName.size());
+  return info;
+}
+
+IndexUnitWriter makeIndexUnitWriter(const CompilerInstance &CI,
+                                    const EmitIndexOptions &RecordOpts,
+                                    StringRef OutputFile,
+                                    const FileEntry *MainFile,
+                                    Module *UnitModule) {
+  assert(MainFile == nullptr || UnitModule == nullptr);
+
+  SmallString<256> AbsOutputFilePath(OutputFile);
+  CI.getFileManager().makeAbsolutePath(AbsOutputFilePath);
+
+  llvm::Optional<ModuleIndexData> ModuleData =
+      [UnitModule]() -> llvm::Optional<ModuleIndexData> {
+    if (UnitModule) {
+      return ModuleIndexData{UnitModule->getFullModuleName(),
+                             static_cast<bool>(UnitModule->IsSystem)};
+    }
+    return llvm::None;
+  }();
+
+  SmallString<128> WorkDir;
+  if (!CI.getFileManager().getFileSystemOpts().WorkingDir.empty()) {
+    WorkDir = CI.getFileManager().getFileSystemOpts().WorkingDir;
+    if (!llvm::sys::path::is_absolute(WorkDir)) {
+      llvm::sys::fs::make_absolute(WorkDir);
+    }
+  } else {
+    std::error_code EC = llvm::sys::fs::current_path(WorkDir);
+    if (EC) {
+      // FIXME: error handling
+      llvm_unreachable("failed to determine current working directory");
+    }
+  }
+
+  return IndexUnitWriter{CI.getFileManager(),
+                         RecordOpts,
+                         "clang",
+                         getClangVersion(),
+                         OutputFile,
+                         AbsOutputFilePath,
+                         MainFile,
+                         ModuleData,
+                         CI.getCodeGenOpts().OptimizationLevel == 0,
+                         CI.getTargetOpts().Triple,
+                         WorkDir.str(),
+                         CI.getHeaderSearchOpts().Sysroot,
+                         getModuleInfo};
+}
+
+bool initIndexDirectory(const EmitIndexOptions &RecordingOpts,
+                        std::string &Error /* <- kill */) {
+  using namespace llvm::sys;
+  auto CreateDir = [&](StringRef DirPath) {
+    std::error_code EC = fs::create_directories(DirPath);
+    if (EC) {
+      llvm::raw_string_ostream Err(Error);
+      Err << "failed to create directory '" << DirPath << "': " << EC.message();
+      return true;
+    }
+    return false;
+  };
+  if (CreateDir(RecordingOpts.IndexDir))
+    return true;
+  if (CreateDir(RecordingOpts.RecordDir))
+    return true;
+  if (CreateDir(RecordingOpts.UnitDir))
+    return true;
+
+  return false;
+}
+
+llvm::Optional<bool>
+doesIndexUnitForModuleExist(StringRef ModuleFilePath,
+                            const FileManager &FileMgr,
+                            const EmitIndexOptions &RecOpts) {
+  SmallString<256> UnitPath(
+      StringRef(llvm::Twine(RecOpts.UnitDir)
+                    .concat(llvm::sys::path::get_separator().str())
+                    .str()));
+
+  SmallString<256> AbsPath(ModuleFilePath);
+  FileMgr.makeAbsolutePath(AbsPath);
+
+  getUnitNameForOutputFile(AbsPath, UnitPath);
+
+  llvm::sys::fs::file_status UnitStat;
+  if (std::error_code EC = llvm::sys::fs::status(UnitPath.c_str(), UnitStat)) {
+    if (EC == llvm::errc::no_such_file_or_directory)
+      return false;
+
+    return None;
+  }
+  return true;
+}
+
+} // namespace index
+} // namespace clang
 No newline at end of file

clang/test/Index/Store/Inputs/head.h

+
+extern void test1_func(void);
+extern void test2_func(void);

clang/test/Index/Store/Inputs/json.c.json

+{
+  "files": [
+    "/test1.o",
+    "/Inputs/test1.c",
+    "/Inputs/head.h",
+    "/test2.o",
+    "/Inputs/test2.c",
+    "/test3.o",
+    "/Inputs/test3.cpp"
+  ],
+  "symbols": [
+    {
+      "kind": "function",
+      "lang": "C",
+      "usr": "c:@F@test1_func",
+      "name": "test1_func",
+      "codegen": "_test1_func",
+      "roles": "Decl,Def"
+    },
+    {
+      "kind": "function",
+      "lang": "C",
+      "usr": "c:@F@test2_func",
+      "name": "test2_func",
+      "codegen": "_test2_func",
+      "roles": "Decl,Def"
+    },
+    {
+      "kind": "class",
+      "lang": "C++",
+      "usr": "c:@S@Base",
+      "name": "Base",
+      "roles": "Def,Ref,RelBase,RelCont"
+    },
+    {
+      "kind": "class",
+      "lang": "C++",
+      "usr": "c:@S@Sub",
+      "name": "Sub",
+      "roles": "Def",
+      "rel-roles": "RelBase,RelCont"
+    }
+  ],
+  "records": [
+    {
+      "occurrences": [
+        {
+          "symbol": 0,
+          "line": 3,
+          "col": 6,
+          "roles": "Def"
+        }
+      ]
+    },
+    {
+      "occurrences": [
+        {
+          "symbol": 0,
+          "line": 2,
+          "col": 13,
+          "roles": "Decl"
+        },
+        {
+          "symbol": 1,
+          "line": 3,
+          "col": 13,
+          "roles": "Decl"
+        }
+      ]
+    },
+    {
+      "occurrences": [
+        {
+          "symbol": 1,
+          "line": 3,
+          "col": 6,
+          "roles": "Def"
+        }
+      ]
+    },
+    {
+      "occurrences": [
+        {
+          "symbol": 2,
+          "line": 1,
+          "col": 7,
+          "roles": "Def"
+        },
+        {
+          "symbol": 3,
+          "line": 2,
+          "col": 7,
+          "roles": "Def"
+        },
+        {
+          "symbol": 2,
+          "line": 2,
+          "col": 20,
+          "roles": "Ref,RelBase,RelCont",
+          "relations": [
+            {
+              "symbol": 3,
+              "rel-roles": "RelBase,RelCont"
+            }
+          ]
+
+        }
+      ]
+    }
+  ],
+  "units": [
+    {
+      "triple": "x86_64-apple-macosx10.7.0",
+      "out-file": 0,
+      "sources": [
+        {
+          "file": 1,
+          "records": [0]
+        },
+        {
+          "file": 2,
+          "records": [1]
+        }
+      ]
+    },
+    {
+      "triple": "x86_64-apple-macosx10.7.0",
+      "out-file": 3,
+      "sources": [
+        {
+          "file": 4,
+          "records": [2]
+        },
+        {
+          "file": 2,
+          "records": [1]
+        }
+      ]
+    },
+    {
+      "triple": "x86_64-apple-macosx10.7.0",
+      "out-file": 5,
+      "sources": [
+        {
+          "file": 6,
+          "records": [3]
+        }
+      ]
+    }
+  ]
+}

clang/test/Index/Store/Inputs/module/ModDep.h

+#include "ModTop.h"
+
+void ModDep_func(ModTopStruct s);

clang/test/Index/Store/Inputs/module/ModSystem.h

+
+typedef struct {} ModSystemStruct;
+
+void ModSystem_func(void);

clang/test/Index/Store/Inputs/module/ModTop.h

+
+typedef struct {} ModTopStruct;
+
+void ModTop_func(void);

clang/test/Index/Store/Inputs/module/ModTopSub1.h

+void ModTopSub1_func(void);

clang/test/Index/Store/Inputs/module/ModTopSub2.h

+// This header has no symbols, intended to show up as file dependency.

clang/test/Index/Store/Inputs/module/module.modulemap

+module ModTop {
+  header "ModTop.h"
+  export *
+  module Sub1 {
+    header "ModTopSub1.h"
+  }
+  module Sub2 {
+    header "ModTopSub2.h"
+  }
+}
+module ModDep { header "ModDep.h" export * }
+module ModSystem [system] { header "ModSystem.h" export * }

clang/test/Index/Store/Inputs/overlay.yaml

+{
+  'version': 0,
+  'roots': [{ 'type': 'file', 'name': 'OUT_DIR/using-overlay.h',
+             'external-contents': 'INPUT_DIR/using-overlay.h'
+           }]
+}

clang/test/Index/Store/Inputs/print-unit.h

+#include "head.h"
+#include "using-overlay.h"

clang/test/Index/Store/Inputs/sys/another.h

+
+extern void sys_another_func(void);

clang/test/Index/Store/Inputs/sys/syshead.h

+
+#include "another.h"
+
+extern void sys_test1_func(void);

clang/test/Index/Store/Inputs/test1.c

+#include "head.h"
+
+void test1_func(void) {}

clang/test/Index/Store/Inputs/test2.c

+#include "head.h"
+
+void test2_func(void) {}

clang/test/Index/Store/Inputs/test3.cpp

+class Base {};
+class Sub : public Base {};

clang/test/Index/Store/Inputs/using-overlay.h

+void using_overlay(void);

clang/test/Index/Store/assembly-invocation.c

+// Make sure it doesn't crash.
+// RUN: %clang -target x86_64-apple-macosx10.7 -S %s -o %t.s
+// RUN: env CLANG_PROJECT_INDEX_PATH=%t.idx %clang -target x86_64-apple-macosx10.7 -c %t.s -o %t.o

clang/test/Index/Store/empty-unit.c

+void foo(int i);
+
+// RUN: rm -rf %t/idx
+// RUN: %clang_cc1 -index-store-path %t/idx %s -o %t.o
+// RUN: touch %t.empty
+
+// RUN: cp %t.empty $(find %t/idx -name "empty-unit.c*o*")
+// RUN: not c-index-test core -print-unit %t/idx 2> %t.err
+// RUN: FileCheck %s -input-file %t.err -check-prefix ERR-UNIT
+// ERR-UNIT: error loading unit: empty file
+
+// Also check for empty record files.
+// RUN: rm -rf %t/idx2
+// RUN: %clang_cc1 -index-store-path %t/idx2 %s -o %t.o
+// RUN: cp %t.empty $(find %t/idx2 -name "empty-unit.c-*")
+// RUN: not c-index-test core -print-record %t/idx2 2> %t2.err
+// RUN: FileCheck %s -input-file %t2.err -check-prefix ERR-RECORD
+// ERR-RECORD: error loading record: empty file

clang/test/Index/Store/external-source-symbol-hash.m

+// RUN: rm -rf %t.idx
+// RUN: %clang_cc1 %s -index-store-path %t.idx -D USE_EXTERNAL
+// RUN: c-index-test core -print-record %t.idx | FileCheck %s
+// RUN: %clang_cc1 %s -index-store-path %t.idx
+// RUN: find %t.idx/*/records -name "external-source-symbol-hash*" | count 2
+
+#ifdef USE_EXTERNAL
+#  define EXT_DECL(mod_name) __attribute__((external_source_symbol(language="Swift", defined_in=mod_name)))
+#else
+#  define EXT_DECL(mod_name)
+#endif
+
+#define NS_ENUM(_name, _type) enum _name:_type _name; enum _name : _type
+
+// Forward declarations should pick up the attribute from later decls
+@protocol P1;
+// CHECK: [[@LINE-1]]:11 | protocol/Swift | c:@M@some_module@objc(pl)P1 | Ref | rel: 0
+@class I2;
+// CHECK: [[@LINE-1]]:8 | class/Swift | c:@M@other_module@objc(cs)I2 | Ref | rel: 0
+enum E3: int;
+// CHECK: [[@LINE-1]]:6 | enum/Swift | c:@M@third_module@E@E3 | Decl | rel: 0
+
+void test(id<P1> first, I2 *second, enum E3 third) {}
+// CHECK: [[@LINE-1]]:14 | protocol/Swift | c:@M@some_module@objc(pl)P1 | Ref,RelCont | rel: 1
+// CHECK: [[@LINE-2]]:25 | class/Swift | c:@M@other_module@objc(cs)I2 | Ref,RelCont | rel: 1
+// CHECK: [[@LINE-3]]:42 | enum/Swift | c:@M@third_module@E@E3 | Ref,RelCont | rel: 1
+
+EXT_DECL("some_module")
+@protocol P1
+// CHECK: [[@LINE-1]]:11 | protocol/Swift | c:@M@some_module@objc(pl)P1 | Decl | rel: 0
+-(void)method;
+// CHECK: [[@LINE-1]]:8 | instance-method(protocol)/Swift | c:@M@some_module@objc(pl)P1(im)method | Decl,Dyn,RelChild | rel: 1
+@end
+
+EXT_DECL("other_module")
+@interface I2
+// CHECK: [[@LINE-1]]:12 | class/Swift | c:@M@other_module@objc(cs)I2 | Decl | rel: 0
+-(void)method;
+// CHECK: [[@LINE-1]]:8 | instance-method/Swift | c:@M@other_module@objc(cs)I2(im)method | Decl,Dyn,RelChild | rel: 1
+@end
+
+
+typedef NS_ENUM(E3, int) {
+// CHECK: [[@LINE-1]]:17 | enum/Swift | c:@M@third_module@E@E3 | Def | rel: 0
+  firstCase = 1,
+  // CHECK: [[@LINE-1]]:3 | enumerator/Swift | c:@M@third_module@E@E3@firstCase | Def,RelChild | rel: 1
+} EXT_DECL("third_module");

clang/test/Index/Store/handle-prebuilt-module.m

+// RUN: rm -rf %t
+// RUN: mkdir %t
+// RUN: %clang -arch x86_64 -mmacosx-version-min=10.7 -fsyntax-only %s -o %t.o -index-store-path %t/idx1 -fmodules -fmodules-cache-path=%t/mcp -I %S/Inputs/module -Rindex-store 2> %t.err1
+// RUN: %clang -arch x86_64 -mmacosx-version-min=10.7 -fsyntax-only %s -o %t.o -index-store-path %t/idx2 -fmodules -fmodules-cache-path=%t/mcp -I %S/Inputs/module -Rindex-store 2> %t.err2
+// RUN: %clang -arch x86_64 -mmacosx-version-min=10.7 -fsyntax-only %s -o %t.o -index-store-path %t/idx2 -fmodules -fmodules-cache-path=%t/mcp -I %S/Inputs/module -Rindex-store 2> %t.err3
+// RUN: FileCheck -input-file=%t.err1 -check-prefix=CREATING_MODULES %s -allow-empty
+// RUN: FileCheck -input-file=%t.err2 -check-prefix=CREATING_INDEX_DATA_FROM_MODULE_FILES %s
+// RUN: FileCheck -input-file=%t.err3 -check-prefix=EXISTING_INDEX_DATA_FROM_MODULE_FILES %s -allow-empty
+// RUN: c-index-test core -print-unit %t/idx1 | sort > %t/all-units1.txt
+// RUN: c-index-test core -print-unit %t/idx2 | sort > %t/all-units2.txt
+// RUN: c-index-test core -print-record %t/idx1 | sort > %t/all-records1.txt
+// RUN: c-index-test core -print-record %t/idx2 | sort > %t/all-records2.txt
+// RUN: diff -u %t/all-units1.txt %t/all-units2.txt
+// RUN: diff -u %t/all-records1.txt %t/all-records2.txt
+
+@import ModDep;
+
+// CREATING_MODULES-NOT: remark:
+
+// CREATING_INDEX_DATA_FROM_MODULE_FILES: remark: producing index data for module file {{.*}}ModDep{{.*}}.pcm
+// CREATING_INDEX_DATA_FROM_MODULE_FILES: remark: producing index data for module file {{.*}}ModTop{{.*}}.pcm
+
+// EXISTING_INDEX_DATA_FROM_MODULE_FILES-NOT: remark:

clang/test/Index/Store/json-with-module.m

+// RUN: rm -rf %t.idx %t.mcp
+// RUN: %clang -target x86_64-apple-darwin -arch x86_64 -mmacosx-version-min=10.7 -c %s -o %t.o -index-store-path %t.idx -fmodules -fmodules-cache-path=%t.mcp -Xclang -fdisable-module-hash -I %S/Inputs/module
+// RUN: c-index-test core -aggregate-json %t.idx -o %t.json
+// RUN: sed -e "s:%S::g" -e "s:%T::g" %t.json > %t.final.json
+// RUN: diff -u %s.json %t.final.json
+
+@import ModDep;

clang/test/Index/Store/json-with-module.m.json

+{
+  "files": [
+    "/json-with-module.m.tmp.mcp/ModDep.pcm",
+    "/json-with-module.m.tmp.mcp/ModTop.pcm",
+    "/Inputs/module/ModDep.h",
+    "/Inputs/module/ModTop.h",
+    "/Inputs/module/ModTopSub1.h",
+    "/Inputs/module/ModTopSub2.h",
+    "/json-with-module.m.tmp.o",
+    "/json-with-module.m",
+    "/Inputs/module/module.modulemap"
+  ],
+  "symbols": [
+    {
+      "kind": "function",
+      "lang": "C",
+      "usr": "c:@F@ModDep_func",
+      "name": "ModDep_func",
+      "roles": "Decl",
+      "rel-roles": "RelCont"
+    },
+    {
+      "kind": "type-alias",
+      "lang": "C",
+      "usr": "c:@T@ModTopStruct",
+      "name": "ModTopStruct",
+      "roles": "Def,Ref,RelCont"
+    },
+    {
+      "kind": "struct",
+      "lang": "C",
+      "usr": "c:@SA@ModTopStruct",
+      "name": "",
+      "roles": "Def"
+    },
+    {
+      "kind": "function",
+      "lang": "C",
+      "usr": "c:@F@ModTop_func",
+      "name": "ModTop_func",
+      "roles": "Decl"
+    },
+    {
+      "kind": "function",
+      "lang": "C",
+      "usr": "c:@F@ModTopSub1_func",
+      "name": "ModTopSub1_func",
+      "roles": "Decl"
+    }
+  ],
+  "records": [
+    {
+      "occurrences": [
+        {
+          "symbol": 0,
+          "line": 3,
+          "col": 6,
+          "roles": "Decl"
+        },
+        {
+          "symbol": 1,
+          "line": 3,
+          "col": 18,
+          "roles": "Ref,RelCont",
+          "relations": [
+            {
+              "symbol": 0,
+              "rel-roles": "RelCont"
+            }
+          ]
+
+        }
+      ]
+    },
+    {
+      "occurrences": [
+        {
+          "symbol": 2,
+          "line": 2,
+          "col": 9,
+          "roles": "Def"
+        },
+        {
+          "symbol": 1,
+          "line": 2,
+          "col": 19,
+          "roles": "Def"
+        },
+        {
+          "symbol": 3,
+          "line": 4,
+          "col": 6,
+          "roles": "Decl"
+        }
+      ]
+    },
+    {
+      "occurrences": [
+        {
+          "symbol": 4,
+          "line": 1,
+          "col": 6,
+          "roles": "Decl"
+        }
+      ]
+    }
+  ],
+  "units": [
+    {
+      "triple": "x86_64-apple-macosx10.7.0",
+      "out-file": 0,
+      "unit-dependencies": [1],
+      "sources": [
+        {
+          "file": 2,
+          "records": [0]
+        }
+      ]
+    },
+    {
+      "triple": "x86_64-apple-macosx10.7.0",
+      "out-file": 1,
+      "sources": [
+        {
+          "file": 3,
+          "records": [1]
+        },
+        {
+          "file": 4,
+          "records": [2]
+        },
+        {
+          "file": 5
+        }
+      ]
+    },
+    {
+      "triple": "x86_64-apple-macosx10.7.0",
+      "out-file": 6,
+      "unit-dependencies": [0],
+      "sources": [
+        {
+          "file": 7
+        },
+        {
+          "file": 8
+        }
+      ]
+    }
+  ]
+}

clang/test/Index/Store/json-with-pch.c

+// RUN: rm -rf %t.idx
+// RUN: %clang -target x86_64-apple-darwin -arch x86_64 -mmacosx-version-min=10.7 -x c-header %S/Inputs/head.h -o %t.h.pch -index-store-path %t.idx
+// RUN: %clang -target x86_64-apple-darwin -arch x86_64 -mmacosx-version-min=10.7 -c %s -o %t.o -index-store-path %t.idx -include %t.h -Werror
+// RUN: c-index-test core -aggregate-json %t.idx -o %t.json
+// RUN: sed -e "s:%S::g" -e "s:%T::g" %t.json > %t.final.json
+// RUN: diff -u %s.json %t.final.json
+
+int main() {
+  test1_func();
+}

clang/test/Index/Store/json-with-pch.c.json

+{
+  "files": [
+    "/json-with-pch.c.tmp.h.pch",
+    "/Inputs/head.h",
+    "/json-with-pch.c.tmp.o",
+    "/json-with-pch.c"
+  ],
+  "symbols": [
+    {
+      "kind": "function",
+      "lang": "C",
+      "usr": "c:@F@test1_func",
+      "name": "test1_func",
+      "roles": "Decl,Ref,Call,RelCall,RelCont"
+    },
+    {
+      "kind": "function",
+      "lang": "C",
+      "usr": "c:@F@test2_func",
+      "name": "test2_func",
+      "roles": "Decl"
+    },
+    {
+      "kind": "function",
+      "lang": "C",
+      "usr": "c:@F@main",
+      "name": "main",
+      "roles": "Def",
+      "rel-roles": "RelCall,RelCont"
+    }
+  ],
+  "records": [
+    {
+      "occurrences": [
+        {
+          "symbol": 0,
+          "line": 2,
+          "col": 13,
+          "roles": "Decl"
+        },
+        {
+          "symbol": 1,
+          "line": 3,
+          "col": 13,
+          "roles": "Decl"
+        }
+      ]
+    },
+    {
+      "occurrences": [
+        {
+          "symbol": 2,
+          "line": 8,
+          "col": 5,
+          "roles": "Def"
+        },
+        {
+          "symbol": 0,
+          "line": 9,
+          "col": 3,
+          "roles": "Ref,Call,RelCall,RelCont",
+          "relations": [
+            {
+              "symbol": 2,
+              "rel-roles": "RelCall,RelCont"
+            }
+          ]
+
+        }
+      ]
+    }
+  ],
+  "units": [
+    {
+      "triple": "x86_64-apple-macosx10.7.0",
+      "out-file": 0,
+      "sources": [
+        {
+          "file": 1,
+          "records": [0]
+        }
+      ]
+    },
+    {
+      "triple": "x86_64-apple-macosx10.7.0",
+      "out-file": 2,
+      "unit-dependencies": [0],
+      "sources": [
+        {
+          "file": 3,
+          "records": [1]
+        }
+      ]
+    }
+  ]
+}

clang/test/Index/Store/json.c

+// RUN: rm -rf %t.idx
+// RUN: mkdir -p %t.o
+// RUN: env CLANG_PROJECT_INDEX_PATH=%t.idx %clang -target x86_64-apple-macosx10.7 -arch x86_64 -mmacosx-version-min=10.7 -c %S/Inputs/test1.c -o %t.o/test1.o
+// RUN: env CLANG_PROJECT_INDEX_PATH=%t.idx %clang -target x86_64-apple-macosx10.7 -arch x86_64 -mmacosx-version-min=10.7 -c %S/Inputs/test2.c -o %t.o/test2.o
+// RUN: env CLANG_PROJECT_INDEX_PATH=%t.idx %clang -target x86_64-apple-macosx10.7 -arch x86_64 -mmacosx-version-min=10.7 -c %S/Inputs/test3.cpp -o %t.o/test3.o
+// RUN: c-index-test core -aggregate-json %t.idx -o %t.json
+// RUN: sed -e "s:%S::g" -e "s:%t.o::g" %t.json > %t.final.json
+// RUN: diff -u %S/Inputs/json.c.json %t.final.json

clang/test/Index/Store/print-record.mm

+// RUN: rm -rf %t.idx
+// RUN: %clang_cc1 %s -index-store-path %t.idx
+// RUN: c-index-test core -print-record %t.idx | FileCheck %s
+
+
+
+@class MyCls;
+
+@interface MyCls
+@end
+
+// CHECK: [[@LINE+2]]:6 | function/C | c:@F@foo#*$objc(cs)MyCls# | Decl | rel: 0
+// CHECK: [[@LINE+1]]:10 | class/ObjC | c:objc(cs)MyCls | Ref,RelCont | rel: 1
+void foo(MyCls *p);
+
+
+// RANGE-NOT: before_range
+void before_range();
+
+// RANGE: [[@LINE+1]]:6 | function/C | c:@F@in_range1# | Decl
+void in_range1();
+// RANGE: [[@LINE+1]]:6 | function/C | c:@F@in_range2# | Decl
+void in_range2();
+
+// RANGE-NOT: after_range
+void after_range();
+
+// RUN: c-index-test core -print-record %t.idx -filepath %s:21:23 | FileCheck -check-prefix=RANGE %s

clang/test/Index/Store/print-unit.c

+
+
+#include "print-unit.h"
+#include "syshead.h"
+
+void foo(int i);
+
+// RUN: rm -rf %t
+// RUN: %clang_cc1 -I %S/Inputs -isystem %S/Inputs/sys -index-store-path %t/idx %s -triple x86_64-apple-macosx10.8
+// RUN: c-index-test core -print-unit %t/idx | FileCheck %s
+// RUN: %clang_cc1 -I %S/Inputs -isystem %S/Inputs/sys -index-store-path %t/idx_opt1 %s -triple x86_64-apple-macosx10.8 -O2
+// RUN: c-index-test core -print-unit %t/idx_opt1 | FileCheck %s -check-prefix=OPT
+// RUN: %clang_cc1 -I %S/Inputs -isystem %S/Inputs/sys -index-store-path %t/idx_opt2 %s -triple x86_64-apple-macosx10.8 -Os
+// RUN: c-index-test core -print-unit %t/idx_opt2 | FileCheck %s -check-prefix=OPT
+
+// CHECK: print-unit.c.o
+// CHECK: provider: clang-
+// CHECK: is-system: 0
+// CHECK: has-main: 1
+// CHECK: main-path: {{.*}}/print-unit.c
+// CHECK: out-file: {{.*}}/print-unit.c.o
+// CHECK: target: x86_64-apple-macosx10.8
+// CHECK: is-debug: 1
+// CHECK: DEPEND START
+// CHECK: Record | user | {{.*}}/print-unit.c | print-unit.c-
+// CHECK: Record | user | {{.*}}/Inputs/head.h | head.h-
+// CHECK: Record | user | {{.*}}/Inputs/using-overlay.h | using-overlay.h-
+// CHECK: Record | system | {{.*}}/Inputs/sys/syshead.h | syshead.h-
+// CHECK: Record | system | {{.*}}/Inputs/sys/another.h | another.h-
+// CHECK: File | user | {{.*}}/Inputs/print-unit.h{{$}}
+// CHECK: DEPEND END (6)
+// CHECK: INCLUDE START
+// CHECK-DAG: {{.*}}/print-unit.c:3 | {{.*}}/Inputs/print-unit.h
+// CHECK-DAG: {{.*}}/print-unit.c:4 | {{.*}}/Inputs/sys/syshead.h
+// CHECK-DAG: {{.*}}/Inputs/print-unit.h:1 | {{.*}}/Inputs/head.h
+// CHECK-DAG: {{.*}}/Inputs/print-unit.h:2 | {{.*}}/Inputs/using-overlay.h
+// CHECK: INCLUDE END (4)
+
+// OPT: is-debug: 0

clang/test/Index/Store/print-units-with-modules.m

+// RUN: rm -rf %t.idx %t.mcp
+// RUN: %clang -arch x86_64 -mmacosx-version-min=10.7 -c %s -o %t.o -index-store-path %t.idx -fmodules -fmodules-cache-path=%t.mcp -Xclang -fdisable-module-hash -I %S/Inputs/module
+// RUN: c-index-test core -print-unit %t.idx | FileCheck %s
+
+@import ModDep;
+@import ModSystem;
+
+// CHECK: ModDep.pcm
+// CHECK: provider: clang-
+// CHECK: is-system: 0
+// CHECK: is-module: 1
+// CHECK: module-name: ModDep
+// CHECK: has-main: 0
+// CHECK: main-path: {{$}}
+// CHECK: out-file: {{.*}}/ModDep.pcm
+// CHECK: DEPEND START
+// CHECK: Unit | user | ModTop | {{.*}}/ModTop.pcm | ModTop.pcm
+// CHECK: Record | user | ModDep | {{.*}}/Inputs/module/ModDep.h | ModDep.h
+// CHECK: DEPEND END (2)
+
+// CHECK: ModSystem.pcm
+// CHECK: is-system: 1
+// CHECK: is-module: 1
+// CHECK: module-name: ModSystem
+// CHECK: has-main: 0
+// CHECK: main-path: {{$}}
+// CHECK: out-file: {{.*}}/ModSystem.pcm
+// CHECK: DEPEND START
+// CHECK: Record | system | ModSystem | {{.*}}/Inputs/module/ModSystem.h | ModSystem.h
+// CHECK: DEPEND END (1)
+
+// CHECK: ModTop.pcm
+// CHECK: is-system: 0
+// CHECK: is-module: 1
+// CHECK: module-name: ModTop
+// CHECK: has-main: 0
+// CHECK: main-path: {{$}}
+// CHECK: out-file: {{.*}}/ModTop.pcm
+// CHECK: DEPEND START
+// CHECK: Record | user | ModTop | {{.*}}/Inputs/module/ModTop.h | ModTop.h
+// CHECK: Record | user | ModTop.Sub1 | {{.*}}/Inputs/module/ModTopSub1.h | ModTopSub1.h
+// CHECK: File | user | ModTop.Sub2 | {{.*}}/Inputs/module/ModTopSub2.h{{$}}
+// CHECK: DEPEND END (3)
+
+// CHECK: print-units-with-modules.m.tmp.o
+// CHECK: is-system: 0
+// CHECK: is-module: 0
+// CHECK: module-name: <none>
+// CHECK: has-main: 1
+// CHECK: main-path: {{.*}}/print-units-with-modules.m
+// CHECK: out-file: {{.*}}/print-units-with-modules.m.tmp.o
+// CHECK: DEPEND START
+// CHECK: Unit | user | ModDep | {{.*}}/ModDep.pcm | ModDep.pcm
+// CHECK: Unit | system | ModSystem | {{.*}}/ModSystem.pcm | ModSystem.pcm
+// CHECK: File | user | {{.*}}/print-units-with-modules.m{{$}}
+// CHECK: File | user | {{.*}}/Inputs/module/module.modulemap{{$}}
+// CHECK: DEPEND END (4)

clang/test/Index/Store/print-units-with-pch.c

+// RUN: rm -rf %t.idx
+// RUN: %clang -target x86_64-apple-darwin -arch x86_64 -mmacosx-version-min=10.7 -x c-header %S/Inputs/head.h -o %t.h.pch -index-store-path %t.idx
+// RUN: %clang -target x86_64-apple-darwin -arch x86_64 -mmacosx-version-min=10.7 -c %s -o %t.o -index-store-path %t.idx -include %t.h -Werror
+// RUN: c-index-test core -print-unit %t.idx | FileCheck %s
+
+int main() {
+  test1_func();
+}
+
+// CHECK: print-units-with-pch.c.tmp.h.pch
+// CHECK: is-system: 0
+// CHECK: has-main: 0
+// CHECK: main-path: {{$}}
+// CHECK: out-file: {{.*}}/print-units-with-pch.c.tmp.h.pch
+// CHECK: DEPEND START
+// CHECK: Record | user | {{.*}}/Inputs/head.h | head.h
+// CHECK: DEPEND END (1)
+
+// CHECK: print-units-with-pch.c.tmp.o
+// CHECK: is-system: 0
+// CHECK: has-main: 1
+// CHECK: main-path: {{.*}}/print-units-with-pch.c
+// CHECK: out-file: {{.*}}/print-units-with-pch.c.tmp.o
+// CHECK: DEPEND START
+// CHECK: Unit | user | {{.*}}/print-units-with-pch.c.tmp.h.pch | print-units-with-pch.c.tmp.h.pch
+// CHECK: Record | user | {{.*}}/print-units-with-pch.c | print-units-with-pch.c
+// CHECK: DEPEND END (2)

clang/test/Index/Store/record-hash-crash-invalid-name.cpp

+// Makes sure it doesn't crash.
+
+// RUN: rm -rf %t
+// RUN: not %clang_cc1 %s -index-store-path %t/idx -std=c++14
+// RUN: c-index-test core -print-record %t/idx | FileCheck %s
+
+namespace rdar32474406 {
+void foo();
+typedef void (*Func_t)();
+// CHECK: [[@LINE+4]]:1 | type-alias/C | c:record-hash-crash-invalid-name.cpp@N@rdar32474406@T@Func_t | Ref,RelCont | rel: 1
+// CHECK-NEXT: RelCont | c:@N@rdar32474406
+// CHECK: [[@LINE+2]]:14 | function/C | c:@N@rdar32474406@F@foo# | Ref,RelCont | rel: 1
+// CHECK-NEXT: RelCont | c:@N@rdar32474406
+Func_t[] = { foo }; // invalid decomposition
+}

clang/test/Index/Store/record-hash-crash.cpp

+// Makes sure it doesn't crash.
+
+// RUN: rm -rf %t
+// RUN: %clang_cc1 %s -index-store-path %t/idx -std=c++14
+// RUN: c-index-test core -print-record %t/idx | FileCheck %s
+
+namespace crash1 {
+// CHECK: [[@LINE+1]]:6 | function/C
+auto getit() { return []() {}; }
+}
+
+namespace crash2 {
+// CHECK: [[@LINE+2]]:7 | class(Gen)/C++ | c:@N@crash2@ST>1#T@Foo | Decl,RelChild | rel: 1
+template <typename T>
+class Foo; // canonical decl
+
+// CHECK: [[@LINE+2]]:7 | class(Gen)/C++ | c:@N@crash2@ST>1#T@Foo | Def,RelChild | rel: 1
+template <typename T>
+class Foo {};
+
+// CHECK: [[@LINE+2]]:8 | struct(Gen)/C++ | c:@N@crash2@ST>1#t>1#pT@Wrapper | Def,RelChild | rel: 1
+template <template <typename... ARGS> class TYPE>
+struct Wrapper {};
+
+// CHECK: [[@LINE+3]]:8 | struct(Gen,TS)/C++ | c:@N@crash2@S@Wrapper>#@N@crash2@ST>1#T@Foo | Def,RelChild,RelSpecialization | rel: 2
+// CHECK: RelSpecialization | c:@N@crash2@ST>1#t>1#pT@Wrapper
+template <>
+struct Wrapper<Foo> {};
+}

clang/test/Index/Store/record-hash-using.cpp

+// RUN: rm -rf %t
+// RUN: %clang_cc1 %s -std=c++11 -index-store-path %t/idx -DTYPE1=A -DTYPE2=A -DTYPE3=T -DTYPE4=T
+// RUN: %clang_cc1 %s -std=c++11 -index-store-path %t/idx -DTYPE1=B -DTYPE2=A -DTYPE3=T -DTYPE4=T
+// RUN: %clang_cc1 %s -std=c++11 -index-store-path %t/idx -DTYPE1=A -DTYPE2=B -DTYPE3=T -DTYPE4=T
+// RUN: %clang_cc1 %s -std=c++11 -index-store-path %t/idx -DTYPE1=B -DTYPE2=B -DTYPE3=T -DTYPE4=T
+// RUN: find %t/idx/*/records -name "record-hash*" | count 4
+//
+// RUN: rm -rf %t
+// RUN: %clang_cc1 %s -std=c++11 -index-store-path %t/idx -DTYPE1=A -DTYPE2=A -DTYPE3=T -DTYPE4=T
+// RUN: %clang_cc1 %s -std=c++11 -index-store-path %t/idx -DTYPE1=A -DTYPE2=A -DTYPE3=U -DTYPE4=T
+// RUN: %clang_cc1 %s -std=c++11 -index-store-path %t/idx -DTYPE1=A -DTYPE2=A -DTYPE3=T -DTYPE4=U
+// RUN: %clang_cc1 %s -std=c++11 -index-store-path %t/idx -DTYPE1=A -DTYPE2=A -DTYPE3=U -DTYPE4=U
+// RUN: find %t/idx/*/records -name "record-hash*" | count 4
+
+template<typename T>
+struct A {
+  typedef int X;
+  void foo();
+};
+
+template<typename T>
+struct B : public A<T> {
+  typedef float X;
+  void foo(int);
+};
+
+template<typename T>
+struct C : public B<T> {
+// This should result in different records, due to the different types.
+  using TYPE1<T>::X;
+  using TYPE2<T>::foo;
+};
+
+template <typename T>
+struct D {
+  typedef T X;
+  void foo(T);
+};
+template <typename T, typename U>
+struct E : public D<T>, public D<U> {
+// This should result in different records, due to the different template parameter.
+  using D<TYPE3>::X;
+  using D<TYPE4>::foo;
+};

clang/test/Index/Store/record-hash.cpp

+// RUN: rm -rf %t
+// RUN: %clang_cc1 %s -index-store-path %t/idx -D THE_TYPE=long
+// RUN: %clang_cc1 %s -index-store-path %t/idx -D THE_TYPE=char
+// RUN: find %t/idx/*/records -name "record-hash*" | count 2
+
+template<typename T>
+class TC {};
+
+// This should result in different records, due to the different template parameter type.
+void some_func(TC<THE_TYPE>);

clang/test/Index/Store/relative-out-path.c

+// Needs 'find'.
+// REQUIRES: shell
+
+// RUN: rm -rf %t
+// RUN: mkdir -p %t
+// RUN: %clang %s -index-store-path %t/idx1 -c -o %t/outfile.o
+// RUN: cd %t
+// RUN: %clang %s -index-store-path %t/idx2 -c -o outfile.o
+// RUN: cd ..
+// RUN: %clang %s -index-store-path %t/idx3 -fsyntax-only -o outfile.o -working-directory=%t
+// RUN: diff -r -u %t/idx2 %t/idx3
+
+// RUN: find %t/idx1 -name '*outfile.o*' > %t/hashes.txt
+// RUN: find %t/idx3 -name '*outfile.o*' >> %t/hashes.txt
+// RUN: FileCheck %s --input-file=%t/hashes.txt
+// CHECK:      outfile.o[[OUT_HASH:.*$]]
+// CHECK-NEXT: outfile.o[[OUT_HASH]]
+
+void foo();

clang/test/Index/Store/syntax-only.c

+// RUN: rm -rf %t.idx
+// RUN: %clang -fsyntax-only %s -index-store-path %t.idx -o %T/syntax-only.c.myoutfile
+// RUN: c-index-test core -print-unit %t.idx | FileCheck %s -check-prefix=CHECK-UNIT
+// RUN: c-index-test core -print-record %t.idx | FileCheck %s -check-prefix=CHECK-RECORD
+
+// CHECK-UNIT: out-file: {{.*}}/syntax-only.c.myoutfile
+// CHECK-RECORD: function/C | foo | c:@F@foo
+
+void foo();

clang/test/Index/Store/unit-with-vfs.c

+// RUN: sed -e "s:INPUT_DIR:%S/Inputs:g" -e "s:OUT_DIR:%t:g" %S/Inputs/overlay.yaml > %t.yaml
+// REQUIRES: shell
+
+#include "using-overlay.h"
+
+// RUN: rm -rf %t.idx
+// RUN: %clang_cc1 %s -index-store-path %t.idx -I %t -ivfsoverlay %t.yaml
+// RUN: c-index-test core -print-unit %t.idx | FileCheck %s
+
+// CHECK: Record | user | {{.*}}test/Index/Store/Inputs/using-overlay.h

clang/test/Index/Store/unit-workdir-prefix.c

+#include "header.h"
+
+void foo(void) {
+  bar();
+}
+
+// RUN: rm -rf %t
+// RUN: mkdir -p %t/Directory
+// RUN: mkdir -p %t/Directory.surprise
+// RUN: mkdir -p %t/sdk
+// RUN: mkdir -p %t/sdk_other
+// RUN: echo "void bar(void);" > %t/sdk_other/header.h
+// RUN: cp %s %t/Directory.surprise/main.c
+//
+// RUN: %clang_cc1 -isystem %t/sdk_other -isysroot %t/sdk -index-store-path %t/idx %t/Directory.surprise/main.c -triple x86_64-apple-macosx10.8 -working-directory %t/Directory
+// RUN: c-index-test core -print-unit %t/idx | FileCheck %s
+
+// CHECK: main.c.o
+// CHECK: provider: clang-
+// CHECK: is-system: 0
+// CHECK: has-main: 1
+// CHECK: main-path: {{.*}}Directory.surprise{{/|\\}}main.c
+// CHECK: out-file: {{.*}}Directory.surprise{{/|\\}}main.c.o
+// CHECK: target: x86_64-apple-macosx10.8
+// CHECK: is-debug: 1
+// CHECK: DEPEND START
+// CHECK: Record | user | {{.*}}Directory.surprise{{/|\\}}main.c | main.c-
+// CHECK: Record | system | {{.*}}sdk_other{{/|\\}}header.h | header.h-

clang/test/Index/Store/using-libstdcpp-arc.mm

+// Test to make sure we don't crash, rdar://30816887&36162712.
+
+// RUN: mkdir -p %t/include
+// RUN: echo 'module Foo { header "test.h" }' > %t/include/module.modulemap
+// RUN: echo '' > %t/include/test.h
+
+// RUN: rm -rf %t.idx %t.mcp
+// RUN: %clang_cc1 %s -index-store-path %t.idx -I %t/include -fobjc-arc -fobjc-arc-cxxlib=libstdc++ -fmodules -fmodules-cache-path=%t.mcp -fimplicit-module-maps
+// RUN: c-index-test core -print-record %t.idx | FileCheck %s
+
+// RUN: rm -rf %t.idx2
+// RUN: %clang_cc1 %s -index-store-path %t.idx2 -I %t/include -fobjc-arc -fobjc-arc-cxxlib=libstdc++ -fmodules -fmodules-cache-path=%t.mcp -fimplicit-module-maps
+// RUN: c-index-test core -print-record %t.idx2 | FileCheck %s
+
+@import Foo;
+
+// CHECK: [[@LINE+1]]:6 | function/C
+void test1(void);

clang/tools/CMakeLists.txt

 create_subdirectory_options(CLANG TOOL)
 
 add_clang_subdirectory(diagtool)
 add_clang_subdirectory(driver)
 add_clang_subdirectory(clang-diff)
 add_clang_subdirectory(clang-format)
 add_clang_subdirectory(clang-format-vs)
 add_clang_subdirectory(clang-fuzzer)
 add_clang_subdirectory(clang-import-test)
 add_clang_subdirectory(clang-offload-bundler)
 add_clang_subdirectory(clang-offload-wrapper)
 add_clang_subdirectory(clang-scan-deps)
 
 add_clang_subdirectory(c-index-test)
+add_clang_subdirectory(IndexStore)
 
 add_clang_subdirectory(clang-rename)
 add_clang_subdirectory(clang-refactor)
 if(UNIX)
   add_clang_subdirectory(clang-shlib)
 endif()
 
 if(CLANG_ENABLE_ARCMT)

clang/tools/IndexStore/CMakeLists.txt

+include(CheckIncludeFiles)
+
+set(SOURCES
+  IndexStore.cpp
+
+  ADDITIONAL_HEADERS
+  ../../include/indexstore/indexstore.h
+  ../../include/indexstore/IndexStoreCXX.h
+  )
+
+set(LIBS
+  clangIndex
+  clangIndexSerialization
+  clangIndexDataStore
+)
+
+if(NOT CMAKE_SYSTEM_NAME STREQUAL Windows)
+  set(LLVM_EXPORTED_SYMBOL_FILE ${CMAKE_CURRENT_SOURCE_DIR}/IndexStore.exports)
+endif()
+
+if(WIN32)
+  set(output_name "libIndexStore")
+else()
+  set(output_name "IndexStore")
+endif()
+
+add_clang_library(IndexStore SHARED
+  OUTPUT_NAME ${output_name}
+  ${SOURCES}
+
+  LINK_LIBS
+  ${LIBS}
+
+  LINK_COMPONENTS
+  ${LLVM_TARGETS_TO_BUILD}
+  Core
+  Support
+  )
+
+if(APPLE)
+  set(INDEXSTORE_LIBRARY_VERSION "${LLVM_VERSION_MAJOR}.${LLVM_VERSION_MINOR}.${LLVM_VERSION_PATCH}")
+
+  set(INDEXSTORE_LINK_FLAGS " -Wl,-compatibility_version -Wl,1")
+  set(INDEXSTORE_LINK_FLAGS "${INDEXSTORE_LINK_FLAGS} -Wl,-current_version -Wl,${INDEXSTORE_LIBRARY_VERSION}")
+
+  check_include_files("CoreServices/CoreServices.h" HAVE_CORESERVICES_H)
+  if(HAVE_CORESERVICES_H)
+    set(INDEXSTORE_LINK_FLAGS "${INDEXSTORE_LINK_FLAGS} -framework CoreServices")
+  endif()
+
+  set_property(TARGET IndexStore APPEND_STRING PROPERTY
+               LINK_FLAGS ${INDEXSTORE_LINK_FLAGS})
+endif()
+
+if (LLVM_INSTALL_TOOLCHAIN_ONLY)
+  install(TARGETS IndexStore
+    COMPONENT IndexStore
+    LIBRARY DESTINATION lib${LLVM_LIBDIR_SUFFIX}
+    ARCHIVE DESTINATION lib${LLVM_LIBDIR_SUFFIX}
+    RUNTIME DESTINATION bin)
+
+  if (NOT CMAKE_CONFIGURATION_TYPES)
+    add_llvm_install_targets(install-IndexStore
+                             DEPENDS IndexStore
+                             COMPONENT IndexStore)
+  endif()
+endif()
+
+set(INDEXSTORE_HEADERS_INSTALL_DESTINATION "local/include")
+
+install(DIRECTORY ../../include/indexstore
+  COMPONENT IndexStore
+  DESTINATION "${INDEXSTORE_HEADERS_INSTALL_DESTINATION}"
+  FILES_MATCHING
+  PATTERN "*.h"
+  PATTERN ".svn" EXCLUDE
+  )

clang/tools/IndexStore/IndexStore.cpp

+//===- IndexStore.cpp - Index store API -----------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements the API for the index store.
+//
+//===----------------------------------------------------------------------===//
+
+#include "indexstore/indexstore.h"
+#include "indexstore/Conversions.h"
+#include "clang/DirectoryWatcher/DirectoryWatcher.h"
+#include "clang/IndexDataStore/IndexDataStore.h"
+#include "clang/IndexSerialization/IndexDataFormat.h"
+#include "clang/IndexSerialization/IndexRecordReader.h"
+#include "clang/IndexSerialization/IndexUnitReader.h"
+#include "clang/IndexSerialization/IndexUnitWriter.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/Optional.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/Support/Chrono.h"
+#include "llvm/Support/ManagedStatic.h"
+
+#if INDEXSTORE_HAS_BLOCKS
+#include <Block.h>
+#endif
+
+using namespace clang;
+using namespace clang::index;
+using namespace llvm;
+
+static indexstore_string_ref_t toIndexStoreString(StringRef str) {
+  return indexstore_string_ref_t{str.data(), str.size()};
+}
+
+static timespec toTimeSpec(sys::TimePoint<> tp) {
+  std::chrono::seconds sec =
+      std::chrono::time_point_cast<std::chrono::seconds>(tp).time_since_epoch();
+  std::chrono::nanoseconds nsec =
+      std::chrono::time_point_cast<std::chrono::nanoseconds>(tp - sec)
+          .time_since_epoch();
+  timespec ts;
+  ts.tv_sec = sec.count();
+  ts.tv_nsec = nsec.count();
+  return ts;
+}
+
+//===----------------------------------------------------------------------===//
+// Fatal error handling
+//===----------------------------------------------------------------------===//
+
+static void fatal_error_handler(void *user_data, const std::string &reason,
+                                bool gen_crash_diag) {
+  // Write the result out to stderr avoiding errs() because raw_ostreams can
+  // call report_fatal_error.
+  fprintf(stderr, "INDEXSTORE FATAL ERROR: %s\n", reason.c_str());
+  ::abort();
+}
+
+namespace {
+struct RegisterFatalErrorHandler {
+  RegisterFatalErrorHandler() {
+    llvm::install_fatal_error_handler(fatal_error_handler, nullptr);
+  }
+};
+} // namespace
+
+static llvm::ManagedStatic<RegisterFatalErrorHandler>
+    RegisterFatalErrorHandlerOnce;
+
+//===----------------------------------------------------------------------===//
+// C API
+//===----------------------------------------------------------------------===//
+
+namespace {
+
+struct IndexStoreError {
+  std::string Error;
+};
+
+} // anonymous namespace
+
+const char *indexstore_error_get_description(indexstore_error_t err) {
+  return static_cast<IndexStoreError *>(err)->Error.c_str();
+}
+
+void indexstore_error_dispose(indexstore_error_t err) {
+  delete static_cast<IndexStoreError *>(err);
+}
+
+unsigned indexstore_format_version(void) {
+  return IndexDataStore::getFormatVersion();
+}
+
+indexstore_t indexstore_store_create(const char *store_path,
+                                     indexstore_error_t *c_error) {
+  // Look through the managed static to trigger construction of the managed
+  // static which registers our fatal error handler. This ensures it is only
+  // registered once.
+  (void)*RegisterFatalErrorHandlerOnce;
+
+  std::unique_ptr<IndexDataStore> store;
+  std::string error;
+  store = IndexDataStore::create(store_path, error);
+  if (!store) {
+    if (c_error)
+      *c_error = new IndexStoreError{error};
+    return nullptr;
+  }
+  return store.release();
+}
+
+void indexstore_store_dispose(indexstore_t store) {
+  delete static_cast<IndexDataStore *>(store);
+}
+
+#if INDEXSTORE_HAS_BLOCKS
+bool indexstore_store_units_apply(
+    indexstore_t c_store, unsigned sorted,
+    INDEXSTORE_NOESCAPE bool (^applier)(indexstore_string_ref_t unit_name)) {
+  IndexDataStore *store = static_cast<IndexDataStore *>(c_store);
+  return store->foreachUnitName(sorted, [&](StringRef unitName) -> bool {
+    return applier(toIndexStoreString(unitName));
+  });
+}
+#endif
+
+bool indexstore_store_units_apply_f(
+    indexstore_t c_store, unsigned sorted, void *context,
+    INDEXSTORE_NOESCAPE bool (*applier)(void *context,
+                                        indexstore_string_ref_t unit_name)) {
+  IndexDataStore *store = static_cast<IndexDataStore *>(c_store);
+  return store->foreachUnitName(sorted, [&](StringRef unitName) -> bool {
+    return applier(context, toIndexStoreString(unitName));
+  });
+}
+
+size_t indexstore_unit_event_notification_get_events_count(
+    indexstore_unit_event_notification_t c_evtnote) {
+  auto *evtnote =
+      static_cast<IndexDataStore::UnitEventNotification *>(c_evtnote);
+  return evtnote->Events.size();
+}
+
+indexstore_unit_event_t indexstore_unit_event_notification_get_event(
+    indexstore_unit_event_notification_t c_evtnote, size_t index) {
+  auto *evtnote =
+      static_cast<IndexDataStore::UnitEventNotification *>(c_evtnote);
+  return (indexstore_unit_event_t)&evtnote->Events[index];
+}
+
+bool indexstore_unit_event_notification_is_initial(
+    indexstore_unit_event_notification_t c_evtnote) {
+  auto *evtnote =
+      static_cast<IndexDataStore::UnitEventNotification *>(c_evtnote);
+  return evtnote->IsInitial;
+}
+
+indexstore_unit_event_kind_t
+indexstore_unit_event_get_kind(indexstore_unit_event_t c_evt) {
+  auto *evt = static_cast<IndexDataStore::UnitEvent *>(c_evt);
+  indexstore_unit_event_kind_t k;
+  switch (evt->Kind) {
+  case IndexDataStore::UnitEventKind::Removed:
+    k = INDEXSTORE_UNIT_EVENT_REMOVED;
+    break;
+  case IndexDataStore::UnitEventKind::Modified:
+    k = INDEXSTORE_UNIT_EVENT_MODIFIED;
+    break;
+  case IndexDataStore::UnitEventKind::DirectoryDeleted:
+    k = INDEXSTORE_UNIT_EVENT_DIRECTORY_DELETED;
+    break;
+  case IndexDataStore::UnitEventKind::Failure:
+    k = INDEXSTORE_UNIT_EVENT_FAILURE;
+    break;
+  }
+  return k;
+}
+
+indexstore_string_ref_t
+indexstore_unit_event_get_unit_name(indexstore_unit_event_t c_evt) {
+  auto *evt = static_cast<IndexDataStore::UnitEvent *>(c_evt);
+  return toIndexStoreString(evt->UnitName);
+}
+
+#if INDEXSTORE_HAS_BLOCKS
+void indexstore_store_set_unit_event_handler(
+    indexstore_t c_store, indexstore_unit_event_handler_t blk_handler) {
+  IndexDataStore *store = static_cast<IndexDataStore *>(c_store);
+  if (!blk_handler) {
+    store->setUnitEventHandler(nullptr);
+    return;
+  }
+
+  class BlockWrapper {
+    indexstore_unit_event_handler_t blk_handler;
+
+  public:
+    BlockWrapper(indexstore_unit_event_handler_t handler) {
+      blk_handler = Block_copy(handler);
+    }
+    BlockWrapper(const BlockWrapper &other) {
+      blk_handler = Block_copy(other.blk_handler);
+    }
+    ~BlockWrapper() { Block_release(blk_handler); }
+
+    void operator()(indexstore_unit_event_notification_t evt_note) const {
+      blk_handler(evt_note);
+    }
+  };
+
+  BlockWrapper handler(blk_handler);
+
+  store->setUnitEventHandler(
+      [handler](IndexDataStore::UnitEventNotification evtNote) {
+        handler(&evtNote);
+      });
+}
+#endif
+
+void indexstore_store_set_unit_event_handler_f(
+    indexstore_t c_store, void *context,
+    void (*fn_handler)(void *context, indexstore_unit_event_notification_t),
+    void (*finalizer)(void *context)) {
+  IndexDataStore *store = static_cast<IndexDataStore *>(c_store);
+  if (!fn_handler) {
+    store->setUnitEventHandler(nullptr);
+    return;
+  }
+
+  class BlockWrapper {
+    void *context;
+    void (*fn_handler)(void *context, indexstore_unit_event_notification_t);
+    void (*finalizer)(void *context);
+
+  public:
+    BlockWrapper(void *_context,
+                 void (*_fn_handler)(void *context,
+                                     indexstore_unit_event_notification_t),
+                 void (*_finalizer)(void *context))
+        : context(_context), fn_handler(_fn_handler), finalizer(_finalizer) {}
+
+    ~BlockWrapper() {
+      if (finalizer) {
+        finalizer(context);
+      }
+    }
+
+    void operator()(indexstore_unit_event_notification_t evt_note) const {
+      fn_handler(context, evt_note);
+    }
+  };
+
+  auto handler = std::make_shared<BlockWrapper>(context, fn_handler, finalizer);
+
+  store->setUnitEventHandler(
+      [handler](IndexDataStore::UnitEventNotification evtNote) {
+        (*handler)(&evtNote);
+      });
+}
+
+bool indexstore_store_start_unit_event_listening(
+    indexstore_t c_store, indexstore_unit_event_listen_options_t *client_opts,
+    size_t listen_options_struct_size, indexstore_error_t *c_error) {
+  IndexDataStore *store = static_cast<IndexDataStore *>(c_store);
+  indexstore_unit_event_listen_options_t listen_opts;
+  memset(&listen_opts, 0, sizeof(listen_opts));
+  unsigned clientOptSize = listen_options_struct_size < sizeof(listen_opts)
+                               ? listen_options_struct_size
+                               : sizeof(listen_opts);
+  memcpy(&listen_opts, client_opts, clientOptSize);
+
+  std::string error;
+  bool err = store->startEventListening(listen_opts.wait_initial_sync, error);
+  if (err && c_error)
+    *c_error = new IndexStoreError{error};
+  return err;
+}
+
+void indexstore_store_stop_unit_event_listening(indexstore_t c_store) {
+  IndexDataStore *store = static_cast<IndexDataStore *>(c_store);
+  store->stopEventListening();
+}
+
+void indexstore_store_discard_unit(indexstore_t c_store,
+                                   const char *unit_name) {
+  IndexDataStore *store = static_cast<IndexDataStore *>(c_store);
+  store->discardUnit(unit_name);
+}
+
+void indexstore_store_discard_record(indexstore_t c_store,
+                                     const char *record_name) {
+  IndexDataStore *store = static_cast<IndexDataStore *>(c_store);
+  store->discardRecord(record_name);
+}
+
+void indexstore_store_purge_stale_data(indexstore_t c_store) {
+  IndexDataStore *store = static_cast<IndexDataStore *>(c_store);
+  store->purgeStaleData();
+}
+
+indexstore_symbol_kind_t indexstore_symbol_get_kind(indexstore_symbol_t sym) {
+  return getIndexStoreKind(static_cast<IndexRecordDecl *>(sym)->SymInfo.Kind);
+}
+
+indexstore_symbol_subkind_t
+indexstore_symbol_get_subkind(indexstore_symbol_t sym) {
+  return getIndexStoreSubKind(
+      static_cast<IndexRecordDecl *>(sym)->SymInfo.SubKind);
+}
+
+indexstore_symbol_language_t
+indexstore_symbol_get_language(indexstore_symbol_t sym) {
+  return getIndexStoreLang(static_cast<IndexRecordDecl *>(sym)->SymInfo.Lang);
+}
+
+uint64_t indexstore_symbol_get_properties(indexstore_symbol_t sym) {
+  return getIndexStoreProperties(
+      static_cast<IndexRecordDecl *>(sym)->SymInfo.Properties);
+}
+
+uint64_t indexstore_symbol_get_roles(indexstore_symbol_t sym) {
+  return getIndexStoreRoles(static_cast<IndexRecordDecl *>(sym)->Roles);
+}
+
+uint64_t indexstore_symbol_get_related_roles(indexstore_symbol_t sym) {
+  return getIndexStoreRoles(static_cast<IndexRecordDecl *>(sym)->RelatedRoles);
+}
+
+indexstore_string_ref_t indexstore_symbol_get_name(indexstore_symbol_t sym) {
+  auto *D = static_cast<IndexRecordDecl *>(sym);
+  return toIndexStoreString(D->Name);
+}
+
+indexstore_string_ref_t indexstore_symbol_get_usr(indexstore_symbol_t sym) {
+  auto *D = static_cast<IndexRecordDecl *>(sym);
+  return toIndexStoreString(D->USR);
+}
+
+indexstore_string_ref_t
+indexstore_symbol_get_codegen_name(indexstore_symbol_t sym) {
+  auto *D = static_cast<IndexRecordDecl *>(sym);
+  return toIndexStoreString(D->CodeGenName);
+}
+
+uint64_t
+indexstore_symbol_relation_get_roles(indexstore_symbol_relation_t sym_rel) {
+  return getIndexStoreRoles(static_cast<IndexRecordRelation *>(sym_rel)->Roles);
+}
+
+indexstore_symbol_t
+indexstore_symbol_relation_get_symbol(indexstore_symbol_relation_t sym_rel) {
+  return (indexstore_symbol_t) static_cast<IndexRecordRelation *>(sym_rel)->Dcl;
+}
+
+indexstore_symbol_t
+indexstore_occurrence_get_symbol(indexstore_occurrence_t occur) {
+  return (indexstore_symbol_t) static_cast<IndexRecordOccurrence *>(occur)->Dcl;
+}
+
+#if INDEXSTORE_HAS_BLOCKS
+bool indexstore_occurrence_relations_apply(
+    indexstore_occurrence_t occur,
+    INDEXSTORE_NOESCAPE bool (^applier)(
+        indexstore_symbol_relation_t symbol_rel)) {
+  auto *recOccur = static_cast<IndexRecordOccurrence *>(occur);
+  for (auto &rel : recOccur->Relations) {
+    if (!applier(&rel))
+      return false;
+  }
+  return true;
+}
+#endif
+
+bool indexstore_occurrence_relations_apply_f(
+    indexstore_occurrence_t occur, void *context,
+    INDEXSTORE_NOESCAPE bool (*applier)(
+        void *context, indexstore_symbol_relation_t symbol_rel)) {
+  auto *recOccur = static_cast<IndexRecordOccurrence *>(occur);
+  for (auto &rel : recOccur->Relations) {
+    if (!applier(context, &rel))
+      return false;
+  }
+  return true;
+}
+
+uint64_t indexstore_occurrence_get_roles(indexstore_occurrence_t occur) {
+  return getIndexStoreRoles(static_cast<IndexRecordOccurrence *>(occur)->Roles);
+}
+
+void indexstore_occurrence_get_line_col(indexstore_occurrence_t occur,
+                                        unsigned *line, unsigned *column) {
+  auto *recOccur = static_cast<IndexRecordOccurrence *>(occur);
+  if (line)
+    *line = recOccur->Line;
+  if (column)
+    *column = recOccur->Column;
+}
+
+typedef void *indexstore_record_reader_t;
+
+indexstore_record_reader_t
+indexstore_record_reader_create(indexstore_t c_store, const char *record_name,
+                                indexstore_error_t *c_error) {
+  IndexDataStore *store = static_cast<IndexDataStore *>(c_store);
+  std::unique_ptr<IndexRecordReader> reader;
+  std::string error;
+  reader = IndexRecordReader::createWithRecordFilename(
+      record_name, store->getFilePath(), error);
+  if (!reader) {
+    if (c_error)
+      *c_error = new IndexStoreError{error};
+    return nullptr;
+  }
+  return reader.release();
+}
+
+void indexstore_record_reader_dispose(indexstore_record_reader_t rdr) {
+  auto *reader = static_cast<IndexRecordReader *>(rdr);
+  delete reader;
+}
+
+#if INDEXSTORE_HAS_BLOCKS
+/// Goes through the symbol data and passes symbols to \c receiver, for the
+/// symbol data that \c filter returns true on.
+///
+/// This allows allocating memory only for the record symbols that the caller is
+/// interested in.
+bool indexstore_record_reader_search_symbols(
+    indexstore_record_reader_t rdr,
+    INDEXSTORE_NOESCAPE bool (^filter)(indexstore_symbol_t symbol, bool *stop),
+    INDEXSTORE_NOESCAPE void (^receiver)(indexstore_symbol_t symbol)) {
+  auto *reader = static_cast<IndexRecordReader *>(rdr);
+
+  auto filterFn =
+      [&](const IndexRecordDecl &D) -> IndexRecordReader::DeclSearchReturn {
+    bool stop = false;
+    bool accept = filter((indexstore_symbol_t)&D, &stop);
+    return {accept, !stop};
+  };
+  auto receiverFn = [&](const IndexRecordDecl *D) {
+    receiver((indexstore_symbol_t)D);
+  };
+
+  return reader->searchDecls(filterFn, receiverFn);
+}
+
+bool indexstore_record_reader_symbols_apply(
+    indexstore_record_reader_t rdr, bool nocache,
+    INDEXSTORE_NOESCAPE bool (^applier)(indexstore_symbol_t symbol)) {
+  auto *reader = static_cast<IndexRecordReader *>(rdr);
+  auto receiverFn = [&](const IndexRecordDecl *D) -> bool {
+    return applier((indexstore_symbol_t)D);
+  };
+  return reader->foreachDecl(nocache, receiverFn);
+}
+
+bool indexstore_record_reader_occurrences_apply(
+    indexstore_record_reader_t rdr,
+    INDEXSTORE_NOESCAPE bool (^applier)(indexstore_occurrence_t occur)) {
+  auto *reader = static_cast<IndexRecordReader *>(rdr);
+  auto receiverFn = [&](const IndexRecordOccurrence &RO) -> bool {
+    return applier((indexstore_occurrence_t)&RO);
+  };
+  return reader->foreachOccurrence(receiverFn);
+}
+
+bool indexstore_record_reader_occurrences_in_line_range_apply(
+    indexstore_record_reader_t rdr, unsigned line_start, unsigned line_count,
+    INDEXSTORE_NOESCAPE bool (^applier)(indexstore_occurrence_t occur)) {
+  auto *reader = static_cast<IndexRecordReader *>(rdr);
+  auto receiverFn = [&](const IndexRecordOccurrence &RO) -> bool {
+    return applier((indexstore_occurrence_t)&RO);
+  };
+  return reader->foreachOccurrenceInLineRange(line_start, line_count,
+                                              receiverFn);
+}
+
+/// \param symbols if non-zero \c symbols_count, indicates the list of symbols
+/// that we want to get occurrences for. An empty array indicates that we want
+/// occurrences for all symbols.
+/// \param related_symbols Same as \c symbols but for related symbols.
+bool indexstore_record_reader_occurrences_of_symbols_apply(
+    indexstore_record_reader_t rdr, indexstore_symbol_t *symbols,
+    size_t symbols_count, indexstore_symbol_t *related_symbols,
+    size_t related_symbols_count,
+    INDEXSTORE_NOESCAPE bool (^applier)(indexstore_occurrence_t occur)) {
+  auto *reader = static_cast<IndexRecordReader *>(rdr);
+  auto receiverFn = [&](const IndexRecordOccurrence &RO) -> bool {
+    return applier((indexstore_occurrence_t)&RO);
+  };
+  return reader->foreachOccurrence(
+      receiverFn, {(IndexRecordDecl **)symbols, symbols_count},
+      {(IndexRecordDecl **)related_symbols, related_symbols_count});
+}
+#endif
+
+bool indexstore_record_reader_search_symbols_f(
+    indexstore_record_reader_t rdr, void *filter_ctx,
+    INDEXSTORE_NOESCAPE bool (*filter)(void *filter_ctx,
+                                       indexstore_symbol_t symbol, bool *stop),
+    void *receiver_ctx,
+    INDEXSTORE_NOESCAPE void (*receiver)(void *receiver_ctx,
+                                         indexstore_symbol_t symbol)) {
+  auto *reader = static_cast<IndexRecordReader *>(rdr);
+
+  auto filterFn =
+      [&](const IndexRecordDecl &D) -> IndexRecordReader::DeclSearchReturn {
+    bool stop = false;
+    bool accept = filter(filter_ctx, (indexstore_symbol_t)&D, &stop);
+    return {accept, !stop};
+  };
+  auto receiverFn = [&](const IndexRecordDecl *D) {
+    receiver(receiver_ctx, (indexstore_symbol_t)D);
+  };
+
+  return reader->searchDecls(filterFn, receiverFn);
+}
+
+bool indexstore_record_reader_symbols_apply_f(
+    indexstore_record_reader_t rdr, bool nocache, void *context,
+    INDEXSTORE_NOESCAPE bool (*applier)(void *context,
+                                        indexstore_symbol_t symbol)) {
+  auto *reader = static_cast<IndexRecordReader *>(rdr);
+  auto receiverFn = [&](const IndexRecordDecl *D) -> bool {
+    return applier(context, (indexstore_symbol_t)D);
+  };
+  return reader->foreachDecl(nocache, receiverFn);
+}
+
+bool indexstore_record_reader_occurrences_apply_f(
+    indexstore_record_reader_t rdr, void *context,
+    INDEXSTORE_NOESCAPE bool (*applier)(void *context,
+                                        indexstore_occurrence_t occur)) {
+  auto *reader = static_cast<IndexRecordReader *>(rdr);
+  auto receiverFn = [&](const IndexRecordOccurrence &RO) -> bool {
+    return applier(context, (indexstore_occurrence_t)&RO);
+  };
+  return reader->foreachOccurrence(receiverFn);
+}
+
+bool indexstore_record_reader_occurrences_in_line_range_apply_f(
+    indexstore_record_reader_t rdr, unsigned line_start, unsigned line_count,
+    void *context,
+    INDEXSTORE_NOESCAPE bool (*applier)(void *context,
+                                        indexstore_occurrence_t occur)) {
+  auto *reader = static_cast<IndexRecordReader *>(rdr);
+  auto receiverFn = [&](const IndexRecordOccurrence &RO) -> bool {
+    return applier(context, (indexstore_occurrence_t)&RO);
+  };
+  return reader->foreachOccurrenceInLineRange(line_start, line_count,
+                                              receiverFn);
+}
+
+bool indexstore_record_reader_occurrences_of_symbols_apply_f(
+    indexstore_record_reader_t rdr, indexstore_symbol_t *symbols,
+    size_t symbols_count, indexstore_symbol_t *related_symbols,
+    size_t related_symbols_count, void *context,
+    INDEXSTORE_NOESCAPE bool (*applier)(void *context,
+                                        indexstore_occurrence_t occur)) {
+  auto *reader = static_cast<IndexRecordReader *>(rdr);
+  auto receiverFn = [&](const IndexRecordOccurrence &RO) -> bool {
+    return applier(context, (indexstore_occurrence_t)&RO);
+  };
+  return reader->foreachOccurrence(
+      receiverFn, {(IndexRecordDecl **)symbols, symbols_count},
+      {(IndexRecordDecl **)related_symbols, related_symbols_count});
+}
+
+size_t indexstore_store_get_unit_name_from_output_path(indexstore_t store,
+                                                       const char *output_path,
+                                                       char *name_buf,
+                                                       size_t buf_size) {
+  llvm::SmallString<256> unitName;
+  clang::index::store::getUnitNameForOutputFile(output_path, unitName);
+  size_t nameLen = unitName.size();
+  if (buf_size != 0) {
+    strncpy(name_buf, unitName.c_str(), buf_size - 1);
+    name_buf[buf_size - 1] = '\0';
+  }
+  return nameLen;
+}
+
+bool indexstore_store_get_unit_modification_time(indexstore_t c_store,
+                                                 const char *unit_name,
+                                                 int64_t *seconds,
+                                                 int64_t *nanoseconds,
+                                                 indexstore_error_t *c_error) {
+  IndexDataStore *store = static_cast<IndexDataStore *>(c_store);
+  std::string error;
+  // FIXME: This provides mod time with second-only accuracy.
+  auto optModTime = IndexUnitReader::getModificationTimeForUnit(
+      unit_name, store->getFilePath(), error);
+  if (!optModTime) {
+    if (c_error)
+      *c_error = new IndexStoreError{error};
+    return true;
+  }
+
+  timespec ts = toTimeSpec(*optModTime);
+  if (seconds)
+    *seconds = ts.tv_sec;
+  if (nanoseconds)
+    *nanoseconds = ts.tv_nsec;
+
+  return false;
+}
+
+indexstore_unit_reader_t
+indexstore_unit_reader_create(indexstore_t c_store, const char *unit_name,
+                              indexstore_error_t *c_error) {
+  IndexDataStore *store = static_cast<IndexDataStore *>(c_store);
+  std::unique_ptr<IndexUnitReader> reader;
+  std::string error;
+  reader = IndexUnitReader::createWithUnitFilename(unit_name,
+                                                   store->getFilePath(), error);
+  if (!reader) {
+    if (c_error)
+      *c_error = new IndexStoreError{error};
+    return nullptr;
+  }
+  return reader.release();
+}
+
+void indexstore_unit_reader_dispose(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  delete reader;
+}
+
+indexstore_string_ref_t
+indexstore_unit_reader_get_provider_identifier(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return toIndexStoreString(reader->getProviderIdentifier());
+}
+
+indexstore_string_ref_t
+indexstore_unit_reader_get_provider_version(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return toIndexStoreString(reader->getProviderVersion());
+}
+
+void indexstore_unit_reader_get_modification_time(indexstore_unit_reader_t rdr,
+                                                  int64_t *seconds,
+                                                  int64_t *nanoseconds) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  // FIXME: This provides mod time with second-only accuracy.
+  sys::TimePoint<> timeVal = reader->getModificationTime();
+  timespec ts = toTimeSpec(timeVal);
+  if (seconds)
+    *seconds = ts.tv_sec;
+  if (nanoseconds)
+    *nanoseconds = ts.tv_nsec;
+}
+
+bool indexstore_unit_reader_is_system_unit(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return reader->isSystemUnit();
+}
+
+bool indexstore_unit_reader_is_module_unit(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return reader->isModuleUnit();
+}
+
+bool indexstore_unit_reader_is_debug_compilation(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return reader->isDebugCompilation();
+}
+
+bool indexstore_unit_reader_has_main_file(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return reader->hasMainFile();
+}
+
+indexstore_string_ref_t
+indexstore_unit_reader_get_main_file(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return toIndexStoreString(reader->getMainFilePath());
+}
+
+indexstore_string_ref_t
+indexstore_unit_reader_get_module_name(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return toIndexStoreString(reader->getModuleName());
+}
+
+indexstore_string_ref_t
+indexstore_unit_reader_get_working_dir(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return toIndexStoreString(reader->getWorkingDirectory());
+}
+
+indexstore_string_ref_t
+indexstore_unit_reader_get_output_file(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return toIndexStoreString(reader->getOutputFile());
+}
+
+indexstore_string_ref_t
+indexstore_unit_reader_get_sysroot_path(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return toIndexStoreString(reader->getSysrootPath());
+}
+
+indexstore_string_ref_t
+indexstore_unit_reader_get_target(indexstore_unit_reader_t rdr) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return toIndexStoreString(reader->getTarget());
+}
+
+indexstore_unit_dependency_kind_t
+indexstore_unit_dependency_get_kind(indexstore_unit_dependency_t c_dep) {
+  auto dep = static_cast<const IndexUnitReader::DependencyInfo *>(c_dep);
+  switch (dep->Kind) {
+  case IndexUnitReader::DependencyKind::Unit:
+    return INDEXSTORE_UNIT_DEPENDENCY_UNIT;
+  case IndexUnitReader::DependencyKind::Record:
+    return INDEXSTORE_UNIT_DEPENDENCY_RECORD;
+  case IndexUnitReader::DependencyKind::File:
+    return INDEXSTORE_UNIT_DEPENDENCY_FILE;
+  }
+}
+
+bool indexstore_unit_dependency_is_system(indexstore_unit_dependency_t c_dep) {
+  auto dep = static_cast<const IndexUnitReader::DependencyInfo *>(c_dep);
+  return dep->IsSystem;
+}
+
+indexstore_string_ref_t
+indexstore_unit_dependency_get_filepath(indexstore_unit_dependency_t c_dep) {
+  auto dep = static_cast<const IndexUnitReader::DependencyInfo *>(c_dep);
+  return toIndexStoreString(dep->FilePath);
+}
+
+indexstore_string_ref_t
+indexstore_unit_dependency_get_modulename(indexstore_unit_dependency_t c_dep) {
+  auto dep = static_cast<const IndexUnitReader::DependencyInfo *>(c_dep);
+  return toIndexStoreString(dep->ModuleName);
+}
+
+indexstore_string_ref_t
+indexstore_unit_dependency_get_name(indexstore_unit_dependency_t c_dep) {
+  auto dep = static_cast<const IndexUnitReader::DependencyInfo *>(c_dep);
+  return toIndexStoreString(dep->UnitOrRecordName);
+}
+
+indexstore_string_ref_t
+indexstore_unit_include_get_source_path(indexstore_unit_include_t c_inc) {
+  auto inc = static_cast<const IndexUnitReader::IncludeInfo *>(c_inc);
+  return toIndexStoreString(inc->SourcePath);
+}
+
+indexstore_string_ref_t
+indexstore_unit_include_get_target_path(indexstore_unit_include_t c_inc) {
+  auto inc = static_cast<const IndexUnitReader::IncludeInfo *>(c_inc);
+  return toIndexStoreString(inc->TargetPath);
+}
+
+unsigned
+indexstore_unit_include_get_source_line(indexstore_unit_include_t c_inc) {
+  auto inc = static_cast<const IndexUnitReader::IncludeInfo *>(c_inc);
+  return inc->SourceLine;
+}
+
+#if INDEXSTORE_HAS_BLOCKS
+bool indexstore_unit_reader_dependencies_apply(
+    indexstore_unit_reader_t rdr,
+    INDEXSTORE_NOESCAPE bool (^applier)(indexstore_unit_dependency_t)) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return reader->foreachDependency(
+      [&](const IndexUnitReader::DependencyInfo &depInfo) -> bool {
+        return applier((void *)&depInfo);
+      });
+}
+
+bool indexstore_unit_reader_includes_apply(
+    indexstore_unit_reader_t rdr,
+    INDEXSTORE_NOESCAPE bool (^applier)(indexstore_unit_include_t)) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return reader->foreachInclude(
+      [&](const IndexUnitReader::IncludeInfo &incInfo) -> bool {
+        return applier((void *)&incInfo);
+      });
+}
+#endif
+
+bool indexstore_unit_reader_dependencies_apply_f(
+    indexstore_unit_reader_t rdr, void *context,
+    INDEXSTORE_NOESCAPE bool (*applier)(void *context,
+                                        indexstore_unit_dependency_t)) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return reader->foreachDependency(
+      [&](const IndexUnitReader::DependencyInfo &depInfo) -> bool {
+        return applier(context, (void *)&depInfo);
+      });
+}
+
+bool indexstore_unit_reader_includes_apply_f(
+    indexstore_unit_reader_t rdr, void *context,
+    INDEXSTORE_NOESCAPE bool (*applier)(void *context,
+                                        indexstore_unit_include_t)) {
+  auto reader = static_cast<IndexUnitReader *>(rdr);
+  return reader->foreachInclude(
+      [&](const IndexUnitReader::IncludeInfo &incInfo) -> bool {
+        return applier(context, (void *)&incInfo);
+      });
+}

clang/tools/IndexStore/IndexStore.exports

+indexstore_error_get_description
+indexstore_error_dispose
+indexstore_format_version
+indexstore_store_create
+indexstore_store_dispose
+indexstore_store_get_unit_modification_time
+indexstore_store_get_unit_name_from_output_path
+indexstore_store_units_apply
+indexstore_store_units_apply_f
+indexstore_store_set_unit_event_handler
+indexstore_store_set_unit_event_handler_f
+indexstore_store_start_unit_event_listening
+indexstore_store_stop_unit_event_listening
+indexstore_store_discard_unit
+indexstore_store_discard_record
+indexstore_store_purge_stale_data
+indexstore_symbol_get_kind
+indexstore_symbol_get_language
+indexstore_symbol_get_properties
+indexstore_symbol_get_roles
+indexstore_symbol_get_related_roles
+indexstore_symbol_get_subkind
+indexstore_symbol_get_name
+indexstore_symbol_get_usr
+indexstore_symbol_get_codegen_name
+indexstore_symbol_relation_get_roles
+indexstore_symbol_relation_get_symbol
+indexstore_occurrence_get_symbol
+indexstore_occurrence_get_roles
+indexstore_occurrence_get_line_col
+indexstore_occurrence_relations_apply
+indexstore_occurrence_relations_apply_f
+indexstore_record_reader_create
+indexstore_record_reader_dispose
+indexstore_record_reader_search_symbols
+indexstore_record_reader_search_symbols_f
+indexstore_record_reader_symbols_apply
+indexstore_record_reader_symbols_apply_f
+indexstore_record_reader_occurrences_apply
+indexstore_record_reader_occurrences_apply_f
+indexstore_record_reader_occurrences_in_line_range_apply
+indexstore_record_reader_occurrences_in_line_range_apply_f
+indexstore_record_reader_occurrences_of_symbols_apply
+indexstore_record_reader_occurrences_of_symbols_apply_f
+indexstore_unit_dependency_get_kind
+indexstore_unit_dependency_get_filepath
+indexstore_unit_dependency_get_modulename
+indexstore_unit_dependency_get_name
+indexstore_unit_dependency_is_system
+indexstore_unit_event_get_kind
+indexstore_unit_event_get_unit_name
+indexstore_unit_event_notification_get_event
+indexstore_unit_event_notification_get_events_count
+indexstore_unit_event_notification_is_initial
+indexstore_unit_reader_create
+indexstore_unit_reader_dispose
+indexstore_unit_reader_get_main_file
+indexstore_unit_reader_get_modification_time
+indexstore_unit_reader_get_module_name
+indexstore_unit_reader_get_provider_identifier
+indexstore_unit_reader_get_provider_version
+indexstore_unit_reader_get_working_dir
+indexstore_unit_reader_get_output_file
+indexstore_unit_reader_get_sysroot_path
+indexstore_unit_reader_get_target
+indexstore_unit_reader_dependencies_apply
+indexstore_unit_reader_dependencies_apply_f
+indexstore_unit_reader_includes_apply
+indexstore_unit_reader_includes_apply_f
+indexstore_unit_reader_has_main_file
+indexstore_unit_reader_is_debug_compilation
+indexstore_unit_reader_is_module_unit
+indexstore_unit_reader_is_system_unit
+indexstore_unit_include_get_source_path
+indexstore_unit_include_get_target_path
+indexstore_unit_include_get_source_line

clang/tools/c-index-test/CMakeLists.txt

+include(CheckIncludeFiles)
+
 set(LLVM_LINK_COMPONENTS
   support
 )
 
 add_clang_executable(c-index-test
   c-index-test.c
   core_main.cpp
+  JSONAggregation.cpp
   )
 
+set(INDEXSTORE_LIB)
+set(CINDEXTEST_LIBS)
+set(INDEXSTORE_LIB IndexStore)
+
 if(NOT MSVC)
   set_property(
     SOURCE c-index-test.c
     PROPERTY COMPILE_FLAGS "-std=gnu89"
     )
 endif()
 
 if (LLVM_BUILD_STATIC)
   target_link_libraries(c-index-test
     PRIVATE
     libclang_static
     clangCodeGen
     clangIndex
+    clangIndexSerialization
+    ${CINDEXTEST_LIBS}
   )
 else()
   target_link_libraries(c-index-test
     PRIVATE
     libclang
+    ${INDEXSTORE_LIB}
     clangAST
     clangBasic
     clangCodeGen
+    clangDirectoryWatcher
+    clangIndexSerialization
     clangFrontend
     clangIndex
     clangSerialization
+    ${CINDEXTEST_LIBS}
   )
 endif()
 
 set_target_properties(c-index-test
   PROPERTIES
   LINKER_LANGUAGE CXX)
 
 # If libxml2 is available, make it available for c-index-test.
 if (CLANG_HAVE_LIBXML)
   if ((CMAKE_OSX_SYSROOT) AND (EXISTS ${CMAKE_OSX_SYSROOT}/${LIBXML2_INCLUDE_DIR}))
     include_directories(SYSTEM ${CMAKE_OSX_SYSROOT}/${LIBXML2_INCLUDE_DIR})
   else()
     include_directories(SYSTEM ${LIBXML2_INCLUDE_DIR})
   endif()
   target_link_libraries(c-index-test PRIVATE ${LIBXML2_LIBRARIES})
 endif()
 
+if(APPLE)
+  check_include_files("CoreServices/CoreServices.h" HAVE_CORESERVICES_H)
+  if(HAVE_CORESERVICES_H)
+    target_link_libraries(c-index-test PRIVATE "-framework CoreServices")
+  endif()
+endif()
+
 if (NOT LLVM_INSTALL_TOOLCHAIN_ONLY)
   if(INTERNAL_INSTALL_PREFIX)
     set(INSTALL_DESTINATION "${INTERNAL_INSTALL_PREFIX}/bin")
     set_property(TARGET c-index-test APPEND PROPERTY INSTALL_RPATH
        "@executable_path/../../lib")
   else()
     set(INSTALL_DESTINATION bin)
   endif()

clang/tools/c-index-test/JSONAggregation.h

+//===--- JSONAggregation.h - Index data aggregation in JSON format --------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_TOOLS_CINDEXTEST_JSONAGGREGATION_H
+#define LLVM_CLANG_TOOLS_CINDEXTEST_JSONAGGREGATION_H
+
+#include "clang/Basic/LLVM.h"
+
+namespace clang {
+namespace index {
+
+/// Returns true if an error occurred, false otherwise.
+bool aggregateDataAsJSON(StringRef StorePath, raw_ostream &OS);
+
+} // end namespace index
+} // end namespace clang
+
+#endif

clang/tools/c-index-test/JSONAggregation.cpp

+//===--- JSONAggregation.cpp - Index data aggregation in JSON format ------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "JSONAggregation.h"
+#include "indexstore/Conversions.h"
+#include "indexstore/IndexStoreCXX.h"
+#include "clang/Frontend/Utils.h"
+#include "llvm/ADT/StringMap.h"
+#include "llvm/Support/Allocator.h"
+#include "llvm/Support/BuryPointer.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/raw_ostream.h"
+
+using namespace clang;
+using namespace clang::index;
+using namespace indexstore;
+using namespace llvm;
+
+namespace {
+
+typedef size_t FilePathIndex;
+typedef size_t RecordIndex;
+typedef size_t SymbolIndex;
+
+struct UnitSourceInfo {
+  FilePathIndex FilePath;
+  SmallVector<RecordIndex, 2> AssociatedRecords;
+};
+
+struct UnitInfo {
+  std::string Name;
+  SmallVector<UnitSourceInfo, 8> Sources;
+  SmallVector<std::string, 3> UnitDepends;
+  FilePathIndex OutFile;
+  StringRef Triple;
+};
+
+struct SymbolInfo {
+  SymbolKind Kind;
+  SymbolLanguage Lang;
+  StringRef USR;
+  StringRef Name;
+  StringRef CodegenName;
+  SymbolRoleSet Roles = 0;
+  SymbolRoleSet RelatedRoles = 0;
+};
+
+struct SymbolRelationInfo {
+  SymbolIndex RelatedSymbol;
+  SymbolRoleSet Roles;
+  SymbolRelationInfo(SymbolIndex relSymbol, SymbolRoleSet roles)
+      : RelatedSymbol(relSymbol), Roles(roles) {}
+};
+
+struct SymbolOccurrenceInfo {
+  SymbolIndex Symbol;
+  SymbolRoleSet Roles = 0;
+  std::vector<SymbolRelationInfo> Relations;
+  unsigned Line;
+  unsigned Column;
+};
+
+struct RecordInfo {
+  SmallVector<SymbolOccurrenceInfo, 8> Occurrences;
+};
+
+class Aggregator {
+  IndexStore Store;
+
+  BumpPtrAllocator Allocator;
+
+  StringMap<FilePathIndex, BumpPtrAllocator &> FilePathIndices;
+  std::vector<StringRef> FilePaths;
+  StringMap<char, BumpPtrAllocator &> Triples;
+
+  std::vector<std::unique_ptr<UnitInfo>> Units;
+
+  StringMap<RecordIndex, BumpPtrAllocator &> RecordIndices;
+  std::vector<std::unique_ptr<RecordInfo>> Records;
+
+  StringMap<SymbolIndex, BumpPtrAllocator &> SymbolIndices;
+  std::vector<SymbolInfo> Symbols;
+
+public:
+  explicit Aggregator(IndexStore store)
+      : Store(std::move(store)), FilePathIndices(Allocator), Triples(Allocator),
+        RecordIndices(Allocator), SymbolIndices(Allocator) {}
+
+  bool process();
+  void processUnit(StringRef name, IndexUnitReader &UnitReader);
+  void dumpJSON(raw_ostream &OS);
+
+private:
+  StringRef copyStr(StringRef str) {
+    if (str.empty())
+      return StringRef();
+    char *buf = Allocator.Allocate<char>(str.size());
+    std::copy(str.begin(), str.end(), buf);
+    return StringRef(buf, str.size());
+  }
+
+  StringRef getTripleString(StringRef inputTriple) {
+    return Triples.insert(std::make_pair(inputTriple, 0)).first->first();
+  }
+
+  FilePathIndex getFilePathIndex(StringRef path, StringRef workingDir);
+  RecordIndex getRecordIndex(StringRef recordFile);
+  SymbolIndex getSymbolIndex(IndexRecordSymbol sym);
+  std::unique_ptr<RecordInfo> processRecord(StringRef recordFile);
+};
+
+} // anonymous namespace
+
+bool Aggregator::process() {
+  bool succ =
+      Store.foreachUnit(/*sorted=*/true, [&](StringRef unitName) -> bool {
+        std::string error;
+        auto unitReader = IndexUnitReader(Store, unitName, error);
+        if (!unitReader) {
+          errs() << "error opening unit file '" << unitName << "': " << error
+                 << '\n';
+          return false;
+        }
+
+        processUnit(unitName, unitReader);
+        return true;
+      });
+
+  return !succ;
+}
+
+void Aggregator::processUnit(StringRef name, IndexUnitReader &UnitReader) {
+  auto workDir = UnitReader.getWorkingDirectory();
+  auto unit = std::make_unique<UnitInfo>();
+  unit->Name = name.str();
+  unit->Triple = getTripleString(UnitReader.getTarget());
+  unit->OutFile = getFilePathIndex(UnitReader.getOutputFile(), workDir);
+
+  struct DepInfo {
+    UnitSourceInfo source;
+    std::string unitName;
+  };
+  SmallVector<DepInfo, 32> Deps;
+  UnitReader.foreachDependency([&](IndexUnitDependency dep) -> bool {
+    Deps.resize(Deps.size() + 1);
+    auto &depInfo = Deps.back();
+    switch (dep.getKind()) {
+    case IndexUnitDependency::DependencyKind::Unit: {
+      depInfo.unitName = dep.getName().str();
+      StringRef filePath = dep.getFilePath();
+      if (!filePath.empty())
+        depInfo.source.FilePath = getFilePathIndex(filePath, workDir);
+      break;
+    }
+    case IndexUnitDependency::DependencyKind::Record: {
+      depInfo.source.FilePath = getFilePathIndex(dep.getFilePath(), workDir);
+      RecordIndex recIndex = getRecordIndex(dep.getName());
+      depInfo.source.AssociatedRecords.push_back(recIndex);
+      break;
+    }
+    case IndexUnitDependency::DependencyKind::File:
+      depInfo.source.FilePath = getFilePathIndex(dep.getFilePath(), workDir);
+    }
+    return true;
+  });
+
+  unit->Sources.reserve(Deps.size());
+  for (auto &dep : Deps) {
+    if (!dep.unitName.empty()) {
+      unit->UnitDepends.emplace_back(std::move(dep.unitName));
+    } else {
+      unit->Sources.push_back(std::move(dep.source));
+    }
+  }
+
+  Units.push_back(std::move(unit));
+}
+
+FilePathIndex Aggregator::getFilePathIndex(StringRef path,
+                                           StringRef workingDir) {
+  StringRef absPath;
+  SmallString<128> absPathBuf;
+  if (sys::path::is_absolute(path) || workingDir.empty()) {
+    absPath = path;
+  } else {
+    absPathBuf = workingDir;
+    sys::path::append(absPathBuf, path);
+    absPath = absPathBuf.str();
+  }
+
+  auto pair = FilePathIndices.insert(std::make_pair(absPath, FilePaths.size()));
+  bool wasInserted = pair.second;
+  if (wasInserted) {
+    FilePaths.push_back(pair.first->first());
+  }
+  return pair.first->second;
+}
+
+RecordIndex Aggregator::getRecordIndex(StringRef recordFile) {
+  auto pair = RecordIndices.insert(std::make_pair(recordFile, Records.size()));
+  bool wasInserted = pair.second;
+  if (wasInserted) {
+    Records.push_back(processRecord(recordFile));
+  }
+  return pair.first->second;
+}
+
+std::unique_ptr<RecordInfo> Aggregator::processRecord(StringRef recordFile) {
+  std::string error;
+  auto recordReader = IndexRecordReader(Store, recordFile, error);
+  if (!recordReader) {
+    errs() << "failed reading record file: " << recordFile << '\n';
+    ::exit(1);
+  }
+  auto record = std::make_unique<RecordInfo>();
+  recordReader.foreachOccurrence([&](IndexRecordOccurrence idxOccur) -> bool {
+    SymbolIndex symIdx = getSymbolIndex(idxOccur.getSymbol());
+    SymbolInfo &symInfo = Symbols[symIdx];
+    symInfo.Roles |= getSymbolRoles(idxOccur.getRoles());
+    SymbolOccurrenceInfo occurInfo;
+    occurInfo.Symbol = symIdx;
+    idxOccur.foreachRelation([&](IndexSymbolRelation rel) -> bool {
+      SymbolIndex relsymIdx = getSymbolIndex(rel.getSymbol());
+      SymbolInfo &relsymInfo = Symbols[relsymIdx];
+      relsymInfo.RelatedRoles |= getSymbolRoles(rel.getRoles());
+      occurInfo.Relations.emplace_back(relsymIdx,
+                                       getSymbolRoles(rel.getRoles()));
+      return true;
+    });
+    occurInfo.Roles = getSymbolRoles(idxOccur.getRoles());
+    std::tie(occurInfo.Line, occurInfo.Column) = idxOccur.getLineCol();
+    record->Occurrences.push_back(std::move(occurInfo));
+    return true;
+  });
+  return record;
+}
+
+SymbolIndex Aggregator::getSymbolIndex(IndexRecordSymbol sym) {
+  auto pair =
+      SymbolIndices.insert(std::make_pair(sym.getUSR(), Symbols.size()));
+  bool wasInserted = pair.second;
+  if (wasInserted) {
+    SymbolInfo symInfo;
+    symInfo.Kind = getSymbolKind(sym.getKind());
+    symInfo.Lang = getSymbolLanguage(sym.getLanguage());
+    symInfo.USR = pair.first->first();
+    symInfo.Name = copyStr(sym.getName());
+    symInfo.CodegenName = copyStr(sym.getCodegenName());
+    Symbols.push_back(std::move(symInfo));
+  }
+  return pair.first->second;
+}
+
+void Aggregator::dumpJSON(raw_ostream &OS) {
+  OS << "{\n";
+  OS.indent(2) << "\"files\": [\n";
+  for (unsigned i = 0, e = FilePaths.size(); i != e; ++i) {
+    OS.indent(4) << '\"' << FilePaths[i] << '\"';
+    if (i < e - 1)
+      OS << ',';
+    OS << '\n';
+  }
+  OS.indent(2) << "],\n";
+
+  OS.indent(2) << "\"symbols\": [\n";
+  for (unsigned i = 0, e = Symbols.size(); i != e; ++i) {
+    OS.indent(4) << "{\n";
+    SymbolInfo &symInfo = Symbols[i];
+    OS.indent(6) << "\"kind\": \"" << getSymbolKindString(symInfo.Kind)
+                 << "\",\n";
+    OS.indent(6) << "\"lang\": \"" << getSymbolLanguageString(symInfo.Lang)
+                 << "\",\n";
+    OS.indent(6) << "\"usr\": \"" << symInfo.USR << "\",\n";
+    OS.indent(6) << "\"name\": \"" << symInfo.Name << "\",\n";
+    if (!symInfo.CodegenName.empty())
+      OS.indent(6) << "\"codegen\": \"" << symInfo.CodegenName << "\",\n";
+    OS.indent(6) << "\"roles\": \"";
+    printSymbolRoles(symInfo.Roles, OS);
+    OS << '\"';
+    if (symInfo.RelatedRoles != 0) {
+      OS << ",\n";
+      OS.indent(6) << "\"rel-roles\": \"";
+      printSymbolRoles(symInfo.RelatedRoles, OS);
+      OS << '\"';
+    }
+    OS << '\n';
+    OS.indent(4) << "}";
+    if (i < e - 1)
+      OS << ',';
+    OS << '\n';
+  }
+  OS.indent(2) << "],\n";
+
+  OS.indent(2) << "\"records\": [\n";
+  for (unsigned i = 0, e = Records.size(); i != e; ++i) {
+    OS.indent(4) << "{\n";
+    RecordInfo &recInfo = *Records[i];
+    OS.indent(6) << "\"occurrences\": [\n";
+    for (unsigned oi = 0, oe = recInfo.Occurrences.size(); oi != oe; ++oi) {
+      OS.indent(8) << "{\n";
+      SymbolOccurrenceInfo &occurInfo = recInfo.Occurrences[oi];
+      OS.indent(10) << "\"symbol\": " << occurInfo.Symbol << ",\n";
+      OS.indent(10) << "\"line\": " << occurInfo.Line << ",\n";
+      OS.indent(10) << "\"col\": " << occurInfo.Column << ",\n";
+      OS.indent(10) << "\"roles\": \"";
+      printSymbolRoles(occurInfo.Roles, OS);
+      OS << '\"';
+      if (!occurInfo.Relations.empty()) {
+        OS << ",\n";
+        OS.indent(10) << "\"relations\": [\n";
+        for (unsigned ri = 0, re = occurInfo.Relations.size(); ri != re; ++ri) {
+          OS.indent(12) << "{\n";
+          SymbolRelationInfo &relInfo = occurInfo.Relations[ri];
+          OS.indent(14) << "\"symbol\": " << relInfo.RelatedSymbol << ",\n";
+          OS.indent(14) << "\"rel-roles\": \"";
+          printSymbolRoles(relInfo.Roles, OS);
+          OS << "\"\n";
+          OS.indent(12) << "}";
+          if (ri < re - 1)
+            OS << ',';
+          OS << '\n';
+        }
+        OS.indent(10) << "]\n";
+      }
+      OS << '\n';
+      OS.indent(8) << "}";
+      if (oi < oe - 1)
+        OS << ',';
+      OS << '\n';
+    }
+    OS.indent(6) << "]\n";
+    OS.indent(4) << "}";
+    if (i < e - 1)
+      OS << ',';
+    OS << '\n';
+  }
+  OS.indent(2) << "],\n";
+
+  StringMap<size_t> UnitIndicesByName;
+  for (unsigned i = 0, e = Units.size(); i != e; ++i) {
+    UnitInfo &unit = *Units[i];
+    UnitIndicesByName[unit.Name] = i;
+  }
+
+  OS.indent(2) << "\"units\": [\n";
+  for (unsigned i = 0, e = Units.size(); i != e; ++i) {
+    OS.indent(4) << "{\n";
+    UnitInfo &unit = *Units[i];
+    OS.indent(6) << "\"triple\": \"" << unit.Triple << "\",\n";
+    OS.indent(6) << "\"out-file\": " << unit.OutFile << ",\n";
+    if (!unit.UnitDepends.empty()) {
+      OS.indent(6) << "\"unit-dependencies\": [";
+      for (unsigned ui = 0, ue = unit.UnitDepends.size(); ui != ue; ++ui) {
+        OS << UnitIndicesByName[unit.UnitDepends[ui]];
+        if (ui < ue - 1)
+          OS << ", ";
+      }
+      OS << "],\n";
+    }
+    OS.indent(6) << "\"sources\": [\n";
+    for (unsigned si = 0, se = unit.Sources.size(); si != se; ++si) {
+      OS.indent(8) << "{\n";
+      UnitSourceInfo &source = unit.Sources[si];
+      OS.indent(10) << "\"file\": " << source.FilePath;
+      if (!source.AssociatedRecords.empty()) {
+        OS << ",\n";
+        OS.indent(10) << "\"records\": [";
+        for (unsigned ri = 0, re = source.AssociatedRecords.size(); ri != re;
+             ++ri) {
+          OS << source.AssociatedRecords[ri];
+          if (ri < re - 1)
+            OS << ", ";
+        }
+        OS << ']';
+      }
+      OS << '\n';
+      OS.indent(8) << "}";
+      if (si < se - 1)
+        OS << ',';
+      OS << '\n';
+    }
+    OS.indent(6) << "]\n";
+    OS.indent(4) << "}";
+    if (i < e - 1)
+      OS << ',';
+    OS << '\n';
+  }
+  OS.indent(2) << "]\n";
+  OS << "}\n";
+}
+
+bool index::aggregateDataAsJSON(StringRef StorePath, raw_ostream &OS) {
+  std::string error;
+  auto dataStore = IndexStore(StorePath, error);
+  if (!dataStore) {
+    errs() << "error opening store path '" << StorePath << "': " << error
+           << '\n';
+    return true;
+  }
+
+  // Explicitely avoid doing any memory cleanup for aggregator since the process
+  // is going to exit when we are done.
+  Aggregator *aggregator = new Aggregator(std::move(dataStore));
+  bool err = aggregator->process();
+  if (err)
+    return true;
+  aggregator->dumpJSON(OS);
+  llvm::BuryPointer(aggregator);
+  return false;
+}

clang/tools/c-index-test/core_main.cpp

 //===-- core_main.cpp - Core Index Tool testbed ---------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
+#include "JSONAggregation.h"
+#include "indexstore/Conversions.h"
+#include "indexstore/IndexStoreCXX.h"
 #include "clang/AST/Mangle.h"
 #include "clang/Basic/LangOptions.h"
 #include "clang/CodeGen/ObjectFilePCHContainerOperations.h"
+#include "clang/DirectoryWatcher/DirectoryWatcher.h"
 #include "clang/Frontend/ASTUnit.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/CompilerInvocation.h"
 #include "clang/Frontend/FrontendAction.h"
-#include "clang/Index/IndexingAction.h"
 #include "clang/Index/IndexDataConsumer.h"
+#include "clang/Index/IndexingAction.h"
 #include "clang/Index/USRGeneration.h"
+#include "clang/IndexSerialization/IndexRecordReader.h"
+#include "clang/IndexSerialization/IndexUnitReader.h"
 #include "clang/Lex/Preprocessor.h"
 #include "clang/Serialization/ASTReader.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/FileSystem.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/PrettyStackTrace.h"
 #include "llvm/Support/Signals.h"
 #include "llvm/Support/raw_ostream.h"
-#include "llvm/Support/PrettyStackTrace.h"
+#include <thread>
 
 using namespace clang;
 using namespace clang::index;
 using namespace llvm;
 
 extern "C" int indextest_core_main(int argc, const char **argv);
 
 namespace {
 
 enum class ActionType {
   None,
   PrintSourceSymbols,
+  PrintRecord,
+  PrintUnit,
+  PrintStoreFormatVersion,
+  AggregateAsJSON,
+  WatchDir,
 };
 
 namespace options {
 
 static cl::OptionCategory IndexTestCoreCategory("index-test-core options");
 
-static cl::opt<ActionType>
-Action(cl::desc("Action:"), cl::init(ActionType::None),
-       cl::values(
-          clEnumValN(ActionType::PrintSourceSymbols,
-                     "print-source-symbols", "Print symbols from source")),
-       cl::cat(IndexTestCoreCategory));
+static cl::opt<ActionType> Action(
+    cl::desc("Action:"), cl::init(ActionType::None),
+        clEnumValN(ActionType::PrintSourceSymbols, "print-source-symbols",
+                   "Print symbols from source"),
+        clEnumValN(ActionType::PrintRecord, "print-record",
+                   "Print record info"),
+        clEnumValN(ActionType::PrintUnit, "print-unit", "Print unit info"),
+        clEnumValN(ActionType::PrintStoreFormatVersion,
+                   "print-store-format-version", "Print store format version"),
+        clEnumValN(ActionType::AggregateAsJSON, "aggregate-json",
+                   "Aggregate index data in JSON format"),
+        clEnumValN(ActionType::WatchDir, "watch-dir",
+                   "Watch directory for file events")),
 
+static cl::opt<std::string> OutputFile("o", cl::desc("output file"),
+                                       cl::cat(IndexTestCoreCategory));
+
+static cl::list<std::string> InputFiles(cl::Positional,
+                                        cl::desc("<filename>..."));
+
 static cl::extrahelp MoreHelp(
   "\nAdd \"-- <compiler arguments>\" at the end to setup the compiler "
   "invocation\n"
 );
 
 static cl::opt<bool>
 DumpModuleImports("dump-imported-module-files",
                cl::desc("Print symbols and input files from imported modules"));
 
 static cl::opt<bool>
 IncludeLocals("include-locals", cl::desc("Print local symbols"));
 
 static cl::opt<std::string>
 ModuleFilePath("module-file",
                cl::desc("Path to module file to print symbols from"));
 static cl::opt<std::string>
   ModuleFormat("fmodule-format", cl::init("raw"),
         cl::desc("Container format for clang modules and PCH, 'raw' or 'obj'"));
 
+static cl::opt<std::string> FilePathAndRange(
+    "filepath", cl::desc("File path that can optionally include a line range"));
 }
 } // anonymous namespace
 
 static void printSymbolInfo(SymbolInfo SymInfo, raw_ostream &OS);
 static void printSymbolNameAndUSR(const Decl *D, ASTContext &Ctx,
                                   raw_ostream &OS);
 static void printSymbolNameAndUSR(const clang::Module *Mod, raw_ostream &OS);
 
static bool printSourceSymbolsFromModule(StringRef modulePath,
   PrintIndexDataConsumer DataConsumer(outs());
   IndexingOptions IndexOpts;
   indexASTUnit(*AU, DataConsumer, IndexOpts);
 
   return false;
 }
 
 //===----------------------------------------------------------------------===//
+// Print Record
+//===----------------------------------------------------------------------===//
+
+static void printSymbol(const IndexRecordDecl &Rec, raw_ostream &OS);
+static void printSymbol(const IndexRecordOccurrence &Rec, raw_ostream &OS);
+
+static int printRecord(StringRef Filename, raw_ostream &OS) {
+  std::string Error;
+  auto Reader = IndexRecordReader::createWithFilePath(Filename, Error);
+  if (!Reader) {
+    errs() << Error << '\n';
+    return true;
+  }
+
+  Reader->foreachDecl(/*noCache=*/true,
+                      [&](const IndexRecordDecl *Rec) -> bool {
+                        printSymbol(*Rec, OS);
+                        return true;
+                      });
+  OS << "------------\n";
+  Reader->foreachOccurrence([&](const IndexRecordOccurrence &Rec) -> bool {
+    printSymbol(Rec, OS);
+    return true;
+  });
+
+  return false;
+};
+
+//===----------------------------------------------------------------------===//
+// Print Store Records
+//===----------------------------------------------------------------------===//
+
+static void printSymbol(indexstore::IndexRecordSymbol Sym, raw_ostream &OS);
+static void printSymbol(indexstore::IndexRecordOccurrence Occur,
+                        raw_ostream &OS);
+
+static bool printStoreRecord(indexstore::IndexStore &Store, StringRef RecName,
+                             StringRef FilePath, raw_ostream &OS) {
+  std::string Error;
+  indexstore::IndexRecordReader Reader(Store, RecName, Error);
+  if (!Reader) {
+    errs() << "error loading record: " << Error << "\n";
+    return true;
+  }
+
+  StringRef Filename = sys::path::filename(FilePath);
+  OS << Filename << '\n';
+  OS << "------------\n";
+  Reader.foreachSymbol(/*noCache=*/true,
+                       [&](indexstore::IndexRecordSymbol Sym) -> bool {
+                         printSymbol(Sym, OS);
+                         return true;
+                       });
+  OS << "------------\n";
+  Reader.foreachOccurrence(
+      [&](indexstore::IndexRecordOccurrence Occur) -> bool {
+        printSymbol(Occur, OS);
+        return true;
+      });
+
+  return false;
+}
+
+static int printStoreRecords(StringRef StorePath, raw_ostream &OS) {
+  std::string Error;
+  indexstore::IndexStore Store(StorePath, Error);
+  if (!Store) {
+    errs() << "error loading store: " << Error << "\n";
+    return 1;
+  }
+
+  bool Success =
+      Store.foreachUnit(/*sorted=*/true, [&](StringRef UnitName) -> bool {
+        indexstore::IndexUnitReader Reader(Store, UnitName, Error);
+        if (!Reader) {
+          errs() << "error loading unit: " << Error << "\n";
+          return false;
+        }
+        return Reader.foreachDependency(
+            [&](indexstore::IndexUnitDependency Dep) -> bool {
+              if (Dep.getKind() ==
+                  indexstore::IndexUnitDependency::DependencyKind::Record) {
+                bool Err = printStoreRecord(Store, Dep.getName(),
+                                            Dep.getFilePath(), OS);
+                OS << '\n';
+                return !Err;
+              }
+              return true;
+            });
+      });
+
+  return !Success;
+}
+
+static std::string findRecordNameForFile(indexstore::IndexStore &store,
+                                         StringRef filePath) {
+  std::string recName;
+  store.foreachUnit(/*sorted=*/false, [&](StringRef unitName) -> bool {
+    std::string error;
+    indexstore::IndexUnitReader Reader(store, unitName, error);
+    if (!Reader) {
+      errs() << "error loading unit: " << error << "\n";
+      return false;
+    }
+    Reader.foreachDependency([&](indexstore::IndexUnitDependency Dep) -> bool {
+      if (Dep.getKind() ==
+          indexstore::IndexUnitDependency::DependencyKind::Record) {
+        if (Dep.getFilePath() == filePath) {
+          recName = Dep.getName().str();
+          return false;
+        }
+        return true;
+      }
+      return true;
+    });
+    return true;
+  });
+  return recName;
+}
+
+static int printStoreFileRecord(StringRef storePath, StringRef filePath,
+                                Optional<unsigned> lineStart,
+                                unsigned lineCount, raw_ostream &OS) {
+  std::string error;
+  indexstore::IndexStore store(storePath, error);
+  if (!store) {
+    errs() << "error loading store: " << error << "\n";
+    return 1;
+  }
+
+  std::string recName = findRecordNameForFile(store, filePath);
+  if (recName.empty()) {
+    errs() << "could not find record for '" << filePath << "'\n";
+    return 1;
+  }
+
+  if (!lineStart.hasValue())
+    return printStoreRecord(store, recName, filePath, OS);
+
+  indexstore::IndexRecordReader Reader(store, recName, error);
+  if (!Reader) {
+    errs() << "error loading record: " << error << "\n";
+    return 1;
+  }
+
+  Reader.foreachOccurrenceInLineRange(
+      *lineStart, lineCount,
+      [&](indexstore::IndexRecordOccurrence Occur) -> bool {
+        printSymbol(Occur, OS);
+        return true;
+      });
+
+  return 0;
+}
+
+//===----------------------------------------------------------------------===//
+// Print Unit
+//===----------------------------------------------------------------------===//
+
+static int printUnit(StringRef Filename, raw_ostream &OS) {
+  std::string Error;
+  auto Reader = IndexUnitReader::createWithFilePath(Filename, Error);
+  if (!Reader) {
+    errs() << Error << '\n';
+    return true;
+  }
+
+  OS << "provider: " << Reader->getProviderIdentifier() << '-'
+     << Reader->getProviderVersion() << '\n';
+  OS << "is-system: " << Reader->isSystemUnit() << '\n';
+  OS << "is-module: " << Reader->isModuleUnit() << '\n';
+  OS << "module-name: "
+     << (Reader->getModuleName().empty() ? "<none>" : Reader->getModuleName())
+     << '\n';
+  OS << "has-main: " << Reader->hasMainFile() << '\n';
+  OS << "main-path: " << Reader->getMainFilePath() << '\n';
+  OS << "work-dir: " << Reader->getWorkingDirectory() << '\n';
+  OS << "out-file: " << Reader->getOutputFile() << '\n';
+  OS << "target: " << Reader->getTarget() << '\n';
+  OS << "is-debug: " << Reader->isDebugCompilation() << '\n';
+  OS << "DEPEND START\n";
+  unsigned NumDepends = 0;
+  Reader->foreachDependency(
+      [&](const IndexUnitReader::DependencyInfo &Dep) -> bool {
+        switch (Dep.Kind) {
+        case IndexUnitReader::DependencyKind::Unit:
+          OS << "Unit | ";
+          break;
+        case IndexUnitReader::DependencyKind::Record:
+          OS << "Record | ";
+          break;
+        case IndexUnitReader::DependencyKind::File:
+          OS << "File | ";
+          break;
+        }
+        OS << (Dep.IsSystem ? "system" : "user");
+        OS << " | ";
+        if (!Dep.ModuleName.empty())
+          OS << Dep.ModuleName << " | ";
+        OS << Dep.FilePath;
+        if (!Dep.UnitOrRecordName.empty())
+          OS << " | " << Dep.UnitOrRecordName;
+        OS << '\n';
+        ++NumDepends;
+        return true;
+      });
+  OS << "DEPEND END (" << NumDepends << ")\n";
+  OS << "INCLUDE START\n";
+  unsigned NumIncludes = 0;
+  Reader->foreachInclude([&](const IndexUnitReader::IncludeInfo &Inc) -> bool {
+    OS << Inc.SourcePath << ":" << Inc.SourceLine << " | ";
+    OS << Inc.TargetPath << '\n';
+    ++NumIncludes;
+    return true;
+  });
+  OS << "INCLUDE END (" << NumIncludes << ")\n";
+
+  return false;
+};
+
+//===----------------------------------------------------------------------===//
+// Print Store Units
+//===----------------------------------------------------------------------===//
+
+static bool printStoreUnit(indexstore::IndexStore &Store, StringRef UnitName,
+                           raw_ostream &OS) {
+  std::string Error;
+  indexstore::IndexUnitReader Reader(Store, UnitName, Error);
+  if (!Reader) {
+    errs() << "error loading unit: " << Error << "\n";
+    return true;
+  }
+
+  OS << "provider: " << Reader.getProviderIdentifier() << '-'
+     << Reader.getProviderVersion() << '\n';
+  OS << "is-system: " << Reader.isSystemUnit() << '\n';
+  OS << "is-module: " << Reader.isModuleUnit() << '\n';
+  OS << "module-name: "
+     << (Reader.getModuleName().empty() ? "<none>" : Reader.getModuleName())
+     << '\n';
+  OS << "has-main: " << Reader.hasMainFile() << '\n';
+  OS << "main-path: " << Reader.getMainFilePath() << '\n';
+  OS << "work-dir: " << Reader.getWorkingDirectory() << '\n';
+  OS << "out-file: " << Reader.getOutputFile() << '\n';
+  OS << "target: " << Reader.getTarget() << '\n';
+  OS << "is-debug: " << Reader.isDebugCompilation() << '\n';
+  OS << "DEPEND START\n";
+  unsigned NumDepends = 0;
+  Reader.foreachDependency([&](indexstore::IndexUnitDependency Dep) -> bool {
+    switch (Dep.getKind()) {
+    case indexstore::IndexUnitDependency::DependencyKind::Unit:
+      OS << "Unit | ";
+      break;
+    case indexstore::IndexUnitDependency::DependencyKind::Record:
+      OS << "Record | ";
+      break;
+    case indexstore::IndexUnitDependency::DependencyKind::File:
+      OS << "File | ";
+      break;
+    }
+    OS << (Dep.isSystem() ? "system" : "user");
+    OS << " | ";
+    if (!Dep.getModuleName().empty())
+      OS << Dep.getModuleName() << " | ";
+    OS << Dep.getFilePath();
+    if (!Dep.getName().empty())
+      OS << " | " << Dep.getName();
+    OS << '\n';
+    ++NumDepends;
+    return true;
+  });
+  OS << "DEPEND END (" << NumDepends << ")\n";
+  OS << "INCLUDE START\n";
+  unsigned NumIncludes = 0;
+  Reader.foreachInclude([&](indexstore::IndexUnitInclude Inc) -> bool {
+    OS << Inc.getSourcePath() << ":" << Inc.getSourceLine() << " | ";
+    OS << Inc.getTargetPath() << '\n';
+    ++NumIncludes;
+    return true;
+  });
+  OS << "INCLUDE END (" << NumIncludes << ")\n";
+
+  return false;
+}
+
+static int printStoreUnits(StringRef StorePath, raw_ostream &OS) {
+  std::string Error;
+  indexstore::IndexStore Store(StorePath, Error);
+  if (!Store) {
+    errs() << "error loading store: " << Error << "\n";
+    return 1;
+  }
+
+  bool Success =
+      Store.foreachUnit(/*sorted=*/true, [&](StringRef UnitName) -> bool {
+        OS << UnitName << '\n';
+        OS << "--------\n";
+        bool err = printStoreUnit(Store, UnitName, OS);
+        OS << '\n';
+        return !err;
+      });
+
+  return !Success;
+}
+
+//===----------------------------------------------------------------------===//
 // Helper Utils
 //===----------------------------------------------------------------------===//
 
 static void printSymbolInfo(SymbolInfo SymInfo, raw_ostream &OS) {
   OS << getSymbolKindString(SymInfo.Kind);
   if (SymInfo.SubKind != SymbolSubKind::None)
     OS << '/' << getSymbolSubKindString(SymInfo.SubKind);
   if (SymInfo.Properties) {

 }
 
 static void printSymbolNameAndUSR(const clang::Module *Mod, raw_ostream &OS) {
   assert(Mod);
   OS << Mod->getFullModuleName() << " | ";
   generateFullUSRForModule(Mod, OS);
 }
 
+static void printSymbol(const IndexRecordDecl &Rec, raw_ostream &OS) {
+  printSymbolInfo(Rec.SymInfo, OS);
+  OS << " | ";
+
+  if (Rec.Name.empty())
+    OS << "<no-name>";
+  else
+    OS << Rec.Name;
+  OS << " | ";
+
+  if (Rec.USR.empty())
+    OS << "<no-usr>";
+  else
+    OS << Rec.USR;
+  OS << " | ";
+
+  if (Rec.CodeGenName.empty())
+    OS << "<no-cgname>";
+  else
+    OS << Rec.CodeGenName;
+  OS << " | ";
+
+  printSymbolRoles(Rec.Roles, OS);
+  OS << " - ";
+  printSymbolRoles(Rec.RelatedRoles, OS);
+  OS << '\n';
+}
+
+static void printSymbol(const IndexRecordOccurrence &Rec, raw_ostream &OS) {
+  OS << Rec.Line << ':' << Rec.Column << " | ";
+  printSymbolInfo(Rec.Dcl->SymInfo, OS);
+  OS << " | ";
+
+  if (Rec.Dcl->USR.empty())
+    OS << "<no-usr>";
+  else
+    OS << Rec.Dcl->USR;
+  OS << " | ";
+
+  printSymbolRoles(Rec.Roles, OS);
+  OS << " | ";
+  OS << "rel: " << Rec.Relations.size() << '\n';
+  for (auto &Rel : Rec.Relations) {
+    OS << '\t';
+    printSymbolRoles(Rel.Roles, OS);
+    OS << " | ";
+    if (Rel.Dcl->USR.empty())
+      OS << "<no-usr>";
+    else
+      OS << Rel.Dcl->USR;
+    OS << '\n';
+  }
+}
+
+static void printSymbol(indexstore::IndexRecordSymbol Sym, raw_ostream &OS) {
+  SymbolInfo SymInfo{getSymbolKind(Sym.getKind()),
+                     getSymbolSubKind(Sym.getSubKind()),
+                     getSymbolLanguage(Sym.getLanguage()),
+                     SymbolPropertySet(Sym.getProperties())};
+
+  printSymbolInfo(SymInfo, OS);
+  OS << " | ";
+
+  if (Sym.getName().empty())
+    OS << "<no-name>";
+  else
+    OS << Sym.getName();
+  OS << " | ";
+
+  if (Sym.getUSR().empty())
+    OS << "<no-usr>";
+  else
+    OS << Sym.getUSR();
+  OS << " | ";
+
+  if (Sym.getCodegenName().empty())
+    OS << "<no-cgname>";
+  else
+    OS << Sym.getCodegenName();
+  OS << " | ";
+
+  printSymbolRoles(getSymbolRoles(Sym.getRoles()), OS);
+  OS << " - ";
+  printSymbolRoles(getSymbolRoles(Sym.getRelatedRoles()), OS);
+  OS << '\n';
+}
+
+static void printSymbol(indexstore::IndexRecordOccurrence Occur,
+                        raw_ostream &OS) {
+  OS << Occur.getLineCol().first << ':' << Occur.getLineCol().second << " | ";
+  auto Sym = Occur.getSymbol();
+  SymbolInfo SymInfo{getSymbolKind(Sym.getKind()),
+                     getSymbolSubKind(Sym.getSubKind()),
+                     getSymbolLanguage(Sym.getLanguage()),
+                     SymbolPropertySet(Sym.getProperties())};
+
+  printSymbolInfo(SymInfo, OS);
+  OS << " | ";
+
+  if (Sym.getUSR().empty())
+    OS << "<no-usr>";
+  else
+    OS << Sym.getUSR();
+  OS << " | ";
+
+  unsigned NumRelations = 0;
+  Occur.foreachRelation([&](indexstore::IndexSymbolRelation) {
+    ++NumRelations;
+    return true;
+  });
+
+  printSymbolRoles(getSymbolRoles(Occur.getRoles()), OS);
+  OS << " | ";
+  OS << "rel: " << NumRelations << '\n';
+  Occur.foreachRelation([&](indexstore::IndexSymbolRelation Rel) {
+    OS << '\t';
+    printSymbolRoles(getSymbolRoles(Rel.getRoles()), OS);
+    OS << " | ";
+    auto Sym = Rel.getSymbol();
+    if (Sym.getUSR().empty())
+      OS << "<no-usr>";
+    else
+      OS << Sym.getUSR();
+    OS << '\n';
+    return true;
+  });
+}
+
+static int watchDirectory(StringRef dirPath) {
+  raw_ostream &OS = outs();
+  auto receiver = [&](ArrayRef<DirectoryWatcher::Event> Events,
+                      bool isInitial) {
+    OS << "-- " << Events.size() << " :\n";
+    for (auto evt : Events) {
+      switch (evt.Kind) {
+      case DirectoryWatcher::Event::EventKind::Modified:
+        OS << "modified: ";
+        break;
+      case DirectoryWatcher::Event::EventKind::Removed:
+        OS << "removed: ";
+        break;
+      case DirectoryWatcher::Event::EventKind::WatchedDirRemoved:
+        OS << "dir deleted: ";
+        break;
+      case DirectoryWatcher::Event::EventKind::WatcherGotInvalidated:
+        OS << "watcher got invalidated: ";
+        break;
+      }
+      OS << evt.Filename << '\n';
+    }
+  };
+  auto watcher = DirectoryWatcher::create(dirPath, receiver,
+                                          /*waitInitialSync=*/true);
+  if (!watcher) {
+    errs() << "failed creating directory watcher" << '\n';
+    return 1;
+  }
+
+  while (1) {
+    std::this_thread::yield();
+  }
+}
+
 //===----------------------------------------------------------------------===//
 // Command line processing.
 //===----------------------------------------------------------------------===//
 
+bool deconstructPathAndRange(StringRef input, std::string &filepath,
+                             Optional<unsigned> &lineStart,
+                             unsigned &lineCount) {
+  StringRef path, range;
+  std::tie(path, range) = input.split(':');
+  StringRef start, end;
+  std::tie(start, end) = range.split(':');
+  filepath = path.str();
+  lineCount = 0;
+  if (start.empty())
+    return false;
+  unsigned num;
+  if (start.getAsInteger(10, num)) {
+    errs() << "couldn't convert to integer: " << start << '\n';
+    return true;
+  }
+  lineStart = num;
+  if (end.empty())
+    return false;
+  if (end.getAsInteger(10, num)) {
+    errs() << "couldn't convert to integer: " << end << '\n';
+    return true;
+  }
+  lineCount = num - lineStart.getValue();
+  return false;
+}
+
 int indextest_core_main(int argc, const char **argv) {
   sys::PrintStackTraceOnErrorSignal(argv[0]);
   PrettyStackTraceProgram X(argc, argv);
   void *MainAddr = (void*) (intptr_t) indextest_core_main;
   std::string Executable = llvm::sys::fs::getMainExecutable(argv[0], MainAddr);
 
   assert(argv[1] == StringRef("core"));
   ++argv;
if (CompArgs.empty()) {
       errs() << "error: missing compiler args; pass '-- <compiler arguments>'\n";
       return 1;
     }
     return printSourceSymbols(Executable.c_str(), CompArgs,
                               options::DumpModuleImports,
                               options::IncludeLocals);
   }
 
+  if (options::Action == ActionType::PrintRecord) {
+    if (!options::FilePathAndRange.empty()) {
+      std::string filepath;
+      Optional<unsigned> lineStart;
+      unsigned lineCount;
+      if (deconstructPathAndRange(options::FilePathAndRange, filepath,
+                                  lineStart, lineCount))
+        return 1;
+
+      if (options::InputFiles.empty()) {
+        errs() << "error: missing index store path\n";
+        return 1;
+      }
+      return printStoreFileRecord(options::InputFiles[0], filepath, lineStart,
+                                  lineCount, outs());
+    }
+
+    if (options::InputFiles.empty()) {
+      errs() << "error: missing input file or directory\n";
+      return 1;
+    }
+
+    if (sys::fs::is_directory(options::InputFiles[0]))
+      return printStoreRecords(options::InputFiles[0], outs());
+    else
+      return printRecord(options::InputFiles[0], outs());
+  }
+
+  if (options::Action == ActionType::PrintUnit) {
+    if (options::InputFiles.empty()) {
+      errs() << "error: missing input file or directory\n";
+      return 1;
+    }
+
+    if (sys::fs::is_directory(options::InputFiles[0]))
+      return printStoreUnits(options::InputFiles[0], outs());
+    else
+      return printUnit(options::InputFiles[0], outs());
+  }
+
+  if (options::Action == ActionType::PrintStoreFormatVersion) {
+    outs() << indexstore::IndexStore::formatVersion() << '\n';
+  }
+
+  if (options::Action == ActionType::AggregateAsJSON) {
+    if (options::InputFiles.empty()) {
+      errs() << "error: missing input data store directory\n";
+      return 1;
+    }
+    StringRef storePath = options::InputFiles[0];
+    if (options::OutputFile.empty())
+      return aggregateDataAsJSON(storePath, outs());
+    std::error_code EC;
+    raw_fd_ostream OS(options::OutputFile, EC, llvm::sys::fs::F_None);
+    if (EC) {
+      errs() << "failed to open output file: " << EC.message() << '\n';
+      return 1;
+    }
+    return aggregateDataAsJSON(storePath, OS);
+  }
+
+  if (options::Action == ActionType::WatchDir) {
+    if (options::InputFiles.empty()) {
+      errs() << "error: missing directory path\n";
+      return 1;
+    }
+    return watchDirectory(options::InputFiles[0]);
+  }
+
   return 0;
 }

clang/unittests/Index/IndexTests.cpp

llvm::raw_ostream &operator<<(llvm::raw_ostream &OS, const TestSymbol &S) {
   return OS << S.QName << '[' << S.WrittenPos << ']' << '@' << S.DeclPos << '('
             << static_cast<unsigned>(S.SymInfo.Kind) << ')';
 }
 
 class Indexer : public IndexDataConsumer {
 public:
   void initialize(ASTContext &Ctx) override {
     AST = &Ctx;
-    IndexDataConsumer::initialize(Ctx);
   }
 
   bool handleDeclOccurrence(const Decl *D, SymbolRoleSet Roles,
                             ArrayRef<SymbolRelation>, SourceLocation Loc,
                             ASTNodeInfo) override {
     const auto *ND = llvm::dyn_cast<NamedDecl>(D);
     if (!ND)
       return true;