[Index] Add indexing support for MACROs.

[sammccall]

I worry this is a trap: the indexing infrastructure here is designed so you can run it as a frontendaction, on an ASTUnit, or by passing a set of top level decls.
However the macro functionality necessarily only works when running as a frontend action, so the same consumer would have different semantics when using these functions.
Moreover, clangd does call indexTopLevelDecls(), so this API might actually be awkward for us to use.

Alternatives would be:

• write lots of loud documentation and hope people read it
• punt on generalization and just do what we need in clangd with PPCallbacks directly
• offer a peer API here for consuming macros, and have the indexTopLevelDecls() etc only take the symbol consumer, createIndexingAction() would create both (and you could have a createIndexingPPCallbacks() that takes only the macro consumer).

WDYT?

include/clang/Index/IndexDataConsumer.h

52

I know this file isn't heavy on documentation, but I think Undefined needs some explanation/example where it might be true. And consider flipping to remove the negation.

Why a default parameter? Virtual + default is slightly confusing, and the #callers here should be few I think.

include/clang/Index/IndexSymbol.h

141

this should take an arg - maybe macroinfo?
There are potentially subkinds here (function-like macros vs object-like ones) and it'll be harder to update callsites once they exist :-)

[ioeric]

• Addressed review comments. Expose create a PPCallbacks for indexing macros.

[ioeric]

Thanks for the review!

In D48961#1152999, @sammccall wrote:

I worry this is a trap: the indexing infrastructure here is designed so you can run it as a frontendaction, on an ASTUnit, or by passing a set of top level decls.
However the macro functionality necessarily only works when running as a frontend action, so the same consumer would have different semantics when using these functions.
Moreover, clangd does call indexTopLevelDecls(), so this API might actually be awkward for us to use.

Alternatives would be:

• write lots of loud documentation and hope people read it
• punt on generalization and just do what we need in clangd with PPCallbacks directly
• offer a peer API here for consuming macros, and have the indexTopLevelDecls() etc only take the symbol consumer, createIndexingAction() would create both (and you could have a createIndexingPPCallbacks() that takes only the macro consumer).

WDYT?

As chatted offline, the fact that indexTopLevelDecls etc can't index macros is unfortunate, but it seems to be by the current design and probably due to how preprocessor and AST work. As you suggested, I added some comments to make this more explicit and also exposed a function that returns a PPCallbacks for indexing macros, which should be useful for clangd's dynamic index. Hope this would make these functions harder to misuse.

[sammccall]

include/clang/Index/IndexDataConsumer.h

50

Ah, now I understand :-)

This doesn't seem like another orthogonal bool parameter, it seems more like a SymbolRole (we have Definition there, logically Undefinition seems like it should be there too, even if it only applies to some symbols).

It looks like it would be safe to insert this as bit 9, and move all the others down by 1 bit? Only the libclang API needs to be stable, and that ends at bit 8.

include/clang/Index/IndexingAction.h

44

nit: indexes all symbols (macros and AST decls).
i.e. if there were other types in the future, they'd get indexed too.

51

nit: can you break before the "Note" here and below, to make this easier to find?

62

I found this second part hard to understand - I think it's what I'd expect (Preprocessor knows about PPCallbacks, but not the other way around), but I had to look up what "set the preprocessor" meant.

Maybe "The caller is responsible for calling Consumer.setPreprocessor()"?

67

are you sure it doesn't index macros? is this a bug? maybe this should be FIXME instead of a note

lib/Index/IndexSymbol.cpp

357

I'm not sure what the intent of this comment is - what's the behavior you *want* here?
One principled option is to introduce a new SymbolLanguage for the preprocessor (it's reasonable to consider this a separate language).
Another is to always consider macros C symbol, but then this shouldn't be a fixme.

lib/Index/IndexingAction.cpp

49

making these shared_ptr everywhere feels a bit like giving up :-)
Can't we keep this one and IndexPPCallbacks as references?

[ioeric]

• addressed review comments.

[ioeric]

include/clang/Index/IndexDataConsumer.h

50

Sounds good. Done.

include/clang/Index/IndexingAction.h

67

It looks like so from the current implementation:

for (const Decl *D : Reader.getModuleFileLevelDecls(Mod)) {
  IndexCtx.indexTopLevelDecl(D);
}

Changed to FIXME.

lib/Index/IndexSymbol.cpp

357

I'm not sure what the intent of this comment is - what's the behavior you *want* here?

I'm not really sure what I want... On one hand, it's possible to get the actual language from LangOptions of the compiler instance; on the other hand, the index library doesn't seem to care about LangOptions for decls (maybe intentionally?), and I'm a bit hesitant to fix them.

Another is to always consider macros C symbol, but then this shouldn't be a fixme.

Sounds good. This aligns with the current behavior of the library. Removed FIXME.

lib/Index/IndexingAction.cpp

49

It's unclear how exactly IndexingContext should be used :( In the current design, the context is owned by IndexActionBase and referenced/shared by AST consumer and PPCallbacks for the IndexAction use case. The new indexMacroCallbacks function would also need to create a context that needs to be owned by PPCallbacks. Ideally, the ASTConsumer and PPCallbacks can own their own context, but it's unclear whether this would fit into the design of IndexingContext. So shared_ptr seems to be reasonable given the current design.

[ioeric]

• removed Undefined parameter.

[sammccall]

include/clang/Index/IndexDataConsumer.h

48

the change from pointer->reference seems maybe unneccesary, and inconsistent with other callbacks? up to you though, doesn't seem like a big deal either way.

include/clang/Index/IndexSymbol.h

105

maybe mention that macro occurrences aren't currently reported by libclang? as it's related.

lib/Index/IndexingContext.cpp

422

this should be Undefinition (and drop the Undefined flag).

[ioeric]

• addressed review comments.

[marcobubke]

This changset is not handling #ifdef MACRO, defined(MACRO) etc. in he PPCallbacks. Is there any intention to omit them or do you simply forgot them?

include/clang/Index/IndexDataConsumer.h

public:
   virtual void setPreprocessor(std::shared_ptr<Preprocessor> PP) {}
 
   /// \returns true to continue indexing, or false to abort.
   virtual bool handleDeclOccurence(const Decl *D, SymbolRoleSet Roles,
                                    ArrayRef<SymbolRelation> Relations,
                                    SourceLocation Loc, ASTNodeInfo ASTNode);
 
   /// \returns true to continue indexing, or false to abort.
-  virtual bool handleMacroOccurence(const IdentifierInfo *Name,
-                                    const MacroInfo *MI, SymbolRoleSet Roles,
+  virtual bool handleMacroOccurence(const IdentifierInfo &Name,
+                                    const MacroInfo &MI, SymbolRoleSet Roles,
                                     SourceLocation Loc);
 
   /// \returns true to continue indexing, or false to abort.
   virtual bool handleModuleOccurence(const ImportDecl *ImportD,
                                      SymbolRoleSet Roles, SourceLocation Loc);
 
   virtual void finish() {}
 };
 
 } // namespace index
 } // namespace clang
 
 #endif

include/clang/Index/IndexSymbol.h

 //===--- IndexSymbol.h - Types and functions for indexing symbols ---------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_INDEX_INDEXSYMBOL_H
 #define LLVM_CLANG_INDEX_INDEXSYMBOL_H
 
 #include "clang/Basic/LLVM.h"
+#include "clang/Lex/MacroInfo.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/Support/DataTypes.h"
 
 namespace clang {
   class Decl;
   class LangOptions;
 
 namespace index {

 };
 static const unsigned SymbolPropertyBitNum = 8;
 
 /// Set of roles that are attributed to symbol occurrences.
 ///
 /// Low 9 bits of clang-c/include/Index.h CXSymbolRole mirrors this enum.
 enum class SymbolRole : uint32_t {
   Declaration = 1 << 0,
-  Definition  = 1 << 1,
-  Reference   = 1 << 2,
-  Read        = 1 << 3,
-  Write       = 1 << 4,
-  Call        = 1 << 5,
-  Dynamic     = 1 << 6,
-  AddressOf   = 1 << 7,
-  Implicit    = 1 << 8,
+  Definition = 1 << 1,
+  Reference = 1 << 2,
+  Read = 1 << 3,
+  Write = 1 << 4,
+  Call = 1 << 5,
+  Dynamic = 1 << 6,
+  AddressOf = 1 << 7,
+  Implicit = 1 << 8,
+  // FIXME: this is not mirrored in CXSymbolRole.
+  Undefinition = 1 << 9, // macro #undef
 
   // Relation roles.
-  RelationChildOf     = 1 << 9,
-  RelationBaseOf      = 1 << 10,
-  RelationOverrideOf  = 1 << 11,
-  RelationReceivedBy  = 1 << 12,
-  RelationCalledBy    = 1 << 13,
-  RelationExtendedBy  = 1 << 14,
-  RelationAccessorOf  = 1 << 15,
-  RelationContainedBy = 1 << 16,
-  RelationIBTypeOf    = 1 << 17,
-  RelationSpecializationOf = 1 << 18,
+  RelationChildOf = 1 << 10,
+  RelationBaseOf = 1 << 11,
+  RelationOverrideOf = 1 << 12,
+  RelationReceivedBy = 1 << 13,
+  RelationCalledBy = 1 << 14,
+  RelationExtendedBy = 1 << 15,
+  RelationAccessorOf = 1 << 16,
+  RelationContainedBy = 1 << 17,
+  RelationIBTypeOf = 1 << 18,
+  RelationSpecializationOf = 1 << 19,
 };
-static const unsigned SymbolRoleBitNum = 19;
+static const unsigned SymbolRoleBitNum = 20;
 typedef unsigned SymbolRoleSet;
 
 /// Represents a relation to another symbol for a symbol occurrence.
 struct SymbolRelation {
   SymbolRoleSet Roles;
   const Decl *RelatedSymbol;
 
   SymbolRelation(SymbolRoleSet Roles, const Decl *Sym)
     : Roles(Roles), RelatedSymbol(Sym) {}
 };
 
 struct SymbolInfo {
   SymbolKind Kind;
   SymbolSubKind SubKind;
   SymbolLanguage Lang;
   SymbolPropertySet Properties;
 };
 
 SymbolInfo getSymbolInfo(const Decl *D);
 
+SymbolInfo getSymbolInfoForMacro(const MacroInfo &MI);
+
 bool isFunctionLocalSymbol(const Decl *D);
 
 void applyForEachSymbolRole(SymbolRoleSet Roles,
                             llvm::function_ref<void(SymbolRole)> Fn);
 bool applyForEachSymbolRoleInterruptible(SymbolRoleSet Roles,
                             llvm::function_ref<bool(SymbolRole)> Fn);
 void printSymbolRoles(SymbolRoleSet Roles, raw_ostream &OS);
 

include/clang/Index/IndexingAction.h

 //===--- IndexingAction.h - Frontend index action -------------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_INDEX_INDEXINGACTION_H
 #define LLVM_CLANG_INDEX_INDEXINGACTION_H
 
 #include "clang/Basic/LLVM.h"
+#include "clang/Lex/PPCallbacks.h"
 #include "llvm/ADT/ArrayRef.h"
 #include <memory>
 
 namespace clang {
   class ASTContext;
   class ASTReader;
   class ASTUnit;
   class Decl;
enum class SystemSymbolFilterKind {
     All,
   };
 
   SystemSymbolFilterKind SystemSymbolFilter
     = SystemSymbolFilterKind::DeclarationsOnly;
   bool IndexFunctionLocals = false;
 };
 
+/// Creates a frontend action that indexes all symbols (macros and AST decls).
 /// \param WrappedAction another frontend action to wrap over or null.
 std::unique_ptr<FrontendAction>
 createIndexingAction(std::shared_ptr<IndexDataConsumer> DataConsumer,
                      IndexingOptions Opts,
                      std::unique_ptr<FrontendAction> WrappedAction);
 
+/// Recursively indexes all decls in the AST.
+/// Note that this does not index macros.
 void indexASTUnit(ASTUnit &Unit, IndexDataConsumer &DataConsumer,
                   IndexingOptions Opts);
 
+/// Recursively indexes \p Decls.
+/// Note that this does not index macros.
 void indexTopLevelDecls(ASTContext &Ctx, ArrayRef<const Decl *> Decls,
                         IndexDataConsumer &DataConsumer, IndexingOptions Opts);
 
+/// Creates a PPCallbacks that indexes macros and feeds macros to \p Consumer.
+/// The caller is responsible for calling `Consumer.setPreprocessor()`.
+std::unique_ptr<PPCallbacks> indexMacrosCallback(IndexDataConsumer &Consumer,
+                                                 IndexingOptions Opts);
+
+/// Recursively indexes all top-level decls in the module.
+/// FIXME: make this index macros as well.
 void indexModuleFile(serialization::ModuleFile &Mod, ASTReader &Reader,
                      IndexDataConsumer &DataConsumer, IndexingOptions Opts);
 
 } // namespace index
 } // namespace clang
 
 #endif

lib/Index/IndexSymbol.cpp

 //===--- IndexSymbol.cpp - Types and functions for indexing symbols -------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "clang/Index/IndexSymbol.h"
 #include "clang/AST/DeclCXX.h"
 #include "clang/AST/DeclObjC.h"
 #include "clang/AST/DeclTemplate.h"
 #include "clang/AST/PrettyPrinter.h"
+#include "clang/Lex/MacroInfo.h"
 
 using namespace clang;
 using namespace clang::index;
 
 /// \returns true if \c D is a subclass of 'XCTestCase'.
 static bool isUnitTestCase(const ObjCInterfaceDecl *D) {
   if (!D)
     return false;
SymbolInfo index::getSymbolInfo(const Decl *D) {
   if (auto *attr = D->getExternalSourceSymbolAttr()) {
     if (attr->getLanguage() == "Swift")
       Info.Lang = SymbolLanguage::Swift;
   }
 
   return Info;
 }
 
+SymbolInfo index::getSymbolInfoForMacro(const MacroInfo &) {
+  SymbolInfo Info;
+  Info.Kind = SymbolKind::Macro;
+  Info.SubKind = SymbolSubKind::None;
+  Info.Properties = SymbolPropertySet();
+  Info.Lang = SymbolLanguage::C;
+  return Info;
+}
+
 bool index::applyForEachSymbolRoleInterruptible(SymbolRoleSet Roles,
                                    llvm::function_ref<bool(SymbolRole)> Fn) {
 #define APPLY_FOR_ROLE(Role) \
   if (Roles & (unsigned)SymbolRole::Role) \
     if (!Fn(SymbolRole::Role)) \
       return false;
 
   APPLY_FOR_ROLE(Declaration);
   APPLY_FOR_ROLE(Definition);
   APPLY_FOR_ROLE(Reference);
   APPLY_FOR_ROLE(Read);
   APPLY_FOR_ROLE(Write);
   APPLY_FOR_ROLE(Call);
   APPLY_FOR_ROLE(Dynamic);
   APPLY_FOR_ROLE(AddressOf);
   APPLY_FOR_ROLE(Implicit);
+  APPLY_FOR_ROLE(Undefinition);
   APPLY_FOR_ROLE(RelationChildOf);
   APPLY_FOR_ROLE(RelationBaseOf);
   APPLY_FOR_ROLE(RelationOverrideOf);
   APPLY_FOR_ROLE(RelationReceivedBy);
   APPLY_FOR_ROLE(RelationCalledBy);
   APPLY_FOR_ROLE(RelationExtendedBy);
   APPLY_FOR_ROLE(RelationAccessorOf);
   APPLY_FOR_ROLE(RelationContainedBy);
applyForEachSymbolRole(Roles, [&](SymbolRole Role) {
     case SymbolRole::Definition: OS << "Def"; break;
     case SymbolRole::Reference: OS << "Ref"; break;
     case SymbolRole::Read: OS << "Read"; break;
     case SymbolRole::Write: OS << "Writ"; break;
     case SymbolRole::Call: OS << "Call"; break;
     case SymbolRole::Dynamic: OS << "Dyn"; break;
     case SymbolRole::AddressOf: OS << "Addr"; break;
     case SymbolRole::Implicit: OS << "Impl"; break;
+    case SymbolRole::Undefinition: OS << "Undef"; break;
     case SymbolRole::RelationChildOf: OS << "RelChild"; break;
     case SymbolRole::RelationBaseOf: OS << "RelBase"; break;
     case SymbolRole::RelationOverrideOf: OS << "RelOver"; break;
     case SymbolRole::RelationReceivedBy: OS << "RelRec"; break;
     case SymbolRole::RelationCalledBy: OS << "RelCall"; break;
     case SymbolRole::RelationExtendedBy: OS << "RelExt"; break;
     case SymbolRole::RelationAccessorOf: OS << "RelAcc"; break;
     case SymbolRole::RelationContainedBy: OS << "RelCont"; break;

lib/Index/IndexingAction.cpp

 //===- IndexingAction.cpp - Frontend index action -------------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "clang/Index/IndexingAction.h"
 #include "IndexingContext.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/FrontendAction.h"
 #include "clang/Frontend/MultiplexConsumer.h"
 #include "clang/Index/IndexDataConsumer.h"
+#include "clang/Lex/PPCallbacks.h"
 #include "clang/Lex/Preprocessor.h"
 #include "clang/Serialization/ASTReader.h"
+#include "llvm/ADT/STLExtras.h"
+#include <memory>
 
 using namespace clang;
 using namespace clang::index;
 
 bool IndexDataConsumer::handleDeclOccurence(const Decl *D, SymbolRoleSet Roles,
                                             ArrayRef<SymbolRelation> Relations,
                                             SourceLocation Loc,
                                             ASTNodeInfo ASTNode) {
   return true;
 }
 
-bool IndexDataConsumer::handleMacroOccurence(const IdentifierInfo *Name,
-                                             const MacroInfo *MI,
+bool IndexDataConsumer::handleMacroOccurence(const IdentifierInfo &Name,
+                                             const MacroInfo &MI,
                                              SymbolRoleSet Roles,
                                              SourceLocation Loc) {
   return true;
 }
 
 bool IndexDataConsumer::handleModuleOccurence(const ImportDecl *ImportD,
                                               SymbolRoleSet Roles,
                                               SourceLocation Loc) {
   return true;
 }
 
 namespace {
 
 class IndexASTConsumer : public ASTConsumer {
   std::shared_ptr<Preprocessor> PP;
-  IndexingContext &IndexCtx;
+  std::shared_ptr<IndexingContext> IndexCtx;
 
 public:
-  IndexASTConsumer(std::shared_ptr<Preprocessor> PP, IndexingContext &IndexCtx)
-      : PP(std::move(PP)), IndexCtx(IndexCtx) {}
+  IndexASTConsumer(std::shared_ptr<Preprocessor> PP,
+                   std::shared_ptr<IndexingContext> IndexCtx)
+      : PP(std::move(PP)), IndexCtx(std::move(IndexCtx)) {}
 
 protected:
   void Initialize(ASTContext &Context) override {
-    IndexCtx.setASTContext(Context);
-    IndexCtx.getDataConsumer().initialize(Context);
-    IndexCtx.getDataConsumer().setPreprocessor(PP);
+    IndexCtx->setASTContext(Context);
+    IndexCtx->getDataConsumer().initialize(Context);
+    IndexCtx->getDataConsumer().setPreprocessor(PP);
   }
 
   bool HandleTopLevelDecl(DeclGroupRef DG) override {
-    return IndexCtx.indexDeclGroupRef(DG);
+    return IndexCtx->indexDeclGroupRef(DG);
   }
 
   void HandleInterestingDecl(DeclGroupRef DG) override {
     // Ignore deserialized decls.
   }
 
   void HandleTopLevelDeclInObjCContainer(DeclGroupRef DG) override {
-    IndexCtx.indexDeclGroupRef(DG);
+    IndexCtx->indexDeclGroupRef(DG);
   }
 
   void HandleTranslationUnit(ASTContext &Ctx) override {
   }
 };
 
+class IndexPPCallbacks : public PPCallbacks {
+  std::shared_ptr<IndexingContext> IndexCtx;
+
+public:
+  IndexPPCallbacks(std::shared_ptr<IndexingContext> IndexCtx)
+      : IndexCtx(std::move(IndexCtx)) {}
+
+  void MacroExpands(const Token &MacroNameTok, const MacroDefinition &MD,
+                    SourceRange Range, const MacroArgs *Args) override {
+    IndexCtx->handleMacroReference(*MacroNameTok.getIdentifierInfo(),
+                                   Range.getBegin(), *MD.getMacroInfo());
+  }
+
+  void MacroDefined(const Token &MacroNameTok,
+                    const MacroDirective *MD) override {
+    IndexCtx->handleMacroDefined(*MacroNameTok.getIdentifierInfo(),
+                                 MacroNameTok.getLocation(),
+                                 *MD->getMacroInfo());
+  }
+
+  void MacroUndefined(const Token &MacroNameTok, const MacroDefinition &MD,
+                      const MacroDirective *Undef) override {
+    IndexCtx->handleMacroUndefined(*MacroNameTok.getIdentifierInfo(),
+                                   MacroNameTok.getLocation(),
+                                   *MD.getMacroInfo());
+  }
+};
+
 class IndexActionBase {
 protected:
   std::shared_ptr<IndexDataConsumer> DataConsumer;
-  IndexingContext IndexCtx;
+  std::shared_ptr<IndexingContext> IndexCtx;
 
   IndexActionBase(std::shared_ptr<IndexDataConsumer> dataConsumer,
                   IndexingOptions Opts)
       : DataConsumer(std::move(dataConsumer)),
-      IndexCtx(Opts, *DataConsumer) {}
+        IndexCtx(new IndexingContext(Opts, *DataConsumer)) {}
 
   std::unique_ptr<IndexASTConsumer>
   createIndexASTConsumer(CompilerInstance &CI) {
     return llvm::make_unique<IndexASTConsumer>(CI.getPreprocessorPtr(),
                                                IndexCtx);
   }
 
+  std::unique_ptr<PPCallbacks> createIndexPPCallbacks() {
+    return llvm::make_unique<IndexPPCallbacks>(IndexCtx);
+  }
+
   void finish() {
     DataConsumer->finish();
   }
 };
 
 class IndexAction : public ASTFrontendAction, IndexActionBase {
 public:
   IndexAction(std::shared_ptr<IndexDataConsumer> DataConsumer,
               IndexingOptions Opts)
     : IndexActionBase(std::move(DataConsumer), Opts) {}
 
 protected:
   std::unique_ptr<ASTConsumer> CreateASTConsumer(CompilerInstance &CI,
                                                  StringRef InFile) override {
     return createIndexASTConsumer(CI);
   }
 
+  bool BeginSourceFileAction(clang::CompilerInstance &CI) override {
+    CI.getPreprocessor().addPPCallbacks(createIndexPPCallbacks());
+    return true;
+  }
+
   void EndSourceFileAction() override {
     FrontendAction::EndSourceFileAction();
     finish();
   }
 };
 
 class WrappingIndexAction : public WrapperFrontendAction, IndexActionBase {
   bool IndexActionFailed = false;
 
 public:
   WrappingIndexAction(std::unique_ptr<FrontendAction> WrappedAction,
                       std::shared_ptr<IndexDataConsumer> DataConsumer,
                       IndexingOptions Opts)
     : WrapperFrontendAction(std::move(WrappedAction)),
       IndexActionBase(std::move(DataConsumer), Opts) {}
 
 protected:
   std::unique_ptr<ASTConsumer> CreateASTConsumer(CompilerInstance &CI,
-                                                 StringRef InFile) override;
-  void EndSourceFileAction() override;
-};
-
-} // anonymous namespace
-
-void WrappingIndexAction::EndSourceFileAction() {
-  // Invoke wrapped action's method.
-  WrapperFrontendAction::EndSourceFileAction();
-  if (!IndexActionFailed)
-    finish();
-}
-
-std::unique_ptr<ASTConsumer>
-WrappingIndexAction::CreateASTConsumer(CompilerInstance &CI, StringRef InFile) {
+                                                 StringRef InFile) override {
     auto OtherConsumer = WrapperFrontendAction::CreateASTConsumer(CI, InFile);
     if (!OtherConsumer) {
       IndexActionFailed = true;
       return nullptr;
     }
 
     std::vector<std::unique_ptr<ASTConsumer>> Consumers;
     Consumers.push_back(std::move(OtherConsumer));
     Consumers.push_back(createIndexASTConsumer(CI));
     return llvm::make_unique<MultiplexConsumer>(std::move(Consumers));
   }
 
+  bool BeginSourceFileAction(clang::CompilerInstance &CI) override {
+    WrapperFrontendAction::BeginSourceFileAction(CI);
+    CI.getPreprocessor().addPPCallbacks(createIndexPPCallbacks());
+    return true;
+  }
+
+  void EndSourceFileAction() override {
+    // Invoke wrapped action's method.
+    WrapperFrontendAction::EndSourceFileAction();
+    if (!IndexActionFailed)
+      finish();
+  }
+};
+
+} // anonymous namespace
+
 std::unique_ptr<FrontendAction>
 index::createIndexingAction(std::shared_ptr<IndexDataConsumer> DataConsumer,
                             IndexingOptions Opts,
                             std::unique_ptr<FrontendAction> WrappedAction) {
   if (WrappedAction)
     return llvm::make_unique<WrappingIndexAction>(std::move(WrappedAction),
                                                   std::move(DataConsumer),
                                                   Opts);
   return llvm::make_unique<IndexAction>(std::move(DataConsumer), Opts);
 }
 
-
 static bool topLevelDeclVisitor(void *context, const Decl *D) {
   IndexingContext &IndexCtx = *static_cast<IndexingContext*>(context);
   return IndexCtx.indexTopLevelDecl(D);
 }
 
 static void indexTranslationUnit(ASTUnit &Unit, IndexingContext &IndexCtx) {
   Unit.visitLocalTopLevelDecls(&IndexCtx, topLevelDeclVisitor);
 }
void index::indexTopLevelDecls(ASTContext &Ctx, ArrayRef<const Decl *> Decls,
   IndexCtx.setASTContext(Ctx);
 
   DataConsumer.initialize(Ctx);
   for (const Decl *D : Decls)
     IndexCtx.indexTopLevelDecl(D);
   DataConsumer.finish();
 }
 
+std::unique_ptr<PPCallbacks>
+index::indexMacrosCallback(IndexDataConsumer &Consumer, IndexingOptions Opts) {
+  return llvm::make_unique<IndexPPCallbacks>(
+      std::make_shared<IndexingContext>(Opts, Consumer));
+}
+
 void index::indexModuleFile(serialization::ModuleFile &Mod, ASTReader &Reader,
                             IndexDataConsumer &DataConsumer,
                             IndexingOptions Opts) {
   ASTContext &Ctx = Reader.getContext();
   IndexingContext IndexCtx(Opts, DataConsumer);
   IndexCtx.setASTContext(Ctx);
   DataConsumer.initialize(Ctx);
 
   for (const Decl *D : Reader.getModuleFileLevelDecls(Mod)) {
     IndexCtx.indexTopLevelDecl(D);
   }
   DataConsumer.finish();
 }

lib/Index/IndexingContext.h

 //===- IndexingContext.h - Indexing context data ----------------*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_LIB_INDEX_INDEXINGCONTEXT_H
 #define LLVM_CLANG_LIB_INDEX_INDEXINGCONTEXT_H
 
+#include "clang/Basic/IdentifierTable.h"
 #include "clang/Basic/LLVM.h"
 #include "clang/Index/IndexSymbol.h"
 #include "clang/Index/IndexingAction.h"
+#include "clang/Lex/MacroInfo.h"
 #include "llvm/ADT/ArrayRef.h"
 
 namespace clang {
   class ASTContext;
   class Decl;
   class DeclGroupRef;
   class ImportDecl;
   class TagDecl;
public:
   bool handleReference(const NamedDecl *D, SourceLocation Loc,
                        const NamedDecl *Parent,
                        const DeclContext *DC,
                        SymbolRoleSet Roles = SymbolRoleSet(),
                        ArrayRef<SymbolRelation> Relations = None,
                        const Expr *RefE = nullptr,
                        const Decl *RefD = nullptr);
 
+  void handleMacroDefined(const IdentifierInfo &Name, SourceLocation Loc,
+                          const MacroInfo &MI);
+
+  void handleMacroUndefined(const IdentifierInfo &Name, SourceLocation Loc,
+                            const MacroInfo &MI);
+
+  void handleMacroReference(const IdentifierInfo &Name, SourceLocation Loc,
+                            const MacroInfo &MD);
+
   bool importedModule(const ImportDecl *ImportD);
 
   bool indexDecl(const Decl *D);
 
   void indexTagDecl(const TagDecl *D,
                     ArrayRef<SymbolRelation> Relations = None);
 
   void indexTypeSourceInfo(TypeSourceInfo *TInfo, const NamedDecl *Parent,

lib/Index/IndexingContext.cpp

 //===- IndexingContext.cpp - Indexing context data ------------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "IndexingContext.h"
+#include "clang/Basic/SourceLocation.h"
 #include "clang/Index/IndexDataConsumer.h"
 #include "clang/AST/ASTContext.h"
 #include "clang/AST/DeclTemplate.h"
 #include "clang/AST/DeclObjC.h"
 #include "clang/Basic/SourceManager.h"
 
 using namespace clang;
 using namespace index;
applyForEachSymbolRoleInterruptible(roles, [&accept](SymbolRole r) -> bool {
       case SymbolRole::Definition:
       case SymbolRole::Reference:
       case SymbolRole::Read:
       case SymbolRole::Write:
       case SymbolRole::Call:
       case SymbolRole::Dynamic:
       case SymbolRole::AddressOf:
       case SymbolRole::Implicit:
+      case SymbolRole::Undefinition:
       case SymbolRole::RelationReceivedBy:
       case SymbolRole::RelationCalledBy:
       case SymbolRole::RelationContainedBy:
       case SymbolRole::RelationSpecializationOf:
         return true;
       }
       llvm_unreachable("Unsupported SymbolRole value!");
     });
bool IndexingContext::handleDeclOccurrence(const Decl *D, SourceLocation Loc,
   for (auto &Rel : Relations) {
     addRelation(SymbolRelation(Rel.Roles,
                                Rel.RelatedSymbol->getCanonicalDecl()));
   }
 
   IndexDataConsumer::ASTNodeInfo Node{OrigE, OrigD, Parent, ContainerDC};
   return DataConsumer.handleDeclOccurence(D, Roles, FinalRelations, Loc, Node);
 }
+
+void IndexingContext::handleMacroDefined(const IdentifierInfo &Name,
+                                         SourceLocation Loc,
+                                         const MacroInfo &MI) {
+  SymbolRoleSet Roles = (unsigned)SymbolRole::Definition;
+  DataConsumer.handleMacroOccurence(Name, MI, Roles, Loc);
+}
+
+void IndexingContext::handleMacroUndefined(const IdentifierInfo &Name,
+                                           SourceLocation Loc,
+                                           const MacroInfo &MI) {
+  SymbolRoleSet Roles = (unsigned)SymbolRole::Undefinition;
+  DataConsumer.handleMacroOccurence(Name, MI, Roles, Loc);
+}
+
+void IndexingContext::handleMacroReference(const IdentifierInfo &Name,
+                                           SourceLocation Loc,
+                                           const MacroInfo &MI) {
+  SymbolRoleSet Roles = (unsigned)SymbolRole::Reference;
+  DataConsumer.handleMacroOccurence(Name, MI, Roles, Loc);
+}

test/Index/Core/index-macros.c

+// RUN: c-index-test core -print-source-symbols -- %s | FileCheck %s
+
+// CHECK: [[@LINE+1]]:9 | macro/C | X1 | c:index-macros.c@157@macro@X1 | Def |
+#define X1 1
+// CHECK: [[@LINE+1]]:9 | macro/C | DEF | c:index-macros.c@251@macro@DEF | Def |
+#define DEF(x) int x
+// CHECK: [[@LINE+1]]:8 | macro/C | X1 | c:index-macros.c@157@macro@X1 | Undef |
+#undef X1
+
+// CHECK: [[@LINE+2]]:1 | macro/C | DEF | c:index-macros.c@251@macro@DEF | Ref |
+// CHECK: [[@LINE+1]]:5 | variable/C | i | c:@i | i | Def | rel: 0
+DEF(i);

tools/c-index-test/core_main.cpp

 //===-- core_main.cpp - Core Index Tool testbed ---------------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
+#include "clang/Basic/LangOptions.h"
 #include "clang/CodeGen/ObjectFilePCHContainerOperations.h"
 #include "clang/Frontend/ASTUnit.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/CompilerInvocation.h"
 #include "clang/Frontend/FrontendAction.h"
 #include "clang/Index/IndexingAction.h"
 #include "clang/Index/IndexDataConsumer.h"
 #include "clang/Index/USRGeneration.h"
 #include "clang/Index/CodegenNameGenerator.h"
+#include "clang/Lex/Preprocessor.h"
 #include "clang/Serialization/ASTReader.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Signals.h"
 #include "llvm/Support/raw_ostream.h"
 #include "llvm/Support/PrettyStackTrace.h"
 
 using namespace clang;
 using namespace clang::index;

 static void printSymbolNameAndUSR(const Decl *D, ASTContext &Ctx,
                                   raw_ostream &OS);
 
 namespace {
 
 class PrintIndexDataConsumer : public IndexDataConsumer {
   raw_ostream &OS;
   std::unique_ptr<CodegenNameGenerator> CGNameGen;
+  std::shared_ptr<Preprocessor> PP;
 
 public:
   PrintIndexDataConsumer(raw_ostream &OS) : OS(OS) {
   }
 
   void initialize(ASTContext &Ctx) override {
     CGNameGen.reset(new CodegenNameGenerator(Ctx));
   }
 
+  void setPreprocessor(std::shared_ptr<Preprocessor> PP) override {
+    this->PP = std::move(PP);
+  }
+
   bool handleDeclOccurence(const Decl *D, SymbolRoleSet Roles,
                            ArrayRef<SymbolRelation> Relations,
                            SourceLocation Loc, ASTNodeInfo ASTNode) override {
     ASTContext &Ctx = D->getASTContext();
     SourceManager &SM = Ctx.getSourceManager();
 
     Loc = SM.getFileLoc(Loc);
     FileID FID = SM.getFileID(Loc);
bool handleModuleOccurence(const ImportDecl *ImportD, SymbolRoleSet Roles,
 
     OS << ImportD->getImportedModule()->getFullModuleName() << " | ";
 
     printSymbolRoles(Roles, OS);
     OS << " |\n";
 
     return true;
   }
+
+  bool handleMacroOccurence(const IdentifierInfo &Name, const MacroInfo &MI,
+                            SymbolRoleSet Roles, SourceLocation Loc) override {
+    assert(PP);
+    SourceManager &SM = PP->getSourceManager();
+
+    Loc = SM.getFileLoc(Loc);
+    FileID FID = SM.getFileID(Loc);
+    unsigned Line = SM.getLineNumber(FID, SM.getFileOffset(Loc));
+    unsigned Col = SM.getColumnNumber(FID, SM.getFileOffset(Loc));
+    OS << Line << ':' << Col << " | ";
+
+    printSymbolInfo(getSymbolInfoForMacro(MI), OS);
+    OS << " | ";
+
+    OS << Name.getName();
+    OS << " | ";
+
+    SmallString<256> USRBuf;
+    if (generateUSRForMacro(Name.getName(), MI.getDefinitionLoc(), SM,
+                            USRBuf)) {
+      OS << "<no-usr>";
+    } else {
+      OS << USRBuf;
+    }
+    OS << " | ";
+
+    printSymbolRoles(Roles, OS);
+    OS << " |\n";
+    return true;
+  }
 };
 
 } // anonymous namespace
 
 //===----------------------------------------------------------------------===//
 // Print Source Symbols
 //===----------------------------------------------------------------------===//