[clang][extract-api] Add support for macros

[dang]

Summary

To achieve this we hook into the preprocessor during the
ExtractAPIAction and record definitions for macros that don't get
undefined during preprocessing.

[zixuw]

Still going through the declaration fragments builder and ExtractAPIVisitor/Consumer/Action changes, but it seems that the patch needs a rebase onto the latest main right now as it's missing several already landed changes.

clang/include/clang/ExtractAPI/API.h

117

Now that we are re-ordering it, would it look more organized if we push GVKind Kind further down so that the two StringRefs could be adjacent, and we also keep a common list of arguments (Name, USR, Loc, Availability, ..., SubHeading) the same across all kinds of records upfront? So maybe push to the end after SubHeading and before Signature.

117

Also it seems that it becomes a requirement for the *Record c'tors that Name should be the first parameter in order to use the addTopLevelRecord template, though that's hidden in an anonymous namespace in the implementation file.
How can we communicate or document this clearly for future extensions?

195

Don't we also need sub-headings for macros?

201

Need out-of-line virtual anchor as this struct inherits a virtual table.

[zixuw]

clang/include/clang/ExtractAPI/Serialization/SymbolGraphSerializer.h

22

Why do we need this include?

clang/lib/ExtractAPI/API.cpp

32

Does it make sense to just directly make the *RecordMap types in APISet as a template using?

clang/lib/ExtractAPI/ExtractAPIConsumer.cpp

446–457

Worth moving this change and the APISet move and relevant structural changes into its own patch, as mentioned in D122446.

clang/lib/ExtractAPI/Serialization/SymbolGraphSerializer.cpp

17

Why do we need this include?

[dang]

clang/include/clang/ExtractAPI/API.h

117

Good catch! Correct, it isn't really a requirement in the sense that we could pass it in twice but since this was the only place that needed the change I felt it would improve the ergonomics of addTopLevelRecord to make the change here.

The way I see it it depends on whether we want to make use of addTopLevelRecord to associate stuff in the relevant map using a key that isn't the name of the record. If we don't want to do that, it might make sense to make the maps sets, which would convey the intent a bit better IMHO, but that would be a follow up patch.

Alternatively if we want to keep the structure as is, the easiest thing to do is to put a comment on top of the APIRecord base class that says that derived classes need to put the Name parameter as the first thing.

clang/lib/ExtractAPI/API.cpp

32

Yup agreed!

[dang]

clang/lib/ExtractAPI/ExtractAPIConsumer.cpp

446–457

https://reviews.llvm.org/D122648

[dang]

Address review feedback.

[zixuw]

clang/include/clang/ExtractAPI/API.h

325

Extra/should-be-empty line here?

clang/lib/ExtractAPI/API.cpp

19

Do we need to include declaration fragments from API.cpp?

32

Now can we use APISet::RecordMap<RecordTy> as the type of the first param here?
Can also get rid of including MapVector.h

[dang]

Update with code review feedback and the rebased code.

[zixuw]

LGTM!

[dyung]

This change is causing 9 test failures on a buildbot, can you please take a look and revert if you need time to investigate?

https://lab.llvm.org/buildbot/#/builders/139/builds/19441

[thakis]

Looks like either this or your other commit at the same time broke tests on windows: http://45.33.8.238/win/55345/step_7.txt

Please take a look, and revert for now if it takes a while to fix.

(Also, please add a "Differential Revision: https://reviews.llvm.org/D122611" line to the bottom of your commit, to make it easy to find the review from the commit message.)

[dang]

I have what should be a NFC fix that should fix the issue, just checking everything still works and will commit it.

[dang]

Committed a revert of the changes in that particular function in commit 985eaa1a3da2a1b88ea70a65ffd5783aa82ea65e

[dang]

@thakis I am UK based so I am logging off as it's 9pm over here. The fix I committed has fixed the issue on other buildbots that were failing due to this. If it still fails on Windows, would you be able to revert the change for me?

[thakis]

My windows bot cycled green, thanks!

[dang]

529a0570f7e8c5144bd3ad057e43f00e3af58d1b

clang/include/clang/ExtractAPI/API.h

struct APIRecord {
 
   /// Discriminator for LLVM-style RTTI (dyn_cast<> et al.)
   enum RecordKind {
     RK_Global,
     RK_EnumConstant,
     RK_Enum,
     RK_StructField,
     RK_Struct,
+    RK_MacroDefinition,
   };
 
 private:
   const RecordKind Kind;
 
 public:
   RecordKind getKind() const { return Kind; }
 

 
 /// This holds information associated with global variables or functions.
 struct GlobalRecord : APIRecord {
   GVKind GlobalKind;
 
   /// The function signature of the record if it is a function.
   FunctionSignature Signature;
 
-  GlobalRecord(GVKind Kind, StringRef Name, StringRef USR, PresumedLoc Loc,
+  GlobalRecord(StringRef Name, GVKind Kind, StringRef USR, PresumedLoc Loc,
                const AvailabilityInfo &Availability, LinkageInfo Linkage,
                const DocComment &Comment, DeclarationFragments Declaration,
                DeclarationFragments SubHeading, FunctionSignature Signature)
       : APIRecord(RK_Global, Name, USR, Loc, Availability, Linkage, Comment,
                   Declaration, SubHeading),
         GlobalKind(Kind), Signature(Signature) {}
 
   static bool classof(const APIRecord *Record) {
StructRecord(StringRef Name, StringRef USR, PresumedLoc Loc,
       : APIRecord(RK_Struct, Name, USR, Loc, Availability, LinkageInfo::none(),
                   Comment, Declaration, SubHeading) {}
 
   static bool classof(const APIRecord *Record) {
     return Record->getKind() == RK_Struct;
   }
 };
 
+struct MacroDefinitionRecord : APIRecord {
+  MacroDefinitionRecord(StringRef Name, StringRef USR, PresumedLoc Loc,
+                        DeclarationFragments DeclarationFragments)
+      : APIRecord(RK_MacroDefinition, Name, USR, Loc, AvailabilityInfo(),
+                  LinkageInfo(), {}, DeclarationFragments, {}) {}
+
+  static bool classof(const APIRecord *Record) {
+    return Record->getKind() == RK_MacroDefinition;
+  }
+};
+
 /// APISet holds the set of API records collected from given inputs.
 class APISet {
 public:
   /// Create and add a GlobalRecord of kind \p Kind into the API set.
   ///
   /// Note: the caller is responsible for keeping the StringRef \p Name and
   /// \p USR alive. APISet::copyString provides a way to copy strings into
   /// APISet itself, and APISet::recordUSR(const Decl *D) is a helper method
public:
   /// to generate the USR for \c D and keep it alive in APISet.
   StructFieldRecord *addStructField(StructRecord *Struct, StringRef Name,
                                     StringRef USR, PresumedLoc Loc,
                                     const AvailabilityInfo &Availability,
                                     const DocComment &Comment,
                                     DeclarationFragments Declaration,
                                     DeclarationFragments SubHeading);
 
+  /// Create a macro definition record into the API set.
+  ///
+  /// Note: the caller is responsible for keeping the StringRef \p Name and
+  /// \p USR alive. APISet::copyString provides a way to copy strings into
+  /// APISet itself, and APISet::recordUSRForMacro(StringRef Name,
+  /// SourceLocation SL, const SourceManager &SM) is a helper method to generate
+  /// the USR for the macro and keep it alive in APISet.
+  MacroDefinitionRecord *
+  addMacroDefinition(StringRef Name, StringRef USR, PresumedLoc Loc,
+                     DeclarationFragments DeclarationFragments);
+
   /// Create and add a struct record into the API set.
   ///
   /// Note: the caller is responsible for keeping the StringRef \p Name and
   /// \p USR alive. APISet::copyString provides a way to copy strings into
   /// APISet itself, and APISet::recordUSR(const Decl *D) is a helper method
   /// to generate the USR for \c D and keep it alive in APISet.
   StructRecord *addStruct(StringRef Name, StringRef USR, PresumedLoc Loc,
                           const AvailabilityInfo &Availability,
public:
   /// key.
   using EnumRecordMap = llvm::MapVector<StringRef, std::unique_ptr<EnumRecord>>;
 
   /// A map to store the set of StructRecord%s with the declaration name as the
   /// key.
   using StructRecordMap =
       llvm::MapVector<StringRef, std::unique_ptr<StructRecord>>;
 
+  /// A map to store the set of MacroDefinitionRecord%s with the macro name as
+  /// the key.
+  using MacroDefinitionRecordMap =
+      llvm::MapVector<StringRef, std::unique_ptr<MacroDefinitionRecord>>;
+
   /// Get the target triple for the ExtractAPI invocation.
   const llvm::Triple &getTarget() const { return Target; }
 
   /// Get the language options used to parse the APIs.
   const LangOptions &getLangOpts() const { return LangOpts; }
 
   const GlobalRecordMap &getGlobals() const { return Globals; }
   const EnumRecordMap &getEnums() const { return Enums; }
   const StructRecordMap &getStructs() const { return Structs; }
+  const MacroDefinitionRecordMap &getMacros() const { return Macros; }
 
   /// Generate and store the USR of declaration \p D.
   ///
   /// Note: The USR string is stored in and owned by Allocator.
   ///
   /// \returns a StringRef of the generated USR string.
   StringRef recordUSR(const Decl *D);
 
+  /// Generate and store the USR for a macro \p Name.
+  ///
+  /// Note: The USR string is stored in and owned by Allocator.
+  ///
+  /// \returns a StringRef to the generate USR string.
+  StringRef recordUSRForMacro(StringRef Name, SourceLocation SL,
+                              const SourceManager &SM);
+
   /// Copy \p String into the Allocator in this APISet.
   ///
   /// \returns a StringRef of the copied string in APISet::Allocator.
   StringRef copyString(StringRef String);
 
   APISet(const llvm::Triple &Target, const LangOptions &LangOpts)
       : Target(Target), LangOpts(LangOpts) {}
 
 private:
   /// BumpPtrAllocator to store generated/copied strings.
   ///
   /// Note: The main use for this is being able to deduplicate strings.
   llvm::BumpPtrAllocator StringAllocator;
 
   const llvm::Triple Target;
   const LangOptions LangOpts;
 
   GlobalRecordMap Globals;
   EnumRecordMap Enums;
   StructRecordMap Structs;
+  MacroDefinitionRecordMap Macros;
 };
 
 } // namespace extractapi
 } // namespace clang
 
 #endif // LLVM_CLANG_EXTRACTAPI_API_H

clang/include/clang/ExtractAPI/DeclarationFragments.h

 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_EXTRACTAPI_DECLARATION_FRAGMENTS_H
 #define LLVM_CLANG_EXTRACTAPI_DECLARATION_FRAGMENTS_H
 
 #include "clang/AST/ASTContext.h"
 #include "clang/AST/Decl.h"
 #include "clang/AST/DeclCXX.h"
+#include "clang/Lex/MacroInfo.h"
 #include "llvm/ADT/StringRef.h"
 #include <vector>
 
 namespace clang {
 namespace extractapi {
 
 /// DeclarationFragments is a vector of tagged important parts of a symbol's
 /// declaration.
public:
   static DeclarationFragments getFragmentsForEnum(const EnumDecl *);
 
   /// Build DeclarationFragments for a field declaration FieldDecl.
   static DeclarationFragments getFragmentsForField(const FieldDecl *);
 
   /// Build DeclarationFragments for a struct record declaration RecordDecl.
   static DeclarationFragments getFragmentsForStruct(const RecordDecl *);
 
+  /// Build DeclarationFragments for a macro.
+  ///
+  /// \param Name name of the macro.
+  /// \param MD the associated MacroDirective.
+  static DeclarationFragments getFragmentsForMacro(StringRef Name,
+                                                   const MacroDirective *MD);
+
   /// Build sub-heading fragments for a NamedDecl.
   static DeclarationFragments getSubHeading(const NamedDecl *);
 
   /// Build FunctionSignature for a function declaration FunctionDecl.
   static FunctionSignature getFunctionSignature(const FunctionDecl *);
 
 private:
   DeclarationFragmentsBuilder() = delete;

clang/include/clang/ExtractAPI/FrontendActions.h

 //===- ExtractAPI/FrontendActions.h -----------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 ///
 /// \file
 /// This file defines the ExtractAPIAction frontend action.
 ///
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_EXTRACTAPI_FRONTEND_ACTIONS_H
 #define LLVM_CLANG_EXTRACTAPI_FRONTEND_ACTIONS_H
 
+#include "clang/ExtractAPI/API.h"
 #include "clang/Frontend/FrontendAction.h"
 
 namespace clang {
 
 /// ExtractAPIAction sets up the output file and creates the ExtractAPIVisitor.
 class ExtractAPIAction : public ASTFrontendAction {
 protected:
   std::unique_ptr<ASTConsumer> CreateASTConsumer(CompilerInstance &CI,
                                                  StringRef InFile) override;
 
 private:
-  /// The synthesized input buffer that contains all the provided input header
-  /// files.
-  std::unique_ptr<llvm::MemoryBuffer> Buffer;
+  /// A representation of the APIs this action extracts.
+  std::unique_ptr<extractapi::APISet> API;
+
+  /// A stream to the output file of this action.
+  std::unique_ptr<raw_pwrite_stream> OS;
+
+  /// The product this action is extracting API information for.
+  std::string ProductName;
 
-public:
   /// Prepare to execute the action on the given CompilerInstance.
   ///
   /// This is called before executing the action on any inputs. This generates a
   /// single header that includes all of CI's inputs and replaces CI's input
   /// list with it before actually executing the action.
   bool PrepareToExecuteAction(CompilerInstance &CI) override;
 
+  /// Called after executing the action on the synthesized input buffer.
+  ///
+  /// Note: Now that we have gathered all the API definitions to surface we can
+  /// emit them in this callback.
+  void EndSourceFileAction() override;
+
   static std::unique_ptr<llvm::raw_pwrite_stream>
   CreateOutputFile(CompilerInstance &CI, StringRef InFile);
+
   static StringRef getInputBufferName() { return "<extract-api-includes>"; }
 };
 
 } // namespace clang
 
 #endif // LLVM_CLANG_EXTRACTAPI_FRONTEND_ACTIONS_H

clang/include/clang/ExtractAPI/Serialization/SymbolGraphSerializer.h

 ///
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_EXTRACTAPI_SERIALIZATION_SYMBOLGRAPHSERIALIZER_H
 #define LLVM_CLANG_EXTRACTAPI_SERIALIZATION_SYMBOLGRAPHSERIALIZER_H
 
 #include "clang/ExtractAPI/API.h"
 #include "clang/ExtractAPI/Serialization/SerializerBase.h"
+#include "clang/Lex/PreprocessingRecord.h"
 #include "llvm/Support/JSON.h"
 #include "llvm/Support/VersionTuple.h"
 #include "llvm/Support/raw_ostream.h"
 
 namespace clang {
 namespace extractapi {
 
 using namespace llvm::json;
private:
   void serializeGlobalRecord(const GlobalRecord &Record);
 
   /// Serialize an enum record.
   void serializeEnumRecord(const EnumRecord &Record);
 
   /// Serialize a struct record.
   void serializeStructRecord(const StructRecord &Record);
 
+  /// Serialize a macro defintion record.
+  void serializeMacroDefinitionRecord(const MacroDefinitionRecord &Record);
+
 public:
   SymbolGraphSerializer(const APISet &API, StringRef ProductName,
                         APISerializerOption Options = {})
       : APISerializer(API, ProductName, Options) {}
 };
 
 } // namespace extractapi
 } // namespace clang
 
 #endif // LLVM_CLANG_EXTRACTAPI_SERIALIZATION_SYMBOLGRAPHSERIALIZER_H

clang/lib/ExtractAPI/API.cpp

 /// and the APISet class.
 ///
 //===----------------------------------------------------------------------===//
 
 #include "clang/ExtractAPI/API.h"
 #include "clang/AST/CommentCommandTraits.h"
 #include "clang/AST/CommentLexer.h"
 #include "clang/AST/RawCommentList.h"
 #include "clang/Index/USRGeneration.h"
+#include "llvm/ADT/MapVector.h"
+#include "llvm/ADT/STLFunctionalExtras.h"
+#include "llvm/ADT/StringRef.h"
 #include <memory>
 
 using namespace clang::extractapi;
 using namespace llvm;
 
+namespace {
+
+template <typename RecordTy, typename... CtorArgsTy>
+RecordTy *
+addTopLevelRecord(MapVector<StringRef, std::unique_ptr<RecordTy>> &RecordMap,
+                  StringRef Name, CtorArgsTy &&...CtorArgs) {
+  auto Result = RecordMap.insert({Name, nullptr});
+
+  // Create the record if it does not already exist
+  if (Result.second)
+    Result.first->second =
+        std::make_unique<RecordTy>(Name, std::forward<CtorArgsTy>(CtorArgs)...);
+
+  return Result.first->second.get();
+}
+
+} // namespace
+
 GlobalRecord *APISet::addGlobal(GVKind Kind, StringRef Name, StringRef USR,
                                 PresumedLoc Loc,
                                 const AvailabilityInfo &Availability,
                                 LinkageInfo Linkage, const DocComment &Comment,
                                 DeclarationFragments Fragments,
                                 DeclarationFragments SubHeading,
                                 FunctionSignature Signature) {
-  auto Result = Globals.insert({Name, nullptr});
-  if (Result.second) {
-    // Create the record if it does not already exist.
-    auto Record = std::make_unique<GlobalRecord>(
-        Kind, Name, USR, Loc, Availability, Linkage, Comment, Fragments,
-        SubHeading, Signature);
-    Result.first->second = std::move(Record);
-  }
-  return Result.first->second.get();
+  return addTopLevelRecord(Globals, Name, Kind, USR, Loc, Availability, Linkage,
+                           Comment, Fragments, SubHeading, Signature);
 }
 
 GlobalRecord *
 APISet::addGlobalVar(StringRef Name, StringRef USR, PresumedLoc Loc,
                      const AvailabilityInfo &Availability, LinkageInfo Linkage,
                      const DocComment &Comment, DeclarationFragments Fragments,
                      DeclarationFragments SubHeading) {
   return addGlobal(GVKind::Variable, Name, USR, Loc, Availability, Linkage,
EnumConstantRecord *APISet::addEnumConstant(
   return Enum->Constants.emplace_back(std::move(Record)).get();
 }
 
 EnumRecord *APISet::addEnum(StringRef Name, StringRef USR, PresumedLoc Loc,
                             const AvailabilityInfo &Availability,
                             const DocComment &Comment,
                             DeclarationFragments Declaration,
                             DeclarationFragments SubHeading) {
-  auto Result = Enums.insert({Name, nullptr});
-  if (Result.second) {
-    // Create the record if it does not already exist.
-    auto Record = std::make_unique<EnumRecord>(
-        Name, USR, Loc, Availability, Comment, Declaration, SubHeading);
-    Result.first->second = std::move(Record);
-  }
-  return Result.first->second.get();
+  return addTopLevelRecord(Enums, Name, USR, Loc, Availability, Comment,
+                           Declaration, SubHeading);
 }
 
 StructFieldRecord *APISet::addStructField(StructRecord *Struct, StringRef Name,
                                           StringRef USR, PresumedLoc Loc,
                                           const AvailabilityInfo &Availability,
                                           const DocComment &Comment,
                                           DeclarationFragments Declaration,
                                           DeclarationFragments SubHeading) {
   auto Record = std::make_unique<StructFieldRecord>(
       Name, USR, Loc, Availability, Comment, Declaration, SubHeading);
   return Struct->Fields.emplace_back(std::move(Record)).get();
 }
 
 StructRecord *APISet::addStruct(StringRef Name, StringRef USR, PresumedLoc Loc,
                                 const AvailabilityInfo &Availability,
                                 const DocComment &Comment,
                                 DeclarationFragments Declaration,
                                 DeclarationFragments SubHeading) {
-  auto Result = Structs.insert({Name, nullptr});
-  if (Result.second) {
-    // Create the record if it does not already exist.
-    auto Record = std::make_unique<StructRecord>(
-        Name, USR, Loc, Availability, Comment, Declaration, SubHeading);
-    Result.first->second = std::move(Record);
-  }
-  return Result.first->second.get();
+  return addTopLevelRecord(Structs, Name, USR, Loc, Availability, Comment,
+                           Declaration, SubHeading);
+
+MacroDefinitionRecord *
+APISet::addMacroDefinition(StringRef Name, StringRef USR, PresumedLoc Loc,
+                           DeclarationFragments DeclarationFragments) {
+  auto *MD = addTopLevelRecord(Macros, Name, USR, Loc, DeclarationFragments);
+  return MD;
 }
 
 StringRef APISet::recordUSR(const Decl *D) {
   SmallString<128> USR;
   index::generateUSRForDecl(D, USR);
   return copyString(USR);
 }
 
+StringRef APISet::recordUSRForMacro(StringRef Name, SourceLocation SL,
+                                    const SourceManager &SM) {
+  SmallString<128> USR;
+  index::generateUSRForMacro(Name, SL, SM, USR);
+  return copyString(USR);
+}
+
 StringRef APISet::copyString(StringRef String) {
   if (String.empty())
     return {};
 
   // No need to allocate memory and copy if the string has already been stored.
   if (StringAllocator.identifyObject(String.data()))
     return String;
 
   void *Ptr = StringAllocator.Allocate(String.size(), 1);
   memcpy(Ptr, String.data(), String.size());
   return StringRef(reinterpret_cast<const char *>(Ptr), String.size());
 }
 
 APIRecord::~APIRecord() {}
 
 void GlobalRecord::anchor() {}

clang/lib/ExtractAPI/DeclarationFragments.cpp

DeclarationFragmentsBuilder::getFragmentsForStruct(const RecordDecl *Record) {
   // struct just use the declaration fragments of the typedef decl.
 
   DeclarationFragments Fragments;
   Fragments.append("struct", DeclarationFragments::FragmentKind::Keyword);
 
   if (!Record->getName().empty())
     Fragments.appendSpace().append(
         Record->getName(), DeclarationFragments::FragmentKind::Identifier);
+  return Fragments;
+}
+
+DeclarationFragments
+DeclarationFragmentsBuilder::getFragmentsForMacro(StringRef Name,
+                                                  const MacroDirective *MD) {
+  DeclarationFragments Fragments;
+  Fragments.append("#define", DeclarationFragments::FragmentKind::Keyword)
+      .appendSpace();
+  Fragments.append(Name, DeclarationFragments::FragmentKind::Identifier);
 
+  auto *MI = MD->getMacroInfo();
+
+  if (MI->isFunctionLike()) {
+    Fragments.append("(", DeclarationFragments::FragmentKind::Text);
+    unsigned numParameters = MI->getNumParams();
+    if (MI->isC99Varargs())
+      --numParameters;
+    for (unsigned i = 0; i < numParameters; ++i) {
+      if (i)
+        Fragments.append(", ", DeclarationFragments::FragmentKind::Text);
+      Fragments.append(MI->params()[i]->getName(),
+                       DeclarationFragments::FragmentKind::InternalParam);
+    }
+    if (MI->isVariadic()) {
+      if (numParameters && MI->isC99Varargs())
+        Fragments.append(", ", DeclarationFragments::FragmentKind::Text);
+      Fragments.append("...", DeclarationFragments::FragmentKind::Text);
+    }
+    Fragments.append(")", DeclarationFragments::FragmentKind::Text);
+  }
   return Fragments;
 }
 
 FunctionSignature
 DeclarationFragmentsBuilder::getFunctionSignature(const FunctionDecl *Func) {
   FunctionSignature Signature;
 
   for (const auto *Param : Func->parameters()) {

clang/lib/ExtractAPI/ExtractAPIConsumer.cpp

 #include "clang/ExtractAPI/API.h"
 #include "clang/ExtractAPI/AvailabilityInfo.h"
 #include "clang/ExtractAPI/DeclarationFragments.h"
 #include "clang/ExtractAPI/FrontendActions.h"
 #include "clang/ExtractAPI/Serialization/SymbolGraphSerializer.h"
 #include "clang/Frontend/ASTConsumers.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/FrontendOptions.h"
+#include "clang/Lex/MacroInfo.h"
+#include "clang/Lex/PPCallbacks.h"
+#include "clang/Lex/PreprocessorOptions.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/SetVector.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/raw_ostream.h"
 
 using namespace clang;
 using namespace extractapi;
 
 namespace {
 
 /// The RecursiveASTVisitor to traverse symbol declarations and collect API
 /// information.
 class ExtractAPIVisitor : public RecursiveASTVisitor<ExtractAPIVisitor> {
 public:
-  explicit ExtractAPIVisitor(ASTContext &Context)
-      : Context(Context),
-        API(Context.getTargetInfo().getTriple(), Context.getLangOpts()) {}
+  ExtractAPIVisitor(ASTContext &Context, APISet &API)
+      : Context(Context), API(API) {}
 
   const APISet &getAPI() const { return API; }
 
   bool VisitVarDecl(const VarDecl *Decl) {
     // Skip function parameters.
     if (isa<ParmVarDecl>(Decl))
       return true;
 
for (const auto *Field : Fields) {
           DeclarationFragmentsBuilder::getSubHeading(Field);
 
       API.addStructField(StructRecord, Name, USR, Loc, Availability, Comment,
                          Declaration, SubHeading);
     }
   }
 
   ASTContext &Context;
-  APISet API;
+  APISet &API;
 };
 
 class ExtractAPIConsumer : public ASTConsumer {
 public:
-  ExtractAPIConsumer(ASTContext &Context, StringRef ProductName,
-                     std::unique_ptr<raw_pwrite_stream> OS)
-      : Visitor(Context), ProductName(ProductName), OS(std::move(OS)) {}
+  ExtractAPIConsumer(ASTContext &Context, APISet &API)
+      : Visitor(Context, API) {}
 
   void HandleTranslationUnit(ASTContext &Context) override {
     // Use ExtractAPIVisitor to traverse symbol declarations in the context.
     Visitor.TraverseDecl(Context.getTranslationUnitDecl());
-
-    // Setup a SymbolGraphSerializer to write out collected API information in
-    // the Symbol Graph format.
-    // FIXME: Make the kind of APISerializer configurable.
-    SymbolGraphSerializer SGSerializer(Visitor.getAPI(), ProductName);
-    SGSerializer.serialize(*OS);
   }
 
 private:
   ExtractAPIVisitor Visitor;
-  std::string ProductName;
-  std::unique_ptr<raw_pwrite_stream> OS;
+};
+
+class MacroCallback : public PPCallbacks {
+public:
+  MacroCallback(const SourceManager &SM, APISet &API) : SM(SM), API(API) {}
+
+  void MacroDefined(const Token &MacroNameToken,
+                    const MacroDirective *MD) override {
+    auto *MacroInfo = MD->getMacroInfo();
+
+    if (MacroInfo->isBuiltinMacro())
+      return;
+
+    auto SourceLoc = MacroNameToken.getLocation();
+    if (SM.isWrittenInBuiltinFile(SourceLoc) ||
+        SM.isWrittenInCommandLineFile(SourceLoc))
+      return;
+
+    PendingMacros.emplace_back(MacroNameToken, MD);
+  }
+
+  // If a macro gets undefined at some point during preprocessing of the inputs
+  // it means that it isn't an exposed API and we should therefore not add a
+  // macro definition for it.
+  void MacroUndefined(const Token &MacroNameToken, const MacroDefinition &MD,
+                      const MacroDirective *Undef) override {
+    llvm::erase_if(PendingMacros, [&MD](const PendingMacro &PM) {
+      return MD.getMacroInfo()->getDefinitionLoc() ==
+             PM.MD->getMacroInfo()->getDefinitionLoc();
+    });
+  }
+
+  void EndOfMainFile() override {
+    for (auto &PM : PendingMacros) {
+      // `isUsedForHeaderGuard` is only set when the preprocessor leaves the
+      // file so check for it here.
+      if (PM.MD->getMacroInfo()->isUsedForHeaderGuard())
+        continue;
+
+      StringRef Name = PM.MacroNameToken.getIdentifierInfo()->getName();
+      PresumedLoc Loc = SM.getPresumedLoc(PM.MacroNameToken.getLocation());
+      StringRef USR =
+          API.recordUSRForMacro(Name, PM.MacroNameToken.getLocation(), SM);
+
+      API.addMacroDefinition(
+          API.copyString(Name), USR, Loc,
+          DeclarationFragmentsBuilder::getFragmentsForMacro(Name, PM.MD));
+    }
+
+    PendingMacros.clear();
+  }
+
+private:
+  struct PendingMacro {
+    Token MacroNameToken;
+    const MacroDirective *MD;
+
+    PendingMacro(const Token &MacroNameToken, const MacroDirective *MD)
+        : MacroNameToken(MacroNameToken), MD(MD) {}
+  };
+
+  const SourceManager &SM;
+  APISet &API;
+  llvm::SmallVector<PendingMacro> PendingMacros;
 };
 
 } // namespace
 
 std::unique_ptr<ASTConsumer>
 ExtractAPIAction::CreateASTConsumer(CompilerInstance &CI, StringRef InFile) {
-  std::unique_ptr<raw_pwrite_stream> OS = CreateOutputFile(CI, InFile);
+  OS = CreateOutputFile(CI, InFile);
   if (!OS)
     return nullptr;
-  return std::make_unique<ExtractAPIConsumer>(
-      CI.getASTContext(), CI.getInvocation().getFrontendOpts().ProductName,
-      std::move(OS));
+
+  ProductName = CI.getFrontendOpts().ProductName;
+
+  // Now that we have enough information about the language options and the
+  // target triple, let's create the APISet before anyone uses it.
+  API = std::make_unique<APISet>(CI.getTarget().getTriple(), CI.getLangOpts());
+
+  // Register preprocessor callbacks that will add macro definitions to API.
+  CI.getPreprocessor().addPPCallbacks(
+      std::make_unique<MacroCallback>(CI.getSourceManager(), *API));
+
+  return std::make_unique<ExtractAPIConsumer>(CI.getASTContext(), *API);
 }
 
 bool ExtractAPIAction::PrepareToExecuteAction(CompilerInstance &CI) {
   auto &Inputs = CI.getFrontendOpts().Inputs;
   if (Inputs.empty())
     return true;
 
   auto Kind = Inputs[0].getKind();
 
   // Convert the header file inputs into a single input buffer.
   SmallString<256> HeaderContents;
   for (const FrontendInputFile &FIF : Inputs) {
     if (Kind.isObjectiveC())
       HeaderContents += "#import";
     else
       HeaderContents += "#include";
     HeaderContents += " \"";
     HeaderContents += FIF.getFile();
     HeaderContents += "\"\n";
   }
 
-  Buffer = llvm::MemoryBuffer::getMemBufferCopy(HeaderContents,
+  auto Buffer = llvm::MemoryBuffer::getMemBufferCopy(HeaderContents,
                                                      getInputBufferName());
 
   // Set that buffer up as our "real" input in the CompilerInstance.
   Inputs.clear();
   Inputs.emplace_back(Buffer->getMemBufferRef(), Kind, /*IsSystem*/ false);
 
+  // Tell the processor about the input file.
+  CI.getPreprocessorOpts().addRemappedFile(Buffer->getBufferIdentifier(),
+                                           Buffer.release());
+
   return true;
 }
 
+void ExtractAPIAction::EndSourceFileAction() {
+  if (!OS)
+    return;
+
+  // Setup a SymbolGraphSerializer to write out collected API information in
+  // the Symbol Graph format.
+  // FIXME: Make the kind of APISerializer configurable.
+  SymbolGraphSerializer SGSerializer(*API, ProductName);
+  SGSerializer.serialize(*OS);
+  OS->flush();
+}
+
 std::unique_ptr<raw_pwrite_stream>
 ExtractAPIAction::CreateOutputFile(CompilerInstance &CI, StringRef InFile) {
   std::unique_ptr<raw_pwrite_stream> OS =
       CI.createDefaultOutputFile(/*Binary=*/false, InFile, /*Extension=*/"json",
                                  /*RemoveFileOnSignal=*/false);
   if (!OS)
     return nullptr;
   return OS;
 }

clang/lib/ExtractAPI/Serialization/SymbolGraphSerializer.cpp

 //===- ExtractAPI/Serialization/SymbolGraphSerializer.cpp -------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 ///
 /// \file
 /// This file implements the SymbolGraphSerializer.
 ///
 //===----------------------------------------------------------------------===//
 
 #include "clang/ExtractAPI/Serialization/SymbolGraphSerializer.h"
 #include "clang/Basic/Version.h"
 #include "clang/ExtractAPI/API.h"
+#include "clang/Lex/PreprocessingRecord.h"
 #include "llvm/Support/JSON.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/VersionTuple.h"
 
 using namespace clang;
 using namespace clang::extractapi;
 using namespace llvm;
 using namespace llvm::json;
Object serializeSymbolKind(const APIRecord &Record,
   case APIRecord::RK_StructField:
     Kind["identifier"] = AddLangPrefix("property");
     Kind["displayName"] = "Instance Property";
     break;
   case APIRecord::RK_Struct:
     Kind["identifier"] = AddLangPrefix("struct");
     Kind["displayName"] = "Structure";
     break;
+  case APIRecord::RK_MacroDefinition:
+    Kind["identifier"] = AddLangPrefix("macro");
+    Kind["displayName"] = "Macro";
   }
 
   return Kind;
 }
 
 } // namespace
 
 void SymbolGraphSerializer::anchor() {}
for (const auto &Field : Record.Fields) {
     if (!StructField)
       continue;
 
     Symbols.emplace_back(std::move(*StructField));
     serializeRelationship(RelationshipKind::MemberOf, *Field, Record);
   }
 }
 
+void SymbolGraphSerializer::serializeMacroDefinitionRecord(
+    const MacroDefinitionRecord &Record) {
+  auto Macro = serializeAPIRecord(Record);
+  if (!Macro)
+    return;
+
+  Symbols.emplace_back(std::move(*Macro));
+}
+
 Object SymbolGraphSerializer::serialize() {
   Object Root;
   serializeObject(Root, "metadata", serializeMetadata());
   serializeObject(Root, "module", serializeModule());
 
   // Serialize global records in the API set.
   for (const auto &Global : API.getGlobals())
     serializeGlobalRecord(*Global.second);
 
   // Serialize enum records in the API set.
   for (const auto &Enum : API.getEnums())
     serializeEnumRecord(*Enum.second);
 
   // Serialize struct records in the API set.
   for (const auto &Struct : API.getStructs())
     serializeStructRecord(*Struct.second);
 
+  for (const auto &Macro : API.getMacros())
+    serializeMacroDefinitionRecord(*Macro.second);
+
   Root["symbols"] = std::move(Symbols);
   Root["relationhips"] = std::move(Relationships);
 
   return Root;
 }
 
 void SymbolGraphSerializer::serialize(raw_ostream &os) {
   Object root = serialize();
   if (Options.Compact)
     os << formatv("{0}", Value(std::move(root))) << "\n";
   else
     os << formatv("{0:2}", Value(std::move(root))) << "\n";
 }

clang/test/ExtractAPI/macro_undefined.c

+// RUN: rm -rf %t
+// RUN: split-file %s %t
+// RUN: sed -e "s@INPUT_DIR@%/t@g" %t/reference.output.json.in >> \
+// RUN: %t/reference.output.json
+// RUN: %clang -extract-api --product-name=Macros -target arm64-apple-macosx \
+// RUN: -x objective-c-header %t/input.h -o %t/output.json | FileCheck -allow-empty %s
+
+// Generator version is not consistent across test runs, normalize it.
+// RUN: sed -e "s@\"generator\": \".*\"@\"generator\": \"?\"@g" \
+// RUN: %t/output.json >> %t/output-normalized.json
+// RUN: diff %t/reference.output.json %t/output-normalized.json
+
+// CHECK-NOT: error:
+// CHECK-NOT: warning:
+
+//--- input.h
+#define HELLO 1
+#define FUNC_GEN(NAME, ...) void NAME(__VA_ARGS__);
+FUNC_GEN(foo)
+FUNC_GEN(bar, const int *, size_t);
+#undef FUNC_GEN
+
+//--- reference.output.json.in
+{
+  "metadata": {
+    "formatVersion": {
+      "major": 0,
+      "minor": 5,
+      "patch": 3
+    },
+    "generator": "?"
+  },
+  "module": {
+    "name": "Macros",
+    "platform": {
+      "architecture": "arm64",
+      "operatingSystem": {
+        "minimumVersion": {
+          "major": 11,
+          "minor": 0,
+          "patch": 0
+        },
+        "name": "macosx"
+      },
+      "vendor": "apple"
+    }
+  },
+  "relationhips": [],
+  "symbols": [
+    {
+      "declarationFragments": [
+        {
+          "kind": "typeIdentifier",
+          "preciseIdentifier": "c:v",
+          "spelling": "void"
+        },
+        {
+          "kind": "text",
+          "spelling": " "
+        },
+        {
+          "kind": "identifier",
+          "spelling": "foo"
+        },
+        {
+          "kind": "text",
+          "spelling": "()"
+        }
+      ],
+      "identifier": {
+        "interfaceLanguage": "c",
+        "precise": "c:@F@foo"
+      },
+      "kind": {
+        "displayName": "Function",
+        "identifier": "c.func"
+      },
+      "location": {
+        "character": 1,
+        "line": 3,
+        "uri": "file://INPUT_DIR/input.h"
+      },
+      "names": {
+        "subHeading": [
+          {
+            "kind": "identifier",
+            "spelling": "foo"
+          }
+        ],
+        "title": "foo"
+      },
+      "parameters": {
+        "returns": [
+          {
+            "kind": "typeIdentifier",
+            "preciseIdentifier": "c:v",
+            "spelling": "void"
+          }
+        ]
+      }
+    },
+    {
+      "declarationFragments": [
+        {
+          "kind": "typeIdentifier",
+          "preciseIdentifier": "c:v",
+          "spelling": "void"
+        },
+        {
+          "kind": "text",
+          "spelling": " "
+        },
+        {
+          "kind": "identifier",
+          "spelling": "bar"
+        },
+        {
+          "kind": "text",
+          "spelling": "("
+        },
+        {
+          "kind": "keyword",
+          "spelling": "const"
+        },
+        {
+          "kind": "text",
+          "spelling": " "
+        },
+        {
+          "kind": "typeIdentifier",
+          "preciseIdentifier": "c:I",
+          "spelling": "int"
+        },
+        {
+          "kind": "text",
+          "spelling": " *"
+        },
+        {
+          "kind": "internalParam",
+          "spelling": ""
+        },
+        {
+          "kind": "text",
+          "spelling": ", "
+        },
+        {
+          "kind": "typeIdentifier",
+          "preciseIdentifier": "c:I",
+          "spelling": "int"
+        },
+        {
+          "kind": "text",
+          "spelling": " "
+        },
+        {
+          "kind": "internalParam",
+          "spelling": "size_t"
+        },
+        {
+          "kind": "text",
+          "spelling": ")"
+        }
+      ],
+      "identifier": {
+        "interfaceLanguage": "c",
+        "precise": "c:@F@bar"
+      },
+      "kind": {
+        "displayName": "Function",
+        "identifier": "c.func"
+      },
+      "location": {
+        "character": 1,
+        "line": 4,
+        "uri": "file://INPUT_DIR/input.h"
+      },
+      "names": {
+        "subHeading": [
+          {
+            "kind": "identifier",
+            "spelling": "bar"
+          }
+        ],
+        "title": "bar"
+      },
+      "parameters": {
+        "parameters": [
+          {
+            "declarationFragments": [
+              {
+                "kind": "keyword",
+                "spelling": "const"
+              },
+              {
+                "kind": "text",
+                "spelling": " "
+              },
+              {
+                "kind": "typeIdentifier",
+                "preciseIdentifier": "c:I",
+                "spelling": "int"
+              },
+              {
+                "kind": "text",
+                "spelling": " *"
+              },
+              {
+                "kind": "internalParam",
+                "spelling": ""
+              }
+            ],
+            "name": ""
+          },
+          {
+            "declarationFragments": [
+              {
+                "kind": "typeIdentifier",
+                "preciseIdentifier": "c:I",
+                "spelling": "int"
+              },
+              {
+                "kind": "text",
+                "spelling": " "
+              },
+              {
+                "kind": "internalParam",
+                "spelling": "size_t"
+              }
+            ],
+            "name": "size_t"
+          }
+        ],
+        "returns": [
+          {
+            "kind": "typeIdentifier",
+            "preciseIdentifier": "c:v",
+            "spelling": "void"
+          }
+        ]
+      }
+    },
+    {
+      "declarationFragments": [
+        {
+          "kind": "keyword",
+          "spelling": "#define"
+        },
+        {
+          "kind": "text",
+          "spelling": " "
+        },
+        {
+          "kind": "identifier",
+          "spelling": "HELLO"
+        }
+      ],
+      "identifier": {
+        "interfaceLanguage": "c",
+        "precise": "c:input.h@8@macro@HELLO"
+      },
+      "kind": {
+        "displayName": "Macro",
+        "identifier": "c.macro"
+      },
+      "location": {
+        "character": 9,
+        "line": 1,
+        "uri": "file://INPUT_DIR/input.h"
+      },
+      "names": {
+        "title": "HELLO"
+      }
+    }
+  ]
+}

clang/test/ExtractAPI/macros.c

+// RUN: rm -rf %t
+// RUN: split-file %s %t
+// RUN: sed -e "s@INPUT_DIR@%/t@g" %t/reference.output.json.in >> \
+// RUN: %t/reference.output.json
+// RUN: %clang -extract-api --product-name=Macros -target arm64-apple-macosx \
+// RUN: -x objective-c-header %t/input.h -o %t/output.json | FileCheck -allow-empty %s
+
+// Generator version is not consistent across test runs, normalize it.
+// RUN: sed -e "s@\"generator\": \".*\"@\"generator\": \"?\"@g" \
+// RUN: %t/output.json >> %t/output-normalized.json
+// RUN: diff %t/reference.output.json %t/output-normalized.json
+
+// CHECK-NOT: error:
+// CHECK-NOT: warning:
+
+//--- input.h
+#define HELLO 1
+#define WORLD 2
+#define MACRO_FUN(x) x x
+#define FUN(x, y, z) x + y + z
+#define FUNC99(x, ...)
+#define FUNGNU(x...)
+
+//--- reference.output.json.in
+{
+  "metadata": {
+    "formatVersion": {
+      "major": 0,
+      "minor": 5,
+      "patch": 3
+    },
+    "generator": "?"
+  },
+  "module": {
+    "name": "Macros",
+    "platform": {
+      "architecture": "arm64",
+      "operatingSystem": {
+        "minimumVersion": {
+          "major": 11,
+          "minor": 0,
+          "patch": 0
+        },
+        "name": "macosx"
+      },
+      "vendor": "apple"
+    }
+  },
+  "relationhips": [],
+  "symbols": [
+    {
+      "declarationFragments": [
+        {
+          "kind": "keyword",
+          "spelling": "#define"
+        },
+        {
+          "kind": "text",
+          "spelling": " "
+        },
+        {
+          "kind": "identifier",
+          "spelling": "HELLO"
+        }
+      ],
+      "identifier": {
+        "interfaceLanguage": "c",
+        "precise": "c:input.h@8@macro@HELLO"
+      },
+      "kind": {
+        "displayName": "Macro",
+        "identifier": "c.macro"
+      },
+      "location": {
+        "character": 9,
+        "line": 1,
+        "uri": "file://INPUT_DIR/input.h"
+      },
+      "names": {
+        "title": "HELLO"
+      }
+    },
+    {
+      "declarationFragments": [
+        {
+          "kind": "keyword",
+          "spelling": "#define"
+        },
+        {
+          "kind": "text",
+          "spelling": " "
+        },
+        {
+          "kind": "identifier",
+          "spelling": "WORLD"
+        }
+      ],
+      "identifier": {
+        "interfaceLanguage": "c",
+        "precise": "c:input.h@24@macro@WORLD"
+      },
+      "kind": {
+        "displayName": "Macro",
+        "identifier": "c.macro"
+      },
+      "location": {
+        "character": 9,
+        "line": 2,
+        "uri": "file://INPUT_DIR/input.h"
+      },
+      "names": {
+        "title": "WORLD"
+      }
+    },
+    {
+      "declarationFragments": [
+        {
+          "kind": "keyword",
+          "spelling": "#define"
+        },
+        {
+          "kind": "text",
+          "spelling": " "
+        },
+        {
+          "kind": "identifier",
+          "spelling": "MACRO_FUN"
+        },
+        {
+          "kind": "text",
+          "spelling": "("
+        },
+        {
+          "kind": "internalParam",
+          "spelling": "x"
+        },
+        {
+          "kind": "text",
+          "spelling": ")"
+        }
+      ],
+      "identifier": {
+        "interfaceLanguage": "c",
+        "precise": "c:input.h@40@macro@MACRO_FUN"
+      },
+      "kind": {
+        "displayName": "Macro",
+        "identifier": "c.macro"
+      },
+      "location": {
+        "character": 9,
+        "line": 3,
+        "uri": "file://INPUT_DIR/input.h"
+      },
+      "names": {
+        "title": "MACRO_FUN"
+      }
+    },
+    {
+      "declarationFragments": [
+        {
+          "kind": "keyword",
+          "spelling": "#define"
+        },
+        {
+          "kind": "text",
+          "spelling": " "
+        },
+        {
+          "kind": "identifier",
+          "spelling": "FUN"
+        },
+        {
+          "kind": "text",
+          "spelling": "("
+        },
+        {
+          "kind": "internalParam",
+          "spelling": "x"
+        },
+        {
+          "kind": "text",
+          "spelling": ", "
+        },
+        {
+          "kind": "internalParam",
+          "spelling": "y"
+        },
+        {
+          "kind": "text",
+          "spelling": ", "
+        },
+        {
+          "kind": "internalParam",
+          "spelling": "z"
+        },
+        {
+          "kind": "text",
+          "spelling": ")"
+        }
+      ],
+      "identifier": {
+        "interfaceLanguage": "c",
+        "precise": "c:input.h@65@macro@FUN"
+      },
+      "kind": {
+        "displayName": "Macro",
+        "identifier": "c.macro"
+      },
+      "location": {
+        "character": 9,
+        "line": 4,
+        "uri": "file://INPUT_DIR/input.h"
+      },
+      "names": {
+        "title": "FUN"
+      }
+    },
+    {
+      "declarationFragments": [
+        {
+          "kind": "keyword",
+          "spelling": "#define"
+        },
+        {
+          "kind": "text",
+          "spelling": " "
+        },
+        {
+          "kind": "identifier",
+          "spelling": "FUNC99"
+        },
+        {
+          "kind": "text",
+          "spelling": "("
+        },
+        {
+          "kind": "internalParam",
+          "spelling": "x"
+        },
+        {
+          "kind": "text",
+          "spelling": ", ...)"
+        }
+      ],
+      "identifier": {
+        "interfaceLanguage": "c",
+        "precise": "c:input.h@96@macro@FUNC99"
+      },
+      "kind": {
+        "displayName": "Macro",
+        "identifier": "c.macro"
+      },
+      "location": {
+        "character": 9,
+        "line": 5,
+        "uri": "file://INPUT_DIR/input.h"
+      },
+      "names": {
+        "title": "FUNC99"
+      }
+    },
+    {
+      "declarationFragments": [
+        {
+          "kind": "keyword",
+          "spelling": "#define"
+        },
+        {
+          "kind": "text",
+          "spelling": " "
+        },
+        {
+          "kind": "identifier",
+          "spelling": "FUNGNU"
+        },
+        {
+          "kind": "text",
+          "spelling": "("
+        },
+        {
+          "kind": "internalParam",
+          "spelling": "x"
+        },
+        {
+          "kind": "text",
+          "spelling": "...)"
+        }
+      ],
+      "identifier": {
+        "interfaceLanguage": "c",
+        "precise": "c:input.h@119@macro@FUNGNU"
+      },
+      "kind": {
+        "displayName": "Macro",
+        "identifier": "c.macro"
+      },
+      "location": {
+        "character": 9,
+        "line": 6,
+        "uri": "file://INPUT_DIR/input.h"
+      },
+      "names": {
+        "title": "FUNGNU"
+      }
+    }
+  ]
+}