[clang][extract-api] Refactor ExtractAPI and improve docs

[zixuw]

Summary

• The name SymbolGraph is inappropriate and confusing for the new library for clang-extract-api. Refactor and rename things to make it clear that ExtractAPI is the core functionality and SymbolGraph is one serializer for the API information.
• Add documentation comments to ExtractAPI classes and methods to improve readability and clearness of the ExtractAPI work.

[dang]

clang/include/clang/ExtractAPI/API.h

34

You should use the actual name of the type or "This"

37

If it wouldn't make things simpler we wouldn't have it. IMHO it is better to use definitive terms in documentation. here it would "This simplifies calculating [...]"

103

Nit: instead of "of a" maybe use "associated with"

123

This is new so I don't think it belongs in this patch.

129

why is stuff reordered in the patch?

clang/include/clang/ExtractAPI/DeclarationFragments.h

108

Not sure an explanation of the process is needed. Anyway "quicking" is not a word ;) I would suggest "Other is moved from and can not be used after a call to this method".

clang/include/clang/ExtractAPI/Serialization/SerializerBase.h

33

If we are going for a base class, I am not sure that the base should be JSON specific.

clang/include/clang/ExtractAPI/Serialization/SymbolGraphSerializer.h

34

Not sure why we need this.

clang/lib/ExtractAPI/API.cpp

34

"Create the record if it does not already exist."

clang/lib/ExtractAPI/ExtractAPIConsumer.cpp

195

The kind of serializer should be made configurable somehow.

clang/lib/ExtractAPI/Serialization/SymbolGraphSerializer.cpp

26

This is unnecessary imho.

[tschuett]

Would you mind putting a high level description of the intent at the top of the main header.

Maybe even a link at https://github.com/apple/swift-docc-symbolkit

[zixuw]

In D122160#3397366, @tschuett wrote:

Would you mind putting a high level description of the intent at the top of the main header.

Maybe even a link at https://github.com/apple/swift-docc-symbolkit

Sure thing!

[zixuw]

Address review issues.

clang/include/clang/ExtractAPI/API.h

123

This is to follow the LLVM Coding Standards guideline of providing an out-of-line virtual method anchor for a class defined in a header file and has a vtable. Still coding style/standard fix/improvement.

129

I looked into some other class definitions inside Clang, and was trying to list "interesting" APIs upfront and less-interesting stuff like constructors near the end for large class definitions to make them more readable.

clang/include/clang/ExtractAPI/DeclarationFragments.h

108

Whops didn't realize that I typed that. Friday's work huh? :p

clang/include/clang/ExtractAPI/Serialization/SymbolGraphSerializer.h

34

As above. Refactor to follow LLVM Coding Standards.

clang/lib/ExtractAPI/ExtractAPIConsumer.cpp

195

Yes for sure. I didn't include that change here because it would probably involve adding command line options and changing the driver, which should have its own patch later.
I added a FIXME here to point it out for now.

[zixuw]

• Minor Doxygen auto-link string adjustments
• Forgot to rename the test directory

[dang]

LGTM once the last few bits of feedback I left are considered.

clang/include/clang/ExtractAPI/API.h

129

OK that is a nice change then, I wasn't aware that this is a thing. I was just complaining because it made the diff harder to parse for me ;)

137

Maybe we should document that the caller is meant to keep Name and USR alive. Name is expected to refer to the appropriate Decl member so it shouldn't be an issue, but this usage should be documented. WRT USR these comments would be a good place to mention the existence of recordUSR and the copyString method as a way of getting USR strings that are guaranteed to have the right lifetime.

192

Should this be a private member function? The overload that doesn't take in an allocator makes sense since it uses the APISet's allocator, but this one does not seem like it should be publicly accessible, since it just ties the lifetime of the returned String to that of an arbitrary allocator?

clang/lib/ExtractAPI/ExtractAPIConsumer.cpp

195

Yeah that makes sense, we can do this as a follow up patch, definitely not urgent as SymbolGraph is the only use case at the moment. However, we should do this relatively soon so that the structure of the existing command line arguments is stable before we are fully done with this.

clang/lib/ExtractAPI/Serialization/SymbolGraphSerializer.cpp

30

You don't need both static an anonymous namespace afaik.

[zixuw]

Address review issues:

• Documenting the responsibility of keeping StringRefs alive for APISet
• Remove unnecessary static for helper functions in an anonymous namespace

[zixuw]

Address review issue: remove unnecessary copyString overload

[dang]

LGTM

[zixuan-wu]

clang/lib/ExtractAPI/Serialization/SymbolGraphSerializer.cpp

159

It's same name as Language variable above, and it cause compile error. @zixuw
Maybe the error depends on host compiler version so that it does not report immediately.

[zixuw]

clang/lib/ExtractAPI/Serialization/SymbolGraphSerializer.cpp

159

Hi! Thanks for the report! Do you have a link to the failed build or detailed error output and compile configuration?

[zixuw]

clang/lib/ExtractAPI/Serialization/SymbolGraphSerializer.cpp

159

Renamed the variable in 826e661a96a2aa7eb309dbca16c85a3d05edcec8

clang/include/clang/ExtractAPI/API.h

-//===- SymbolGraph/API.h ----------------------------------------*- C++ -*-===//
+//===- ExtractAPI/API.h -----------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 ///
 /// \file
-/// \brief Defines SymbolGraph API records.
+/// This file defines the data representation of APIs used by ExtractAPI:
+/// APIRecord and derived record structs, and the APISet class that holds the
+/// records.
 ///
 //===----------------------------------------------------------------------===//
 
-#ifndef LLVM_CLANG_SYMBOLGRAPH_API_H
-#define LLVM_CLANG_SYMBOLGRAPH_API_H
+#ifndef LLVM_CLANG_EXTRACTAPI_API_H
+#define LLVM_CLANG_EXTRACTAPI_API_H
 
 #include "clang/AST/Decl.h"
 #include "clang/AST/RawCommentList.h"
 #include "clang/Basic/SourceLocation.h"
-#include "clang/SymbolGraph/AvailabilityInfo.h"
-#include "clang/SymbolGraph/DeclarationFragments.h"
+#include "clang/ExtractAPI/AvailabilityInfo.h"
+#include "clang/ExtractAPI/DeclarationFragments.h"
 #include "llvm/ADT/MapVector.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/Triple.h"
 #include "llvm/Support/Allocator.h"
 #include "llvm/Support/Casting.h"
 #include <memory>
 
 namespace clang {
-namespace symbolgraph {
+namespace extractapi {
 
+/// Documentation comment is a vector of RawComment::CommentLine.
+///
+/// Each line represents one line of striped documentation comment,
+/// with source range information. This could simplify calculating the source
+/// location of a character in the doc comment for pointing back to the source
+/// file.
+/// e.g.
+/// \code
+///   /// This is a documentation comment
+///       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~'  First line.
+///   ///     with multiple lines.
+///       ^~~~~~~~~~~~~~~~~~~~~~~'         Second line.
+/// \endcode
 using DocComment = std::vector<RawComment::CommentLine>;
 
+/// The base representation of an API record. Holds common symbol information.
 struct APIRecord {
   StringRef Name;
   StringRef USR;
   PresumedLoc Location;
   AvailabilityInfo Availability;
   LinkageInfo Linkage;
+
+  /// Documentation comment lines attached to this symbol declaration.
   DocComment Comment;
+
+  /// Declaration fragments of this symbol declaration.
   DeclarationFragments Declaration;
+
+  /// SubHeading provides a more detailed representation than the plain
+  /// declaration name.
+  ///
+  /// SubHeading is an array of declaration fragments of tagged declaration
+  /// name, with potentially more tokens (for example the \c +/- symbol for
+  /// Objective-C class/instance methods).
   DeclarationFragments SubHeading;
 
   /// Discriminator for LLVM-style RTTI (dyn_cast<> et al.)
   enum RecordKind {
     RK_Global,
   };
 
 private:
APIRecord(RecordKind Kind, StringRef Name, StringRef USR,
       : Name(Name), USR(USR), Location(Location), Availability(Availability),
         Linkage(Linkage), Comment(Comment), Declaration(Declaration),
         SubHeading(SubHeading), Kind(Kind) {}
 
   // Pure virtual destructor to make APIRecord abstract
   virtual ~APIRecord() = 0;
 };
 
+/// The kind of a global record.
 enum class GVKind : uint8_t {
   Unknown = 0,
   Variable = 1,
   Function = 2,
 };
 
+/// GlobalRecord holds information of a global variable or a function.
 struct GlobalRecord : APIRecord {
   GVKind GlobalKind;
+
+  /// The function signature of the record if it is a function.
   FunctionSignature Signature;
 
   GlobalRecord(GVKind Kind, StringRef Name, StringRef USR, PresumedLoc Loc,
                const AvailabilityInfo &Availability, LinkageInfo Linkage,
                const DocComment &Comment, DeclarationFragments Declaration,
                DeclarationFragments SubHeading, FunctionSignature Signature)
       : APIRecord(RK_Global, Name, USR, Loc, Availability, Linkage, Comment,
                   Declaration, SubHeading),
         GlobalKind(Kind), Signature(Signature) {}
 
   static bool classof(const APIRecord *Record) {
     return Record->getKind() == RK_Global;
   }
+
+private:
+  virtual void anchor();
 };
 
+/// APISet holds the set of API records collected from given inputs.
 class APISet {
 public:
-  APISet(const llvm::Triple &Target, const LangOptions &LangOpts)
-      : Target(Target), LangOpts(LangOpts) {}
-
-  const llvm::Triple &getTarget() const { return Target; }
-  const LangOptions &getLangOpts() const { return LangOpts; }
-
+  /// Create and add a GlobalRecord of kind \p Kind into the API set.
   GlobalRecord *addGlobal(GVKind Kind, StringRef Name, StringRef USR,
                           PresumedLoc Loc, const AvailabilityInfo &Availability,
                           LinkageInfo Linkage, const DocComment &Comment,
                           DeclarationFragments Declaration,
                           DeclarationFragments SubHeading,
                           FunctionSignature Signature);
 
+  /// Create and add a global variable record into the API set.
   GlobalRecord *addGlobalVar(StringRef Name, StringRef USR, PresumedLoc Loc,
                              const AvailabilityInfo &Availability,
                              LinkageInfo Linkage, const DocComment &Comment,
                              DeclarationFragments Declaration,
                              DeclarationFragments SubHeading);
 
+  /// Create and add a function record into the API set.
   GlobalRecord *addFunction(StringRef Name, StringRef USR, PresumedLoc Loc,
                             const AvailabilityInfo &Availability,
                             LinkageInfo Linkage, const DocComment &Comment,
                             DeclarationFragments Declaration,
                             DeclarationFragments SubHeading,
                             FunctionSignature Signature);
 
-  StringRef recordUSR(const Decl *D);
-  StringRef copyString(StringRef String, llvm::BumpPtrAllocator &Allocator);
-  StringRef copyString(StringRef String);
-
 private:
   /// \brief A custom deleter used for ``std::unique_ptr`` to APIRecords stored
   /// in the BumpPtrAllocator.
   ///
   /// \tparam T the exact type of the APIRecord subclass.
   template <typename T> struct UniquePtrBumpPtrAllocatorDeleter {
     void operator()(T *Instance) { Instance->~T(); }
   };
 
 public:
   /// A unique pointer to an APIRecord stored in the BumpPtrAllocator.
   ///
   /// \tparam T the exact type of the APIRecord subclass.
   template <typename T>
   using APIRecordUniquePtr =
       std::unique_ptr<T, UniquePtrBumpPtrAllocatorDeleter<T>>;
 
+  /// A map to store the set of GlobalRecords with the declaration name as the
+  /// key.
   using GlobalRecordMap =
       llvm::MapVector<StringRef, APIRecordUniquePtr<GlobalRecord>>;
 
+  /// Get the target triple for the ExtractAPI invocation.
+  const llvm::Triple &getTarget() const { return Target; }
+
+  /// Get the language options used to parse the APIs.
+  const LangOptions &getLangOpts() const { return LangOpts; }
+
   const GlobalRecordMap &getGlobals() const { return Globals; }
 
+  /// Generate and store the USR of declaration \p D.
+  ///
+  /// Note: The USR string is stored in and owned by Allocator.
+  ///
+  /// \returns a StringRef of the generated USR string.
+  StringRef recordUSR(const Decl *D);
+
+  /// Copy \p String into \p Allocator to keep it alive.
+  ///
+  /// \returns a StringRef of the copied string in the \p Allocator.
+  StringRef copyString(StringRef String, llvm::BumpPtrAllocator &Allocator);
+
+  /// Copy \p String into the Allocator in this APISet.
+  ///
+  /// \returns a StringRef of the copied string in the Allocator.
+  StringRef copyString(StringRef String);
+
+  APISet(const llvm::Triple &Target, const LangOptions &LangOpts)
+      : Target(Target), LangOpts(LangOpts) {}
+
 private:
+  /// BumpPtrAllocator to store APIRecords and generated/copied strings.
   llvm::BumpPtrAllocator Allocator;
+
   const llvm::Triple Target;
   const LangOptions LangOpts;
 
   GlobalRecordMap Globals;
 };
 
-} // namespace symbolgraph
+} // namespace extractapi
 } // namespace clang
 
-#endif // LLVM_CLANG_SYMBOLGRAPH_API_H
+#endif // LLVM_CLANG_EXTRACTAPI_API_H

clang/include/clang/ExtractAPI/AvailabilityInfo.h

-//===- SymbolGraph/AvailabilityInfo.h - Availability Info -------*- C++ -*-===//
+//===- ExtractAPI/AvailabilityInfo.h - Availability Info --------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 ///
 /// \file
-/// \brief Defines the Availability Info for a declaration.
+/// This file defines the AvailabilityInfo struct that collects availability
+/// attributes of a symbol.
 ///
 //===----------------------------------------------------------------------===//
 
-#ifndef LLVM_CLANG_SYMBOLGRAPH_AVAILABILITY_INFO_H
-#define LLVM_CLANG_SYMBOLGRAPH_AVAILABILITY_INFO_H
+#ifndef LLVM_CLANG_EXTRACTAPI_AVAILABILITY_INFO_H
+#define LLVM_CLANG_EXTRACTAPI_AVAILABILITY_INFO_H
 
 #include "llvm/Support/Error.h"
 #include "llvm/Support/VersionTuple.h"
 #include "llvm/Support/raw_ostream.h"
 
 using llvm::VersionTuple;
 
 namespace clang {
-namespace symbolgraph {
+namespace extractapi {
 
+/// Stores availability attributes of a symbol.
 struct AvailabilityInfo {
   VersionTuple Introduced;
   VersionTuple Deprecated;
   VersionTuple Obsoleted;
   bool Unavailable{false};
   bool UnconditionallyDeprecated{false};
   bool UnconditionallyUnavailable{false};
 
-  explicit AvailabilityInfo(bool Unavailable = false)
-      : Unavailable(Unavailable) {}
-
-  AvailabilityInfo(VersionTuple I, VersionTuple D, VersionTuple O, bool U,
-                   bool UD, bool UU)
-      : Introduced(I), Deprecated(D), Obsoleted(O), Unavailable(U),
-        UnconditionallyDeprecated(UD), UnconditionallyUnavailable(UU) {}
-
+  /// Determine if this AvailabilityInfo represents the default availability.
   bool isDefault() const { return *this == AvailabilityInfo(); }
+
+  /// Check if the symbol is unavailable.
   bool isUnavailable() const { return Unavailable; }
+
+  /// Check if the symbol is unconditionally deprecated.
+  ///
+  /// i.e. \code __attribute__((deprecated)) \endcode
   bool isUnconditionallyDeprecated() const { return UnconditionallyDeprecated; }
+
+  /// Check if the symbol is unconditionally unavailable.
+  ///
+  /// i.e. \code __attribute__((unavailable)) \endcode
   bool isUnconditionallyUnavailable() const {
     return UnconditionallyUnavailable;
   }
 
+  AvailabilityInfo() = default;
+
+  AvailabilityInfo(VersionTuple I, VersionTuple D, VersionTuple O, bool U,
+                   bool UD, bool UU)
+      : Introduced(I), Deprecated(D), Obsoleted(O), Unavailable(U),
+        UnconditionallyDeprecated(UD), UnconditionallyUnavailable(UU) {}
+
   friend bool operator==(const AvailabilityInfo &Lhs,
                          const AvailabilityInfo &Rhs);
 };
 
 inline bool operator==(const AvailabilityInfo &Lhs,
                        const AvailabilityInfo &Rhs) {
   return std::tie(Lhs.Introduced, Lhs.Deprecated, Lhs.Obsoleted,
                   Lhs.Unavailable, Lhs.UnconditionallyDeprecated,
                   Lhs.UnconditionallyUnavailable) ==
          std::tie(Rhs.Introduced, Rhs.Deprecated, Rhs.Obsoleted,
                   Rhs.Unavailable, Rhs.UnconditionallyDeprecated,
                   Rhs.UnconditionallyUnavailable);
 }
 
-} // namespace symbolgraph
+} // namespace extractapi
 } // namespace clang
 
-#endif // LLVM_CLANG_SYMBOLGRAPH_AVAILABILITY_INFO_H
+#endif // LLVM_CLANG_EXTRACTAPI_AVAILABILITY_INFO_H

clang/include/clang/ExtractAPI/DeclarationFragments.h

-//===- SymbolGraph/DeclarationFragments.h -----------------------*- C++ -*-===//
+//===- ExtractAPI/DeclarationFragments.h ------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 ///
 /// \file
-/// \brief Defines SymbolGraph Declaration Fragments related classes.
+/// This file defines the Declaration Fragments related classes.
 ///
 //===----------------------------------------------------------------------===//
 
-#ifndef LLVM_CLANG_SYMBOLGRAPH_DECLARATION_FRAGMENTS_H
-#define LLVM_CLANG_SYMBOLGRAPH_DECLARATION_FRAGMENTS_H
+#ifndef LLVM_CLANG_EXTRACTAPI_DECLARATION_FRAGMENTS_H
+#define LLVM_CLANG_EXTRACTAPI_DECLARATION_FRAGMENTS_H
 
 #include "clang/AST/ASTContext.h"
 #include "clang/AST/Decl.h"
 #include "clang/AST/DeclCXX.h"
 #include "llvm/ADT/StringRef.h"
 #include <vector>
 
 namespace clang {
-namespace symbolgraph {
+namespace extractapi {
 
+/// DeclarationFragments is a vector of tagged important parts of a symbol's
+/// declaration.
+///
+/// The fragments sequence can be joined to form spans of declaration text, with
+/// attached information useful for purposes like syntax-highlighting etc.
+/// For example:
+/// \code
+///   const -> keyword    "const"
+///   int   -> type       "int"
+///   pi;   -> identifier "pi"
+/// \endcode
 class DeclarationFragments {
 public:
   DeclarationFragments() = default;
 
+  /// The kind of a fragment.
   enum class FragmentKind {
+    /// Unknown fragment kind.
     None,
+
     Keyword,
     Attribute,
     NumberLiteral,
     StringLiteral,
     Identifier,
+
+    /// Identifier that refers to a type in the context.
     TypeIdentifier,
+
+    /// Parameter that's used as generics in the context. For example template
+    /// parameters.
     GenericParameter,
+
+    /// External parameters in Objective-C methods.
+    ///
+    /// For example, \c forKey in
+    /// \code - (void) setValue:(Value)value forKey(Key)key \endcode
     ExternalParam,
+
+    /// Internal/local parameters in Objective-C methods.
+    ///
+    /// For example, \c key in
+    /// \code - (void) setValue:(Value)value forKey(Key)key \endcode
     InternalParam,
+
     Text,
   };
 
+  /// Fragment holds information of a single fragment.
   struct Fragment {
     std::string Spelling;
     FragmentKind Kind;
+
+    /// The USR of the fragment symbol, if applicable.
     std::string PreciseIdentifier;
 
     Fragment(StringRef Spelling, FragmentKind Kind, StringRef PreciseIdentifier)
         : Spelling(Spelling), Kind(Kind), PreciseIdentifier(PreciseIdentifier) {
     }
   };
 
   const std::vector<Fragment> &getFragments() const { return Fragments; }
 
+  /// Append a new Fragment to the end of the Fragments.
+  ///
+  /// \returns a reference to the DeclarationFragments object itself after
+  /// appending to chain up consecutive appends.
   DeclarationFragments &append(StringRef Spelling, FragmentKind Kind,
                                StringRef PreciseIdentifier = "") {
     if (Kind == FragmentKind::Text && !Fragments.empty() &&
         Fragments.back().Kind == FragmentKind::Text) {
+      // If appending a text fragment, and the last fragment is also text,
+      // merge into the last fragment.
       Fragments.back().Spelling.append(Spelling.data(), Spelling.size());
     } else {
       Fragments.emplace_back(Spelling, Kind, PreciseIdentifier);
     }
     return *this;
   }
 
+  /// Append another DeclarationFragments to the end.
+  ///
+  /// A helper function to quicking move the fragments in another
+  /// DeclarationFragments and concatenate.
+  ///
+  /// \returns a reference to the DeclarationFragments object itself after
+  /// appending to chain up consecutive appends.
   DeclarationFragments &append(DeclarationFragments &&Other) {
     Fragments.insert(Fragments.end(),
                      std::make_move_iterator(Other.Fragments.begin()),
                      std::make_move_iterator(Other.Fragments.end()));
     Other.Fragments.clear();
     return *this;
   }
 
+  /// Append a text Fragment of a space character.
+  ///
+  /// \returns a reference to the DeclarationFragments object itself after
+  /// appending to chain up consecutive appends.
   DeclarationFragments &appendSpace();
 
+  /// Get the string description of a FragmentKind \p Kind.
   static StringRef getFragmentKindString(FragmentKind Kind);
+
+  /// Get the corresponding FragmentKind from string \p S.
   static FragmentKind parseFragmentKindFromString(StringRef S);
 
 private:
   std::vector<Fragment> Fragments;
 };
 
+/// Store function signature information with DeclarationFragments of the
+/// return type and parameters.
 class FunctionSignature {
 public:
   FunctionSignature() = default;
 
+  /// Parameter holds the name and DeclarationFragments of a single parameter.
   struct Parameter {
     std::string Name;
     DeclarationFragments Fragments;
 
     Parameter(StringRef Name, DeclarationFragments Fragments)
         : Name(Name), Fragments(Fragments) {}
   };
 
   const std::vector<Parameter> &getParameters() const { return Parameters; }
   const DeclarationFragments &getReturnType() const { return ReturnType; }
 
   FunctionSignature &addParameter(StringRef Name,
                                   DeclarationFragments Fragments) {
     Parameters.emplace_back(Name, Fragments);
     return *this;
   }
 
   void setReturnType(DeclarationFragments RT) { ReturnType = RT; }
 
+  /// Determine if the FunctionSignature is empty.
+  ///
+  /// \returns true if the return type DeclarationFragments is empty and there
+  /// is no parameter, otherwise false.
   bool empty() const {
     return Parameters.empty() && ReturnType.getFragments().empty();
   }
 
 private:
   std::vector<Parameter> Parameters;
   DeclarationFragments ReturnType;
 };
 
+/// A factory class to build DeclarationFragments for different kinds of Decl.
 class DeclarationFragmentsBuilder {
 public:
+  /// Build DeclarationFragments for a variable declaration VarDecl.
   static DeclarationFragments getFragmentsForVar(const VarDecl *);
+
+  /// Build DeclarationFragments for a function declaration FunctionDecl.
   static DeclarationFragments getFragmentsForFunction(const FunctionDecl *);
+
+  /// Build sub-heading fragments for a NamedDecl.
   static DeclarationFragments getSubHeading(const NamedDecl *);
+
+  /// Build FunctionSignature for a function declaration FunctionDecl.
   static FunctionSignature getFunctionSignature(const FunctionDecl *);
 
 private:
   DeclarationFragmentsBuilder() = delete;
 
+  /// Build DeclarationFragments for a QualType.
   static DeclarationFragments getFragmentsForType(const QualType, ASTContext &,
                                                   DeclarationFragments &);
+
+  /// Build DeclarationFragments for a Type.
   static DeclarationFragments getFragmentsForType(const Type *, ASTContext &,
                                                   DeclarationFragments &);
+
+  /// Build DeclarationFragments for a NestedNameSpecifier.
   static DeclarationFragments getFragmentsForNNS(const NestedNameSpecifier *,
                                                  ASTContext &,
                                                  DeclarationFragments &);
+
+  /// Build DeclarationFragments for Qualifiers.
   static DeclarationFragments getFragmentsForQualifiers(const Qualifiers quals);
+
+  /// Build DeclarationFragments for a parameter variable declaration
+  /// ParmVarDecl.
   static DeclarationFragments getFragmentsForParam(const ParmVarDecl *);
 };
 
-} // namespace symbolgraph
+} // namespace extractapi
 } // namespace clang
 
-#endif // LLVM_CLANG_SYMBOLGRAPH_DECLARATION_FRAGMENTS_H
+#endif // LLVM_CLANG_EXTRACTAPI_DECLARATION_FRAGMENTS_H

clang/include/clang/ExtractAPI/FrontendActions.h

-//===- SymbolGraph/FrontendActions.h -----------------------*- C++ -*-===//
+//===- ExtractAPI/FrontendActions.h -----------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 ///
 /// \file
-/// \brief Defines SymbolGraph frontend actions.
+/// This file defines the ExtractAPIAction frontend action.
 ///
 //===----------------------------------------------------------------------===//
 
-#ifndef LLVM_CLANG_SYMBOLGRAPH_FRONTEND_ACTIONS_H
-#define LLVM_CLANG_SYMBOLGRAPH_FRONTEND_ACTIONS_H
+#ifndef LLVM_CLANG_EXTRACTAPI_FRONTEND_ACTIONS_H
+#define LLVM_CLANG_EXTRACTAPI_FRONTEND_ACTIONS_H
 
 #include "clang/Frontend/FrontendAction.h"
 
 namespace clang {
 
+/// ExtractAPIAction sets up the output file and creates the ExtractAPIVisitor.
 class ExtractAPIAction : public ASTFrontendAction {
 protected:
   std::unique_ptr<ASTConsumer> CreateASTConsumer(CompilerInstance &CI,
                                                  StringRef InFile) override;
 
 public:
   static std::unique_ptr<llvm::raw_pwrite_stream>
   CreateOutputFile(CompilerInstance &CI, StringRef InFile);
 };
 
 } // namespace clang
 
-#endif // LLVM_CLANG_SYMBOLGRAPH_FRONTEND_ACTIONS_H
+#endif // LLVM_CLANG_EXTRACTAPI_FRONTEND_ACTIONS_H

clang/include/clang/ExtractAPI/Serialization/SerializerBase.h

+//===- ExtractAPI/Serialization/SerializerBase.h ----------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// This file defines the ExtractAPI APIJSONSerializer interface.
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_EXTRACTAPI_SERIALIZATION_SERIALIZERBASE_H
+#define LLVM_CLANG_EXTRACTAPI_SERIALIZATION_SERIALIZERBASE_H
+
+#include "clang/ExtractAPI/API.h"
+#include "llvm/Support/JSON.h"
+#include "llvm/Support/raw_ostream.h"
+
+namespace clang {
+namespace extractapi {
+
+using namespace llvm::json;
+
+/// Common options to customize the JSON serializer output.
+struct APIJSONSerializerOption {
+  /// Do not include unnecessary whitespaces to save space.
+  bool Compact;
+};
+
+/// The base interface of JSON serializers for API information.
+class APIJSONSerializer {
+public:
+  /// Serialize the API information into a JSON object.
+  virtual Object serialize() = 0;
+
+  /// Wrap \c serialize() and write out the API information to \p os.
+  void serialize(raw_ostream &os);
+
+protected:
+  const APISet &API;
+  APIJSONSerializerOption Options;
+
+public:
+  APIJSONSerializer() = delete;
+  APIJSONSerializer(const APIJSONSerializer &) = delete;
+  APIJSONSerializer(APIJSONSerializer &&) = delete;
+  APIJSONSerializer &operator=(const APIJSONSerializer &) = delete;
+  APIJSONSerializer &operator=(APIJSONSerializer &&) = delete;
+
+protected:
+  APIJSONSerializer(const APISet &API, APIJSONSerializerOption Options = {})
+      : API(API), Options(Options) {}
+
+  virtual ~APIJSONSerializer() = default;
+};
+
+} // namespace extractapi
+} // namespace clang
+
+#endif // LLVM_CLANG_EXTRACTAPI_SERIALIZATION_SERIALIZERBASE_H

clang/include/clang/ExtractAPI/Serialization/SymbolGraphSerializer.h

+//===- ExtractAPI/Serialization/SymbolGraphSerializer.h ---------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// This file defines the SymbolGraphSerializer class.
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_EXTRACTAPI_SERIALIZATION_SYMBOLGRAPHSERIALIZER_H
+#define LLVM_CLANG_EXTRACTAPI_SERIALIZATION_SYMBOLGRAPHSERIALIZER_H
+
+#include "clang/ExtractAPI/API.h"
+#include "clang/ExtractAPI/Serialization/SerializerBase.h"
+#include "llvm/Support/JSON.h"
+#include "llvm/Support/VersionTuple.h"
+#include "llvm/Support/raw_ostream.h"
+
+namespace clang {
+namespace extractapi {
+
+using namespace llvm::json;
+
+/// The serializer that organizes API information in the Symbol Graph format.
+///
+/// The Symbol Graph format (https://github.com/apple/swift-docc-symbolkit)
+/// models an API set as a directed graph, where nodes are symbol declarations,
+/// and edges are relationships between the connected symbols.
+class SymbolGraphSerializer : public APIJSONSerializer {
+  virtual void anchor();
+
+  /// A JSON array of formatted symbols in \c APISet.
+  Array Symbols;
+
+  /// A JSON array of formatted symbol relationships in \c APISet.
+  Array Relationships;
+
+  /// The Symbol Graph format version used by this serializer.
+  static const VersionTuple FormatVersion;
+
+public:
+  /// Bring in the \c serialize(raw_ostream &) helper method from base.
+  using APIJSONSerializer::serialize;
+
+  /// Serialize the APIs in \c APISet in the Symbol Graph format.
+  ///
+  /// \returns a JSON object that contains the root of the formatted
+  /// Symbol Graph.
+  Object serialize() override;
+
+private:
+  /// Synthesize the metadata section of the Symbol Graph format.
+  ///
+  /// The metadata section describes information about the Symbol Graph itself,
+  /// including the format version and the generator information.
+  Object serializeMetadata() const;
+
+  /// Synthesize the module section of the Symbol Graph format.
+  ///
+  /// The module section contains information about the product that is defined
+  /// by the given API set.
+  /// Note that "module" here is not to be confused with the Clang/C++ module
+  /// concept.
+  Object serializeModule() const;
+
+  /// Determine if the given \p Record should be skipped during serialization.
+  bool shouldSkip(const APIRecord &Record) const;
+
+  /// Format the common API information for \p Record.
+  ///
+  /// This handles the shared information of all kinds of API records,
+  /// for example identifier and source location. The resulting object is then
+  /// augmented with kind-specific symbol information by the caller.
+  /// This method also checks if the given \p Record should be skipped during
+  /// serialization.
+  ///
+  /// \returns \c None if this \p Record should be skipped, or a JSON object
+  /// containing common symbol information of \p Record.
+  Optional<Object> serializeAPIRecord(const APIRecord &Record) const;
+
+  /// Serialize a global record.
+  void serializeGlobalRecord(const GlobalRecord &Record);
+
+public:
+  SymbolGraphSerializer(const APISet &API, APIJSONSerializerOption Options = {})
+      : APIJSONSerializer(API, Options) {}
+};
+
+} // namespace extractapi
+} // namespace clang
+
+#endif // LLVM_CLANG_EXTRACTAPI_SERIALIZATION_SYMBOLGRAPHSERIALIZER_H

clang/include/clang/SymbolGraph/Serialization.h

-//===- SymbolGraph/Serialization.h ------------------------------*- C++ -*-===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-///
-/// \file
-/// \brief Defines the SymbolGraph serializer and parser.
-///
-//===----------------------------------------------------------------------===//
-
-#ifndef LLVM_CLANG_SYMBOLGRAPH_SERIALIZATION_H
-#define LLVM_CLANG_SYMBOLGRAPH_SERIALIZATION_H
-
-#include "clang/SymbolGraph/API.h"
-#include "llvm/Support/JSON.h"
-#include "llvm/Support/VersionTuple.h"
-#include "llvm/Support/raw_ostream.h"
-
-namespace clang {
-namespace symbolgraph {
-
-using namespace llvm::json;
-
-struct SerializerOption {
-  bool Compact;
-};
-
-class Serializer {
-public:
-  Serializer(const APISet &API, SerializerOption Options = {})
-      : API(API), Options(Options) {}
-
-  Object serialize();
-  void serialize(raw_ostream &os);
-
-private:
-  Object serializeMetadata() const;
-  Object serializeModule() const;
-  Optional<Object> serializeAPIRecord(const APIRecord &Record) const;
-  void serializeGlobalRecord(const GlobalRecord &Record);
-
-  bool shouldSkip(const APIRecord &Record) const;
-
-  const APISet &API;
-  SerializerOption Options;
-  Array Symbols;
-  Array Relationships;
-
-  static const VersionTuple FormatVersion;
-};
-
-} // namespace symbolgraph
-} // namespace clang
-
-#endif // LLVM_CLANG_SYMBOLGRAPH_SERIALIZATION_H

clang/lib/CMakeLists.txt

 add_subdirectory(Headers)
 add_subdirectory(Basic)
 add_subdirectory(APINotes)
 add_subdirectory(Lex)
 add_subdirectory(Parse)
 add_subdirectory(AST)
 add_subdirectory(ASTMatchers)
 add_subdirectory(CrossTU)
 add_subdirectory(Sema)
 add_subdirectory(CodeGen)
 add_subdirectory(Analysis)
 add_subdirectory(Edit)
+add_subdirectory(ExtractAPI)
 add_subdirectory(Rewrite)
 if(CLANG_ENABLE_ARCMT)
   add_subdirectory(ARCMigrate)
 endif()
 add_subdirectory(Driver)
 add_subdirectory(Serialization)
 add_subdirectory(Frontend)
 add_subdirectory(FrontendTool)
 add_subdirectory(Tooling)
 add_subdirectory(DirectoryWatcher)
 add_subdirectory(Index)
 add_subdirectory(IndexSerialization)
 add_subdirectory(StaticAnalyzer)
-add_subdirectory(SymbolGraph)
 add_subdirectory(Format)
 add_subdirectory(Testing)
 add_subdirectory(Interpreter)

clang/lib/ExtractAPI/API.cpp

-//===- SymbolGraph/API.cpp --------------------------------------*- C++ -*-===//
+//===- ExtractAPI/API.cpp ---------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 ///
 /// \file
-/// \brief Defines SymbolGraph API records.
+/// This file implements the APIRecord and derived record structs,
+/// and the APISet class.
 ///
 //===----------------------------------------------------------------------===//
 
-#include "clang/SymbolGraph/API.h"
+#include "clang/ExtractAPI/API.h"
 #include "clang/AST/CommentCommandTraits.h"
 #include "clang/AST/CommentLexer.h"
 #include "clang/AST/RawCommentList.h"
 #include "clang/Index/USRGeneration.h"
 #include "llvm/Support/Allocator.h"
 
-namespace clang {
-namespace symbolgraph {
-
-APIRecord::~APIRecord() {}
+using namespace clang::extractapi;
+using namespace llvm;
 
 GlobalRecord *APISet::addGlobal(GVKind Kind, StringRef Name, StringRef USR,
                                 PresumedLoc Loc,
                                 const AvailabilityInfo &Availability,
                                 LinkageInfo Linkage, const DocComment &Comment,
                                 DeclarationFragments Fragments,
                                 DeclarationFragments SubHeading,
                                 FunctionSignature Signature) {
   auto Result = Globals.insert({Name, nullptr});
   if (Result.second) {
+    // Create the record if not already exist.
     auto Record = APIRecordUniquePtr<GlobalRecord>(new (Allocator) GlobalRecord{
         Kind, Name, USR, Loc, Availability, Linkage, Comment, Fragments,
         SubHeading, Signature});
     Result.first->second = std::move(Record);
   }
   return Result.first->second.get();
 }
 
StringRef APISet::recordUSR(const Decl *D) {
   return copyString(USR);
 }
 
 StringRef APISet::copyString(StringRef String,
                              llvm::BumpPtrAllocator &Allocator) {
   if (String.empty())
     return {};
 
+  // No need to allocate memory and copy if the string has already been stored.
   if (Allocator.identifyObject(String.data()))
     return String;
 
   void *Ptr = Allocator.Allocate(String.size(), 1);
   memcpy(Ptr, String.data(), String.size());
   return StringRef(reinterpret_cast<const char *>(Ptr), String.size());
 }
 
 StringRef APISet::copyString(StringRef String) {
   return copyString(String, Allocator);
 }
 
-} // namespace symbolgraph
-} // namespace clang
+APIRecord::~APIRecord() {}
+
+void GlobalRecord::anchor() {}

clang/lib/ExtractAPI/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   Support
   )
 
-add_clang_library(clangSymbolGraph
+add_clang_library(clangExtractAPI
   API.cpp
   ExtractAPIConsumer.cpp
   DeclarationFragments.cpp
-  Serialization.cpp
+  Serialization/SerializerBase.cpp
+  Serialization/SymbolGraphSerializer.cpp
 
   LINK_LIBS
   clangAST
   clangBasic
   clangFrontend
   clangIndex
   )

clang/lib/ExtractAPI/DeclarationFragments.cpp

-//===- SymbolGraph/DeclarationFragments.cpp ---------------------*- C++ -*-===//
+//===- ExtractAPI/DeclarationFragments.cpp ----------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 ///
 /// \file
-/// \brief Defines SymbolGraph Declaration Fragments related classes.
+/// This file implements Declaration Fragments related classes.
 ///
 //===----------------------------------------------------------------------===//
 
-#include "clang/SymbolGraph/DeclarationFragments.h"
+#include "clang/ExtractAPI/DeclarationFragments.h"
 #include "clang/Index/USRGeneration.h"
 #include "llvm/ADT/StringSwitch.h"
 
-namespace clang {
-namespace symbolgraph {
+using namespace clang::extractapi;
+using namespace llvm;
 
 DeclarationFragments &DeclarationFragments::appendSpace() {
   if (!Fragments.empty()) {
     Fragment Last = Fragments.back();
     if (Last.Kind == FragmentKind::Text) {
-      if (Last.Spelling.back() != ' ') {
+      // Merge the extra space into the last fragment if the last fragment is
+      // also text.
+      if (Last.Spelling.back() != ' ') { // avoid extra trailing spaces.
         Last.Spelling.push_back(' ');
       }
     } else {
       append(" ", FragmentKind::Text);
     }
   }
 
   return *this;

 DeclarationFragments
 DeclarationFragmentsBuilder::getSubHeading(const NamedDecl *Decl) {
   DeclarationFragments Fragments;
   if (!Decl->getName().empty())
     Fragments.append(Decl->getName(),
                      DeclarationFragments::FragmentKind::Identifier);
   return Fragments;
 }
-
-} // namespace symbolgraph
-} // namespace clang

clang/lib/ExtractAPI/ExtractAPIConsumer.cpp

-//===- ExtractAPIConsumer.cpp -----------------------------------*- C++ -*-===//
+//===- ExtractAPI/ExtractAPIConsumer.cpp ------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 ///
 /// \file
-/// \brief Defines the ExtractAPI AST visitor to collect API information.
+/// This file implements the ExtractAPIAction, and ASTVisitor/Consumer to
+/// collect API information.
 ///
 //===----------------------------------------------------------------------===//
-//
 
 #include "clang/AST/ASTConsumer.h"
 #include "clang/AST/ASTContext.h"
 #include "clang/AST/Decl.h"
 #include "clang/AST/DeclCXX.h"
 #include "clang/AST/ParentMapContext.h"
 #include "clang/AST/RawCommentList.h"
 #include "clang/AST/RecursiveASTVisitor.h"
 #include "clang/Basic/TargetInfo.h"
+#include "clang/ExtractAPI/API.h"
+#include "clang/ExtractAPI/AvailabilityInfo.h"
+#include "clang/ExtractAPI/DeclarationFragments.h"
+#include "clang/ExtractAPI/FrontendActions.h"
+#include "clang/ExtractAPI/Serialization/SymbolGraphSerializer.h"
 #include "clang/Frontend/ASTConsumers.h"
 #include "clang/Frontend/CompilerInstance.h"
-#include "clang/SymbolGraph/API.h"
-#include "clang/SymbolGraph/AvailabilityInfo.h"
-#include "clang/SymbolGraph/DeclarationFragments.h"
-#include "clang/SymbolGraph/FrontendActions.h"
-#include "clang/SymbolGraph/Serialization.h"
 #include "llvm/Support/raw_ostream.h"
 
 using namespace clang;
-using namespace symbolgraph;
+using namespace extractapi;
 
 namespace {
+
+/// The RecursiveASTVisitor to traverse symbol declarations and collect API
+/// information.
 class ExtractAPIVisitor : public RecursiveASTVisitor<ExtractAPIVisitor> {
 public:
   explicit ExtractAPIVisitor(ASTContext &Context)
       : Context(Context),
         API(Context.getTargetInfo().getTriple(), Context.getLangOpts()) {}
 
   const APISet &getAPI() const { return API; }
 
bool VisitVarDecl(const VarDecl *Decl) {
     if (!Decl->isDefinedOutsideFunctionOrMethod())
       return true;
 
     // If this is a template but not specialization or instantiation, skip.
     if (Decl->getASTContext().getTemplateOrSpecializationInfo(Decl) &&
         Decl->getTemplateSpecializationKind() == TSK_Undeclared)
       return true;
 
+    // Collect symbol information.
     StringRef Name = Decl->getName();
     StringRef USR = API.recordUSR(Decl);
     PresumedLoc Loc =
         Context.getSourceManager().getPresumedLoc(Decl->getLocation());
     AvailabilityInfo Availability = getAvailability(Decl);
     LinkageInfo Linkage = Decl->getLinkageAndVisibility();
     DocComment Comment;
     if (auto *RawComment = Context.getRawCommentForDeclNoCache(Decl))
       Comment = RawComment->getFormattedLines(Context.getSourceManager(),
                                               Context.getDiagnostics());
+
+    // Build declaration fragments and sub-heading for the variable.
     DeclarationFragments Declaration =
         DeclarationFragmentsBuilder::getFragmentsForVar(Decl);
     DeclarationFragments SubHeading =
         DeclarationFragmentsBuilder::getSubHeading(Decl);
 
+    // Add the global variable record to the API set.
     API.addGlobalVar(Name, USR, Loc, Availability, Linkage, Comment,
                      Declaration, SubHeading);
     return true;
   }
 
   bool VisitFunctionDecl(const FunctionDecl *Decl) {
     if (const auto *Method = dyn_cast<CXXMethodDecl>(Decl)) {
       // Skip member function in class templates.
case FunctionDecl::TK_FunctionTemplateSpecialization:
           return true;
       }
       break;
     case FunctionDecl::TK_FunctionTemplate:
     case FunctionDecl::TK_DependentFunctionTemplateSpecialization:
       return true;
     }
 
+    // Collect symbol information.
     StringRef Name = Decl->getName();
     StringRef USR = API.recordUSR(Decl);
     PresumedLoc Loc =
         Context.getSourceManager().getPresumedLoc(Decl->getLocation());
     AvailabilityInfo Availability = getAvailability(Decl);
     LinkageInfo Linkage = Decl->getLinkageAndVisibility();
     DocComment Comment;
     if (auto *RawComment = Context.getRawCommentForDeclNoCache(Decl))
       Comment = RawComment->getFormattedLines(Context.getSourceManager(),
                                               Context.getDiagnostics());
+
+    // Build declaration fragments, sub-heading, and signature of the function.
     DeclarationFragments Declaration =
         DeclarationFragmentsBuilder::getFragmentsForFunction(Decl);
     DeclarationFragments SubHeading =
         DeclarationFragmentsBuilder::getSubHeading(Decl);
     FunctionSignature Signature =
         DeclarationFragmentsBuilder::getFunctionSignature(Decl);
 
+    // Add the function record to the API set.
     API.addFunction(Name, USR, Loc, Availability, Linkage, Comment, Declaration,
                     SubHeading, Signature);
     return true;
   }
 
 private:
+  /// Get availability information of the declaration \p D.
   AvailabilityInfo getAvailability(const Decl *D) const {
     StringRef PlatformName = Context.getTargetInfo().getPlatformName();
 
     AvailabilityInfo Availability;
+    // Collect availability attributes from all redeclarations.
     for (const auto *RD : D->redecls()) {
       for (const auto *A : RD->specific_attrs<AvailabilityAttr>()) {
         if (A->getPlatform()->getName() != PlatformName)
           continue;
         Availability = AvailabilityInfo(A->getIntroduced(), A->getDeprecated(),
                                         A->getObsoleted(), A->getUnavailable(),
                                         /* UnconditionallyDeprecated */ false,
                                         /* UnconditionallyUnavailable */ false);

 };
 
 class ExtractAPIConsumer : public ASTConsumer {
 public:
   ExtractAPIConsumer(ASTContext &Context, std::unique_ptr<raw_pwrite_stream> OS)
       : Visitor(Context), OS(std::move(OS)) {}
 
   void HandleTranslationUnit(ASTContext &Context) override {
+    // Use ExtractAPIVisitor to traverse symbol declarations in the context.
     Visitor.TraverseDecl(Context.getTranslationUnitDecl());
-    Serializer Serializer(Visitor.getAPI());
-    Serializer.serialize(*OS);
+
+    // Setup a SymbolGraphSerializer to write out collected API information in
+    // the Symbol Graph format.
+    SymbolGraphSerializer SGSerializer(Visitor.getAPI());
+    SGSerializer.serialize(*OS);
   }
 
 private:
   ExtractAPIVisitor Visitor;
   std::unique_ptr<raw_pwrite_stream> OS;
 };
+
 } // namespace
 
 std::unique_ptr<ASTConsumer>
 ExtractAPIAction::CreateASTConsumer(CompilerInstance &CI, StringRef InFile) {
   std::unique_ptr<raw_pwrite_stream> OS = CreateOutputFile(CI, InFile);
   if (!OS)
     return nullptr;
   return std::make_unique<ExtractAPIConsumer>(CI.getASTContext(),

clang/lib/ExtractAPI/Serialization/SerializerBase.cpp

+//===- ExtractAPI/Serialization/SerializerBase.cpp --------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// This file implements the APIJSONSerializer interface.
+///
+//===----------------------------------------------------------------------===//
+
+#include "clang/ExtractAPI/Serialization/SerializerBase.h"
+#include "llvm/Support/JSON.h"
+#include "llvm/Support/raw_ostream.h"
+
+using namespace clang::extractapi;
+
+void APIJSONSerializer::serialize(llvm::raw_ostream &os) {
+  llvm::json::Object root = serialize();
+  if (Options.Compact)
+    os << llvm::formatv("{0}", Value(std::move(root))) << "\n";
+  else
+    os << llvm::formatv("{0:2}", Value(std::move(root))) << "\n";
+}

clang/lib/ExtractAPI/Serialization/SymbolGraphSerializer.cpp

-//===- SymbolGraph/Serialization.cpp ----------------------------*- C++ -*-===//
+//===- ExtractAPI/Serialization/SymbolGraphSerializer.cpp -------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 ///
 /// \file
-/// \brief Defines the SymbolGraph serializer and parser.
+/// This file implements the SymbolGraphSerializer.
 ///
 //===----------------------------------------------------------------------===//
 
-#include "clang/SymbolGraph/Serialization.h"
+#include "clang/ExtractAPI/Serialization/SymbolGraphSerializer.h"
 #include "clang/Basic/Version.h"
-#include "clang/SymbolGraph/API.h"
+#include "clang/ExtractAPI/API.h"
 #include "llvm/Support/JSON.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/VersionTuple.h"
-#include "llvm/Support/raw_ostream.h"
 
 using namespace clang;
-using namespace clang::symbolgraph;
+using namespace clang::extractapi;
 using namespace llvm;
 using namespace llvm::json;
 
-namespace {
+namespace { // Helper functions.
 
+/// Helper function to inject a JSON object \p Obj into another object \p Paren
+/// at position \p Key.
 static void serializeObject(Object &Paren, StringRef Key,
                             Optional<Object> Obj) {
   if (Obj)
     Paren[Key] = std::move(Obj.getValue());
 }
 
+/// Helper function to inject a JSON array \p Array into object \p Paren at
+/// position \p Key.
 static void serializeArray(Object &Paren, StringRef Key,
                            Optional<Array> Array) {
   if (Array)
     Paren[Key] = std::move(Array.getValue());
 }
 
-// SymbolGraph: SemanticVersion
+/// Serialize a \c VersionTuple \p V with the Symbol Graph semantic version
+/// format.
+///
+/// A semantic version object contains three numeric fields, representing the
+/// \c major, \c minor, and \c patch parts of the version tuple.
+/// For example version tuple 1.0.3 is serialized as:
+/// \code
+///   {
+///     "major" : 1,
+///     "minor" : 0,
+///     "patch" : 3
+///   }
+/// \endcode
+///
+/// \returns \c None if the version \p V is empty, or an \c Object containing
+/// the semantic version representation of \p V.
 static Optional<Object> serializeSemanticVersion(const VersionTuple &V) {
   if (V.empty())
     return None;
 
   Object Version;
   Version["major"] = V.getMajor();
   Version["minor"] = V.getMinor().getValueOr(0);
   Version["patch"] = V.getSubminor().getValueOr(0);
   return Version;
 }
 
+/// Serialize the OS information in the Symbol Graph platform property.
+///
+/// The OS information in Symbol Graph contains the \c name of the OS, and an
+/// optional \c minimumVersion semantic version field.
 static Object serializeOperatingSystem(const Triple &T) {
   Object OS;
   OS["name"] = T.getOSTypeName(T.getOS());
   serializeObject(OS, "minimumVersion",
                   serializeSemanticVersion(T.getMinimumSupportedOSVersion()));
   return OS;
 }
 
-// SymbolGraph: Platform
+/// Serialize the platform information in the Symbol Graph module section.
+///
+/// The platform object describes a target platform triple in corresponding
+/// three fields: \c architecture, \c vendor, and \c operatingSystem.
 static Object serializePlatform(const Triple &T) {
   Object Platform;
   Platform["architecture"] = T.getArchName();
   Platform["vendor"] = T.getVendorName();
   Platform["operatingSystem"] = serializeOperatingSystem(T);
   return Platform;
 }
 
-// SymbolGraph: SourcePosition
+/// Serialize a source location in file.
+///
+/// \param Loc The presumed location to serialize.
+/// \param IncludeFileURI If true, include the file path of \p Loc as a URI.
+/// Defaults to false.
 static Object serializeSourcePosition(const PresumedLoc &Loc,
                                       bool IncludeFileURI = false) {
   assert(Loc.isValid() && "invalid source position");
 
   Object SourcePosition;
   SourcePosition["line"] = Loc.getLine();
   SourcePosition["character"] = Loc.getColumn();
 
   if (IncludeFileURI) {
     std::string FileURI = "file://";
+    // Normalize file path to use forward slashes for the URI.
     FileURI += sys::path::convert_to_slash(Loc.getFilename());
     SourcePosition["uri"] = FileURI;
   }
 
   return SourcePosition;
 }
 
-// SymbolGraph: SourceRange
+/// Serialize a source range with begin and end locations.
 static Object serializeSourceRange(const PresumedLoc &BeginLoc,
                                    const PresumedLoc &EndLoc) {
   Object SourceRange;
   serializeObject(SourceRange, "start", serializeSourcePosition(BeginLoc));
   serializeObject(SourceRange, "end", serializeSourcePosition(EndLoc));
   return SourceRange;
 }
 
-// SymbolGraph: AvailabilityItem
+/// Serialize the availability attributes of a symbol.
+///
+/// Availability information contains the introduced, deprecated, and obsoleted
+/// versions of the symbol as semantic versions, if not default.
+/// Availability information also contains flags to indicate if the symbol is
+/// unconditionally unavailable or deprecated,
+/// i.e. \c __attribute__((unavailable)) and \c __attribute__((deprecated)).
+///
+/// \returns \c None if the symbol has default availability attributes, or
+/// an \c Object containing the formatted availability information.
 static Optional<Object> serializeAvailability(const AvailabilityInfo &Avail) {
   if (Avail.isDefault())
     return None;
 
   Object Availbility;
   serializeObject(Availbility, "introducedVersion",
                   serializeSemanticVersion(Avail.Introduced));
   serializeObject(Availbility, "deprecatedVersion",
                   serializeSemanticVersion(Avail.Deprecated));
   serializeObject(Availbility, "obsoletedVersion",
                   serializeSemanticVersion(Avail.Obsoleted));
   if (Avail.isUnavailable())
     Availbility["isUnconditionallyUnavailable"] = true;
   if (Avail.isUnconditionallyDeprecated())
     Availbility["isUnconditionallyDeprecated"] = true;
 
   return Availbility;
 }
 
+/// Get the short language name string for interface language references.
 static StringRef getLanguageName(const LangOptions &LangOpts) {
   auto Language =
       LangStandard::getLangStandardForKind(LangOpts.LangStd).getLanguage();
   switch (Language) {
   case Language::C:
     return "c";
   case Language::ObjC:
     return "objc";
 
   // Unsupported language currently
   case Language::CXX:
static StringRef getLanguageName(const LangOptions &LangOpts) {
   case Language::Asm:
   case Language::LLVM_IR:
     llvm_unreachable("Unsupported language kind");
   }
 
   llvm_unreachable("Unhandled language kind");
 }
 
-// SymbolGraph: Symbol::identifier
+/// Serialize the identifier object as specified by the Symbol Graph format.
+///
+/// The identifier property of a symbol contains the USR for precise and unique
+/// references, and the interface language name.
 static Object serializeIdentifier(const APIRecord &Record,
                                   const LangOptions &LangOpts) {
   Object Identifier;
   Identifier["precise"] = Record.USR;
   Identifier["interfaceLanguage"] = getLanguageName(LangOpts);
 
   return Identifier;
 }
 
-// SymbolGraph: DocComment
+/// Serialize the documentation comments attached to a symbol, as specified by
+/// the Symbol Graph format.
+///
+/// The Symbol Graph \c docComment object contains an array of lines. Each line
+/// represents one line of striped documentation comment, with source range
+/// information.
+/// e.g.
+/// \code
+///   /// This is a documentation comment
+///       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~'  First line.
+///   ///     with multiple lines.
+///       ^~~~~~~~~~~~~~~~~~~~~~~'         Second line.
+/// \endcode
+///
+/// \returns \c None if \p Comment is empty, or an \c Object containing the
+/// formatted lines.
 static Optional<Object> serializeDocComment(const DocComment &Comment) {
   if (Comment.empty())
     return None;
 
   Object DocComment;
   Array LinesArray;
   for (const auto &CommentLine : Comment) {
     Object Line;
     Line["text"] = CommentLine.Text;
     serializeObject(Line, "range",
                     serializeSourceRange(CommentLine.Begin, CommentLine.End));
     LinesArray.emplace_back(std::move(Line));
   }
   serializeArray(DocComment, "lines", LinesArray);
 
   return DocComment;
 }
 
+/// Serialize the declaration fragments of a symbol.
+///
+/// The Symbol Graph declaration fragments is an array of tagged important
+/// parts of a symbol's declaration. The fragments sequence can be joined to
+/// form spans of declaration text, with attached information useful for
+/// purposes like syntax-highlighting etc. For example:
+/// \code
+///   const int pi; -> "declarationFragments" : [
+///                      {
+///                        "kind" : "keyword",
+///                        "spelling" : "const"
+///                      },
+///                      {
+///                        "kind" : "text",
+///                        "spelling" : " "
+///                      },
+///                      {
+///                        "kind" : "typeIdentifier",
+///                        "preciseIdentifier" : "c:I",
+///                        "spelling" : "int"
+///                      },
+///                      {
+///                        "kind" : "text",
+///                        "spelling" : " "
+///                      },
+///                      {
+///                        "kind" : "identifier",
+///                        "spelling" : "pi"
+///                      }
+///                    ]
+/// \endcode
+///
+/// \returns \c None if \p DF is empty, or an \c Array containing the formatted
+/// declaration fragments array.
 static Optional<Array>
 serializeDeclarationFragments(const DeclarationFragments &DF) {
   if (DF.getFragments().empty())
     return None;
 
   Array Fragments;
   for (const auto &F : DF.getFragments()) {
     Object Fragment;
     Fragment["spelling"] = F.Spelling;
     Fragment["kind"] = DeclarationFragments::getFragmentKindString(F.Kind);
     if (!F.PreciseIdentifier.empty())
       Fragment["preciseIdentifier"] = F.PreciseIdentifier;
     Fragments.emplace_back(std::move(Fragment));
   }
 
   return Fragments;
 }
 
+/// Serialize the function signature field of a function, as specified by the
+/// Symbol Graph format.
+///
+/// The Symbol Graph function signature property contains two arrays.
+///   - The \c returns array is the declaration fragments of the return type;
+///   - The \c parameters array contains names and declaration fragments of the
+///     parameters.
+///
+/// \returns \c None if \p FS is empty, or an \c Object containing the
+/// formatted function signature.
 static Optional<Object>
 serializeFunctionSignature(const FunctionSignature &FS) {
   if (FS.empty())
     return None;
 
   Object Signature;
   serializeArray(Signature, "returns",
                  serializeDeclarationFragments(FS.getReturnType()));
 
   Array Parameters;
   for (const auto &P : FS.getParameters()) {
     Object Parameter;
     Parameter["name"] = P.Name;
     serializeArray(Parameter, "declarationFragments",
                    serializeDeclarationFragments(P.Fragments));
     Parameters.emplace_back(std::move(Parameter));
   }
 
   if (!Parameters.empty())
     Signature["parameters"] = std::move(Parameters);
 
   return Signature;
 }
 
+/// Serialize the \c names field of a symbol as specified by the Symbol Graph
+/// format.
+///
+/// The Symbol Graph names field contains multiple representations of a symbol
+/// that can be used for different applications:
+///   - \c title : The simple declared name of the symbol;
+///   - \c subHeading : An array of declaration fragments that provides tags,
+///     and potentially more tokens (for example the \c +/- symbol for
+///     Objective-C methods). Can be used as sub-headings for documentation.
 static Object serializeNames(const APIRecord &Record) {
   Object Names;
   Names["title"] = Record.Name;
   serializeArray(Names, "subHeading",
                  serializeDeclarationFragments(Record.SubHeading));
 
   return Names;
 }
 
-// SymbolGraph: Symbol::kind
+/// Serialize the symbol kind information.
+///
+/// The Symbol Graph symbol kind property contains a shorthand \c identifier
+/// which is prefixed by the source language name, useful for tooling to parse
+/// the kind, and a \c displayName for rendering human-readable names.
 static Object serializeSymbolKind(const APIRecord &Record,
                                   const LangOptions &LangOpts) {
   Object Kind;
   switch (Record.getKind()) {
   case APIRecord::RK_Global:
     auto *GR = dyn_cast<GlobalRecord>(&Record);
     switch (GR->GlobalKind) {
     case GVKind::Function:
case APIRecord::RK_Global:
     break;
   }
 
   return Kind;
 }
 
 } // namespace
 
-const VersionTuple Serializer::FormatVersion{0, 5, 3};
+void SymbolGraphSerializer::anchor() {}
+
+/// Defines the format version emitted by SymbolGraphSerializer.
+const VersionTuple SymbolGraphSerializer::FormatVersion{0, 5, 3};
 
-Object Serializer::serializeMetadata() const {
+Object SymbolGraphSerializer::serializeMetadata() const {
   Object Metadata;
   serializeObject(Metadata, "formatVersion",
                   serializeSemanticVersion(FormatVersion));
   Metadata["generator"] = clang::getClangFullVersion();
   return Metadata;
 }
 
-Object Serializer::serializeModule() const {
+Object SymbolGraphSerializer::serializeModule() const {
   Object Module;
-  // FIXME: What to put in here?
+  // FIXME: We might not be building a module, some Clang-based languages might
+  // not have a "module" concept. Figure out a way to provide a name to
+  // describe the API set.
   Module["name"] = "";
   serializeObject(Module, "platform", serializePlatform(API.getTarget()));
   return Module;
 }
 
-bool Serializer::shouldSkip(const APIRecord &Record) const {
+bool SymbolGraphSerializer::shouldSkip(const APIRecord &Record) const {
   // Skip unconditionally unavailable symbols
   if (Record.Availability.isUnconditionallyUnavailable())
     return true;
 
   return false;
 }
 
-Optional<Object> Serializer::serializeAPIRecord(const APIRecord &Record) const {
+Optional<Object>
+SymbolGraphSerializer::serializeAPIRecord(const APIRecord &Record) const {
   if (shouldSkip(Record))
     return None;
 
   Object Obj;
   serializeObject(Obj, "identifier",
                   serializeIdentifier(Record, API.getLangOpts()));
   serializeObject(Obj, "kind", serializeSymbolKind(Record, API.getLangOpts()));
   serializeObject(Obj, "names", serializeNames(Record));
   serializeObject(
       Obj, "location",
       serializeSourcePosition(Record.Location, /*IncludeFileURI=*/true));
   serializeObject(Obj, "availbility",
                   serializeAvailability(Record.Availability));
   serializeObject(Obj, "docComment", serializeDocComment(Record.Comment));
   serializeArray(Obj, "declarationFragments",
                  serializeDeclarationFragments(Record.Declaration));
 
   return Obj;
 }
 
-void Serializer::serializeGlobalRecord(const GlobalRecord &Record) {
+void SymbolGraphSerializer::serializeGlobalRecord(const GlobalRecord &Record) {
   auto Obj = serializeAPIRecord(Record);
   if (!Obj)
     return;
 
   if (Record.GlobalKind == GVKind::Function)
     serializeObject(*Obj, "parameters",
                     serializeFunctionSignature(Record.Signature));
 
   Symbols.emplace_back(std::move(*Obj));
 }
 
-Object Serializer::serialize() {
+Object SymbolGraphSerializer::serialize() {
   Object Root;
   serializeObject(Root, "metadata", serializeMetadata());
   serializeObject(Root, "module", serializeModule());
 
+  // Serialize global records in the API set.
   for (const auto &Global : API.getGlobals())
     serializeGlobalRecord(*Global.second);
 
   Root["symbols"] = std::move(Symbols);
   Root["relationhips"] = std::move(Relationships);
 
   return Root;
 }
-
-void Serializer::serialize(raw_ostream &os) {
-  Object root = serialize();
-  if (Options.Compact)
-    os << formatv("{0}", Value(std::move(root))) << "\n";
-  else
-    os << formatv("{0:2}", Value(std::move(root))) << "\n";
-}

clang/lib/FrontendTool/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   Option
   Support
   )
 
 set(link_libs
   clangBasic
   clangCodeGen
   clangDriver
+  clangExtractAPI
   clangFrontend
   clangRewriteFrontend
-  clangSymbolGraph
   )
 
 if(CLANG_ENABLE_ARCMT)
   list(APPEND link_libs
     clangARCMigrate
     )
 endif()
 

clang/lib/FrontendTool/ExecuteCompilerInvocation.cpp

 // minimize the impact of pulling in essentially everything else in Clang.
 //
 //===----------------------------------------------------------------------===//
 
 #include "clang/ARCMigrate/ARCMTActions.h"
 #include "clang/CodeGen/CodeGenAction.h"
 #include "clang/Config/config.h"
 #include "clang/Driver/Options.h"
+#include "clang/ExtractAPI/FrontendActions.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/CompilerInvocation.h"
 #include "clang/Frontend/FrontendActions.h"
 #include "clang/Frontend/FrontendDiagnostic.h"
 #include "clang/Frontend/FrontendPluginRegistry.h"
 #include "clang/Frontend/Utils.h"
 #include "clang/FrontendTool/Utils.h"
 #include "clang/Rewrite/Frontend/FrontendActions.h"
 #include "clang/StaticAnalyzer/Frontend/AnalyzerHelpFlags.h"
 #include "clang/StaticAnalyzer/Frontend/FrontendActions.h"
-#include "clang/SymbolGraph/FrontendActions.h"
 #include "llvm/Option/OptTable.h"
 #include "llvm/Option/Option.h"
 #include "llvm/Support/BuryPointer.h"
 #include "llvm/Support/DynamicLibrary.h"
 #include "llvm/Support/ErrorHandling.h"
 using namespace clang;
 using namespace llvm::opt;