[clang] Add an extract-api driver option

[zixuw]

Summary

This is the initial commit for the clang-extract-api RFC
https://lists.llvm.org/pipermail/cfe-dev/2021-September/068768.html
Add a new driver option -extract-api and associate it with a dummy
(for now) frontend action to set up the initial structure for
incremental works.

[ributzka]

LGTM. Just a few minor nits.

clang/lib/Driver/ToolChains/Clang.cpp

7082

Is this change required for extract-api? I don't think the output file needs to be renamed, because there is only one output in this case.

clang/lib/Frontend/CompilerInvocation.cpp

2431

This looks like an unrelated change.

[zixuw]

clang/lib/Frontend/CompilerInvocation.cpp

2431

Result of clang-format, I can restore this line

[zixuw]

• Remove unrelated whitespace change
• Remove unnecessary code

[QuietMisdreavus]

clang/include/clang/Driver/Types.def

103

Symbol graph files generally have the extension .symbols.json - is that something that should be reflected here?

[zixuw]

clang/include/clang/Driver/Types.def

103

Actually I don't know. Looking at other types in this file, I don't see any existing ones with "two level" suffixes and I don't know if it makes sense (or be any useful) to put that here. The output file name (including the suffix) is still whatever the user put after the -o option. From the comment at the beginning of this file, this suffix here is only used for temporary files created, if any would be during the extract-api pipeline.

[zixuw]

Pre-merge check failure with clang-format complaints :(

[zixuw]

Ping for any additional reviews/comments.
I must admit that I got really confused and lost in the layers of the driver and setting up the phases. I wasn't sure if I made enough (or too many?) changes to make this work. So any comments or advice are welcomed.

[morehouse]

clang/lib/Frontend/ExtractAPIConsumer.cpp

21

ASTContext *Context is unused, and causing buildbot failures: https://lab.llvm.org/buildbot/#/builders/77/builds/13645

/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/clang/lib/Frontend/ExtractAPIConsumer.cpp:20:15: error: private field 'Context' is not used [-Werror,-Wunused-private-field]
  ASTContext *Context;
              ^
1 error generated.

Please fix.

[zixuw]

On it

[zixuw]

@morehouse should be fixed in 98fa46f870e4

[MaskRay]

Thanks for working on such tools but the patch order is not right.
You should implement the functionality first, and in the last, add the driver option.
The driver option is user-facing and the availability makes users think the functionality is ready when it actually isn't.

I am also a bit unsure whether the option name should use the single-dash -e* since -e is a short option taking a value, so a typo in -extract-api cannot be detected.
By convention a two-dash option should be used.
(Yes, I know -emit-llvm is a violation.)

[zixuw]

In D117809#3306785, @MaskRay wrote:

Thanks for working on such tools but the patch order is not right.
You should implement the functionality first, and in the last, add the driver option.
The driver option is user-facing and the availability makes users think the functionality is ready when it actually isn't.

I see. I was working on the functionality at the same time with this driver option, but it took longer and more complicated than I expected so I decided to post this as an initial patch to keep things small and incremental. I do have another patch ready to post very shortly.

I am also a bit unsure whether the option name should use the single-dash -e* since -e is a short option taking a value, so a typo in -extract-api cannot be detected.
By convention a two-dash option should be used.
(Yes, I know -emit-llvm is a violation.)

That's an interesting find, I'll look into it.

clang/include/clang/Driver/Options.td

def emit_interface_stubs : Flag<["-"], "emit-interface-stubs">, Flags<[CC1Option]>, Group<Action_Group>,
   HelpText<"Generate Interface Stub Files.">;
 def emit_merged_ifs : Flag<["-"], "emit-merged-ifs">,
   Flags<[CC1Option]>, Group<Action_Group>,
   HelpText<"Generate Interface Stub Files, emit merged text not binary.">;
 def end_no_unused_arguments : Flag<["--"], "end-no-unused-arguments">, Flags<[CoreOption]>,
   HelpText<"Start emitting warnings for unused driver arguments">;
 def interface_stub_version_EQ : JoinedOrSeparate<["-"], "interface-stub-version=">, Flags<[CC1Option]>;
 def exported__symbols__list : Separate<["-"], "exported_symbols_list">;
+def extract_api : Flag<["-"], "extract-api">, Flags<[CC1Option]>, Group<Action_Group>,
+  HelpText<"Extract API information">;
 def e : JoinedOrSeparate<["-"], "e">, Flags<[LinkerInput]>, Group<Link_Group>;
 def fmax_tokens_EQ : Joined<["-"], "fmax-tokens=">, Group<f_Group>, Flags<[CC1Option]>,
   HelpText<"Max total number of preprocessed tokens for -Wmax-tokens.">,
   MarshallingInfoInt<LangOpts<"MaxTokens">>;
 def fPIC : Flag<["-"], "fPIC">, Group<f_Group>;
 def fno_PIC : Flag<["-"], "fno-PIC">, Group<f_Group>;
 def fPIE : Flag<["-"], "fPIE">, Group<f_Group>;
 def fno_PIE : Flag<["-"], "fno-PIE">, Group<f_Group>;

clang/include/clang/Driver/Types.def

 TYPE("precompiled-header",       PCH,          INVALID,         "gch",    phases::Compile, phases::Backend, phases::Assemble, phases::Link)
 TYPE("object",                   Object,       INVALID,         "o",      phases::Link)
 TYPE("treelang",                 Treelang,     INVALID,         nullptr,  phases::Compile, phases::Backend, phases::Assemble, phases::Link)
 TYPE("image",                    Image,        INVALID,         "out",    phases::Compile, phases::Backend, phases::Assemble, phases::Link)
 TYPE("dSYM",                     dSYM,         INVALID,         "dSYM",   phases::Compile, phases::Backend, phases::Assemble, phases::Link)
 TYPE("dependencies",             Dependencies, INVALID,         "d",      phases::Compile, phases::Backend, phases::Assemble, phases::Link)
 TYPE("cuda-fatbin",              CUDA_FATBIN,  INVALID,         "fatbin", phases::Compile, phases::Backend, phases::Assemble, phases::Link)
 TYPE("hip-fatbin",               HIP_FATBIN,   INVALID,         "hipfb",  phases::Compile, phases::Backend, phases::Assemble, phases::Link)
+TYPE("api-information",          API_INFO,     INVALID,         "json",   phases::Compile)
 TYPE("none",                     Nothing,      INVALID,         nullptr,  phases::Compile, phases::Backend, phases::Assemble, phases::Link)

clang/include/clang/Frontend/FrontendActions.h

 //===-- FrontendActions.h - Useful Frontend Actions -------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_FRONTEND_FRONTENDACTIONS_H
 #define LLVM_CLANG_FRONTEND_FRONTENDACTIONS_H
 
 #include "clang/Frontend/FrontendAction.h"
+#include <memory>
 #include <string>
 #include <vector>
 
 namespace clang {
 
 //===----------------------------------------------------------------------===//
 // Custom Consumer Actions
 //===----------------------------------------------------------------------===//
protected:
   std::unique_ptr<ASTConsumer> CreateASTConsumer(CompilerInstance &,
                                                  StringRef) override {
     return nullptr;
   }
 
   bool usesPreprocessorOnly() const override { return true; }
 };
 
+class ExtractAPIAction : public ASTFrontendAction {
+protected:
+  std::unique_ptr<ASTConsumer> CreateASTConsumer(CompilerInstance &CI,
+                                                 StringRef InFile) override;
+};
+
 //===----------------------------------------------------------------------===//
 // Preprocessor Actions
 //===----------------------------------------------------------------------===//
 
 class DumpRawTokensAction : public PreprocessorFrontendAction {
 protected:
   void ExecuteAction() override;
 };

clang/include/clang/Frontend/FrontendOptions.h

enum ActionKind {
   EmitLLVMOnly,
 
   /// Generate machine code, but don't emit anything.
   EmitCodeGenOnly,
 
   /// Emit a .o file.
   EmitObj,
 
+  // Extract API information
+  ExtractAPI,
+
   /// Parse and apply any fixits to the source.
   FixIt,
 
   /// Generate pre-compiled module from a module map.
   GenerateModule,
 
   /// Generate pre-compiled module from a C++ module interface file.
   GenerateModuleInterface,

clang/lib/Driver/Driver.cpp

 #include "clang/Driver/Compilation.h"
 #include "clang/Driver/DriverDiagnostic.h"
 #include "clang/Driver/InputInfo.h"
 #include "clang/Driver/Job.h"
 #include "clang/Driver/Options.h"
 #include "clang/Driver/SanitizerArgs.h"
 #include "clang/Driver/Tool.h"
 #include "clang/Driver/ToolChain.h"
+#include "clang/Driver/Types.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallSet.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/StringSet.h"
 #include "llvm/ADT/StringSwitch.h"
 #include "llvm/Config/llvm-config.h"
phases::ID Driver::getFinalPhase(const DerivedArgList &DAL,
   } else if ((PhaseArg = DAL.getLastArg(options::OPT_fsyntax_only)) ||
              (PhaseArg = DAL.getLastArg(options::OPT_print_supported_cpus)) ||
              (PhaseArg = DAL.getLastArg(options::OPT_module_file_info)) ||
              (PhaseArg = DAL.getLastArg(options::OPT_verify_pch)) ||
              (PhaseArg = DAL.getLastArg(options::OPT_rewrite_objc)) ||
              (PhaseArg = DAL.getLastArg(options::OPT_rewrite_legacy_objc)) ||
              (PhaseArg = DAL.getLastArg(options::OPT__migrate)) ||
              (PhaseArg = DAL.getLastArg(options::OPT__analyze)) ||
-             (PhaseArg = DAL.getLastArg(options::OPT_emit_ast))) {
+             (PhaseArg = DAL.getLastArg(options::OPT_emit_ast)) ||
+             (PhaseArg = DAL.getLastArg(options::OPT_extract_api))) {
     FinalPhase = phases::Compile;
 
   // -S only runs up to the backend.
   } else if ((PhaseArg = DAL.getLastArg(options::OPT_S))) {
     FinalPhase = phases::Backend;
 
   // -c compilation only runs up to the assembler.
   } else if ((PhaseArg = DAL.getLastArg(options::OPT_c))) {
case phases::Precompile: {
     const char *ModName = nullptr;
     if (OutputTy == types::TY_PCH) {
       if (Arg *A = Args.getLastArg(options::OPT_fmodule_name_EQ))
         ModName = A->getValue();
       if (ModName)
         OutputTy = types::TY_ModuleFile;
     }
 
-    if (Args.hasArg(options::OPT_fsyntax_only)) {
+    if (Args.hasArg(options::OPT_fsyntax_only) ||
+        Args.hasArg(options::OPT_extract_api)) {
       // Syntax checks should not emit a PCH file
       OutputTy = types::TY_Nothing;
     }
 
     if (ModName)
       return C.MakeAction<HeaderModulePrecompileJobAction>(Input, OutputTy,
                                                            ModName);
     return C.MakeAction<PrecompileJobAction>(Input, OutputTy);
case phases::Compile: {
     if (Args.hasArg(options::OPT__migrate))
       return C.MakeAction<MigrateJobAction>(Input, types::TY_Remap);
     if (Args.hasArg(options::OPT_emit_ast))
       return C.MakeAction<CompileJobAction>(Input, types::TY_AST);
     if (Args.hasArg(options::OPT_module_file_info))
       return C.MakeAction<CompileJobAction>(Input, types::TY_ModuleFile);
     if (Args.hasArg(options::OPT_verify_pch))
       return C.MakeAction<VerifyPCHJobAction>(Input, types::TY_Nothing);
+    if (Args.hasArg(options::OPT_extract_api))
+      return C.MakeAction<CompileJobAction>(Input, types::TY_API_INFO);
     return C.MakeAction<CompileJobAction>(Input, types::TY_LLVM_BC);
   }
   case phases::Backend: {
     if (isUsingLTO() && TargetDeviceOffloadKind == Action::OFK_None) {
       types::ID Output =
           Args.hasArg(options::OPT_S) ? types::TY_LTO_IR : types::TY_LTO_BC;
       return C.MakeAction<BackendJobAction>(Input, Output);
     }

clang/lib/Driver/ToolChains/Clang.cpp

 #include "clang/Basic/LangOptions.h"
 #include "clang/Basic/ObjCRuntime.h"
 #include "clang/Basic/Version.h"
 #include "clang/Driver/Distro.h"
 #include "clang/Driver/DriverDiagnostic.h"
 #include "clang/Driver/InputInfo.h"
 #include "clang/Driver/Options.h"
 #include "clang/Driver/SanitizerArgs.h"
+#include "clang/Driver/Types.h"
 #include "clang/Driver/XRayArgs.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/Config/llvm-config.h"
 #include "llvm/Option/ArgList.h"
 #include "llvm/Support/CodeGen.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/Compression.h"
 #include "llvm/Support/FileSystem.h"
if (isa<AnalyzeJobAction>(JA)) {
     } else if (JA.getType() == types::TY_ModuleFile) {
       CmdArgs.push_back("-module-file-info");
     } else if (JA.getType() == types::TY_RewrittenObjC) {
       CmdArgs.push_back("-rewrite-objc");
       rewriteKind = RK_NonFragile;
     } else if (JA.getType() == types::TY_RewrittenLegacyObjC) {
       CmdArgs.push_back("-rewrite-objc");
       rewriteKind = RK_Fragile;
+    } else if (JA.getType() == types::TY_API_INFO) {
+      CmdArgs.push_back("-extract-api");
     } else {
       assert(JA.getType() == types::TY_PP_Asm && "Unexpected output type!");
     }
 
     // Preserve use-list order by default when emitting bitcode, so that
     // loading the bitcode up in 'opt' or 'llc' and running passes gives the
     // same result as running passes here.  For LTO, we don't need to preserve
     // the use-list order, since serialization to bitcode is part of the flow.
if (Output.getType() == types::TY_Dependencies) {
     // Handled with other dependency code.
   } else if (Output.isFilename()) {
     if (Output.getType() == clang::driver::types::TY_IFS_CPP ||
         Output.getType() == clang::driver::types::TY_IFS) {
       SmallString<128> OutputFilename(Output.getFilename());
       llvm::sys::path::replace_extension(OutputFilename, "ifs");
       CmdArgs.push_back("-o");
       CmdArgs.push_back(Args.MakeArgString(OutputFilename));
+    } else if (Output.getType() == types::TY_API_INFO) {
+      SmallString<128> OutputFilename(Output.getFilename());
+      llvm::sys::path::replace_extension(OutputFilename, "json");
+      CmdArgs.push_back("-o");
+      CmdArgs.push_back(Args.MakeArgString(OutputFilename));
     } else {
       CmdArgs.push_back("-o");
       CmdArgs.push_back(Output.getFilename());
     }
   } else {
     assert(Output.isNothing() && "Invalid output.");
   }
 

clang/lib/Driver/Types.cpp

bool types::isAcceptedByClang(ID Id) {
   case TY_CHeader: case TY_PP_CHeader:
   case TY_CLHeader:
   case TY_ObjCHeader: case TY_PP_ObjCHeader:
   case TY_CXXHeader: case TY_PP_CXXHeader:
   case TY_ObjCXXHeader: case TY_PP_ObjCXXHeader:
   case TY_CXXModule: case TY_PP_CXXModule:
   case TY_AST: case TY_ModuleFile: case TY_PCH:
   case TY_LLVM_IR: case TY_LLVM_BC:
+  case TY_API_INFO:
     return true;
   }
 }
 
 bool types::isDerivedFromC(ID Id) {
   switch (Id) {
   default:
     return false;

clang/lib/Frontend/CMakeLists.txt

add_clang_library(clangFrontend
   ChainedDiagnosticConsumer.cpp
   ChainedIncludesSource.cpp
   CompilerInstance.cpp
   CompilerInvocation.cpp
   CreateInvocationFromCommandLine.cpp
   DependencyFile.cpp
   DependencyGraph.cpp
   DiagnosticRenderer.cpp
+  ExtractAPIConsumer.cpp
   FrontendAction.cpp
   FrontendActions.cpp
   FrontendOptions.cpp
   HeaderIncludeGen.cpp
   InitPreprocessor.cpp
   LayoutOverrideSource.cpp
   LogDiagnosticPrinter.cpp
   ModuleDependencyCollector.cpp

clang/lib/Frontend/CompilerInvocation.cpp

static const std::pair<frontend::ActionKind, unsigned> Table[] = {
       {frontend::EmitAssembly, OPT_S},
       {frontend::EmitBC, OPT_emit_llvm_bc},
       {frontend::EmitHTML, OPT_emit_html},
       {frontend::EmitLLVM, OPT_emit_llvm},
       {frontend::EmitLLVMOnly, OPT_emit_llvm_only},
       {frontend::EmitCodeGenOnly, OPT_emit_codegen_only},
       {frontend::EmitCodeGenOnly, OPT_emit_codegen_only},
       {frontend::EmitObj, OPT_emit_obj},
+      {frontend::ExtractAPI, OPT_extract_api},
 
       {frontend::FixIt, OPT_fixit_EQ},
       {frontend::FixIt, OPT_fixit},
 
       {frontend::GenerateModule, OPT_emit_module},
       {frontend::GenerateModuleInterface, OPT_emit_module_interface},
       {frontend::GenerateHeaderModule, OPT_emit_header_module},
       {frontend::GeneratePCH, OPT_emit_pch},
       {frontend::GenerateInterfaceStubs, OPT_emit_interface_stubs},
       {frontend::InitOnly, OPT_init_only},
       {frontend::ParseSyntaxOnly, OPT_fsyntax_only},
       {frontend::ModuleFileInfo, OPT_module_file_info},
       {frontend::VerifyPCH, OPT_verify_pch},
       {frontend::PrintPreamble, OPT_print_preamble},
       {frontend::PrintPreprocessedInput, OPT_E},
       {frontend::TemplightDump, OPT_templight_dump},
       {frontend::RewriteMacros, OPT_rewrite_macros},
       {frontend::RewriteObjC, OPT_rewrite_objc},
       {frontend::RewriteTest, OPT_rewrite_test},
       {frontend::RunAnalysis, OPT_analyze},
       {frontend::MigrateSource, OPT_migrate},
       {frontend::RunPreprocessorOnly, OPT_Eonly},
       {frontend::PrintDependencyDirectivesSourceMinimizerOutput,
-          OPT_print_dependency_directives_minimized_source},
   };
 
   return Table;
 }
 
 /// Maps command line option to frontend action.
 static Optional<frontend::ActionKind> getFrontendAction(OptSpecifier &Opt) {
   for (const auto &ActionOpt : getFrontendActionTable())
static bool isStrictlyPreprocessorAction(frontend::ActionKind Action) {
   case frontend::ASTView:
   case frontend::EmitAssembly:
   case frontend::EmitBC:
   case frontend::EmitHTML:
   case frontend::EmitLLVM:
   case frontend::EmitLLVMOnly:
   case frontend::EmitCodeGenOnly:
   case frontend::EmitObj:
+  case frontend::ExtractAPI:
   case frontend::FixIt:
   case frontend::GenerateModule:
   case frontend::GenerateModuleInterface:
   case frontend::GenerateHeaderModule:
   case frontend::GeneratePCH:
   case frontend::GenerateInterfaceStubs:
   case frontend::ParseSyntaxOnly:
   case frontend::ModuleFileInfo:

clang/lib/Frontend/ExtractAPIConsumer.cpp

+#include "clang/AST/ASTConsumer.h"
+#include "clang/AST/RecursiveASTVisitor.h"
+#include "clang/Frontend/ASTConsumers.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Frontend/FrontendActions.h"
+
+using namespace clang;
+
+namespace {
+class ExtractAPIVisitor : public RecursiveASTVisitor<ExtractAPIVisitor> {
+public:
+  explicit ExtractAPIVisitor(ASTContext *Context) : Context(Context) {}
+
+  bool VisitNamedDecl(NamedDecl *Decl) {
+    llvm::outs() << Decl->getName() << "\n";
+    return true;
+  }
+
+private:
+  ASTContext *Context;
+};
+
+class ExtractAPIConsumer : public ASTConsumer {
+public:
+  explicit ExtractAPIConsumer(ASTContext *Context) : Visitor(Context) {}
+
+  void HandleTranslationUnit(ASTContext &Context) override {
+    Visitor.TraverseDecl(Context.getTranslationUnitDecl());
+  }
+
+private:
+  ExtractAPIVisitor Visitor;
+};
+} // namespace
+
+std::unique_ptr<ASTConsumer>
+ExtractAPIAction::CreateASTConsumer(CompilerInstance &CI, StringRef InFile) {
+  return std::make_unique<ExtractAPIConsumer>(&CI.getASTContext());
+}

clang/lib/FrontendTool/ExecuteCompilerInvocation.cpp

CreateFrontendBaseAction(CompilerInstance &CI) {
   case DumpTokens:             return std::make_unique<DumpTokensAction>();
   case EmitAssembly:           return std::make_unique<EmitAssemblyAction>();
   case EmitBC:                 return std::make_unique<EmitBCAction>();
   case EmitHTML:               return std::make_unique<HTMLPrintAction>();
   case EmitLLVM:               return std::make_unique<EmitLLVMAction>();
   case EmitLLVMOnly:           return std::make_unique<EmitLLVMOnlyAction>();
   case EmitCodeGenOnly:        return std::make_unique<EmitCodeGenOnlyAction>();
   case EmitObj:                return std::make_unique<EmitObjAction>();
+  case ExtractAPI:
+    return std::make_unique<ExtractAPIAction>();
   case FixIt:                  return std::make_unique<FixItAction>();
   case GenerateModule:
     return std::make_unique<GenerateModuleFromModuleMapAction>();
   case GenerateModuleInterface:
     return std::make_unique<GenerateModuleInterfaceAction>();
   case GenerateHeaderModule:
     return std::make_unique<GenerateHeaderModuleAction>();
   case GeneratePCH:            return std::make_unique<GeneratePCHAction>();

clang/test/Driver/extract-api.c

+// RUN: %clang -target x86_64-unknown-unknown -ccc-print-phases -extract-api %s 2> %t
+// RUN: echo 'END' >> %t
+// RUN: FileCheck -check-prefix EXTRACT-API-PHASES -input-file %t %s
+
+// EXTRACT-API-PHASES: 0: input,
+// EXTRACT-API-PHASES: , c
+// EXTRACT-API-PHASES: 1: preprocessor, {0}, cpp-output
+// EXTRACT-API-PHASES: 2: compiler, {1}, api-information
+// EXTRACT-API-PHASES-NOT: 3:
+// EXTRACT-API-PHASES: END
+
+// FIXME: Check for the dummy output now to verify that the custom action was executed.
+// RUN: %clang -extract-api %s | FileCheck -check-prefix DUMMY-OUTPUT %s
+
+void dummy_function();
+// DUMMY-OUTPUT: dummy_function