[clang-scan-deps][static analyzer] Support for clang --analyze in scan-deps

[jkorous]

Summary

This is v2 of https://reviews.llvm.org/D67938 which I reverted as it broke clang-tidy and there wasn't any nice fix.

The full list of goals here is:

• Support commands with --analyze in compilation database for clang-scan-deps.
• Keep the macro spelling in one place only.
• Don't break clang-tidy which relies on the fact that __clang_analyzer__ is defined as built-in macro. Otherwise it's treated as part of the input and exposed in diagnostics - specifically the test was failing because macros should be lowercase or uppercase depending on clang-tidy config.

Two things I'd appreciate feedback on:

• name for the new option
• preferences whether the option should be automatically set to true whenever cc1 action is RunAnalysis or whether it should always be set by driver in case input args contain --analyze

[arphaman]

clang/test/ClangScanDeps/static-analyzer.c

13

Please use existing Inputs/header.h instead of adding a new empty file.

clang/tools/clang-scan-deps/ClangScanDeps.cpp

262

Clang's driver should pass this, otherwise users of the ScanDeps library won't be able to rely on this behavior if they don't use the clang-scan-deps tool, but just the library itself.

[jkorous]

Adressed comments.

[jkorous]

clang/test/ClangScanDeps/static-analyzer.c

13

Good point!

clang/tools/clang-scan-deps/ClangScanDeps.cpp

262

Thanks, I was thinking only about the tool.

[arphaman]

clang/tools/clang-scan-deps/ClangScanDeps.cpp

265

This code now should no longer be necessary.

[jkorous]

Addressed comment.

[hiraditya]

clang/include/clang/Driver/CC1Options.td

849

The name doesn't quite reflect what it does.

[jkorous]

clang/include/clang/Driver/CC1Options.td

849

setup-pp-for-analyzer? I'm open to suggestions.

[NoQ]

clang/include/clang/Driver/CC1Options.td

849

I actually suggest modifying the help text to something like "Behave as if the Static Analyzer is going to be invoked, even if it's not actually going to be invoked (for now this boils down to defining the clang_analyzer macro)" and keep the flag name roughly the same. This is the actual purpose of the option, right? We don't need to specify what precisely takes place when the option is invoked because this may change in the future, but the contract will remain.

[jkorous]

clang/include/clang/Driver/CC1Options.td

849

Sorry, there were quite a few changes in the patch and it deviated from the original idea. After the most recent one the description wouldn't be correct anymore - please take a look at clang/lib/Driver/ToolChains/Clang.cpp below.
The -setup-static-analyzer flag can be passed on its own to frontend (without -analyze) but if you pass --analyze to driver then both -analyze and -setup-static-analyzer are passed to frontend.
Passing just -analyze without -setup-static-analyzer to frontend would result in skipping the __clang_analyzer__ macro definition.

[arphaman]

LGTM

[jkorous]

@NoQ , @hiraditya any suggestions for the option name and/or description?

[NoQ]

Looks great, thanks!

clang-tools-extra/clang-tidy/ClangTidy.cpp

 #include "clang/Frontend/ASTConsumers.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/FrontendActions.h"
 #include "clang/Frontend/FrontendDiagnostic.h"
 #include "clang/Frontend/MultiplexConsumer.h"
 #include "clang/Frontend/TextDiagnosticPrinter.h"
 #include "clang/Lex/PPCallbacks.h"
 #include "clang/Lex/Preprocessor.h"
+#include "clang/Lex/PreprocessorOptions.h"
 #include "clang/Rewrite/Frontend/FixItRewriter.h"
 #include "clang/Rewrite/Frontend/FrontendActions.h"
 #include "clang/Tooling/Core/Diagnostic.h"
 #include "clang/Tooling/DiagnosticsYaml.h"
 #include "clang/Tooling/Refactoring.h"
 #include "clang/Tooling/ReplacementsYaml.h"
 #include "clang/Tooling/Tooling.h"
 #include "llvm/Support/Process.h"
public:
     std::unique_ptr<FrontendAction> create() override {
       return std::make_unique<Action>(&ConsumerFactory);
     }
 
     bool runInvocation(std::shared_ptr<CompilerInvocation> Invocation,
                        FileManager *Files,
                        std::shared_ptr<PCHContainerOperations> PCHContainerOps,
                        DiagnosticConsumer *DiagConsumer) override {
-      // Explicitly set ProgramAction to RunAnalysis to make the preprocessor
-      // define __clang_analyzer__ macro. The frontend analyzer action will not
-      // be called here.
-      Invocation->getFrontendOpts().ProgramAction = frontend::RunAnalysis;
+      // Explicitly ask to define __clang_analyzer__ macro.
+      Invocation->getPreprocessorOpts().SetUpStaticAnalyzer = true;
       return FrontendActionFactory::runInvocation(
           Invocation, Files, PCHContainerOps, DiagConsumer);
     }
 
   private:
     class Action : public ASTFrontendAction {
     public:
       Action(ClangTidyASTConsumerFactory *Factory) : Factory(Factory) {}

clang/include/clang/Driver/CC1Options.td

 
 def chain_include : Separate<["-"], "chain-include">, MetaVarName<"<file>">,
   HelpText<"Include and chain a header file after turning it into PCH">;
 def preamble_bytes_EQ : Joined<["-"], "preamble-bytes=">,
   HelpText<"Assume that the precompiled header is a precompiled preamble "
            "covering the first N bytes of the main file">;
 def detailed_preprocessing_record : Flag<["-"], "detailed-preprocessing-record">,
   HelpText<"include a detailed record of preprocessing actions">;
+def setup_static_analyzer : Flag<["-"], "setup-static-analyzer">,
+  HelpText<"Set up preprocessor for static analyzer (done automatically when static analyzer is run).">;
 
 //===----------------------------------------------------------------------===//
 // OpenCL Options
 //===----------------------------------------------------------------------===//
 
 def cl_ext_EQ : CommaJoined<["-"], "cl-ext=">,
   HelpText<"OpenCL only. Enable or disable OpenCL extensions. The argument is a comma-separated sequence of one or more extension names, each prefixed by '+' or '-'.">;
 

clang/include/clang/Lex/PreprocessorOptions.h

public:
   /// Contains the currently active skipped range mappings for skipping excluded
   /// conditional directives.
   ///
   /// The pointer is passed to the Preprocessor when it's constructed. The
   /// pointer is unowned, the client is responsible for its lifetime.
   ExcludedPreprocessorDirectiveSkipMapping
       *ExcludedConditionalDirectiveSkipMappings = nullptr;
 
+  /// Set up preprocessor for RunAnalysis action.
+  bool SetUpStaticAnalyzer = false;
+
 public:
   PreprocessorOptions() : PrecompiledPreambleBytes(0, false) {}
 
   void addMacroDef(StringRef Name) { Macros.emplace_back(Name, false); }
   void addMacroUndef(StringRef Name) { Macros.emplace_back(Name, true); }
 
   void addRemappedFile(StringRef From, StringRef To) {
     RemappedFiles.emplace_back(From, To);

clang/lib/Driver/ToolChains/Clang.cpp

if (Args.hasArg(options::OPT_static))
     CmdArgs.push_back("-static-define");
 
   if (Args.hasArg(options::OPT_municode))
     CmdArgs.push_back("-DUNICODE");
 
   if (isa<AnalyzeJobAction>(JA))
     RenderAnalyzerOptions(Args, CmdArgs, Triple, Input);
 
+  if (isa<AnalyzeJobAction>(JA) ||
+      (isa<PreprocessJobAction>(JA) && Args.hasArg(options::OPT__analyze)))
+    CmdArgs.push_back("-setup-static-analyzer");
+
   // Enable compatilibily mode to avoid analyzer-config related errors.
   // Since we can't access frontend flags through hasArg, let's manually iterate
   // through them.
   bool FoundAnalyzerConfig = false;
   for (auto Arg : Args.filtered(options::OPT_Xclang))
     if (StringRef(Arg->getValue()) == "-analyzer-config") {
       FoundAnalyzerConfig = true;
       break;

clang/lib/Frontend/CompilerInvocation.cpp

else
       Opts.ObjCXXARCStandardLibrary = (ObjCXXARCStandardLibraryKind)Library;
   }
 
   // Always avoid lexing editor placeholders when we're just running the
   // preprocessor as we never want to emit the
   // "editor placeholder in source file" error in PP only mode.
   if (isStrictlyPreprocessorAction(Action))
     Opts.LexEditorPlaceholders = false;
+
+  Opts.SetUpStaticAnalyzer = Args.hasArg(OPT_setup_static_analyzer);
 }
 
 static void ParsePreprocessorOutputArgs(PreprocessorOutputOptions &Opts,
                                         ArgList &Args,
                                         frontend::ActionKind Action) {
   if (isStrictlyPreprocessorAction(Action))
     Opts.ShowCPP = !Args.hasArg(OPT_dM);
   else

clang/lib/Frontend/InitPreprocessor.cpp

if (LangOpts.ConceptsTS)
     Builder.defineMacro("__cpp_experimental_concepts", "1L");
   if (LangOpts.Coroutines)
     Builder.defineMacro("__cpp_coroutines", "201703L");
 }
 
 static void InitializePredefinedMacros(const TargetInfo &TI,
                                        const LangOptions &LangOpts,
                                        const FrontendOptions &FEOpts,
+                                       const PreprocessorOptions &PPOpts,
                                        MacroBuilder &Builder) {
   // Compiler version introspection macros.
   Builder.defineMacro("__llvm__");  // LLVM Backend
   Builder.defineMacro("__clang__"); // Clang Frontend
 #define TOSTR2(X) #X
 #define TOSTR(X) TOSTR2(X)
   Builder.defineMacro("__clang_major__", TOSTR(CLANG_VERSION_MAJOR));
   Builder.defineMacro("__clang_minor__", TOSTR(CLANG_VERSION_MINOR));
#undef DEFINE_LOCK_FREE_MACRO
 
   if (LangOpts.getStackProtector() == LangOptions::SSPOn)
     Builder.defineMacro("__SSP__");
   else if (LangOpts.getStackProtector() == LangOptions::SSPStrong)
     Builder.defineMacro("__SSP_STRONG__", "2");
   else if (LangOpts.getStackProtector() == LangOptions::SSPReq)
     Builder.defineMacro("__SSP_ALL__", "3");
 
-  // Define a macro that exists only when using the static analyzer.
-  if (FEOpts.ProgramAction == frontend::RunAnalysis)
+  if (PPOpts.SetUpStaticAnalyzer)
     Builder.defineMacro("__clang_analyzer__");
 
   if (LangOpts.FastRelaxedMath)
     Builder.defineMacro("__FAST_RELAXED_MATH__");
 
   if (FEOpts.ProgramAction == frontend::RewriteObjC ||
       LangOpts.getGC() != LangOptions::NonGC) {
     Builder.defineMacro("__weak", "__attribute__((objc_gc(weak)))");
if (!PP.getLangOpts().AsmPreprocessor)
     Builder.append("# 1 \"<built-in>\" 3");
 
   // Install things like __POWERPC__, __GNUC__, etc into the macro table.
   if (InitOpts.UsePredefines) {
     // FIXME: This will create multiple definitions for most of the predefined
     // macros. This is not the right way to handle this.
     if ((LangOpts.CUDA || LangOpts.OpenMPIsDevice) && PP.getAuxTargetInfo())
       InitializePredefinedMacros(*PP.getAuxTargetInfo(), LangOpts, FEOpts,
-                                 Builder);
+                                 PP.getPreprocessorOpts(), Builder);
 
-    InitializePredefinedMacros(PP.getTargetInfo(), LangOpts, FEOpts, Builder);
+    InitializePredefinedMacros(PP.getTargetInfo(), LangOpts, FEOpts,
+                               PP.getPreprocessorOpts(), Builder);
 
     // Install definitions to make Objective-C++ ARC work well with various
     // C++ Standard Library implementations.
     if (LangOpts.ObjC && LangOpts.CPlusPlus &&
         (LangOpts.ObjCAutoRefCount || LangOpts.ObjCWeak)) {
       switch (InitOpts.ObjCXXARCStandardLibrary) {
       case ARCXX_nolib:
       case ARCXX_libcxx:

clang/test/Analysis/preprocessor-setup.c

+// RUN: %clang_cc1 -E -setup-static-analyzer %s
+
+#ifndef __clang_analyzer__
+#error __clang_analyzer__ not defined
+#endif

clang/test/ClangScanDeps/Inputs/static-analyzer-cdb.json

+[
+{
+  "directory": "DIR",
+  "command": "clang --analyze DIR/static-analyzer.c",
+  "file": "DIR/static-analyzer.c"
+}
+]

clang/test/ClangScanDeps/static-analyzer.c

+// RUN: rm -rf %t.dir
+// RUN: rm -rf %t.dir/cdb.json
+// RUN: mkdir -p %t.dir
+// RUN: cp %s %t.dir/static-analyzer.c
+// RUN: mkdir %t.dir/Inputs
+// RUN: cp %S/Inputs/header.h %t.dir/Inputs/analyze_header_input.h
+// RUN: sed -e "s|DIR|%t.dir|g" %S/Inputs/static-analyzer-cdb.json > %t.dir/cdb.json
+//
+// RUN: clang-scan-deps -compilation-database %t.dir/cdb.json -j 1 | FileCheck %s
+
+#ifdef __clang_analyzer__
+#include "Inputs/analyze_header_input.h"
+#endif
+
+// CHECK: analyze_header_input.h
+

clang/tools/clang-scan-deps/ClangScanDeps.cpp

AdjustingCompilations->appendArgumentsAdjuster(
             // FIXME: We are missing the directory unless the -o value is an
             // absolute path.
             AdjustedArgs.push_back(!LastO.empty() ? LastO
                                                   : getObjFilePath(FileName));
           } else {
             AdjustedArgs.push_back(FileName);
           }
         }
+        // Support for static analyzer.
+        auto It = find(AdjustedArgs.begin(), AdjustedArgs.end(), "--analyze");
+        if (It != AdjustedArgs.end())
+          *It = "-Wp,-setup-static-analyzer";
+
         AdjustedArgs.push_back("-Xclang");
         AdjustedArgs.push_back("-Eonly");
         AdjustedArgs.push_back("-Xclang");
         AdjustedArgs.push_back("-sys-header-deps");
         AdjustedArgs.push_back("-Wno-error");
         return AdjustedArgs;
       });
   AdjustingCompilations->appendArgumentsAdjuster(

llvm/utils/lit/lit/llvm/config.py

def use_clang(self, additional_tool_dirs=[], additional_flags=[], required=True):
         if shl:
             self.config.substitutions.append(('%llvmshlibdir', shl))
         if pext:
             self.config.substitutions.append(('%pluginext', pext))
 
         builtin_include_dir = self.get_clang_builtin_include_dir(self.config.clang)
         tool_substitutions = [
             ToolSubst('%clang', command=self.config.clang, extra_args=additional_flags),
-            ToolSubst('%clang_analyze_cc1', command='%clang_cc1', extra_args=['-analyze', '%analyze']+additional_flags),
+            ToolSubst('%clang_analyze_cc1', command='%clang_cc1', extra_args=['-analyze', '%analyze', '-setup-static-analyzer']+additional_flags),
             ToolSubst('%clang_cc1', command=self.config.clang, extra_args=['-cc1', '-internal-isystem', builtin_include_dir, '-nostdsysteminc']+additional_flags),
             ToolSubst('%clang_cpp', command=self.config.clang, extra_args=['--driver-mode=cpp']+additional_flags),
             ToolSubst('%clang_cl', command=self.config.clang, extra_args=['--driver-mode=cl']+additional_flags),
             ToolSubst('%clangxx', command=self.config.clang, extra_args=['--driver-mode=g++']+additional_flags),
             ]
         self.add_tool_substitutions(tool_substitutions)
 
         self.config.substitutions.append(('%itanium_abi_triple',