[WIP] Add support for preprocessing sources for analyzer

[jkorous]

Summary

I'd like to teach clang how to preprocess sources so static analyzer would behave in the same way as if run on raw input sources.

clang help
...
--analyze               Run the static analyzer
...
-E                      Only run the preprocessor

Driver currently ignores --analyze option in presence of -E but I am not sure if that's based on conscious decision or if it's just an implementation detail (getFinalPhase sets the last compilation phase as Preprocess in presence of -E). @dergachev.a could you please confirm?

clang --analyze -E ~/tmp/trivial.cpp
clang: warning: argument unused during compilation: '--analyze' [-Wunused-command-line-argument]

The only relevant effect of running analyzer that I found is that in Frontend __clang_analyzer__ macro is set (see BOOKMARK below).
@dergachev.a Do you know about any other things I should take care of?

I generally see two possible solutions:

1. We generate superset of options for both actions in Driver (probably by calling RenderAnalyzerOptions in Clang::ConstructJob) and deal with that in Frontend.
2. We generate such options in Driver that setup Frontend as necessary. (This seemed way easier implementation-wise so I explored it first.)

@arphaman @dergachev.a WDYT?

[arphaman]

It's best not to have two places that define the macro. Can we always pass in -D from the driver instead?

[NoQ]

That's a nice twist, i like it!

Do you know about any other things I should take care of?

AFAIK this is the only thing that changes in the --analyze mode. We also do a tiny bit of AST editing, but there's no way it affects the preprocessor output.

I'd like to teach clang how to preprocess sources so static analyzer would behave in the same way as if run on raw input sources.

Note that in order to retain the pristine behavior, you'd also need -frewrite-includes, because the analyzer occasionally takes macro expansions into account but they get irreversibly destroyed by the preprocessor. I don't recommend turning it on under --analyze -E though, because it'd be super confusing.

Driver currently ignores --analyzeoption in presence of -E but I am not sure if that's based on conscious decision or if it's just an implementation detail (getFinalPhase sets the last compilation phase as Preprocess in presence of -E). @dergachev.a could you please confirm?

I don't know whether it's conscious or accidental, but i don't see any problems with the existing behavior. When i see a bug in the analyzer, i take the clang run-line that reproduces the bug, append -E to it, and get my reproducer. I don't want to conduct more analysis at this point because i already know the results of it, so i'm happy that -E disables the analysis. The warning for "argument unused during compilation" looks harmless and educational.

clang/lib/Driver/ToolChains/Clang.cpp

3909

It's the first time in my life i hear about --analyze-auto. We should probably remove it completely.

clang/lib/Frontend/InitPreprocessor.cpp

995

This comment looks accidental.

[NoQ]

Accept as soon as Alex's comment is addressed somehow :)

[NoQ]

In D67682#1673332, @arphaman wrote:

Can we always pass in -D from the driver instead?

I think this shouldn't break anything. We're being invoked from the driver anyway. @xazax.hun @Szelethus @dkrupp would that break anything for you?

[Szelethus]

I really don't think so, but I'll ask around in the office.

[jkorous]

@Szelethus @NoQ There's this (fairly vintage) test
https://github.com/llvm/llvm-project/blob/master/clang/test/Analysis/misc-ps.m#L6
that checks specifically for the __clang_analyzer__ macro being set when cc1 is run directly.

#ifndef __clang_analyzer__
#error __clang_analyzer__ not defined
#endif

It's not obvious to me why though.

[jkorous]

Anyways, I created a separate patch:
https://reviews.llvm.org/D67938

clang/lib/Driver/ToolChains/Clang.cpp

if (Args.hasArg(options::OPT_static))
     CmdArgs.push_back("-static-define");
 
   if (Args.hasArg(options::OPT_municode))
     CmdArgs.push_back("-DUNICODE");
 
   if (isa<AnalyzeJobAction>(JA))
     RenderAnalyzerOptions(Args, CmdArgs, Triple, Input);
 
+  if (isa<PreprocessJobAction>(JA) && Args.hasArg(options::OPT__analyze, options::OPT__analyze_auto))
+    CmdArgs.push_back("-D__clang_analyzer__");
+
   // Enable compatilibily mode to avoid analyzer-config related errors.
   // Since we can't access frontend flags through hasArg, let's manually iterate
   // through them.
   bool FoundAnalyzerConfig = false;
   for (auto Arg : Args.filtered(options::OPT_Xclang))
     if (StringRef(Arg->getValue()) == "-analyzer-config") {
       FoundAnalyzerConfig = true;
       break;

clang/lib/Frontend/InitPreprocessor.cpp

if (LangOpts.getStackProtector() == LangOptions::SSPOn)
     Builder.defineMacro("__SSP__");
   else if (LangOpts.getStackProtector() == LangOptions::SSPStrong)
     Builder.defineMacro("__SSP_STRONG__", "2");
   else if (LangOpts.getStackProtector() == LangOptions::SSPReq)
     Builder.defineMacro("__SSP_ALL__", "3");
 
   // Define a macro that exists only when using the static analyzer.
   if (FEOpts.ProgramAction == frontend::RunAnalysis)
-    Builder.defineMacro("__clang_analyzer__");
+    Builder.defineMacro("__clang_analyzer__"); /* BOOKMARK */
 
   if (LangOpts.FastRelaxedMath)
     Builder.defineMacro("__FAST_RELAXED_MATH__");
 
   if (FEOpts.ProgramAction == frontend::RewriteObjC ||
       LangOpts.getGC() != LangOptions::NonGC) {
     Builder.defineMacro("__weak", "__attribute__((objc_gc(weak)))");
     Builder.defineMacro("__strong", "__attribute__((objc_gc(strong)))");

clang/test/Frontend/pp-only-with-analyze.c

+// RUN: %clang -E --analyze %s | FileCheck %s
+// expected-no-diagnostics
+
+#ifdef __clang_analyzer__
+int successfully_passed_the_macro;
+// CHECK: successfully_passed_the_macro
+#endif
+int main() { return 42; }