[CTU][Analyzer]Add DisplayCTUProgress analyzer switch

[martong]

Summary

With a new switch we may be able to print to stderr if a new TU is being loaded
during CTU. This is very important for higher level scripts (like CodeChecker)
to be able to parse this output so they can create e.g. a zip file in case of
a Clang crash which contains all the related TU files.

[Szelethus]

The idea is great!

I think this should rather be an -analyzer-config flag, since the actual analysis changes with a new output (refer to AnalyzerOption's doxygen comments). Please note that I'm about to land some patches that modifies AnalyzerOptions.def quite a bit, but rebasing shouldn't be too bad anyways.

edit: They are now in trunk.

Also, almost everywhere CTU is capitalized, so I guess it should be in the field name too.

test/Analysis/ctu-main.cpp

6

I think these RUN lines would really benefit from introducing line breaks.

[a_sidorin]

Hi Gabor,
In addition to Umann remarks, there is a small comment inline.

lib/CrossTU/CrossTranslationUnit.cpp

239

I think we can remove parens from the message.

[Szelethus]

Also, AnalyzerOptions.def was recently clan-formatted, feel free to run it again after the changes you make in it.

[martong]

• Rename AnalyzerDisplayCtuProgress to Opts.AnalyzerDisplayCTUProgress
• Change the CTU progress message

[martong]

Also, almost everywhere CTU is capitalized, so I guess it should be in the field name too.

Ok, I renamed it to have CTU all capitalized.

lib/CrossTU/CrossTranslationUnit.cpp

239

Ok, I removed the parens. And also changed the text from CTU loaded AST for source file to CTU loaded AST file since we dump the path to the AST, not the path to the source file from which we generated the AST file.

test/Analysis/ctu-main.cpp

6

Yes, I agree. Unfortunately, I could not figure out how to break them. Using a simple "\" at around the 80th column gives Test has unterminated run lines (with '\'). If I use block comments with "\" the same happens. If I use block comments and don't use the "\" then the second line is not interpreted. Is it really possible to break RUN lines? I could not find anything about that in the online docs.

[martong]

• Change the CTU progress message in the test too

[martong]

• Break long RUN line

test/Analysis/ctu-main.cpp

6

Oh, I just have found your other comment to the other patch. So yes, it is indeed possible to break this line. I updated the patch accordingly. The other long lines which are already there I am going to change in an independent patch: (https://reviews.llvm.org/D55129).

[Szelethus]

test/Analysis/ctu-main.cpp

6

Is there any particular reason why we use %clang_cc1 -analyze as opposed to %clang_analyze_cc1? If not, lets change it.

By the way, thanks! This looks neat.

[Szelethus]

The code not directly related to adding the actual flag looks great, but this still should be an -analyzer-config option.

[xazax.hun]

Having an analyzer config option makes sense.

[martong]

• Use clang_analyze_cc1
• Change to be an analyzer config option

[martong]

should be an -analyzer-config option.

Ok, just changed it to be.

[Szelethus]

Thanks! LGTM! :)

[xazax.hun]

While Static Analyzer is the only client of CTU library at the moment, we might have more in the future. I would not use the phrase ANALYZE in the log message. Once this is resolved the rest looks good.

[martong]

While Static Analyzer is the only client of CTU library at the moment, we might have more in the future. I would not use the phrase ANALYZE in the log message. Once this is resolved the rest looks good.

Ok, I removed the "ANALYZE " prefix from the log message.

[martong]

• Remove 'ANALYZE ' prefix from the log message

include/clang/CrossTU/CrossTranslationUnit.h

public:
   ///
   /// \return The declaration with the definition will be returned.
   /// If no suitable definition is found in the index file or multiple
   /// definitions found error will be returned.
   ///
   /// Note that the AST files should also be in the \p CrossTUDir.
   llvm::Expected<const FunctionDecl *>
   getCrossTUDefinition(const FunctionDecl *FD, StringRef CrossTUDir,
-                       StringRef IndexName);
+                       StringRef IndexName, bool DisplayCTUProgress = false);
 
   /// This function loads a function definition from an external AST
   ///        file.
   ///
   /// A function definition with the same declaration will be looked up in the
   /// index file which should be in the \p CrossTUDir directory, called
   /// \p IndexName. In case the declaration is found in the index the
   /// corresponding AST file will be loaded.
   ///
   /// \return Returns an ASTUnit that contains the definition of the looked up
   /// function.
   ///
   /// Note that the AST files should also be in the \p CrossTUDir.
   llvm::Expected<ASTUnit *> loadExternalAST(StringRef LookupName,
                                             StringRef CrossTUDir,
-                                            StringRef IndexName);
+                                            StringRef IndexName,
+                                            bool DisplayCTUProgress = false);
 
   /// This function merges a definition from a separate AST Unit into
   ///        the current one which was created by the compiler instance that
   ///        was passed to the constructor.
   ///
   /// \return Returns the resulting definition or an error.
   llvm::Expected<const FunctionDecl *> importDefinition(const FunctionDecl *FD);
 

include/clang/Driver/CC1Options.td

 def trim_egraph : Flag<["-"], "trim-egraph">,
   HelpText<"Only show error-related paths in the analysis graph">;
 def analyzer_viz_egraph_graphviz : Flag<["-"], "analyzer-viz-egraph-graphviz">,
   HelpText<"Display exploded graph using GraphViz">;
 def analyzer_dump_egraph : Separate<["-"], "analyzer-dump-egraph">,
   HelpText<"Dump exploded graph to the specified file">;
 def analyzer_dump_egraph_EQ : Joined<["-"], "analyzer-dump-egraph=">, Alias<analyzer_dump_egraph>;
 
+def analyzer_display_ctu_progress : Flag<["-"], "analyzer-display-ctu-progress">,
+  HelpText<"Emit verbose output about the analyzer's progress related to ctu">;
+
 def analyzer_inline_max_stack_depth : Separate<["-"], "analyzer-inline-max-stack-depth">,
   HelpText<"Bound on stack depth while inlining (4 by default)">;
 def analyzer_inline_max_stack_depth_EQ : Joined<["-"], "analyzer-inline-max-stack-depth=">,
   Alias<analyzer_inline_max_stack_depth>;
 
 def analyzer_inlining_mode : Separate<["-"], "analyzer-inlining-mode">,
   HelpText<"Specify the function selection heuristic used during inlining">;
 def analyzer_inlining_mode_EQ : Joined<["-"], "analyzer-inlining-mode=">, Alias<analyzer_inlining_mode>;

include/clang/StaticAnalyzer/Core/AnalyzerOptions.h

public:
   /// command-line options will get checked.
   unsigned DisableAllChecks : 1;
 
   unsigned ShowCheckerHelp : 1;
   unsigned ShowEnabledCheckerList : 1;
   unsigned ShowConfigOptionsList : 1;
   unsigned AnalyzeAll : 1;
   unsigned AnalyzerDisplayProgress : 1;
+  unsigned AnalyzerDisplayCTUProgress : 1;
   unsigned AnalyzeNestedBlocks : 1;
 
   unsigned eagerlyAssumeBinOpBifurcation : 1;
 
   unsigned TrimGraph : 1;
   unsigned visualizeExplodedGraphWithGraphViz : 1;
   unsigned UnoptimizedCFG : 1;
   unsigned PrintStats : 1;
#undef ANALYZER_OPTION_DEPENDS_ON_USER_MODE
 
   void initOption(Optional<unsigned> &V, StringRef Name,
                                          unsigned DefaultVal);
 public:
   AnalyzerOptions()
       : DisableAllChecks(false), ShowCheckerHelp(false),
         ShowEnabledCheckerList(false), ShowConfigOptionsList(false),
         AnalyzeAll(false), AnalyzerDisplayProgress(false),
-        AnalyzeNestedBlocks(false), eagerlyAssumeBinOpBifurcation(false),
-        TrimGraph(false), visualizeExplodedGraphWithGraphViz(false),
-        UnoptimizedCFG(false), PrintStats(false), NoRetryExhausted(false) {}
+        AnalyzerDisplayCTUProgress(false), AnalyzeNestedBlocks(false),
+        eagerlyAssumeBinOpBifurcation(false), TrimGraph(false),
+        visualizeExplodedGraphWithGraphViz(false), UnoptimizedCFG(false),
+        PrintStats(false), NoRetryExhausted(false) {}
 
   /// Interprets an option's string value as a boolean. The "true" string is
   /// interpreted as true and the "false" string is interpreted as false.
   ///
   /// If an option value is not provided, returns the given \p DefaultVal.
   /// @param [in] Name Name for option to retrieve.
   /// @param [in] DefaultVal Default value returned if no such option was
   /// specified.

lib/CrossTU/CrossTranslationUnit.cpp

for (const Decl *D : DC->decls()) {
     return ResultDecl;
   }
   return nullptr;
 }
 
 llvm::Expected<const FunctionDecl *>
 CrossTranslationUnitContext::getCrossTUDefinition(const FunctionDecl *FD,
                                                   StringRef CrossTUDir,
-                                                  StringRef IndexName) {
+                                                  StringRef IndexName,
+                                                  bool DisplayCTUProgress) {
   assert(!FD->hasBody() && "FD has a definition in current translation unit!");
   const std::string LookupFnName = getLookupName(FD);
   if (LookupFnName.empty())
     return llvm::make_error<IndexError>(
         index_error_code::failed_to_generate_usr);
   llvm::Expected<ASTUnit *> ASTUnitOrError =
-      loadExternalAST(LookupFnName, CrossTUDir, IndexName);
+      loadExternalAST(LookupFnName, CrossTUDir, IndexName, DisplayCTUProgress);
   if (!ASTUnitOrError)
     return ASTUnitOrError.takeError();
   ASTUnit *Unit = *ASTUnitOrError;
   if (!Unit)
     return llvm::make_error<IndexError>(
         index_error_code::failed_to_get_external_ast);
   assert(&Unit->getFileManager() ==
          &Unit->getASTContext().getSourceManager().getFileManager());
Context.getDiagnostics().Report(diag::err_multiple_def_index)
         << IE.getLineNum();
     break;
   default:
     break;
   }
 }
 
 llvm::Expected<ASTUnit *> CrossTranslationUnitContext::loadExternalAST(
-    StringRef LookupName, StringRef CrossTUDir, StringRef IndexName) {
+    StringRef LookupName, StringRef CrossTUDir, StringRef IndexName,
+    bool DisplayCTUProgress) {
   // FIXME: The current implementation only supports loading functions with
   //        a lookup name from a single translation unit. If multiple
   //        translation units contains functions with the same lookup name an
   //        error will be returned.
   ASTUnit *Unit = nullptr;
   auto FnUnitCacheEntry = FunctionASTUnitMap.find(LookupName);
   if (FnUnitCacheEntry == FunctionASTUnitMap.end()) {
     if (FunctionFileMap.empty()) {
if (ASTCacheEntry == FileASTUnitMap.end()) {
       IntrusiveRefCntPtr<DiagnosticsEngine> Diags(
           new DiagnosticsEngine(DiagID, &*DiagOpts, DiagClient));
 
       std::unique_ptr<ASTUnit> LoadedUnit(ASTUnit::LoadFromASTFile(
           ASTFileName, CI.getPCHContainerOperations()->getRawReader(),
           ASTUnit::LoadEverything, Diags, CI.getFileSystemOpts()));
       Unit = LoadedUnit.get();
       FileASTUnitMap[ASTFileName] = std::move(LoadedUnit);
+      if (DisplayCTUProgress) {
+        llvm::errs() << "ANALYZE CTU loaded AST file: "
+                     << ASTFileName << "\n";
+      }
     } else {
       Unit = ASTCacheEntry->second.get();
     }
     FunctionASTUnitMap[LookupName] = Unit;
   } else {
     Unit = FnUnitCacheEntry->second;
   }
   return Unit;

lib/Frontend/CompilerInvocation.cpp

#include "clang/StaticAnalyzer/Core/Analyses.def"
   Opts.DisableAllChecks = Args.hasArg(OPT_analyzer_disable_all_checks);
 
   Opts.visualizeExplodedGraphWithGraphViz =
     Args.hasArg(OPT_analyzer_viz_egraph_graphviz);
   Opts.DumpExplodedGraphTo = Args.getLastArgValue(OPT_analyzer_dump_egraph);
   Opts.NoRetryExhausted = Args.hasArg(OPT_analyzer_disable_retry_exhausted);
   Opts.AnalyzeAll = Args.hasArg(OPT_analyzer_opt_analyze_headers);
   Opts.AnalyzerDisplayProgress = Args.hasArg(OPT_analyzer_display_progress);
+  Opts.AnalyzerDisplayCTUProgress =
+      Args.hasArg(OPT_analyzer_display_ctu_progress);
   Opts.AnalyzeNestedBlocks =
     Args.hasArg(OPT_analyzer_opt_analyze_nested_blocks);
   Opts.AnalyzeSpecificFunction = Args.getLastArgValue(OPT_analyze_function);
   Opts.UnoptimizedCFG = Args.hasArg(OPT_analysis_UnoptimizedCFG);
   Opts.TrimGraph = Args.hasArg(OPT_trim_egraph);
   Opts.maxBlockVisitOnPath =
       getLastArgIntValue(Args, OPT_analyzer_max_loop, 4, Diags);
   Opts.PrintStats = Args.hasArg(OPT_analyzer_stats);

lib/StaticAnalyzer/Core/CallEvent.cpp

RuntimeDefinition AnyFunctionCall::getRuntimeDefinition() const {
 
   // Try to get CTU definition only if CTUDir is provided.
   if (!Opts.naiveCTUEnabled())
     return {};
 
   cross_tu::CrossTranslationUnitContext &CTUCtx =
       *Engine->getCrossTranslationUnitContext();
   llvm::Expected<const FunctionDecl *> CTUDeclOrError =
-      CTUCtx.getCrossTUDefinition(FD, Opts.getCTUDir(), Opts.getCTUIndexName());
+      CTUCtx.getCrossTUDefinition(FD, Opts.getCTUDir(), Opts.getCTUIndexName(),
+                                  Opts.AnalyzerDisplayCTUProgress);
 
   if (!CTUDeclOrError) {
     handleAllErrors(CTUDeclOrError.takeError(),
                     [&](const cross_tu::IndexError &IE) {
                       CTUCtx.emitCrossTUDiagnostics(IE);
                     });
     return {};
   }

test/Analysis/ctu-main.cpp

 // RUN: mkdir -p %T/ctudir
 // RUN: %clang_cc1 -triple x86_64-pc-linux-gnu -emit-pch -o %T/ctudir/ctu-other.cpp.ast %S/Inputs/ctu-other.cpp
 // RUN: %clang_cc1 -triple x86_64-pc-linux-gnu -emit-pch -o %T/ctudir/ctu-chain.cpp.ast %S/Inputs/ctu-chain.cpp
 // RUN: cp %S/Inputs/externalFnMap.txt %T/ctudir/
 // RUN: %clang_cc1 -triple x86_64-pc-linux-gnu -fsyntax-only -analyze -analyzer-checker=core,debug.ExprInspection -analyzer-config experimental-enable-naive-ctu-analysis=true -analyzer-config ctu-dir=%T/ctudir -verify %s
+// RUN: %clang_cc1 -triple x86_64-pc-linux-gnu -fsyntax-only -analyze -analyzer-checker=core,debug.ExprInspection -analyzer-config experimental-enable-naive-ctu-analysis=true -analyzer-config ctu-dir=%T/ctudir -analyzer-display-ctu-progress 2>&1 %s | FileCheck %s
+
+// CHECK: ANALYZE (CTU loaded AST for source file): {{.*}}/ctu-other.cpp
+// CHECK: ANALYZE (CTU loaded AST for source file): {{.*}}/ctu-chain.cpp
 
 #include "ctu-hdr.h"
 
 void clang_analyzer_eval(int);
 
 int f(int);
 int g(int);
 int h(int);