[analyzer][ctu] Allow loading invocation list from a compilation database automatically detected from the ctu-dir

[OikawaKirie]

Summary

It seems to be simpler and more convenient to reuse the compilation database generated for ClangTool when using clang-check to run the analyzer.
In this patch, when the invocation list file is failed to be loaded, the ASTLoader will try to find a compilation database from the ctu-dir and convert the detected database to an invocation list.

For each compile command objects in the database:

• the "directory" entry is converted to option -working-directory appended to the invocation commands;
• the "file" entry is updated to an absolute path based on the "directory" entry as the index;
• and the "command" entry is converted to the invocation commands.

[martong]

Thank you @OikawaKirie for working on this many CTU related patches! I am going to find time for a thorough review and going to pursue @gamesh411 as well to do the same! On the other hand, it would be really useful if you could build a "Stack" from these patches, I mean could you please set which patch depends from which other (see "Edit related revisions")?

[OikawaKirie]

In D102149#2765402, @martong wrote:

Thank you @OikawaKirie for working on this many CTU related patches! I am going to find time for a thorough review and going to pursue @gamesh411 as well to do the same! On the other hand, it would be really useful if you could build a "Stack" from these patches, I mean could you please set which patch depends from which other (see "Edit related revisions")?

To make it easier to review, all these recently submitted patches are based on the main branch. Their code does not depend on each other.

Logically, they can be separated into two branches: invocation list and ctu index.
The invocation list related patches target on constructing the invocation list in ASTLoader from a compilation database. And the problems of using invocation lists are also fixed.
Whereas the ctu index related patches mainly focus on the delimiter character in the ctu index, which is generated by the clang-extdef-mapping tool.

If the "stack" means the logical relations, I will update that later. Otherwise, it is OK to start with any of the patches.

[martong]

Thank you for the patch!

Though, the idea is nice, there is a serious technical obstacle here: we cannot use the clangTooling lib as a dependency of the CTU lib because that would introduce a circular dependency. Actually, this was the reason for introducing the invocation list yaml file; we could not use the compilation database implementation from tooling (https://reviews.llvm.org/D75665?id=260238#inline-722328).

[steakhal]

This comment was removed by steakhal.

[OikawaKirie]

In D102149#2768541, @martong wrote:

Thank you for the patch!

Though, the idea is nice, there is a serious technical obstacle here: we cannot use the clangTooling lib as a dependency of the CTU lib because that would introduce a circular dependency. Actually, this was the reason for introducing the invocation list yaml file; we could not use the compilation database implementation from tooling (https://reviews.llvm.org/D75665?id=260238#inline-722328).

According to my recent experiences on using clang-check, converting the compilation database to an invocation list is also an acceptable solution.
Currently, a better solution may be adding a tool to handle the conversion, just as what has been mentioned in the revision you presented. Although, it would be a very simple python script such as:

a = dict()
for i in json.load(open("/path/to/compile_commands.json")):
    a[os.path.abspath(os.path.join(i['directory'], i['file']))] = \
        (shlex.split(i['command']) if 'command' in i else i['arguments']) + \
        ['-Xclang', '-working-directory=' + i['directory']]
print(yaml.dump(a));

If you agree with this idea, I will follow the way to create a tool for the conversion and drop this revision.

Actually, I am writing a makefile to schedule the ctu-index generation as well as analyzing the code with the clang-check tool. And the python script snippet presented above is just a part of my makefile.
Maybe I can add my makefile to as a part of the analyzer.

clang/lib/CrossTU/CrossTranslationUnit.cpp

711–712

The approach of converting a compilation database to an invocation list is just simply adding the -working-directory argument for the "directory" entry for each compile command object. Therefore, a script doing the conversion can make it simpler and easier to use.

The main reason why this patch is submitted is I previously do not know the -working-directory argument. Without this argument, it would be very difficult and complex to convert a compilation database to an invocation list.

If you agree that my approach of converting the database (presented in the for loop) is OK, I will continue with making a tool for the conversion.

[OikawaKirie]

It seems that this issue has better solutions outside the CTU module, I will drop this revision as mentioned above.
Sorry for the delay.

clang/include/clang/CrossTU/CrossTranslationUnit.h

enum class index_error_code {
   failed_import,
   failed_to_get_external_ast,
   failed_to_generate_usr,
   triple_mismatch,
   lang_mismatch,
   lang_dialect_mismatch,
   load_threshold_reached,
   invocation_list_ambiguous,
-  invocation_list_file_not_found,
   invocation_list_empty,
   invocation_list_wrong_format,
-  invocation_list_lookup_unsuccessful
+  loader_lookup_unsuccessful,
+  loader_index_not_found,
 };
 
 class IndexError : public llvm::ErrorInfo<IndexError> {
 public:
   static char ID;
   IndexError(index_error_code C) : Code(C), LineNo(0) {}
   IndexError(index_error_code C, std::string FileName, int LineNo = 0)
       : Code(C), FileName(std::move(FileName)), LineNo(LineNo) {}

clang/lib/CrossTU/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   Support
   )
 
 add_clang_library(clangCrossTU
   CrossTranslationUnit.cpp
 
   LINK_LIBS
   clangAST
   clangBasic
   clangFrontend
   clangIndex
+  clangTooling
   )

clang/lib/CrossTU/CrossTranslationUnit.cpp

 #include "clang/AST/Decl.h"
 #include "clang/AST/ParentMapContext.h"
 #include "clang/Basic/TargetInfo.h"
 #include "clang/CrossTU/CrossTUDiagnostic.h"
 #include "clang/Frontend/ASTUnit.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/TextDiagnosticPrinter.h"
 #include "clang/Index/USRGeneration.h"
+#include "clang/Tooling/CompilationDatabase.h"
+#include "clang/Tooling/Tooling.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/ADT/Triple.h"
 #include "llvm/Option/ArgList.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/ManagedStatic.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/YAMLParser.h"
case index_error_code::lang_mismatch:
       return "Language mismatch";
     case index_error_code::lang_dialect_mismatch:
       return "Language dialect mismatch";
     case index_error_code::load_threshold_reached:
       return "Load threshold reached";
     case index_error_code::invocation_list_ambiguous:
       return "Invocation list file contains multiple references to the same "
              "source file.";
-    case index_error_code::invocation_list_file_not_found:
-      return "Invocation list file is not found.";
     case index_error_code::invocation_list_empty:
       return "Invocation list file is empty.";
     case index_error_code::invocation_list_wrong_format:
       return "Invocation list file is in wrong format.";
-    case index_error_code::invocation_list_lookup_unsuccessful:
-      return "Invocation list file does not contain the requested source file.";
+    case index_error_code::loader_lookup_unsuccessful:
+      return "Compile commands for the requested source file is not found.";
+    case index_error_code::loader_index_not_found:
+      return "Invocation list file or loader compilation database not found.";
     }
     llvm_unreachable("Unrecognized index_error_code.");
   }
 };
 
 static llvm::ManagedStatic<IndexErrorCategory> Category;
 } // end anonymous namespace
 
CrossTranslationUnitContext::ASTLoader::loadFromSource(
 
   if (llvm::Error InitError = lazyInitInvocationList())
     return std::move(InitError);
   assert(InvocationList);
 
   auto Invocation = InvocationList->find(SourceFilePath);
   if (Invocation == InvocationList->end())
     return llvm::make_error<IndexError>(
-        index_error_code::invocation_list_lookup_unsuccessful);
+        index_error_code::loader_lookup_unsuccessful);
 
   const InvocationListTy::mapped_type &InvocationCommand = Invocation->second;
 
   SmallVector<const char *, 32> CommandLineArgs(InvocationCommand.size());
   std::transform(InvocationCommand.begin(), InvocationCommand.end(),
                  CommandLineArgs.begin(),
                  [](auto &&CmdPart) { return CmdPart.c_str(); });
 
parseInvocationList(StringRef FileContent, llvm::sys::path::Style PathStyle) {
   return InvocationList;
 }
 
 llvm::Error CrossTranslationUnitContext::ASTLoader::lazyInitInvocationList() {
   /// Lazily initialize the invocation list member used for on-demand parsing.
   if (InvocationList)
     return llvm::Error::success();
 
-  llvm::ErrorOr<std::unique_ptr<llvm::MemoryBuffer>> FileContent =
-      llvm::MemoryBuffer::getFile(InvocationListFilePath);
-  if (!FileContent)
-    return llvm::make_error<IndexError>(
-        index_error_code::invocation_list_file_not_found);
+  llvm::Optional<llvm::Error> InvocationListParsingError;
+  if (llvm::ErrorOr<std::unique_ptr<llvm::MemoryBuffer>> FileContent =
+          llvm::MemoryBuffer::getFile(InvocationListFilePath)) {
     std::unique_ptr<llvm::MemoryBuffer> ContentBuffer = std::move(*FileContent);
-  assert(ContentBuffer && "If no error was produced after loading, the pointer "
-                          "should not be nullptr.");
+    assert(ContentBuffer && "If no error was produced after loading, the "
+                            "pointer should not be nullptr.");
 
     llvm::Expected<InvocationListTy> ExpectedInvocationList =
         parseInvocationList(ContentBuffer->getBuffer(), PathStyle);
 
-  if (!ExpectedInvocationList)
-    return ExpectedInvocationList.takeError();
-
+    if (ExpectedInvocationList) {
       InvocationList = *ExpectedInvocationList;
+      return llvm::Error::success();
+    }
+
+    InvocationListParsingError = ExpectedInvocationList.takeError();
+  }
+
+  std::string CDBLoadingErrorMsg;
+  if (auto CDB = tooling::CompilationDatabase::autoDetectFromDirectory(
+                 CTUDir, CDBLoadingErrorMsg)) {
+    InvocationListTy IL;
+
+    for (auto &CC : CDB->getAllCompileCommands()) {
+      SmallString<256> Filename(CC.Directory);
+      if (llvm::sys::path::is_absolute(CC.Filename)) Filename = CC.Filename;
+      else llvm::sys::path::append(Filename, CC.Filename);
+      llvm::sys::path::remove_dots(Filename);
+
+      if (IL.end() != IL.find(Filename)) {
+        IL.clear();
+        break;
+      }
+
+      auto &List = IL[Filename];
+      for (auto &Argv : CC.CommandLine)
+        List.emplace_back(Argv);
+
+      List.emplace_back("-Xclang");
+      List.emplace_back("-working-directory=" + CC.Directory);
+    }
 
+    if (!IL.empty()) {
+      InvocationList = IL;
+      // Neutralize the stored error during parsing invocation list.
+      if (InvocationListParsingError) {
+        handleAllErrors(std::move(*InvocationListParsingError),
+                        [&](const IndexError &) {});
+      }
       return llvm::Error::success();
     }
+  }
+
+  // If an invocation list parsing error is stored, return this error.
+  // Otherwise, report not found.
+  return InvocationListParsingError
+             ? std::move(*InvocationListParsingError)
+             : llvm::make_error<IndexError>(
+                   index_error_code::loader_index_not_found);
+}
 
 template <typename T>
 llvm::Expected<const T *>
 CrossTranslationUnitContext::importDefinitionImpl(const T *D, ASTUnit *Unit) {
   assert(hasBodyOrInit(D) && "Decls to be imported should have body or init.");
 
   assert(&D->getASTContext() == &Unit->getASTContext() &&
          "ASTContext of Decl and the unit should match.");

clang/test/Analysis/ctu-on-demand-parsing.c

 // RUN: rm -rf %t
 // RUN: mkdir -p %t
 // RUN: cp "%s" "%t/ctu-on-demand-parsing.c"
 // RUN: cp "%S/Inputs/ctu-other.c" "%t/ctu-other.c"
 //
 // Path substitutions on Windows platform could contain backslashes. These are escaped in the json file.
 // compile_commands.json is only needed for extdef_mapping, not for the analysis itself.
 // RUN: echo '[{"directory":"%t","command":"gcc -std=c89 -Wno-visibility ctu-other.c","file":"ctu-other.c"}]' | sed -e 's/\\/\\\\/g' > %t/compile_commands.json
 //
 // RUN: echo '"%t/ctu-other.c": ["gcc", "-std=c89", "-Wno-visibility", "ctu-other.c"]' | sed -e 's/\\/\\\\/g' > %t/invocations.yaml
 //
 // RUN: cd "%t" && %clang_extdef_map "%t/ctu-other.c" > externalDefMap.txt
 //
+// Run with invocation list.
 // RUN: cd "%t" && %clang_cc1 -fsyntax-only -std=c89 -analyze \
 // RUN:   -analyzer-checker=core,debug.ExprInspection \
 // RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
 // RUN:   -analyzer-config ctu-dir=. \
 // RUN:   -analyzer-config ctu-invocation-list=invocations.yaml \
 // RUN:   -verify ctu-on-demand-parsing.c
 //
+// Run with compilation database.
+// RUN: rm %t/invocations.yaml
+// RUN: cd "%t" && %clang_cc1 -fsyntax-only -std=c89 -analyze \
+// RUN:   -analyzer-checker=core,debug.ExprInspection \
+// RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
+// RUN:   -analyzer-config ctu-dir=. \
+// RUN:   -verify ctu-on-demand-parsing.c
+//
 // FIXME: Path handling should work on all platforms.
 // REQUIRES: system-linux
 
 void clang_analyzer_eval(int);
 
 // Test typedef and global variable in function.
 typedef struct {
   int a;

clang/test/Analysis/ctu-on-demand-parsing.cpp

 // RUN: rm -rf %t
 // RUN: mkdir -p %t/Inputs
 // RUN: cp %s %t/ctu-on-demand-parsing.cpp
 // RUN: cp %S/ctu-hdr.h %t/ctu-hdr.h
 // RUN: cp %S/Inputs/ctu-chain.cpp %t/Inputs/ctu-chain.cpp
 // RUN: cp %S/Inputs/ctu-other.cpp %t/Inputs/ctu-other.cpp
 //
 // Path substitutions on Windows platform could contain backslashes. These are escaped in the json file.
 // compile_commands.json is only needed for the extdef_mapping, not for the analysis itself.
 // RUN: echo '[{"directory":"%t/Inputs","command":"clang++ ctu-chain.cpp","file":"ctu-chain.cpp"},{"directory":"%t/Inputs","command":"clang++ ctu-other.cpp","file":"ctu-other.cpp"}]' | sed -e 's/\\/\\\\/g' > %t/compile_commands.json
 //
 // RUN: echo '{"%t/Inputs/ctu-chain.cpp": ["g++", "%t/Inputs/ctu-chain.cpp"], "%t/Inputs/ctu-other.cpp": ["g++", "%t/Inputs/ctu-other.cpp"]}' | sed -e 's/\\/\\\\/g' > %t/invocations.yaml
 //
 // RUN: cd "%t" && %clang_extdef_map Inputs/ctu-chain.cpp Inputs/ctu-other.cpp > externalDefMap.txt
 //
+// Run with invocation list.
 // RUN: cd "%t" && %clang_analyze_cc1 \
 // RUN:   -analyzer-checker=core,debug.ExprInspection \
 // RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
 // RUN:   -analyzer-config ctu-dir=. \
 // RUN:   -analyzer-config ctu-invocation-list=invocations.yaml \
 // RUN:   -verify ctu-on-demand-parsing.cpp
 // RUN: cd "%t" && %clang_analyze_cc1 \
 // RUN:   -analyzer-checker=core,debug.ExprInspection \
 // RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
 // RUN:   -analyzer-config ctu-dir=. \
 // RUN:   -analyzer-config ctu-invocation-list=invocations.yaml \
 // RUN:   -analyzer-config display-ctu-progress=true ctu-on-demand-parsing.cpp 2>&1 | FileCheck %t/ctu-on-demand-parsing.cpp
 //
+// Run with compilation database.
+// RUN: rm %t/invocations.yaml
+// RUN: cd "%t" && %clang_analyze_cc1 \
+// RUN:   -analyzer-checker=core,debug.ExprInspection \
+// RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
+// RUN:   -analyzer-config ctu-dir=. \
+// RUN:   -verify ctu-on-demand-parsing.cpp
+// RUN: cd "%t" && %clang_analyze_cc1 \
+// RUN:   -analyzer-checker=core,debug.ExprInspection \
+// RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
+// RUN:   -analyzer-config ctu-dir=. \
+// RUN:   -analyzer-config display-ctu-progress=true ctu-on-demand-parsing.cpp 2>&1 | FileCheck %t/ctu-on-demand-parsing.cpp
+//
 // CHECK: CTU loaded AST file: {{.*}}ctu-other.cpp
 // CHECK: CTU loaded AST file: {{.*}}ctu-chain.cpp
 //
 // FIXME: Path handling should work on all platforms.
 // REQUIRES: system-linux
 
 #include "ctu-hdr.h"