[driver] add a new option `-gen-cdb-fragment-path` to emit a fragment of a compilation database for each compilation

[arphaman]

Summary

This patch adds a new option called -gen-cdb-fragment-path to the driver, which can be used to specify a directory path to which clang can emit a fragment of a CDB for each compilation it needs to invoke.

The CDB fragment is emitted into a unique file in the specified directory. It contains the -cc1 clang invocation in its command. The file itself is actually a valid standalone CDB (if you disregard the not yet well supported -cc1 innovation by the CDB, which I'll fix). To load the full CDB that can be emitted during a build, I'm going to add a new CDB reader from a directory that reads all fragments and aggregates them into one CDB in a follow-up patch.

This is useful to setup verification infrastructure for clang-scan-deps for some projects that don't have a way of constructing a CDB from their build.

[jkorous]

clang/lib/Driver/Driver.cpp

1044

IIUC this feature might end up being used from some test script. I'm wondering - what would happen had the argument been an empty string?

clang -c foo.c -gen-cdb-fragment-path ""

4868

I am just wondering whether we shouldn't notice user about failures, possibly with some error message.

4896

It feels like this might have been already implemented somewhere else.

Couldn't we use for example llvm::yaml::escape()?

https://llvm.org/doxygen/namespacellvm_1_1yaml.html#aefed9cb3f107aee0cff4d325c7d689ae

[tschuett]

How does it relate to the -MJ option?

[arphaman]

In D66555#1640603, @tschuett wrote:

How does it relate to the -MJ option?

Ah, I wasn't aware that such option existed. I should use it instead.

[arphaman]

Reimplement the new option on top of -MJ. -MJ is still the preferred option if both are specified.

[arphaman]

@jkorous I addressed your comments as well ("" directory is now allowed, error is reported, existing escape used).

[jkorous]

clang/lib/Driver/ToolChains/Clang.cpp

2067 ↗

(On Diff #216717)

Do we still need this now?

2089 ↗

(On Diff #216717)

Just to make sure - the error different on purpose here?

2092 ↗

(On Diff #216717)

I'm not familiar with intended lifecycle of CompilationDatabase member. Just to make sure - we need neither to preserve the original value nor reset the stream after we're done here (as in - nobody is going to write anything after we're done here), right?

[jkorous]

clang/lib/Driver/ToolChains/Clang.h

99 ↗

(On Diff #216717)

BTW this looks kind of funny const InputInfo &Output, const InputInfo &Input.

[arphaman]

Address review comments.

[arphaman]

clang/lib/Driver/ToolChains/Clang.cpp

2067 ↗

(On Diff #216717)

No, the CWD should actually be read in DumpCompilationDatabase. I updated that method.

2089 ↗

(On Diff #216717)

Not really needed I guess, I reused the error.

2092 ↗

(On Diff #216717)

Yes, we should either preserve or not reset the stream once it's created. Multiple clang invocations can be constructed from one driver run, so we want to keep the stream open while the driver is running.

clang/lib/Driver/ToolChains/Clang.h

99 ↗

(On Diff #216717)

Yes, that's unfortunate type naming in Clang :(

clang/include/clang/Driver/Driver.h

private:
 
   /// Arguments originated from command line.
   std::unique_ptr<llvm::opt::InputArgList> CLOptions;
 
   /// Whether to check that input files exist when constructing compilation
   /// jobs.
   unsigned CheckInputsExist : 1;
 
+  /// The directory to which the driver should write out a compilation database
+  /// fragment for its invocation.
+  std::string PathToCDBFragmentDir;
+
 public:
+  /// Returns the directory to which a compilation database fragment should be
+  /// written out for its invocation, or \c None if the fragments should not be
+  /// saved.
+  Optional<StringRef> getPathToCDBFragmentDir() const {
+    if (PathToCDBFragmentDir.empty())
+      return None;
+    return StringRef(PathToCDBFragmentDir);
+  }
+
   /// Force clang to emit reproducer for driver invocation. This is enabled
   /// indirectly by setting FORCE_CLANG_DIAGNOSTICS_CRASH environment variable
   /// or when using the -gen-reproducer driver flag.
   unsigned GenReproducer : 1;
 
 private:
   /// Certain options suppress the 'no input files' warning.
   unsigned SuppressMissingInputWarning : 1;
public:
   /// digits to look for is equal to Digits.size().
   ///
   /// \return True if the entire string was parsed and there are
   /// no extra characters remaining at the end.
   static bool GetReleaseVersion(StringRef Str,
                                 MutableArrayRef<unsigned> Digits);
   /// Compute the default -fmodule-cache-path.
   static void getDefaultModuleCachePath(SmallVectorImpl<char> &Result);
+
+  /// Emits a compilation database fragment to the given directory, that
+  /// contains an entry for the specified filename and arguments.
+  void
+  emitCompilationDatabaseFragment(StringRef DestDir, StringRef Executable,
+                                  StringRef Filename,
+                                  const llvm::opt::ArgStringList &Args) const;
 };
 
 /// \return True if the last defined optimization level is -Ofast.
 /// And False otherwise.
 bool isOptimizationLevelFast(const llvm::opt::ArgList &Args);
 
 } // end namespace driver
 } // end namespace clang
 
 #endif

clang/include/clang/Driver/Job.h

public:
   /// \remark If the environment remains unset, then the environment
   ///         from the parent process will be used.
   void setEnvironment(llvm::ArrayRef<const char *> NewEnvironment);
 
   const char *getExecutable() const { return Executable; }
 
   const llvm::opt::ArgStringList &getArguments() const { return Arguments; }
 
+  const llvm::opt::ArgStringList &getInputFilenames() const {
+    return InputFilenames;
+  }
+
   /// Print a command argument, and optionally quote it.
   static void printArg(llvm::raw_ostream &OS, StringRef Arg, bool Quote);
 
   /// Set whether to print the input filenames when executing.
   void setPrintInputFilenames(bool P) { PrintInputFilenames = P; }
 };
 
 /// Like Command, but with a fallback which is executed in case

clang/include/clang/Driver/Options.td

def ccc_arcmt_migrate : Separate<["-"], "ccc-arcmt-migrate">, InternalDriverOpt,
   HelpText<"Apply modifications and produces temporary files that conform to ARC">;
 def arcmt_migrate_report_output : Separate<["-"], "arcmt-migrate-report-output">,
   HelpText<"Output path for the plist report">,  Flags<[CC1Option]>;
 def arcmt_migrate_emit_arc_errors : Flag<["-"], "arcmt-migrate-emit-errors">,
   HelpText<"Emit ARC errors even if the migrator can fix them">,
   Flags<[CC1Option]>;
 def gen_reproducer: Flag<["-"], "gen-reproducer">, InternalDebugOpt,
   HelpText<"Auto-generates preprocessed source files and a reproduction script">;
+def gen_cdb_fragment_path: Separate<["-"], "gen-cdb-fragment-path">, InternalDebugOpt,
+  HelpText<"Emit a compilation database fragment to the specified directory">;
 
 def _migrate : Flag<["--"], "migrate">, Flags<[DriverOption]>,
   HelpText<"Run the migrator">;
 def ccc_objcmt_migrate : Separate<["-"], "ccc-objcmt-migrate">,
   InternalDriverOpt,
   HelpText<"Apply modifications and produces temporary files to migrate to "
    "modern ObjC syntax">;
 def objcmt_migrate_literals : Flag<["-"], "objcmt-migrate-literals">, Flags<[CC1Option]>,

clang/lib/Driver/Compilation.cpp

if (getDriver().CCPrintOptions && getDriver().CCPrintOptionsFilename) {
       OS = OwnedStream.get();
     }
 
     if (getDriver().CCPrintOptions)
       *OS << "[Logging clang options]";
 
     C.Print(*OS, "\n", /*Quote=*/getDriver().CCPrintOptions);
   }
+  if (auto CDBOutputDir = getDriver().getPathToCDBFragmentDir()) {
+    if (isa<AssembleJobAction>(C.getSource()) ||
+        isa<BackendJobAction>(C.getSource())) {
+      for (const auto &F : C.getInputFilenames())
+        getDriver().emitCompilationDatabaseFragment(
+            *CDBOutputDir, C.getExecutable(), F, C.getArguments());
+    }
+  }
 
   std::string Error;
   bool ExecutionFailed;
   int Res = C.Execute(Redirects, &Error, &ExecutionFailed);
   if (!Error.empty()) {
     assert(Res && "Error string set with 0 result code!");
     getDriver().Diag(diag::err_drv_command_failure) << Error;
   }

clang/lib/Driver/Driver.cpp

Compilation *Driver::BuildCompilation(ArrayRef<const char *> ArgList) {
   // or -b.
   CCCPrintPhases = Args.hasArg(options::OPT_ccc_print_phases);
   CCCPrintBindings = Args.hasArg(options::OPT_ccc_print_bindings);
   if (const Arg *A = Args.getLastArg(options::OPT_ccc_gcc_name))
     CCCGenericGCCName = A->getValue();
   GenReproducer = Args.hasFlag(options::OPT_gen_reproducer,
                                options::OPT_fno_crash_diagnostics,
                                !!::getenv("FORCE_CLANG_DIAGNOSTICS_CRASH"));
+  if (const Arg *A = Args.getLastArg(options::OPT_gen_cdb_fragment_path))
+    PathToCDBFragmentDir = A->getValue();
   // FIXME: TargetTriple is used by the target-prefixed calls to as/ld
   // and getToolChain is const.
   if (IsCLMode()) {
     // clang-cl targets MSVC-style Win32.
     llvm::Triple T(TargetTriple);
     T.setOS(llvm::Triple::Win32);
     T.setVendor(llvm::Triple::PC);
     T.setEnvironment(llvm::Triple::MSVC);
Driver::getIncludeExcludeOptionFlagMasks(bool IsClCompatMode) const {
   }
 
   return std::make_pair(IncludedFlagsBitmask, ExcludedFlagsBitmask);
 }
 
 bool clang::driver::isOptimizationLevelFast(const ArgList &Args) {
   return Args.hasFlag(options::OPT_Ofast, options::OPT_O_Group, false);
 }
+
+void Driver::emitCompilationDatabaseFragment(StringRef DestDir,
+                                             StringRef Executable,
+                                             StringRef Filename,
+                                             const ArgStringList &Args) const {
+  auto CWD = getVFS().getCurrentWorkingDirectory();
+  if (!CWD)
+    return;
+
+  SmallString<256> Path = DestDir;
+  getVFS().makeAbsolute(Path);
+  auto Err = llvm::sys::fs::create_directory(Path, /*IgnoreExisting=*/true);
+  if (Err)
+    return;
+
+  llvm::sys::path::append(Path, Twine(llvm::sys::path::filename(Filename)) +
+                                    ".%%%%.json");
+  int FD;
+  SmallString<256> TempPath;
+  Err = llvm::sys::fs::createUniqueFile(Path, FD, TempPath);
+  if (Err)
+    return;
+
+  llvm::raw_fd_ostream OS(FD, /*ShouldClose=*/true);
+  OS << "[{\n";
+  OS << R"("directory":")" << *CWD << "\",\n";
+  OS << R"("command":")";
+  auto EmitCommandLineFragment = [&OS](StringRef Arg) {
+    for (char C : Arg) {
+      if (C == ' ' || C == '\'' || C == '"' || C == '\\')
+        OS << '\\';
+      OS << C;
+    }
+  };
+  EmitCommandLineFragment(Executable);
+  for (const auto &Arg : Args) {
+    OS << ' ';
+    EmitCommandLineFragment(Arg);
+  }
+  OS << "\",\n";
+  OS << R"("file":")" << Filename << "\"\n";
+  OS << "}]";
+}

clang/test/Driver/gen-cdb-fragment.c

+// REQUIRES: x86-registered-target
+// RUN: rm -rf %t.cdb
+// RUN: %clang -target x86_64-apple-macos10.15 -c %s -o -  -gen-cdb-fragment-path %t.cdb
+// RUN: ls %t.cdb | FileCheck --check-prefix=CHECK-LS %s
+// CHECK-LS: gen-cdb-fragment.c.{{.*}}.json
+
+// RUN: cat %t.cdb/*.json | FileCheck --check-prefix=CHECK %s
+// CHECK: [{
+// CHECK-NEXT: "directory":"{{.*}}",
+// CHECK-NEXT: "command":"{{.*}} -cc1 -triple x86_64-apple-macosx10.15.0 {{.*}}",
+// CHECK-NEXT: "file":"{{.*}}gen-cdb-fragment.c"
+// CHECK: }]
+
+// RUN: rm -rf %t.cdb
+// RUN: mkdir %t.cdb
+// RUN: ls %t.cdb | not FileCheck --check-prefix=CHECK-LS %s
+// RUN: %clang -target x86_64-apple-macos10.15 -S %s -o -  -gen-cdb-fragment-path %t.cdb
+// RUN: ls %t.cdb | FileCheck --check-prefix=CHECK-LS %s