[driver] add a new option `-gen-cdb-fragment-path` to emit a fragment of a compilation database for each compilation

[arphaman]

Summary

This patch adds a new option called -gen-cdb-fragment-path to the driver, which can be used to specify a directory path to which clang can emit a fragment of a CDB for each compilation it needs to invoke.

The CDB fragment is emitted into a unique file in the specified directory. It contains the -cc1 clang invocation in its command. The file itself is actually a valid standalone CDB (if you disregard the not yet well supported -cc1 innovation by the CDB, which I'll fix). To load the full CDB that can be emitted during a build, I'm going to add a new CDB reader from a directory that reads all fragments and aggregates them into one CDB in a follow-up patch.

This is useful to setup verification infrastructure for clang-scan-deps for some projects that don't have a way of constructing a CDB from their build.

[jkorous]

clang/lib/Driver/Driver.cpp

1044 ↗

(On Diff #216466)

IIUC this feature might end up being used from some test script. I'm wondering - what would happen had the argument been an empty string?

clang -c foo.c -gen-cdb-fragment-path ""

4868 ↗

(On Diff #216466)

I am just wondering whether we shouldn't notice user about failures, possibly with some error message.

4896 ↗

(On Diff #216466)

It feels like this might have been already implemented somewhere else.

Couldn't we use for example llvm::yaml::escape()?

https://llvm.org/doxygen/namespacellvm_1_1yaml.html#aefed9cb3f107aee0cff4d325c7d689ae

[tschuett]

How does it relate to the -MJ option?

[arphaman]

In D66555#1640603, @tschuett wrote:

How does it relate to the -MJ option?

Ah, I wasn't aware that such option existed. I should use it instead.

[arphaman]

Reimplement the new option on top of -MJ. -MJ is still the preferred option if both are specified.

[arphaman]

@jkorous I addressed your comments as well ("" directory is now allowed, error is reported, existing escape used).

[jkorous]

clang/lib/Driver/ToolChains/Clang.cpp

2068

Do we still need this now?

2090

Just to make sure - the error different on purpose here?

2093

I'm not familiar with intended lifecycle of CompilationDatabase member. Just to make sure - we need neither to preserve the original value nor reset the stream after we're done here (as in - nobody is going to write anything after we're done here), right?

[jkorous]

clang/lib/Driver/ToolChains/Clang.h

99

BTW this looks kind of funny const InputInfo &Output, const InputInfo &Input.

[arphaman]

Address review comments.

[arphaman]

clang/lib/Driver/ToolChains/Clang.cpp

2068

No, the CWD should actually be read in DumpCompilationDatabase. I updated that method.

2090

Not really needed I guess, I reused the error.

2093

Yes, we should either preserve or not reset the stream once it's created. Multiple clang invocations can be constructed from one driver run, so we want to keep the stream open while the driver is running.

clang/lib/Driver/ToolChains/Clang.h

99

Yes, that's unfortunate type naming in Clang :(

clang/include/clang/Driver/Options.td

def ccc_arcmt_migrate : Separate<["-"], "ccc-arcmt-migrate">, InternalDriverOpt,
   HelpText<"Apply modifications and produces temporary files that conform to ARC">;
 def arcmt_migrate_report_output : Separate<["-"], "arcmt-migrate-report-output">,
   HelpText<"Output path for the plist report">,  Flags<[CC1Option]>;
 def arcmt_migrate_emit_arc_errors : Flag<["-"], "arcmt-migrate-emit-errors">,
   HelpText<"Emit ARC errors even if the migrator can fix them">,
   Flags<[CC1Option]>;
 def gen_reproducer: Flag<["-"], "gen-reproducer">, InternalDebugOpt,
   HelpText<"Auto-generates preprocessed source files and a reproduction script">;
+def gen_cdb_fragment_path: Separate<["-"], "gen-cdb-fragment-path">, InternalDebugOpt,
+  HelpText<"Emit a compilation database fragment to the specified directory">;
 
 def _migrate : Flag<["--"], "migrate">, Flags<[DriverOption]>,
   HelpText<"Run the migrator">;
 def ccc_objcmt_migrate : Separate<["-"], "ccc-objcmt-migrate">,
   InternalDriverOpt,
   HelpText<"Apply modifications and produces temporary files to migrate to "
    "modern ObjC syntax">;
 def objcmt_migrate_literals : Flag<["-"], "objcmt-migrate-literals">, Flags<[CC1Option]>,

clang/lib/Driver/ToolChains/Clang.h

private:
   mutable std::unique_ptr<visualstudio::Compiler> CLFallback;
 
   mutable std::unique_ptr<llvm::raw_fd_ostream> CompilationDatabase = nullptr;
   void DumpCompilationDatabase(Compilation &C, StringRef Filename,
                                StringRef Target,
                                const InputInfo &Output, const InputInfo &Input,
                                const llvm::opt::ArgList &Args) const;
 
+  void DumpCompilationDatabaseFragmentToDir(
+      StringRef Dir, Compilation &C, StringRef Target, const InputInfo &Output,
+      const InputInfo &Input, const llvm::opt::ArgList &Args) const;
+
 public:
   Clang(const ToolChain &TC);
   ~Clang() override;
 
   bool hasGoodDiagnostics() const override { return true; }
   bool hasIntegratedAssembler() const override { return true; }
   bool hasIntegratedCPP() const override { return true; }
   bool canEmitIR() const override { return true; }

clang/lib/Driver/ToolChains/Clang.cpp

if (!CompilationDatabase) {
     if (EC) {
       D.Diag(clang::diag::err_drv_compilationdatabase) << Filename
                                                        << EC.message();
       return;
     }
     CompilationDatabase = std::move(File);
   }
   auto &CDB = *CompilationDatabase;
-  SmallString<128> Buf;
-  if (llvm::sys::fs::current_path(Buf))
-    Buf = ".";
-  CDB << "{ \"directory\": \"" << escape(Buf) << "\"";
+  auto CWD = D.getVFS().getCurrentWorkingDirectory();
+  if (!CWD)
+    CWD = ".";
+  CDB << "{ \"directory\": \"" << escape(*CWD) << "\"";
   CDB << ", \"file\": \"" << escape(Input.getFilename()) << "\"";
   CDB << ", \"output\": \"" << escape(Output.getFilename()) << "\"";
   CDB << ", \"arguments\": [\"" << escape(D.ClangExecutable) << "\"";
+  SmallString<128> Buf;
   Buf = "-x";
   Buf += types::getTypeName(Input.getType());
   CDB << ", \"" << escape(Buf) << "\"";
   if (!D.SysRoot.empty() && !Args.hasArg(options::OPT__sysroot_EQ)) {
     Buf = "--sysroot=";
     Buf += D.SysRoot;
     CDB << ", \"" << escape(Buf) << "\"";
   }
   CDB << ", \"" << escape(Input.getFilename()) << "\"";
   for (auto &A: Args) {
     auto &O = A->getOption();
     // Skip language selection, which is positional.
     if (O.getID() == options::OPT_x)
       continue;
     // Skip writing dependency output and the compilation database itself.
     if (O.getGroup().isValid() && O.getGroup().getID() == options::OPT_M_Group)
       continue;
+    if (O.getID() == options::OPT_gen_cdb_fragment_path)
+      continue;
     // Skip inputs.
     if (O.getKind() == Option::InputClass)
       continue;
     // All other arguments are quoted and appended.
     ArgStringList ASL;
     A->render(Args, ASL);
     for (auto &it: ASL)
       CDB << ", \"" << escape(it) << "\"";
   }
   Buf = "--target=";
   Buf += Target;
   CDB << ", \"" << escape(Buf) << "\"]},\n";
 }
 
+void Clang::DumpCompilationDatabaseFragmentToDir(
+    StringRef Dir, Compilation &C, StringRef Target, const InputInfo &Output,
+    const InputInfo &Input, const llvm::opt::ArgList &Args) const {
+  // If this is a dry run, do not create the compilation database file.
+  if (C.getArgs().hasArg(options::OPT__HASH_HASH_HASH))
+    return;
+
+  if (CompilationDatabase)
+    DumpCompilationDatabase(C, "", Target, Output, Input, Args);
+
+  SmallString<256> Path = Dir;
+  const auto &Driver = C.getDriver();
+  Driver.getVFS().makeAbsolute(Path);
+  auto Err = llvm::sys::fs::create_directory(Path, /*IgnoreExisting=*/true);
+  if (Err) {
+    Driver.Diag(diag::err_drv_compilationdatabase) << Dir << Err.message();
+    return;
+  }
+
+  llvm::sys::path::append(
+      Path,
+      Twine(llvm::sys::path::filename(Input.getFilename())) + ".%%%%.json");
+  int FD;
+  SmallString<256> TempPath;
+  Err = llvm::sys::fs::createUniqueFile(Path, FD, TempPath);
+  if (Err) {
+    Driver.Diag(diag::err_drv_compilationdatabase) << Path << Err.message();
+    return;
+  }
+  CompilationDatabase =
+      std::make_unique<llvm::raw_fd_ostream>(FD, /*shouldClose=*/true);
+  DumpCompilationDatabase(C, "", Target, Output, Input, Args);
+}
+
 static void CollectArgsForIntegratedAssembler(Compilation &C,
                                               const ArgList &Args,
                                               ArgStringList &CmdArgs,
                                               const Driver &D) {
   if (UseRelaxAll(C, Args))
     CmdArgs.push_back("-mrelax-all");
 
   // Only default to -mincremental-linker-compatible if we think we are
   // targeting the MSVC linker.
   bool DefaultIncrementalLinkerCompatible =
       C.getDefaultToolChain().getTriple().isWindowsMSVCEnvironment();
void Clang::ConstructJob(Compilation &C, const JobAction &JA,
 
   // Add the "effective" target triple.
   CmdArgs.push_back("-triple");
   CmdArgs.push_back(Args.MakeArgString(TripleStr));
 
   if (const Arg *MJ = Args.getLastArg(options::OPT_MJ)) {
     DumpCompilationDatabase(C, MJ->getValue(), TripleStr, Output, Input, Args);
     Args.ClaimAllArgs(options::OPT_MJ);
+  } else if (const Arg *GenCDBFragment =
+                 Args.getLastArg(options::OPT_gen_cdb_fragment_path)) {
+    DumpCompilationDatabaseFragmentToDir(GenCDBFragment->getValue(), C,
+                                         TripleStr, Output, Input, Args);
+    Args.ClaimAllArgs(options::OPT_gen_cdb_fragment_path);
   }
 
   if (IsCuda || IsHIP) {
     // We have to pass the triple of the host if compiling for a CUDA/HIP device
     // and vice-versa.
     std::string NormalizedTriple;
     if (JA.isDeviceOffloading(Action::OFK_Cuda) ||
         JA.isDeviceOffloading(Action::OFK_HIP))

clang/test/Driver/gen-cdb-fragment.c

+// REQUIRES: x86-registered-target
+// RUN: rm -rf %t.cdb
+// RUN: %clang -target x86_64-apple-macos10.15 -c %s -o -  -gen-cdb-fragment-path %t.cdb
+// RUN: ls %t.cdb | FileCheck --check-prefix=CHECK-LS %s
+// CHECK-LS: gen-cdb-fragment.c.{{.*}}.json
+
+// RUN: cat %t.cdb/*.json | FileCheck --check-prefix=CHECK %s
+// CHECK: { "directory": "{{.*}}", "file": "{{.*}}gen-cdb-fragment.c", "output": "-", "arguments": [{{.*}}, "--target=x86_64-apple-macos10.15"{{.*}}]},
+// RUN: cat %t.cdb/*.json | FileCheck --check-prefix=CHECK-FLAG %s
+// CHECK-FLAG-NOT: -gen-cdb-fragment-path
+
+// RUN: rm -rf %t.cdb
+// RUN: mkdir %t.cdb
+// RUN: ls %t.cdb | not FileCheck --check-prefix=CHECK-LS %s
+// RUN: %clang -target x86_64-apple-macos10.15 -S %s -o -  -gen-cdb-fragment-path %t.cdb
+// RUN: ls %t.cdb | FileCheck --check-prefix=CHECK-LS %s
+
+// Empty path is equivalent to '.'
+// RUN: rm -rf %t.cdb
+// RUN: mkdir %t.cdb
+// RUN: %clang -target x86_64-apple-macos10.15 -working-directory %t.cdb -c %s -o -  -gen-cdb-fragment-path ""
+// RUN: ls %t.cdb | FileCheck --check-prefix=CHECK-LS %s
+// RUN: cat %t.cdb/*.json | FileCheck --check-prefix=CHECK-CWD %s
+// CHECK-CWD: "directory": "{{.*}}.cdb"
+
+// -### does not emit the CDB fragment
+// RUN: rm -rf %t.cdb
+// RUN: mkdir %t.cdb
+// RUN: %clang -target x86_64-apple-macos10.15 -S %s -o -  -gen-cdb-fragment-path %t.cdb -###
+// RUN: ls %t.cdb | not FileCheck --check-prefix=CHECK-LS %s
+
+// -MJ is preferred over -gen-cdb-fragment-path
+// RUN: rm -rf %t.cdb
+// RUN: mkdir %t.cdb
+// RUN: %clang -target x86_64-apple-macos10.15 -S %s -o -  -gen-cdb-fragment-path %t.cdb -MJ %t.out
+// RUN: ls %t.cdb | not FileCheck --check-prefix=CHECK-LS %s
+// RUN: FileCheck %s < %t.out