Allow clang to write compilation database records

[joerg]

Summary

When integrating compilation database output into existing build systems, two approaches dominate so far. Ad-hoc implementation of the JSON output rules or using compiler wrappers. This patch adds a new option "-MJ foo.json" which gives a slightly cleaned up compilation record. The output is a fragment, i.e. you still need to add the array markers, but it allows multiple files to be easy merged.

This way the only change in a build system is adding the option with potentially a per-target output file and merging the files with something like (echo '['; cat *.o.json; echo ']' > compilation_database.json or filtering out the trailing comma for strict JSON compliance.

[joerg]

Use llvm::yaml::escape.

[joerg]

Move implementation into the Clang class, keep track of the raw_fd_stream. This avoids reopening it on multiple inputs and removes the need for append mode. Short circuit the function when -### is present. Add proper diagnostics on open failure.

Still not completely happy with the place and the associated need for the mutable member.

[klimek]

Adding some folks. One question is whether we can use the additional output stuff doug added at some point for this.

[joerg]

Add test case.

[joerg]

Ping

[arichardson]

It would also be good to append --sysroot=DEFAULT_SYSROOT if DEFAULT_SYSROOT is not empty and there is no explicit --sysroot flag. That way the compilation database can also be consumed by a libclang based too that does not have the same DEFAULT_SYSROOT set.

[klimek]

LG. We can always add more intricate ways later on.

include/clang/Driver/Options.td

def M : Flag<["-"], "M">, Group<M_Group>,
     HelpText<"Like -MD, but also implies -E and writes to stdout by default">;
 def MM : Flag<["-"], "MM">, Group<M_Group>,
     HelpText<"Like -MMD, but also implies -E and writes to stdout by default">;
 def MF : JoinedOrSeparate<["-"], "MF">, Group<M_Group>,
     HelpText<"Write depfile output from -MMD, -MD, -MM, or -M to <file>">,
     MetaVarName<"<file>">;
 def MG : Flag<["-"], "MG">, Group<M_Group>, Flags<[CC1Option]>,
     HelpText<"Add missing headers to depfile">;
+def MJ : JoinedOrSeparate<["-"], "MJ">, Group<M_Group>,
+    HelpText<"Write a compilation database entry per input">;
 def MP : Flag<["-"], "MP">, Group<M_Group>, Flags<[CC1Option]>,
     HelpText<"Create phony target for each dependency (other than main file)">;
 def MQ : JoinedOrSeparate<["-"], "MQ">, Group<M_Group>, Flags<[CC1Option]>,
     HelpText<"Specify name of main file output to quote in depfile">;
 def MT : JoinedOrSeparate<["-"], "MT">, Group<M_Group>, Flags<[CC1Option]>,
     HelpText<"Specify name of main file output in depfile">;
 def MV : Flag<["-"], "MV">, Group<M_Group>, Flags<[CC1Option]>,
     HelpText<"Use NMake/Jom format for the depfile">;

lib/Driver/Tools.cpp

   bool IsPIE;
   std::tie(RelocationModel, PICLevel, IsPIE) =
       ParsePICArgs(ToolChain, ToolChain.getTriple(), Args);
 
   if (RelocationModel != llvm::Reloc::Static)
     CmdArgs.push_back("-KPIC");
 }
 
+static void QuoteJSONString(llvm::raw_fd_ostream &Stream, StringRef Str) {
+  Stream << "\"";
+  if (Str.find_first_of("\\\"") == Str.npos) {
+    Stream << Str;
+  } else {
+    for (auto ch: Str) {
+      if (ch == '\\' || ch == '"')
+        Stream << '\\';
+      Stream << ch;
+    }
+  }
+  Stream << "\"";
+}
+
+static void DumpCompilationDatabase(const Driver &D, StringRef Filename, StringRef Target, const InputInfo &Output,
+                                    const InputInfo &Input, const ArgList &Args) {
+  std::error_code EC;
+  llvm::raw_fd_ostream File(Filename, EC, llvm::sys::fs::F_Text | llvm::sys::fs::F_Append);
+  if (EC) {
+    //errs() << "Failed to open " << Filename << ": " << EC.message() << "\n" ;
+    return;
+  }
+  SmallString<128> Buf;
+  if (llvm::sys::fs::current_path(Buf))
+    Buf = ".";
+  File << "{ \"directory\": ";
+  QuoteJSONString(File, Buf);
+  File << ", \"file\": ";
+  QuoteJSONString(File, Input.getFilename());
+  File << ", \"output\": ";
+  QuoteJSONString(File, Output.getFilename());
+
+  File << ", \"arguments\": [";
+  QuoteJSONString(File, D.ClangExecutable);
+  File << ", ";
+  Buf = "-x";
+  Buf += types::getTypeName(Input.getType());
+  QuoteJSONString(File, Buf);
+  File << ", ";
+  QuoteJSONString(File, Input.getFilename());
+  for (auto &A: Args) {
+    auto &O = A->getOption();
+    // Skip language selection, which is positional.
+    if (O.getID() == options::OPT_x)
+      continue;
+    // Skip writing dependency output and the compiliation database itself.
+    if (O.getGroup().isValid() && O.getGroup().getID() == options::OPT_M_Group)
+      continue;
+    // Skip inputs.
+    if (O.getKind() == Option::InputClass)
+      continue;
+    // All other arguments are quoted and appended.
+    ArgStringList ASL;
+    A->render(Args, ASL);
+    for (auto &it: ASL) {
+      File << ", ";
+      QuoteJSONString(File, it);
+    }
+  }
+  File << ", ";
+  Buf = "--target=";
+  Buf += Target;
+  QuoteJSONString(File, Buf);
+  File << "]},\n";
+}
+
 void Clang::ConstructJob(Compilation &C, const JobAction &JA,
                          const InputInfo &Output, const InputInfoList &Inputs,
                          const ArgList &Args, const char *LinkingOutput) const {
   const llvm::Triple &Triple = getToolChain().getEffectiveTriple();
   const std::string &TripleStr = Triple.getTriple();
 
   bool KernelOrKext =
       Args.hasArg(options::OPT_mkernel, options::OPT_fapple_kext);

   //
   // FIXME: Implement custom jobs for internal actions.
   CmdArgs.push_back("-cc1");
 
   // Add the "effective" target triple.
   CmdArgs.push_back("-triple");
   CmdArgs.push_back(Args.MakeArgString(TripleStr));
 
+  if (const Arg *MJ = Args.getLastArg(options::OPT_MJ))
+    DumpCompilationDatabase(C.getDriver(), MJ->getValue(), TripleStr, Output, Input, Args);
+  Args.ClaimAllArgs(options::OPT_MJ);
+
   if (IsCuda) {
     // We have to pass the triple of the host if compiling for a CUDA device and
     // vice-versa.
     std::string NormalizedTriple;
     if (JA.isDeviceOffloading(Action::OFK_Cuda))
       NormalizedTriple = C.getSingleOffloadToolChain<Action::OFK_Host>()
                              ->getTriple()
                              .normalize();