Generate Libclang invocation reproducers using a new -cc1gen-reproducer option

[arphaman]

Summary

This patch is a follow up to the previous work that recorded Libclang invocations into temporary files: https://reviews.llvm.org/D40527.

It adds a new -cc1 mode to clang: -cc1gen-reproducer. The goal of this mode is to generate Clang reproducer files for Libclang tool invocation. The JSON format in the invocation files is not really intended to be stable, so Libclang and Clang should be of the same version when generating reproducers.
The new mode emits the information about the temporary files in the reproducers to stdout using JSON. It also injects additional Libclang-specific information about the reproducer to the reproducer's .sh files.

Thanks for taking a look!

[bruno]

Hi Alex,

Thanks for improving this.

• Instead of adding -cc1gen-reproducer, why can't you run that through -cc1 and have a flag similar to -###, which just prints the reproducer line?
• I didn't understand how you can use the final output information, can you give an example to illustrate?

include/clang/Driver/Driver.h

394 ↗

(On Diff #126032)

I assume the number of temporary files are usually small, can you switch to SmallVector here?

tools/driver/cc1gen_reproducer_main.cpp

187 ↗

(On Diff #126032)

int Result = 1;
....
(no need for the else clause)

[arphaman]

In D40983#958809, @bruno wrote:

Hi Alex,

Thanks for improving this.

• Instead of adding -cc1gen-reproducer, why can't you run that through -cc1 and have a flag similar to -###, which just prints the reproducer line?

It would be difficult to add this logic to the regular -cc1, as it would need to construct a real compiler invocation on top of the pseudo one which takes the libclang invocation file (because it has to load the actual compiler arguments from the libclang file). This would be difficult and quite disruptive to the code in the cc1 driver and the compiler invocation. The approach in this patch is much simpler to implement and maintain, and does not disrupt the code in the cc1 driver. This patch makes it more of a tool rather than a clang driver invocation.

• I didn't understand how you can use the final output information, can you give an example to illustrate?

Given a sample output like:

REPRODUCER:
{
"files":["/tmp/a.c","/tmp/a.sh"]
}

The tool's client (also a libclang client) will parse the JSON object with the files that make up the reproducer. It will then be able to do whatever it needs with those specific files.

include/clang/Driver/Driver.h

394 ↗

(On Diff #126032)

Yep.

[arphaman]

Address review comments

[bruno]

Makes sense, LGTM.

Should we add documentation explaining how to use this? I'm fine if it comes in a follow up commit.

[arphaman]

In D40983#968796, @bruno wrote:

Makes sense, LGTM.

Should we add documentation explaining how to use this? I'm fine if it comes in a follow up commit.

Sorry, just got time to get back to this now. Thanks for the review! I'll add a document for this new functionality tomorrow.

[arphaman]

In D40983#1059087, @arphaman wrote:

In D40983#968796, @bruno wrote:

Makes sense, LGTM.

Should we add documentation explaining how to use this? I'm fine if it comes in a follow up commit.

Sorry, just got time to get back to this now. Thanks for the review! I'll add a document for this new functionality tomorrow.

• In a follow-up commit tomorrow.

cfe/trunk/include/clang/Driver/Driver.h

public:
   /// arguments and return an appropriate exit code.
   ///
   /// This routine handles additional processing that must be done in addition
   /// to just running the subprocesses, for example reporting errors, setting
   /// up response files, removing temporary files, etc.
   int ExecuteCompilation(Compilation &C,
      SmallVectorImpl< std::pair<int, const Command *> > &FailingCommands);
 
-  /// generateCompilationDiagnostics - Generate diagnostics information 
+  /// Contains the files in the compilation diagnostic report generated by
+  /// generateCompilationDiagnostics.
+  struct CompilationDiagnosticReport {
+    llvm::SmallVector<std::string, 4> TemporaryFiles;
+  };
+
+  /// generateCompilationDiagnostics - Generate diagnostics information
   /// including preprocessed source file(s).
-  /// 
-  void generateCompilationDiagnostics(Compilation &C,
-                                      const Command &FailingCommand);
+  ///
+  void generateCompilationDiagnostics(
+      Compilation &C, const Command &FailingCommand,
+      StringRef AdditionalInformation = "",
+      CompilationDiagnosticReport *GeneratedReport = nullptr);
 
   /// @}
   /// @name Helper Methods
   /// @{
 
   /// PrintActions - Print the list of actions.
   void PrintActions(const Compilation &C) const;
 

cfe/trunk/lib/Driver/Driver.cpp

#endif
   }
 
   return false;
 }
 
 // When clang crashes, produce diagnostic information including the fully
 // preprocessed source file(s).  Request that the developer attach the
 // diagnostic information to a bug report.
-void Driver::generateCompilationDiagnostics(Compilation &C,
-                                            const Command &FailingCommand) {
+void Driver::generateCompilationDiagnostics(
+    Compilation &C, const Command &FailingCommand,
+    StringRef AdditionalInformation, CompilationDiagnosticReport *Report) {
   if (C.getArgs().hasArg(options::OPT_fno_crash_diagnostics))
     return;
 
   // Don't try to generate diagnostics for link or dsymutil jobs.
   if (FailingCommand.getCreator().isLinkJob() ||
       FailingCommand.getCreator().isDsymutilJob())
     return;
 
Diag(clang::diag::note_drv_command_failed_diag_msg)
       << "\n********************\n\n"
          "PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:\n"
          "Preprocessed source(s) and associated run script(s) are located at:";
 
   SmallString<128> VFS;
   SmallString<128> ReproCrashFilename;
   for (const char *TempFile : TempFiles) {
     Diag(clang::diag::note_drv_command_failed_diag_msg) << TempFile;
+    if (Report)
+      Report->TemporaryFiles.push_back(TempFile);
     if (ReproCrashFilename.empty()) {
       ReproCrashFilename = TempFile;
       llvm::sys::path::replace_extension(ReproCrashFilename, ".crash");
     }
     if (StringRef(TempFile).endswith(".cache")) {
       // In some cases (modules) we'll dump extra data to help with reproducing
       // the crash into a directory next to the output.
       VFS = llvm::sys::path::filename(TempFile);
Diag(clang::diag::note_drv_command_failed_diag_msg)
         << "Error generating run script: " + Script + " " + EC.message();
   } else {
     ScriptOS << "# Crash reproducer for " << getClangFullVersion() << "\n"
              << "# Driver args: ";
     printArgList(ScriptOS, C.getInputArgs());
     ScriptOS << "# Original command: ";
     Cmd.Print(ScriptOS, "\n", /*Quote=*/true);
     Cmd.Print(ScriptOS, "\n", /*Quote=*/true, &CrashInfo);
+    if (!AdditionalInformation.empty())
+      ScriptOS << "\n# Additional information: " << AdditionalInformation
+               << "\n";
+    if (Report)
+      Report->TemporaryFiles.push_back(Script);
     Diag(clang::diag::note_drv_command_failed_diag_msg) << Script;
   }
 
   // On darwin, provide information about the .crash diagnostic report.
   if (llvm::Triple(llvm::sys::getProcessTriple()).isOSDarwin()) {
     SmallString<128> CrashDiagDir;
     if (getCrashDiagnosticFile(ReproCrashFilename, CrashDiagDir)) {
       Diag(clang::diag::note_drv_command_failed_diag_msg)

cfe/trunk/test/Index/create-libclang-completion-reproducer.c

+// RUN: rm -rf %t
+// RUN: mkdir %t
+// RUN: env CINDEXTEST_INVOCATION_EMISSION_PATH=%t not c-index-test -code-completion-at=%s:10:1 "-remap-file=%s,%S/Inputs/record-parsing-invocation-remap.c" %s
+// RUN: %clang -cc1gen-reproducer %t/libclang-* -v | FileCheck %s
+
+// Invocation file must be removed by clang:
+// RUN: ls %t | count 0
+
+// CHECK: REPRODUCER METAINFO: {"libclang.operation": "complete", "libclang.opts": "1", "invocation-args": ["-code-completion-at={{.*}}create-libclang-completion-reproducer.c:10:1"]}
+
+// CHECK: REPRODUCER:
+// CHECK-NEXT: {
+// CHECK-NEXT: "files":["{{.*}}.c","{{.*}}.sh"]
+// CHECK-NEXT: }

cfe/trunk/test/Index/create-libclang-parsing-reproducer.c

+// RUN: rm -rf %t
+// RUN: mkdir %t
+// RUN: env CINDEXTEST_INVOCATION_EMISSION_PATH=%t not c-index-test -test-load-source all %s
+// RUN: %clang -cc1gen-reproducer %t/libclang-* -v | FileCheck %s
+
+// Invocation file must be removed by clang:
+// RUN: ls %t | count 0
+
+// RUN: rm -rf %t
+// RUN: mkdir %t
+// RUN: env CINDEXTEST_INVOCATION_EMISSION_PATH=%t not c-index-test -test-load-source all "-remap-file=%s,%S/Inputs/record-parsing-invocation-remap.c" %s
+// RUN: %clang -cc1gen-reproducer %t/libclang-* -v | FileCheck %s
+
+#pragma clang __debug parser_crash
+
+// CHECK: REPRODUCER METAINFO: {"libclang.operation": "parse", "libclang.opts": "1"}
+
+// CHECK: REPRODUCER:
+// CHECK-NEXT: {
+// CHECK-NEXT: "files":["{{.*}}.c","{{.*}}.sh"]
+// CHECK-NEXT: }

cfe/trunk/tools/driver/CMakeLists.txt

 if(NOT CLANG_BUILT_STANDALONE)
   set(tablegen_deps intrinsics_gen)
 endif()
 
 add_clang_tool(clang
   driver.cpp
   cc1_main.cpp
   cc1as_main.cpp
+  cc1gen_reproducer_main.cpp
 
   DEPENDS
   ${tablegen_deps}
   )
 
 target_link_libraries(clang
   PRIVATE
   clangBasic

cfe/trunk/tools/driver/cc1gen_reproducer_main.cpp

+//===-- cc1gen_reproducer_main.cpp - Clang reproducer generator  ----------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This is the entry point to the clang -cc1gen-reproducer functionality, which
+// generates reproducers for invocations for clang-based tools.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Basic/Diagnostic.h"
+#include "clang/Basic/LLVM.h"
+#include "clang/Basic/VirtualFileSystem.h"
+#include "clang/Driver/Compilation.h"
+#include "clang/Driver/Driver.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/TargetSelect.h"
+#include "llvm/Support/YAMLTraits.h"
+#include "llvm/Support/raw_ostream.h"
+
+using namespace clang;
+
+namespace {
+
+struct UnsavedFileHash {
+  std::string Name;
+  std::string MD5;
+};
+
+struct ClangInvocationInfo {
+  std::string Toolchain;
+  std::string LibclangOperation;
+  std::string LibclangOptions;
+  std::vector<std::string> Arguments;
+  std::vector<std::string> InvocationArguments;
+  std::vector<UnsavedFileHash> UnsavedFileHashes;
+  bool Dump = false;
+};
+
+} // end anonymous namespace
+
+LLVM_YAML_IS_SEQUENCE_VECTOR(UnsavedFileHash)
+
+namespace llvm {
+namespace yaml {
+
+template <> struct MappingTraits<UnsavedFileHash> {
+  static void mapping(IO &IO, UnsavedFileHash &Info) {
+    IO.mapRequired("name", Info.Name);
+    IO.mapRequired("md5", Info.MD5);
+  }
+};
+
+template <> struct MappingTraits<ClangInvocationInfo> {
+  static void mapping(IO &IO, ClangInvocationInfo &Info) {
+    IO.mapRequired("toolchain", Info.Toolchain);
+    IO.mapOptional("libclang.operation", Info.LibclangOperation);
+    IO.mapOptional("libclang.opts", Info.LibclangOptions);
+    IO.mapRequired("args", Info.Arguments);
+    IO.mapOptional("invocation-args", Info.InvocationArguments);
+    IO.mapOptional("unsaved_file_hashes", Info.UnsavedFileHashes);
+  }
+};
+
+} // end namespace yaml
+} // end namespace llvm
+
+static std::string generateReproducerMetaInfo(const ClangInvocationInfo &Info) {
+  std::string Result;
+  llvm::raw_string_ostream OS(Result);
+  OS << '{';
+  bool NeedComma = false;
+  auto EmitKey = [&](StringRef Key) {
+    if (NeedComma)
+      OS << ", ";
+    NeedComma = true;
+    OS << '"' << Key << "\": ";
+  };
+  auto EmitStringKey = [&](StringRef Key, StringRef Value) {
+    if (Value.empty())
+      return;
+    EmitKey(Key);
+    OS << '"' << Value << '"';
+  };
+  EmitStringKey("libclang.operation", Info.LibclangOperation);
+  EmitStringKey("libclang.opts", Info.LibclangOptions);
+  if (!Info.InvocationArguments.empty()) {
+    EmitKey("invocation-args");
+    OS << '[';
+    for (const auto &Arg : llvm::enumerate(Info.InvocationArguments)) {
+      if (Arg.index())
+        OS << ',';
+      OS << '"' << Arg.value() << '"';
+    }
+    OS << ']';
+  }
+  OS << '}';
+  // FIXME: Compare unsaved file hashes and report mismatch in the reproducer.
+  if (Info.Dump)
+    llvm::outs() << "REPRODUCER METAINFO: " << OS.str() << "\n";
+  return std::move(OS.str());
+}
+
+/// Generates a reproducer for a set of arguments from a specific invocation.
+static llvm::Optional<driver::Driver::CompilationDiagnosticReport>
+generateReproducerForInvocationArguments(ArrayRef<const char *> Argv,
+                                         const ClangInvocationInfo &Info) {
+  using namespace driver;
+  auto TargetAndMode = ToolChain::getTargetAndModeFromProgramName(Argv[0]);
+
+  IntrusiveRefCntPtr<DiagnosticOptions> DiagOpts = new DiagnosticOptions;
+
+  IntrusiveRefCntPtr<DiagnosticIDs> DiagID(new DiagnosticIDs());
+  DiagnosticsEngine Diags(DiagID, &*DiagOpts, new IgnoringDiagConsumer());
+  ProcessWarningOptions(Diags, *DiagOpts, /*ReportDiags=*/false);
+  Driver TheDriver(Argv[0], llvm::sys::getDefaultTargetTriple(), Diags);
+  TheDriver.setTargetAndMode(TargetAndMode);
+
+  std::unique_ptr<Compilation> C(TheDriver.BuildCompilation(Argv));
+  if (C && !C->containsError()) {
+    for (const auto &J : C->getJobs()) {
+      if (const Command *Cmd = dyn_cast<Command>(&J)) {
+        Driver::CompilationDiagnosticReport Report;
+        TheDriver.generateCompilationDiagnostics(
+            *C, *Cmd, generateReproducerMetaInfo(Info), &Report);
+        return Report;
+      }
+    }
+  }
+
+  return None;
+}
+
+std::string GetExecutablePath(const char *Argv0, bool CanonicalPrefixes);
+
+static void printReproducerInformation(
+    llvm::raw_ostream &OS, const ClangInvocationInfo &Info,
+    const driver::Driver::CompilationDiagnosticReport &Report) {
+  OS << "REPRODUCER:\n";
+  OS << "{\n";
+  OS << R"("files":[)";
+  for (const auto &File : llvm::enumerate(Report.TemporaryFiles)) {
+    if (File.index())
+      OS << ',';
+    OS << '"' << File.value() << '"';
+  }
+  OS << "]\n}\n";
+}
+
+int cc1gen_reproducer_main(ArrayRef<const char *> Argv, const char *Argv0,
+                           void *MainAddr) {
+  if (Argv.size() < 1) {
+    llvm::errs() << "error: missing invocation file\n";
+    return 1;
+  }
+  // Parse the invocation descriptor.
+  StringRef Input = Argv[0];
+  llvm::ErrorOr<std::unique_ptr<llvm::MemoryBuffer>> Buffer =
+      llvm::MemoryBuffer::getFile(Input);
+  if (!Buffer) {
+    llvm::errs() << "error: failed to read " << Input << ": "
+                 << Buffer.getError().message() << "\n";
+    return 1;
+  }
+  llvm::yaml::Input YAML(Buffer.get()->getBuffer());
+  ClangInvocationInfo InvocationInfo;
+  YAML >> InvocationInfo;
+  if (Argv.size() > 1 && Argv[1] == StringRef("-v"))
+    InvocationInfo.Dump = true;
+
+  // Create an invocation that will produce the reproducer.
+  std::vector<const char *> DriverArgs;
+  for (const auto &Arg : InvocationInfo.Arguments)
+    DriverArgs.push_back(Arg.c_str());
+  std::string Path = GetExecutablePath(Argv0, /*CanonicalPrefixes=*/true);
+  DriverArgs[0] = Path.c_str();
+  llvm::Optional<driver::Driver::CompilationDiagnosticReport> Report =
+      generateReproducerForInvocationArguments(DriverArgs, InvocationInfo);
+
+  // Emit the information about the reproduce files to stdout.
+  int Result = 1;
+  if (Report) {
+    printReproducerInformation(llvm::outs(), InvocationInfo, *Report);
+    Result = 0;
+  }
+
+  // Remove the input file.
+  llvm::sys::fs::remove(Input);
+  return Result;
+}

cfe/trunk/tools/driver/driver.cpp

if (*S != '\0')
       ++S;
   }
 }
 
 extern int cc1_main(ArrayRef<const char *> Argv, const char *Argv0,
                     void *MainAddr);
 extern int cc1as_main(ArrayRef<const char *> Argv, const char *Argv0,
                       void *MainAddr);
+extern int cc1gen_reproducer_main(ArrayRef<const char *> Argv,
+                                  const char *Argv0, void *MainAddr);
 
 static void insertTargetAndModeArgs(const ParsedClangName &NameParts,
                                     SmallVectorImpl<const char *> &ArgVector,
                                     std::set<std::string> &SavedStrings) {
   // Put target and mode arguments at the start of argument list so that
   // arguments specified in command line could override them. Avoid putting
   // them at index 0, as an option like '-cc1' must remain the first.
   int InsertionPoint = 0;

 }
 
 static int ExecuteCC1Tool(ArrayRef<const char *> argv, StringRef Tool) {
   void *GetExecutablePathVP = (void *)(intptr_t) GetExecutablePath;
   if (Tool == "")
     return cc1_main(argv.slice(2), argv[0], GetExecutablePathVP);
   if (Tool == "as")
     return cc1as_main(argv.slice(2), argv[0], GetExecutablePathVP);
+  if (Tool == "gen-reproducer")
+    return cc1gen_reproducer_main(argv.slice(2), argv[0], GetExecutablePathVP);
 
   // Reject unknown tools.
   llvm::errs() << "error: unknown integrated tool '" << Tool << "'. "
                << "Valid tools include '-cc1' and '-cc1as'.\n";
   return 1;
 }
 
 int main(int argc_, const char **argv_) {