[libclang] Record parsing invocation to a temporary file when requested by client

[arphaman]

Summary

This patch extends libclang by allowing it to record parsing operations to a temporary JSON file. The file is deleted after parsing succeeds. When a crash happens during parsing, the file is preserved and the client will be able to use it to generate a reproducer for the crash.

These files are not emitted by default, and the client has to specify the invocation emission path first.

rdar://35322543

[nik]

Could you elaborate on "the client will be able to use it to generate a reproducer for the crash"? Having the json file, what would I need to do in order to reproduce the crash?

[arphaman]

In D40527#937282, @nik wrote:

Could you elaborate on "the client will be able to use it to generate a reproducer for the crash"? Having the json file, what would I need to do in order to reproduce the crash?

Sure. I would like to teach clang how to interpret this file so that it can generate a reproducer for a crash (similar to a -gen-reproducer) option in clang. This way external IDE clients could leverage clang's reproducer infrastructure to generate reproducers for libclang crashes. The external client will be responsible for determining when a crash happens and what JSON files have to be passed to clang.

[jkorous-apple]

tools/libclang/CIndexer.cpp

132

Just a thought - since we are not propagating errors from constructor we are not really sure we were able to create the file in the first place (e. g. return from ctor at line 103). Should we still try to delete it?

[jkorous-apple]

tools/libclang/CIndexer.cpp

109

Nit: Maybe capturing &OS is enough?

[arphaman]

tools/libclang/CIndexer.cpp

132

If createUniqueFile failed in the constructor (return on line 103) then File will be empty. Therefore we won't try to remove the file.

[arphaman]

• Capture OS only.
• Guard toolchain path accessor with a mutex.

[jkorous-apple]

LGTM otherwise.

tools/libclang/CIndexer.h

45

I am just wondering - since we anticipate multi-threaded usage, shouldn't we synchronize access to all member variables mutable via public class interface?

Specifically - shouldn't we use one mutex per every mutating/reading method group?

• setInvocationEmissionPath(), getInvocationEmissionPath()
• setCXGlobalOptFlags(), isOptEnabled()
• getClangResourcesPath()

[arphaman]

tools/libclang/CIndexer.h

45

Actually CIndex is not thread safe, so we don't need the mutex. I'll remove it.

include/clang-c/Index.h

  * \brief Gets the general options associated with a CXIndex.
  *
  * \returns A bitmask of options, a bitwise OR of CXGlobalOpt_XXX flags that
  * are associated with the given CXIndex object.
  */
 CINDEX_LINKAGE unsigned clang_CXIndex_getGlobalOptions(CXIndex);
 
 /**
+ * \brief Sets the invocation emission path option in a CXIndex.
+ *
+ * The invocation emission path specifies a path which will contain log
+ * files for certain libclang invocations. A null value (default) implies that
+ * libclang invocations are not logged..
+ */
+CINDEX_LINKAGE void
+clang_CXIndex_setInvocationEmissionPathOption(CXIndex, const char *Path);
+
+/**
  * \defgroup CINDEX_FILES File manipulation routines
  *
  * @{
  */
 
 /**
  * \brief A particular source file that is part of a translation unit.
  */

test/Index/record-parsing-invocation.c

+// RUN: rm -rf %t
+// RUN: mkdir %t
+// RUN: not env CINDEXTEST_INVOCATION_EMISSION_PATH=%t c-index-test -test-load-source all %s
+// RUN: cat %t/libclang-* | FileCheck %s
+
+// RUN: rm -rf %t
+// RUN: mkdir %t
+// RUN: env LIBCLANG_DISABLE_CRASH_RECOVERY=1 CINDEXTEST_INVOCATION_EMISSION_PATH=%t not --crash c-index-test -test-load-source all %s
+// RUN: cat %t/libclang-* | FileCheck %s
+
+// Verify that the file is removed for successful operation:
+// RUN: rm -rf %t
+// RUN: mkdir %t
+// RUN: env CINDEXTEST_INVOCATION_EMISSION_PATH=%t c-index-test -test-load-source all %s -DAVOID_CRASH
+// RUN: ls %t | count 0
+
+#ifndef AVOID_CRASH
+#  pragma clang __debug parser_crash
+#endif
+
+// CHECK: {"toolchain":"{{.*}}","libclang.operation":"parse","libclang.opts":1,"args":["clang","-fno-spell-checking","{{.*}}/record-parsing-invocation.c","-Xclang","-detailed-preprocessing-record","-fallow-editor-placeholders"]}

tools/c-index-test/c-index-test.c

int perform_test_load_source(int argc, const char **argv,
   CXTranslationUnit TU;
   const char *CommentSchemaFile;
   struct CXUnsavedFile *unsaved_files = 0;
   int num_unsaved_files = 0;
   enum CXErrorCode Err;
   int result;
   unsigned Repeats = 0;
   unsigned I;
+  const char *InvocationPath;
 
   Idx = clang_createIndex(/* excludeDeclsFromPCH */
                           (!strcmp(filter, "local") || 
                            !strcmp(filter, "local-display"))? 1 : 0,
                           /* displayDiagnostics=*/1);
+  InvocationPath = getenv("CINDEXTEST_INVOCATION_EMISSION_PATH");
+  if (InvocationPath)
+    clang_CXIndex_setInvocationEmissionPathOption(Idx, InvocationPath);
 
   if ((CommentSchemaFile = parse_comments_schema(argc, argv))) {
     argc--;
     argv++;
   }
 
   if (parse_remapped_files(argc, argv, 0, &unsaved_files, &num_unsaved_files)) {
     clang_disposeIndex(Idx);

tools/libclang/CIndex.cpp

 }
 
 unsigned clang_CXIndex_getGlobalOptions(CXIndex CIdx) {
   if (CIdx)
     return static_cast<CIndexer *>(CIdx)->getCXGlobalOptFlags();
   return 0;
 }
 
+void clang_CXIndex_setInvocationEmissionPathOption(CXIndex CIdx,
+                                                   const char *Path) {
+  if (CIdx)
+    static_cast<CIndexer *>(CIdx)->setInvocationEmissionPath(Path ? Path : "");
+}
+
 void clang_toggleCrashRecovery(unsigned isEnabled) {
   if (isEnabled)
     llvm::CrashRecoveryContext::Enable();
   else
     llvm::CrashRecoveryContext::Disable();
 }
 
 CXTranslationUnit clang_createTranslationUnit(CXIndex CIdx,
clang_parseTranslationUnit_Impl(CXIndex CIdx, const char *source_filename,
 
   unsigned NumErrors = Diags->getClient()->getNumErrors();
   std::unique_ptr<ASTUnit> ErrUnit;
   // Unless the user specified that they want the preamble on the first parse
   // set it up to be created on the first reparse. This makes the first parse
   // faster, trading for a slower (first) reparse.
   unsigned PrecompilePreambleAfterNParses =
       !PrecompilePreamble ? 0 : 2 - CreatePreambleOnFirstParse;
+
+  // FIXME: Record the hash of the unsaved files.
+  LibclangInvocationReporter InvocationReporter(
+      *CXXIdx, LibclangInvocationReporter::OperationKind::ParseOperation,
+      options, llvm::makeArrayRef(*Args));
   std::unique_ptr<ASTUnit> Unit(ASTUnit::LoadFromCommandLine(
       Args->data(), Args->data() + Args->size(),
       CXXIdx->getPCHContainerOperations(), Diags,
       CXXIdx->getClangResourcesPath(), CXXIdx->getOnlyLocalDecls(),
       /*CaptureDiagnostics=*/true, *RemappedFiles.get(),
       /*RemappedFilesKeepOriginalName=*/true, PrecompilePreambleAfterNParses,
       TUKind, CacheCodeCompletionResults, IncludeBriefCommentsInCodeCompletion,
       /*AllowPCHWithCompilerErrors=*/true, SkipFunctionBodies, SingleFileParse,

tools/libclang/CIndexer.h

 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_TOOLS_LIBCLANG_CINDEXER_H
 #define LLVM_CLANG_TOOLS_LIBCLANG_CINDEXER_H
 
 #include "clang-c/Index.h"
 #include "clang/Frontend/PCHContainerOperations.h"
 #include "llvm/ADT/STLExtras.h"
+#include "llvm/Support/Mutex.h"
 #include <utility>
 
 namespace llvm {
   class CrashRecoveryContext;
 }
 
 namespace clang {
 class ASTUnit;
 class MacroInfo;
 class MacroDefinitionRecord;
 class SourceLocation;
 class Token;
 class IdentifierInfo;
 
 class CIndexer {
   bool OnlyLocalDecls;
   bool DisplayDiagnostics;
   unsigned Options; // CXGlobalOptFlags.
 
   std::string ResourcesPath;
   std::shared_ptr<PCHContainerOperations> PCHContainerOps;
 
+  std::string ToolchainPath;
+
+  std::string InvocationEmissionPath;
+
 public:
   CIndexer(std::shared_ptr<PCHContainerOperations> PCHContainerOps =
                std::make_shared<PCHContainerOperations>())
       : OnlyLocalDecls(false), DisplayDiagnostics(false),
         Options(CXGlobalOpt_None), PCHContainerOps(std::move(PCHContainerOps)) {
   }
 
   /// \brief Whether we only want to see "local" declarations (that did not
public:
   void setCXGlobalOptFlags(unsigned options) { Options = options; }
 
   bool isOptEnabled(CXGlobalOptFlags opt) const {
     return Options & opt;
   }
 
   /// \brief Get the path of the clang resource files.
   const std::string &getClangResourcesPath();
+
+  StringRef getClangToolchainPath();
+
+  void setInvocationEmissionPath(StringRef Str) {
+    InvocationEmissionPath = Str;
+  }
+
+  StringRef getInvocationEmissionPath() const { return InvocationEmissionPath; }
+};
+
+/// Logs information about a particular libclang operation like parsing to
+/// a new file in the invocation emission path.
+class LibclangInvocationReporter {
+public:
+  enum class OperationKind { ParseOperation, CompletionOperation };
+
+  LibclangInvocationReporter(CIndexer &Idx, OperationKind Op,
+                             unsigned ParseOptions,
+                             llvm::ArrayRef<const char *> Args);
+  ~LibclangInvocationReporter();
+
+private:
+  std::string File;
 };
 
   /// \brief Return the current size to request for "safety".
   unsigned GetSafetyThreadStackSize();
 
   /// \brief Set the current size to request for "safety" (or 0, if safety
   /// threads should not be used).
   void SetSafetyThreadStackSize(unsigned Value);

tools/libclang/CIndexer.cpp

 //===- CIndex.cpp - Clang-C Source Indexing Library -----------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 //
 // This file implements the Clang-C Source Indexing library.
 //
 //===----------------------------------------------------------------------===//
 
 #include "CIndexer.h"
 #include "clang/Basic/LLVM.h"
 #include "clang/Basic/Version.h"
+#include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/Config/llvm-config.h"
+#include "llvm/Support/MutexGuard.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/Program.h"
 #include <cstdio>
 
 #ifdef __CYGWIN__
 #include <cygwin/version.h>
 #include <sys/cygwin.h>
 #define LLVM_ON_WIN32 1

 #endif
 
   llvm::sys::path::append(LibClangPath, "clang", CLANG_VERSION_STRING);
 
   // Cache our result.
   ResourcesPath = LibClangPath.str();
   return ResourcesPath;
 }
+
+StringRef CIndexer::getClangToolchainPath() {
+  if (!ToolchainPath.empty())
+    return ToolchainPath;
+  StringRef ResourcePath = getClangResourcesPath();
+  ToolchainPath = llvm::sys::path::parent_path(
+      llvm::sys::path::parent_path(llvm::sys::path::parent_path(ResourcePath)));
+  return ToolchainPath;
+}
+
+LibclangInvocationReporter::LibclangInvocationReporter(
+    CIndexer &Idx, OperationKind Op, unsigned ParseOptions,
+    llvm::ArrayRef<const char *> Args) {
+  StringRef Path = Idx.getInvocationEmissionPath();
+  if (Path.empty())
+    return;
+
+  // Create a temporary file for the invocation log.
+  SmallString<256> TempPath;
+  TempPath = Path;
+  llvm::sys::path::append(TempPath, "libclang-%%%%%%%%%%%%");
+  int FD;
+  if (llvm::sys::fs::createUniqueFile(TempPath, FD, TempPath))
+    return;
+  File = std::string(TempPath.begin(), TempPath.end());
+  llvm::raw_fd_ostream OS(FD, /*ShouldClose=*/true);
+
+  // Write out the information about the invocation to it.
+  auto WriteStringKey = [&OS](StringRef Key, StringRef Value) {
+    OS << R"(")" << Key << R"(":")";
+    OS << Value << '"';
+  };
+  OS << '{';
+  WriteStringKey("toolchain", Idx.getClangToolchainPath());
+  OS << ',';
+  WriteStringKey("libclang.operation",
+                 Op == OperationKind::ParseOperation ? "parse" : "complete");
+  OS << ',';
+  OS << R"("libclang.opts":)" << ParseOptions;
+  OS << ',';
+  OS << R"("args":[)";
+  for (const auto &I : llvm::enumerate(Args)) {
+    if (I.index())
+      OS << ',';
+    OS << '"' << I.value() << '"';
+  }
+  OS << "]}";
+}
+
+LibclangInvocationReporter::~LibclangInvocationReporter() {
+  if (!File.empty())
+    llvm::sys::fs::remove(File);
+}

tools/libclang/libclang.exports

 clang_CXCursorSet_contains
 clang_CXCursorSet_insert
 clang_CXIndex_getGlobalOptions
 clang_CXIndex_setGlobalOptions
+clang_CXIndex_setInvocationEmissionPathOption
 clang_CXXConstructor_isConvertingConstructor
 clang_CXXConstructor_isCopyConstructor
 clang_CXXConstructor_isDefaultConstructor
 clang_CXXConstructor_isMoveConstructor
 clang_CXXField_isMutable
 clang_CXXMethod_isDefaulted
 clang_CXXMethod_isConst
 clang_CXXMethod_isPureVirtual