[clangd] [RFC] Use libclang and CXTranslationUnit instead of ASTUnit

[malaperle-ericsson]

Summary

This change uses the higher level libclang library to achieve similar
results as before. It paves the way to be able to use other things
from the library such as clang_codeCompleteAt. Using CXTranslationUnits
while keeping the current ASTUnit would mean parsing the files twice
so this is why this change removes the ASTUnits.

[malaperle-ericsson]

Just to make things clear, I'm really not sure if this is the right approach and I'm looking for opinions before going further in that direction (code completion).

[bkramer]

It's not. the goal is to get rid of ASTUnit inside of clangd in the long term as it's a big problem for extensibility. libclang is just a wrapper for ASTUnit, with even more problems.

If you want to get completion running, just call ASTUnit::CodeComplete.

[malaperle-ericsson]

In D31019#702631, @bkramer wrote:

It's not. the goal is to get rid of ASTUnit inside of clangd in the long term as it's a big problem for extensibility. libclang is just a wrapper for ASTUnit, with even more problems.

If you want to get completion running, just call ASTUnit::CodeComplete.

Thanks a lot. When you say the goal is to get rid of ASTUnit, what would it be replaced with? Also, why would one use libclang over ASTUnit (libclangFrontend) then? Perhaps there's some documentation listing the pros and cons?

[bkramer]

libclang is supposed to be used for bindings to other languages (it's a pure C interface) and its stable. ASTUnit is the C++ side of libclang, libclang is just a thin wrapper around ASTUnit. ASTUnit itself suffers from lots of technical debt because clang changed since ASTUnit was created and ASTUnit was never refactored to accommodate that.

Even in the current state using ASTUnit gives you more control over clang, e.g. it lets you use the VFS which is not exposed in libclang. At some point we're going to need a cleaner replacement for ASTUnit, but currently it's the only place in clang that supports the precompiled preamble needed for speedy code completion.

[malaperle-ericsson]

Perfect. Thanks a lot for the explanation!

[malaperle-ericsson]

Abandoned because of wrong approach.

clangd/ASTManager.h

 #include "DocumentStore.h"
 #include "JSONRPCDispatcher.h"
 #include "Protocol.h"
 #include "clang/Tooling/Core/Replacement.h"
 #include "llvm/ADT/IntrusiveRefCntPtr.h"
 #include <condition_variable>
 #include <deque>
 #include <thread>
+#include <clang-c/Index.h>
 
 namespace clang {
-class ASTUnit;
 class DiagnosticsEngine;
 class PCHContainerOperations;
 namespace tooling {
 class CompilationDatabase;
 } // namespace tooling
 
 namespace clangd {
 

 private:
   JSONOutput &Output;
   DocumentStore &Store;
 
   // Set to true if requests should block instead of being processed
   // asynchronously.
   bool RunSynchronously;
 
+  CXIndex Index;
+  
   /// Loads a compilation database for URI. May return nullptr if it fails. The
   /// database is cached for subsequent accesses.
   clang::tooling::CompilationDatabase *
   getOrCreateCompilationDatabaseForFile(StringRef Uri);
-  // Craetes a new ASTUnit for the document at Uri.
-  // FIXME: This calls chdir internally, which is thread unsafe.
-  std::unique_ptr<clang::ASTUnit>
-  createASTUnitForFile(StringRef Uri, const DocumentStore &Docs);
+  // Creates a new CXTranslationUnit for the document at Uri.
+  CXTranslationUnit
+  createTranslationUnitForFile(StringRef Uri, std::vector<CXUnsavedFile> &unsavedFiles);
 
   void runWorker();
   void parseFileAndPublishDiagnostics(StringRef File);
 
   /// Clang objects.
-  llvm::StringMap<std::unique_ptr<clang::ASTUnit>> ASTs;
+  llvm::StringMap<CXTranslationUnit> CTranslationUnits;
   llvm::StringMap<std::unique_ptr<clang::tooling::CompilationDatabase>>
       CompilationDatabases;
   std::shared_ptr<clang::PCHContainerOperations> PCHs;
 
   typedef std::map<clangd::Diagnostic, std::vector<clang::tooling::Replacement>>
       DiagnosticToReplacementMap;
   DiagnosticToReplacementMap FixIts;
   std::mutex FixItLock;

clangd/ASTManager.cpp

 #include "llvm/Support/Format.h"
 #include "llvm/Support/Path.h"
 #include <mutex>
 #include <thread>
 using namespace clang;
 using namespace clangd;
 
 /// Retrieve a copy of the contents of every file in the store, for feeding into
-/// ASTUnit.
-static std::vector<ASTUnit::RemappedFile>
-getRemappedFiles(const DocumentStore &Docs) {
+/// CXTranslationUnit.
+static std::vector<CXUnsavedFile>
+getUnsavedFiles(const DocumentStore &Docs) {
+  std::vector<CXUnsavedFile> UnsavedFiles;
   // FIXME: Use VFS instead. This would allow us to get rid of the chdir below.
-  std::vector<ASTUnit::RemappedFile> RemappedFiles;
   for (const auto &P : Docs.getAllDocuments()) {
     StringRef FileName = P.first;
     FileName.consume_front("file://");
-    RemappedFiles.push_back(ASTUnit::RemappedFile(
-        FileName,
-        llvm::MemoryBuffer::getMemBufferCopy(P.second, FileName).release()));
+    FileName.consume_front("file:");
+    char * Filename = (char *) malloc(FileName.size() + 1);
+    memcpy(Filename, FileName.data(), FileName.size());
+    Filename[FileName.size()] = 0;
+
+    char * Content = (char *) malloc(P.second.size());
+    memcpy(Content, P.second.data(), P.second.size());
+    UnsavedFiles.push_back(
+        CXUnsavedFile { Filename, Content, P.second.size() });
   }
-  return RemappedFiles;
+  return UnsavedFiles;
 }
 
 /// Convert from clang diagnostic level to LSP severity.
-static int getSeverity(DiagnosticsEngine::Level L) {
-  switch (L) {
-  case DiagnosticsEngine::Remark:
-    return 4;
-  case DiagnosticsEngine::Note:
+static int getSeverity(CXDiagnosticSeverity S) {
+  switch (S) {
+  case CXDiagnostic_Note:
     return 3;
-  case DiagnosticsEngine::Warning:
+  case CXDiagnostic_Warning:
     return 2;
-  case DiagnosticsEngine::Fatal:
-  case DiagnosticsEngine::Error:
+  case CXDiagnostic_Fatal:
+  case CXDiagnostic_Error:
     return 1;
-  case DiagnosticsEngine::Ignored:
+  case CXDiagnostic_Ignored:
     return 0;
   }
   llvm_unreachable("Unknown diagnostic level!");
 }
 
 ASTManager::ASTManager(JSONOutput &Output, DocumentStore &Store,
                        bool RunSynchronously)
     : Output(Output), Store(Store), RunSynchronously(RunSynchronously),
+      Index(clang_createIndex(1, 1)),
       PCHs(std::make_shared<PCHContainerOperations>()),
       ClangWorker([this]() { runWorker(); }) {}
 
 void ASTManager::runWorker() {
   while (true) {
     std::string File;
 
     {
std::string File;
       RequestQueue.pop_back();
     } // unlock.
 
     parseFileAndPublishDiagnostics(File);
   }
 }
 
 void ASTManager::parseFileAndPublishDiagnostics(StringRef File) {
-  auto &Unit = ASTs[File]; // Only one thread can access this at a time.
+  auto UnsavedFiles = getUnsavedFiles(Store);
 
-  if (!Unit) {
-    Unit = createASTUnitForFile(File, this->Store);
+  auto &TU = CTranslationUnits[File];
+  if (!TU) {
+    TU = createTranslationUnitForFile(File, UnsavedFiles);
   } else {
     // Do a reparse if this wasn't the first parse.
     // FIXME: This might have the wrong working directory if it changed in the
     // meantime.
-    Unit->Reparse(PCHs, getRemappedFiles(this->Store));
+    CXUnsavedFile *UnsavedFilesPtr = &*UnsavedFiles.begin();
+    clang_reparseTranslationUnit(TU, UnsavedFiles.size(), UnsavedFilesPtr, CXReparse_None);
   }
 
-  if (!Unit)
+  if (!TU)
     return;
 
   // Send the diagnotics to the editor.
   // FIXME: If the diagnostic comes from a different file, do we want to
   // show them all? Right now we drop everything not coming from the
   // main file.
   std::string Diagnostics;
   DiagnosticToReplacementMap LocalFixIts; // Temporary storage
-  for (ASTUnit::stored_diag_iterator D = Unit->stored_diag_begin(),
-                                     DEnd = Unit->stored_diag_end();
-       D != DEnd; ++D) {
-    if (!D->getLocation().isValid() ||
-        !D->getLocation().getManager().isInMainFile(D->getLocation()))
+  auto const &DiagnosticSet = clang_getDiagnosticSetFromTU(TU);
+  unsigned NumDiagnosticsInSet = clang_getNumDiagnosticsInSet(DiagnosticSet);
+  for (unsigned i = 0; i < NumDiagnosticsInSet; ++i) {
+    auto const &D = clang_getDiagnosticInSet(DiagnosticSet, i);
+    auto const &Location = clang_getDiagnosticLocation(D);
+    if (clang_equalLocations(Location, clang_getNullLocation()) ||
+        !clang_Location_isFromMainFile(Location))
       continue;
+
+    unsigned Line, Column;
+    clang_getSpellingLocation(Location, NULL, &Line, &Column, NULL);
+    enum CXDiagnosticSeverity sev = clang_getDiagnosticSeverity(D);
+    auto const &Spelling = clang_getDiagnosticSpelling(D);
+
     Position P;
-    P.line = D->getLocation().getSpellingLineNumber() - 1;
-    P.character = D->getLocation().getSpellingColumnNumber();
+    P.line = Line - 1;
+    P.character = Column;
     Range R = {P, P};
     Diagnostics +=
         R"({"range":)" + Range::unparse(R) +
-        R"(,"severity":)" + std::to_string(getSeverity(D->getLevel())) +
-        R"(,"message":")" + llvm::yaml::escape(D->getMessage()) +
+        R"(,"severity":)" + std::to_string(getSeverity(sev)) +
+        R"(,"message":")" + llvm::yaml::escape(clang_getCString(Spelling)) +
         R"("},)";
-
     // We convert to Replacements to become independent of the SourceManager.
-    clangd::Diagnostic Diag = {R, getSeverity(D->getLevel()), D->getMessage()};
+    clangd::Diagnostic Diag = {R, getSeverity(sev), clang_getCString(Spelling)};
     auto &FixItsForDiagnostic = LocalFixIts[Diag];
-    for (const FixItHint &Fix : D->getFixIts()) {
-      FixItsForDiagnostic.push_back(clang::tooling::Replacement(
-          Unit->getSourceManager(), Fix.RemoveRange, Fix.CodeToInsert));
+    unsigned FixItNum = clang_getDiagnosticNumFixIts(D);
+    for (unsigned CurFixIt = 0; CurFixIt < FixItNum; ++CurFixIt) {
+      CXSourceRange ReplacementRange;
+      CXString Replacement = clang_getDiagnosticFixIt(D, CurFixIt, &ReplacementRange);
+
+      auto Start = clang_getRangeStart(ReplacementRange);
+      auto End = clang_getRangeEnd(ReplacementRange);
+      unsigned Offset;
+      clang_getSpellingLocation(Start, NULL, NULL, NULL, &Offset);
+      
+      unsigned EndOffset;
+      clang_getSpellingLocation(End, NULL, NULL, NULL, &EndOffset);
+      unsigned Length = EndOffset - Offset;
+      FixItsForDiagnostic.push_back(clang::tooling::Replacement(File,
+          Offset, Length, clang_getCString(Replacement)));
     }
   }
 
   // Put FixIts into place.
   {
     std::lock_guard<std::mutex> Guard(FixItLock);
     FixIts = std::move(LocalFixIts);
   }
ASTManager::getOrCreateCompilationDatabaseForFile(StringRef Uri) {
   Uri.consume_front("file://");
 
   std::string Error;
   I = tooling::CompilationDatabase::autoDetectFromSource(Uri, Error);
   Output.log("Failed to load compilation database: " + Twine(Error) + "\n");
   return I.get();
 }
 
-std::unique_ptr<clang::ASTUnit>
-ASTManager::createASTUnitForFile(StringRef Uri, const DocumentStore &Docs) {
+CXTranslationUnit
+ASTManager::createTranslationUnitForFile(StringRef File, std::vector<CXUnsavedFile> &UnsavedFiles) {
   tooling::CompilationDatabase *CDB =
-      getOrCreateCompilationDatabaseForFile(Uri);
-
+      getOrCreateCompilationDatabaseForFile(File);
+  StringRef Uri(File);
   Uri.consume_front("file://");
+  Uri.consume_front("file:");
   std::vector<tooling::CompileCommand> Commands;
 
   if (CDB) {
     Commands = CDB->getCompileCommands(Uri);
     // chdir. This is thread hostile.
     if (!Commands.empty())
       llvm::sys::fs::set_current_path(Commands.front().Directory);
   }
ASTManager::createTranslationUnitForFile(StringRef File, std::vector<CXUnsavedFile> &UnsavedFiles) {
   IntrusiveRefCntPtr<DiagnosticsEngine> Diags =
       CompilerInstance::createDiagnostics(new DiagnosticOptions);
 
   std::vector<const char *> ArgStrs;
   for (const auto &S : Commands.front().CommandLine)
     ArgStrs.push_back(S.c_str());
 
   auto ArgP = &*ArgStrs.begin();
-  return std::unique_ptr<clang::ASTUnit>(ASTUnit::LoadFromCommandLine(
-      ArgP, ArgP + ArgStrs.size(), PCHs, Diags, ResourceDir,
-      /*OnlyLocalDecls=*/false, /*CaptureDiagnostics=*/true,
-      getRemappedFiles(Docs),
-      /*RemappedFilesKeepOriginalName=*/true,
-      /*PrecompilePreambleAfterNParses=*/1, /*TUKind=*/TU_Complete,
-      /*CacheCodeCompletionResults=*/true,
-      /*IncludeBriefCommentsInCodeCompletion=*/true,
-      /*AllowPCHWithCompilerErrors=*/true));
+  auto unsaved_files_ptr = &*UnsavedFiles.begin();
+  CXTranslationUnit CXUnit;
+  clang_parseTranslationUnit2FullArgv(Index, nullptr, ArgP, ArgStrs.size(), unsaved_files_ptr, UnsavedFiles.size(), CXTranslationUnit_DetailedPreprocessingRecord, &CXUnit);
+  return CXUnit;
 }
 
 std::vector<clang::tooling::Replacement>
 ASTManager::getFixIts(const clangd::Diagnostic &D) {
   std::lock_guard<std::mutex> Guard(FixItLock);
   auto I = FixIts.find(D);
   if (I != FixIts.end())
     return I->second;
   return {};
 }

clangd/CMakeLists.txt

 add_clang_executable(clangd
   ASTManager.cpp
   ClangDMain.cpp
   JSONRPCDispatcher.cpp
   Protocol.cpp
   ProtocolHandlers.cpp
   )
 
 install(TARGETS clangd RUNTIME DESTINATION bin)
 
 target_link_libraries(clangd
   clangBasic
   clangFormat
   clangFrontend
+  libclang
   clangTooling
   clangToolingCore
   LLVMSupport
   )