[clangd] Allow the client to specify multiple compilation databases in the 'initialize' request

[nridge]

Summary

This is useful for multi-project setups where each project has its own
compilation database, and particularly in cases where such databases
are located outside of the respective source directories (for example,
in build directories).

[nridge]

As background, some discussion of this took place on the mailing list a few months ago: http://lists.llvm.org/pipermail/clangd-dev/2019-January/000232.html

[ilya-biryukov]

Could this be handled outside clangd? If not, what are the complications?

We currently have a capability to specify a path to a single compile_commands.json.
I would expect clients using this extension (Theia?) also handle generation of compile_commands.json. If that's the case, could they merge multiple compile_commands.json files into one and ask clangd to load it using APIs we have today?

[nridge]

Thanks for having a look! I filed an issue so some folks who don't necessarily have an LLVM Phabricator account can weigh in, let's continue the discussion there.

[nridge]

Rebase

[nridge]

Ilya / others if interested, I would like to revive this patch.

We left off with a discussion in the corresponding issue, where I made a suggestion for an initial way forward. Could I get some feedback on that please?

[merge_guards_bot]

Unit tests: fail. 62144 tests passed, 5 failed and 811 were skipped.

failed: libc++.std/language_support/cmp/cmp_partialord/partialord.pass.cpp
failed: libc++.std/language_support/cmp/cmp_strongeq/cmp.strongeq.pass.cpp
failed: libc++.std/language_support/cmp/cmp_strongord/strongord.pass.cpp
failed: libc++.std/language_support/cmp/cmp_weakeq/cmp.weakeq.pass.cpp
failed: libc++.std/language_support/cmp/cmp_weakord/weakord.pass.cpp

clang-tidy: fail. clang-tidy found 2 errors and 4 warnings. 4 of them are added as review comments below (why?).

clang-format: pass.

Build artifacts: diff.json, clang-tidy.txt, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

Pre-merge checks is in beta. Report issue. Please join beta or enable it for your project.

clang-tools-extra/clangd/ClangdLSPServer.cpp

else if (Params.rootPath && !Params.rootPath->empty())
     ClangdServerOpts.WorkspaceRoot = *Params.rootPath;
   if (Server)
     return Reply(llvm::make_error<LSPError>("server already initialized",
                                             ErrorCode::InvalidRequest));
   if (const auto &Dir = Params.initializationOptions.compilationDatabasePath)
     CompileCommandsDir = Dir;
   if (UseDirBasedCDB) {
     BaseCDB = std::make_unique<DirectoryBasedGlobalCompilationDatabase>(
-        CompileCommandsDir);
+        CompileCommandsDir,
+        Params.initializationOptions.compilationDatabaseMap);
     BaseCDB = getQueryDriverDatabase(
         llvm::makeArrayRef(ClangdServerOpts.QueryDriverGlobs),
         std::move(BaseCDB));
   }
   auto Mangler = CommandMangler::detect();
   if (ClangdServerOpts.ResourceDir)
     Mangler.ResourceDir = *ClangdServerOpts.ResourceDir;
   CDB.emplace(BaseCDB.get(), Params.initializationOptions.fallbackFlags,
ClangdLSPServer::ClangdLSPServer(
   MsgHandler->bind("textDocument/symbolInfo", &ClangdLSPServer::onSymbolInfo);
   MsgHandler->bind("textDocument/typeHierarchy", &ClangdLSPServer::onTypeHierarchy);
   MsgHandler->bind("typeHierarchy/resolve", &ClangdLSPServer::onResolveTypeHierarchy);
   MsgHandler->bind("textDocument/selectionRange", &ClangdLSPServer::onSelectionRange);
   MsgHandler->bind("textDocument/documentLink", &ClangdLSPServer::onDocumentLink);
   // clang-format on
 }
 
-ClangdLSPServer::~ClangdLSPServer() { IsBeingDestroyed = true;
+ClangdLSPServer::~ClangdLSPServer() {
+  IsBeingDestroyed = true;
   // Explicitly destroy ClangdServer first, blocking on threads it owns.
   // This ensures they don't access any other members.
   Server.reset();
 }
 
 bool ClangdLSPServer::run() {
   // Run the Language Server loop.
   bool CleanExit = true;

clang-tools-extra/clangd/GlobalCompilationDatabase.h

 //===--- GlobalCompilationDatabase.h -----------------------------*- C++-*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_GLOBALCOMPILATIONDATABASE_H
 #define LLVM_CLANG_TOOLS_EXTRA_CLANGD_GLOBALCOMPILATIONDATABASE_H
 
 #include "CompileCommands.h"
 #include "Function.h"
 #include "Path.h"
+#include "Protocol.h"
 #include "clang/Tooling/ArgumentsAdjusters.h"
 #include "clang/Tooling/CompilationDatabase.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/StringMap.h"
 #include <memory>
 #include <mutex>
 #include <vector>
 

 };
 
 /// Gets compile args from tooling::CompilationDatabases built for parent
 /// directories.
 class DirectoryBasedGlobalCompilationDatabase
     : public GlobalCompilationDatabase {
 public:
   DirectoryBasedGlobalCompilationDatabase(
-      llvm::Optional<Path> CompileCommandsDir);
+      llvm::Optional<Path> CompileCommandsDir,
+      llvm::Optional<CompilationDatabaseMap> CDBMap);
   ~DirectoryBasedGlobalCompilationDatabase() override;
 
   /// Scans File's parents looking for compilation databases.
   /// Any extra flags will be added.
   /// Might trigger OnCommandChanged, if CDB wasn't broadcasted yet.
   llvm::Optional<tooling::CompileCommand>
   getCompileCommand(PathRef File) const override;
 
private:
 
   mutable std::mutex Mutex;
   // Keyed by possibly-case-folded directory path.
   mutable llvm::StringMap<CachedCDB> CompilationDatabases;
 
   /// Used for command argument pointing to folder where compile_commands.json
   /// is located.
   llvm::Optional<Path> CompileCommandsDir;
+
+  llvm::Optional<CompilationDatabaseMap> CDBMap;
 };
 
 /// Extracts system include search path from drivers matching QueryDriverGlobs
 /// and adds them to the compile flags. Base may not be nullptr.
 /// Returns Base when \p QueryDriverGlobs is empty.
 std::unique_ptr<GlobalCompilationDatabase>
 getQueryDriverDatabase(llvm::ArrayRef<std::string> QueryDriverGlobs,
                        std::unique_ptr<GlobalCompilationDatabase> Base);
 
-
 /// Wraps another compilation database, and supports overriding the commands
 /// using an in-memory mapping.
 class OverlayCDB : public GlobalCompilationDatabase {
 public:
   // Base may be null, in which case no entries are inherited.
   // FallbackFlags are added to the fallback compile command.
   // Adjuster is applied to all commands, fallback or not.
   OverlayCDB(const GlobalCompilationDatabase *Base,

clang-tools-extra/clangd/GlobalCompilationDatabase.cpp

tooling::CompileCommand Cmd(llvm::sys::path::parent_path(File),
                               llvm::sys::path::filename(File), std::move(Argv),
                               /*Output=*/"");
   Cmd.Heuristic = "clangd fallback";
   return Cmd;
 }
 
 DirectoryBasedGlobalCompilationDatabase::
     DirectoryBasedGlobalCompilationDatabase(
-        llvm::Optional<Path> CompileCommandsDir)
-    : CompileCommandsDir(std::move(CompileCommandsDir)) {}
+        llvm::Optional<Path> CompileCommandsDir,
+        llvm::Optional<CompilationDatabaseMap> CDBMap)
+    : CompileCommandsDir(std::move(CompileCommandsDir)),
+      CDBMap(std::move(CDBMap)) {}
 
 DirectoryBasedGlobalCompilationDatabase::
     ~DirectoryBasedGlobalCompilationDatabase() = default;
 
 llvm::Optional<tooling::CompileCommand>
 DirectoryBasedGlobalCompilationDatabase::getCompileCommand(PathRef File) const {
   CDBLookupRequest Req;
   Req.FileName = File;
DirectoryBasedGlobalCompilationDatabase::lookupCDB(
   bool ShouldBroadcast = false;
   CDBLookupResult Result;
 
   {
     std::lock_guard<std::mutex> Lock(Mutex);
     CachedCDB *Entry = nullptr;
     if (CompileCommandsDir) {
       Entry = &getCDBInDirLocked(*CompileCommandsDir);
+    } else if (CDBMap) {
+      for (const auto &MapEntry : *CDBMap) {
+        if (StringRef(Request.FileName).startswith(MapEntry.sourceDir)) {
+          Entry = &getCDBInDirLocked(MapEntry.dbPath);
+          break;
+        }
+      }
     } else {
       // Traverse the canonical version to prevent false positives. i.e.:
       // src/build/../a.cc can detect a CDB in /src/build if not canonicalized.
       // FIXME(sammccall): this loop is hot, use a union-find-like structure.
       actOnAllParentDirectories(removeDots(Request.FileName),
                                 [&](PathRef Path) {
                                   Entry = &getCDBInDirLocked(Path);
                                   return Entry->CDB != nullptr;

clang-tools-extra/clangd/Protocol.h

 // Some structs also have operator<< serialization. This is for debugging and
 // tests, and is not generally machine-readable.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_PROTOCOL_H
 #define LLVM_CLANG_TOOLS_EXTRA_CLANGD_PROTOCOL_H
 
+#include "Path.h"
 #include "URI.h"
 #include "index/SymbolID.h"
 #include "clang/Index/IndexSymbol.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/Support/JSON.h"
 #include "llvm/Support/raw_ostream.h"
 #include <bitset>
 #include <memory>

 /// LSP defines this type as `any`.
 struct ConfigurationSettings {
   // Changes to the in-memory compilation database.
   // The key of the map is a file name.
   std::map<std::string, ClangdCompileCommand> compilationDatabaseChanges;
 };
 bool fromJSON(const llvm::json::Value &, ConfigurationSettings &);
 
+struct CompilationDatabasePath {
+  Path sourceDir;
+  Path dbPath;
+};
+bool fromJSON(const llvm::json::Value &, CompilationDatabasePath &);
+
+using CompilationDatabaseMap = std::vector<CompilationDatabasePath>;
+
 /// Clangd extension: parameters configurable at `initialize` time.
 /// LSP defines this type as `any`.
 struct InitializationOptions {
   // What we can change throught the didChangeConfiguration request, we can
   // also set through the initialize request (initializationOptions field).
   ConfigurationSettings ConfigSettings;
 
-  llvm::Optional<std::string> compilationDatabasePath;
+  llvm::Optional<Path> compilationDatabasePath;
+
+  // A map from source directories to directories containing compilation
+  // database files. Paths must be absolute.
+  llvm::Optional<CompilationDatabaseMap> compilationDatabaseMap;
+
   // Additional flags to be included in the "fallback command" used when
   // the compilation database doesn't describe an opened file.
   // The command used will be approximately `clang $FILE $fallbackFlags`.
   std::vector<std::string> fallbackFlags;
 
   /// Clients supports show file status for textDocument/clangd.fileStatus.
   bool FileStatus = false;
 };

clang-tools-extra/clangd/Protocol.cpp

 bool fromJSON(const llvm::json::Value &Params, ConfigurationSettings &S) {
   llvm::json::ObjectMapper O(Params);
   if (!O)
     return true; // 'any' type in LSP.
   O.map("compilationDatabaseChanges", S.compilationDatabaseChanges);
   return true;
 }
 
+bool fromJSON(const llvm::json::Value &Params, CompilationDatabasePath &P) {
+  llvm::json::ObjectMapper O(Params);
+  if (!O)
+    return true; // 'any' type in LSP.
+
+  O.map("sourceDir", P.sourceDir);
+  O.map("dbPath", P.dbPath);
+  return true;
+}
+
 bool fromJSON(const llvm::json::Value &Params, InitializationOptions &Opts) {
   llvm::json::ObjectMapper O(Params);
   if (!O)
     return true; // 'any' type in LSP.
 
   fromJSON(Params, Opts.ConfigSettings);
   O.map("compilationDatabasePath", Opts.compilationDatabasePath);
+  O.map("compilationDatabaseMap", Opts.compilationDatabaseMap);
   O.map("fallbackFlags", Opts.fallbackFlags);
   O.map("clangdFileStatus", Opts.FileStatus);
   return true;
 }
 
 bool fromJSON(const llvm::json::Value &E, TypeHierarchyDirection &Out) {
   auto T = E.getAsInteger();
   if (!T)

clang-tools-extra/clangd/unittests/GlobalCompilationDatabaseTests.cpp

 using ::testing::EndsWith;
 using ::testing::HasSubstr;
 using ::testing::IsEmpty;
 using ::testing::Not;
 using ::testing::StartsWith;
 using ::testing::UnorderedElementsAre;
 
 TEST(GlobalCompilationDatabaseTest, FallbackCommand) {
-  DirectoryBasedGlobalCompilationDatabase DB(None);
+  DirectoryBasedGlobalCompilationDatabase DB(None, None);
   auto Cmd = DB.getFallbackCommand(testPath("foo/bar.cc"));
   EXPECT_EQ(Cmd.Directory, testPath("foo"));
   EXPECT_THAT(Cmd.CommandLine, ElementsAre("clang", testPath("foo/bar.cc")));
   EXPECT_EQ(Cmd.Output, "");
 
   // .h files have unknown language, so they are parsed liberally as obj-c++.
   Cmd = DB.getFallbackCommand(testPath("foo/bar.h"));
   EXPECT_THAT(Cmd.CommandLine, ElementsAre("clang", "-xobjective-c++-header",
TEST(GlobalCompilationDatabaseTest, DiscoveryWithNestedCDBs) {
   CleaningFS FS;
   FS.registerFile("compile_commands.json", CDBOuter);
   FS.registerFile("build/compile_commands.json", CDBInner);
   llvm::SmallString<128> File;
 
   // Note that gen2.cc goes missing with our following model, not sure this
   // happens in practice though.
   {
-    DirectoryBasedGlobalCompilationDatabase DB(llvm::None);
+    DirectoryBasedGlobalCompilationDatabase DB(llvm::None, llvm::None);
     std::vector<std::string> DiscoveredFiles;
     auto Sub =
         DB.watch([&DiscoveredFiles](const std::vector<std::string> Changes) {
           DiscoveredFiles = Changes;
         });
 
     File = FS.Root;
     llvm::sys::path::append(File, "build", "..", "a.cc");
     DB.getCompileCommand(File.str());
     EXPECT_THAT(DiscoveredFiles, UnorderedElementsAre(AllOf(
                                      EndsWith("a.cc"), Not(HasSubstr("..")))));
     DiscoveredFiles.clear();
 
     File = FS.Root;
     llvm::sys::path::append(File, "build", "gen.cc");
     DB.getCompileCommand(File.str());
     EXPECT_THAT(DiscoveredFiles, UnorderedElementsAre(EndsWith("gen.cc")));
   }
 
   // With a custom compile commands dir.
   {
-    DirectoryBasedGlobalCompilationDatabase DB(FS.Root.str().str());
+    DirectoryBasedGlobalCompilationDatabase DB(FS.Root.str().str(), llvm::None);
     std::vector<std::string> DiscoveredFiles;
     auto Sub =
         DB.watch([&DiscoveredFiles](const std::vector<std::string> Changes) {
           DiscoveredFiles = Changes;
         });
 
     File = FS.Root;
     llvm::sys::path::append(File, "a.cc");