[clangd] Filter out non-governed files from broadcast

[kadircet]

Summary

This also turns off implicit discovery of additional compilation
databases.

[sammccall]

clang-tools-extra/clangd/GlobalCompilationDatabase.cpp

106

The duplication here is a bit troublesome.

And we have divergences in behavior - probably not important ones, but I can't say I know for sure. (e.g. here we won't create the CDB as a side-effect of getPathInfo() if there's a CompileCommandsDir, and will return the dir even if the CDB doesn't exist)

I think we might want to have some central logic parameterized by a struct, e.g.

struct DirectoryBasedCompilationDatabase::Lookup {
  // inputs
  PathRef File;
  bool ShouldBroadcast;

  // outputs
  tooling::CompilationDatabase *CDB;
  ProjectInfo Info;
  bool EverBroadcast;
};
lookupCDB(Lookup); // replaces getCDBInDirLocked

209

here we're locking/unlocking the mutex maybe thousands of times, to do mostly redundant lookups into the cache

Seems worth doing at least one of:

• deduplicate directories, so we lock once for each distinct directory and build a string->bool map. Then use that map to filter the results
• lock around the whole loop and use getCDBInDirLocked()

clang-tools-extra/clangd/index/Background.cpp

653

This looks like a bad assertion: it should be OK to provide compile commands but not project info.

(Otherwise getProjectInfo should be pure virtual, but I'm concerned about the raciness if we're going to insist they're consistent but not provide any way of synchronizing)

I'd suggest we should just not index such files (for now). Later we can start passing "" to the index storage factory to get the fallback storage (I think today we pass "", but the factory doesn't handle it - oops!)

[kadircet]

• Address comments

[kadircet]

clang-tools-extra/clangd/index/Background.cpp

653

actually we have a nullstorage for that, which just logs a messsage and doesn't persist the shard.

[kadircet]

• Rearrange changes to get a better diff

[sammccall]

clang-tools-extra/clangd/GlobalCompilationDatabase.cpp

109

I don't think we'll ever fix this, feel free to drop the FIXME if you like.

118

return Path == Result.PI.SourceRoot?

125

(Why not do this in the one pass above? Doesn't seem worth two passes to avoid locking while iterating over AllFiles)

134

assumes isn't strong enough here - we're defining it this way, and should say why.

e.g. "A file is only part of this CDB if lookup for the file would find this CDB."

clang-tools-extra/clangd/GlobalCompilationDatabase.h

115

(random note: we should really rename this something like FixedCDBDir. Probably not in this patch, but what a confusing name)

[kadircet]

• Address comments

clang-tools-extra/clangd/GlobalCompilationDatabase.h

 };
 
 /// Provides compilation arguments used for parsing C and C++ files.
 class GlobalCompilationDatabase {
 public:
   virtual ~GlobalCompilationDatabase() = default;
 
   /// If there are any known-good commands for building this file, returns one.
-  /// If the ProjectInfo pointer is set, it will also be populated.
   virtual llvm::Optional<tooling::CompileCommand>
-  getCompileCommand(PathRef File, ProjectInfo * = nullptr) const = 0;
+  getCompileCommand(PathRef File) const = 0;
+
+  /// Finds the closest project to \p File.
+  virtual llvm::Optional<ProjectInfo> getProjectInfo(PathRef File) const {
+    return llvm::None;
+  }
 
   /// Makes a guess at how to build a file.
   /// The default implementation just runs clang on the file.
   /// Clangd should treat the results as unreliable.
   virtual tooling::CompileCommand getFallbackCommand(PathRef File) const;
 
   using CommandChanged = Event<std::vector<std::string>>;
   /// The callback is notified when files may have new compile commands.
class DirectoryBasedGlobalCompilationDatabase
     : public GlobalCompilationDatabase {
 public:
   DirectoryBasedGlobalCompilationDatabase(
       llvm::Optional<Path> CompileCommandsDir);
   ~DirectoryBasedGlobalCompilationDatabase() override;
 
   /// Scans File's parents looking for compilation databases.
   /// Any extra flags will be added.
+  /// Might trigger OnCommandChanged, if CDB wasn't broadcasted yet.
   llvm::Optional<tooling::CompileCommand>
-  getCompileCommand(PathRef File, ProjectInfo * = nullptr) const override;
+  getCompileCommand(PathRef File) const override;
+
+  /// Returns the path to first directory containing a compilation database in
+  /// \p File's parents.
+  llvm::Optional<ProjectInfo> getProjectInfo(PathRef File) const override;
 
 private:
-  tooling::CompilationDatabase *getCDBForFile(PathRef File,
-                                              ProjectInfo *) const;
-  std::pair<tooling::CompilationDatabase *, /*Cached*/ bool>
+  std::pair<tooling::CompilationDatabase *, /*SentBroadcast*/ bool>
   getCDBInDirLocked(PathRef File) const;
 
+  struct CDBLookupRequest {
+    PathRef FileName;
+    // Whether this lookup should trigger discovery of the CDB found.
+    bool ShouldBroadcast = false;
+  };
+  struct CDBLookupResult {
+    tooling::CompilationDatabase *CDB = nullptr;
+    ProjectInfo PI;
+  };
+  llvm::Optional<CDBLookupResult> lookupCDB(CDBLookupRequest Request) const;
+
+  // Performs broadcast on governed files.
+  void broadcastCDB(CDBLookupResult Res) const;
+
   mutable std::mutex Mutex;
   /// Caches compilation databases loaded from directories(keys are
   /// directories).
-  mutable llvm::StringMap<std::unique_ptr<clang::tooling::CompilationDatabase>>
-      CompilationDatabases;
+  struct CachedCDB {
+    std::unique_ptr<clang::tooling::CompilationDatabase> CDB = nullptr;
+    bool SentBroadcast = false;
+  };
+  mutable llvm::StringMap<CachedCDB> CompilationDatabases;
 
   /// Used for command argument pointing to folder where compile_commands.json
   /// is located.
   llvm::Optional<Path> CompileCommandsDir;
 };
 
 /// Extracts system include search path from drivers matching QueryDriverGlobs
 /// and adds them to the compile flags. Base may not be nullptr.
 /// Returns Base when \p QueryDriverGlobs is empty.
 std::unique_ptr<GlobalCompilationDatabase>
 getQueryDriverDatabase(llvm::ArrayRef<std::string> QueryDriverGlobs,
                        std::unique_ptr<GlobalCompilationDatabase> Base);
 
 /// Wraps another compilation database, and supports overriding the commands
 /// using an in-memory mapping.
 class OverlayCDB : public GlobalCompilationDatabase {
 public:
   // Base may be null, in which case no entries are inherited.
   // FallbackFlags are added to the fallback compile command.
   OverlayCDB(const GlobalCompilationDatabase *Base,
              std::vector<std::string> FallbackFlags = {},
              llvm::Optional<std::string> ResourceDir = llvm::None);
 
   llvm::Optional<tooling::CompileCommand>
-  getCompileCommand(PathRef File, ProjectInfo * = nullptr) const override;
+  getCompileCommand(PathRef File) const override;
   tooling::CompileCommand getFallbackCommand(PathRef File) const override;
+  llvm::Optional<ProjectInfo> getProjectInfo(PathRef File) const override;
 
   /// Sets or clears the compilation command for a particular file.
   void
   setCompileCommand(PathRef File,
                     llvm::Optional<tooling::CompileCommand> CompilationCommand);
 
 private:
   mutable std::mutex Mutex;

clang-tools-extra/clangd/GlobalCompilationDatabase.cpp

 //===--- GlobalCompilationDatabase.cpp ---------------------------*- C++-*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "GlobalCompilationDatabase.h"
 #include "Logger.h"
+#include "Path.h"
 #include "clang/Frontend/CompilerInvocation.h"
 #include "clang/Tooling/ArgumentsAdjusters.h"
 #include "clang/Tooling/CompilationDatabase.h"
+#include "llvm/ADT/None.h"
 #include "llvm/ADT/Optional.h"
+#include "llvm/ADT/STLExtras.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Path.h"
+#include <string>
+#include <tuple>
+#include <vector>
 
 namespace clang {
 namespace clangd {
 namespace {
 
 void adjustArguments(tooling::CompileCommand &Cmd,
                      llvm::StringRef ResourceDir) {
   tooling::ArgumentsAdjuster ArgsAdjuster = tooling::combineAdjusters(
if (!ResourceDir.empty())
     Cmd.CommandLine.push_back(("-resource-dir=" + ResourceDir).str());
 }
 
 std::string getStandardResourceDir() {
   static int Dummy; // Just an address in this process.
   return CompilerInvocation::GetResourcesPath("clangd", (void *)&Dummy);
 }
 
+// Runs the given action on all parent directories of filename, starting from
+// deepest directory and going up to root. Stops whenever action succeeds.
+void actOnAllParentDirectories(PathRef FileName,
+                               llvm::function_ref<bool(PathRef)> Action) {
+  for (auto Path = llvm::sys::path::parent_path(FileName);
+       !Path.empty() && !Action(Path);
+       Path = llvm::sys::path::parent_path(Path))
+    ;
+}
+
 } // namespace
 
 static std::string getFallbackClangPath() {
   static int Dummy;
   std::string ClangdExecutable =
       llvm::sys::fs::getMainExecutable("clangd", (void *)&Dummy);
   SmallString<128> ClangPath;
   ClangPath = llvm::sys::path::parent_path(ClangdExecutable);
DirectoryBasedGlobalCompilationDatabase::
     DirectoryBasedGlobalCompilationDatabase(
         llvm::Optional<Path> CompileCommandsDir)
     : CompileCommandsDir(std::move(CompileCommandsDir)) {}
 
 DirectoryBasedGlobalCompilationDatabase::
     ~DirectoryBasedGlobalCompilationDatabase() = default;
 
 llvm::Optional<tooling::CompileCommand>
-DirectoryBasedGlobalCompilationDatabase::getCompileCommand(
-    PathRef File, ProjectInfo *Project) const {
-  if (auto CDB = getCDBForFile(File, Project)) {
-    auto Candidates = CDB->getCompileCommands(File);
-    if (!Candidates.empty()) {
-      return std::move(Candidates.front());
-    }
-  } else {
+DirectoryBasedGlobalCompilationDatabase::getCompileCommand(PathRef File) const {
+  CDBLookupRequest Req;
+  Req.FileName = File;
+  Req.ShouldBroadcast = true;
+
+  auto Res = lookupCDB(Req);
+  if (!Res) {
     log("Failed to find compilation database for {0}", File);
+    return llvm::None;
   }
+
+  auto Candidates = Res->CDB->getCompileCommands(File);
+  if (!Candidates.empty())
+    return std::move(Candidates.front());
+
   return None;
 }
 
-std::pair<tooling::CompilationDatabase *, /*Cached*/ bool>
+std::pair<tooling::CompilationDatabase *, /*SentBroadcast*/ bool>
 DirectoryBasedGlobalCompilationDatabase::getCDBInDirLocked(PathRef Dir) const {
   // FIXME(ibiryukov): Invalidate cached compilation databases on changes
   auto CachedIt = CompilationDatabases.find(Dir);
   if (CachedIt != CompilationDatabases.end())
-    return {CachedIt->second.get(), true};
+    return {CachedIt->second.CDB.get(), CachedIt->second.SentBroadcast};
   std::string Error = "";
-  auto CDB = tooling::CompilationDatabase::loadFromDirectory(Dir, Error);
-  auto Result = CDB.get();
-  CompilationDatabases.insert(std::make_pair(Dir, std::move(CDB)));
+
+  CachedCDB Entry;
+  Entry.CDB = tooling::CompilationDatabase::loadFromDirectory(Dir, Error);
+  auto Result = Entry.CDB.get();
+  CompilationDatabases[Dir] = std::move(Entry);
+
   return {Result, false};
 }
 
-tooling::CompilationDatabase *
-DirectoryBasedGlobalCompilationDatabase::getCDBForFile(
-    PathRef File, ProjectInfo *Project) const {
-  namespace path = llvm::sys::path;
-  assert((path::is_absolute(File, path::Style::posix) ||
-          path::is_absolute(File, path::Style::windows)) &&
+llvm::Optional<DirectoryBasedGlobalCompilationDatabase::CDBLookupResult>
+DirectoryBasedGlobalCompilationDatabase::lookupCDB(
+    CDBLookupRequest Request) const {
+  assert(llvm::sys::path::is_absolute(Request.FileName) &&
          "path must be absolute");
 
-  tooling::CompilationDatabase *CDB = nullptr;
-  bool Cached = false;
+  CDBLookupResult Result;
+  bool SentBroadcast = false;
+
+  {
     std::lock_guard<std::mutex> Lock(Mutex);
     if (CompileCommandsDir) {
-    std::tie(CDB, Cached) = getCDBInDirLocked(*CompileCommandsDir);
-    if (Project && CDB)
-      Project->SourceRoot = *CompileCommandsDir;
+      std::tie(Result.CDB, SentBroadcast) =
+          getCDBInDirLocked(*CompileCommandsDir);
+      Result.PI.SourceRoot = *CompileCommandsDir;
     } else {
-    for (auto Path = path::parent_path(File); !CDB && !Path.empty();
-         Path = path::parent_path(Path)) {
-      std::tie(CDB, Cached) = getCDBInDirLocked(Path);
-      if (Project && CDB)
-        Project->SourceRoot = Path;
-    }
-  }
-  // FIXME: getAllFiles() may return relative paths, we need absolute paths.
-  // Hopefully the fix is to change JSONCompilationDatabase and the interface.
-  if (CDB && !Cached)
-    OnCommandChanged.broadcast(CDB->getAllFiles());
-  return CDB;
+      actOnAllParentDirectories(
+          Request.FileName, [this, &SentBroadcast, &Result](PathRef Path) {
+            std::tie(Result.CDB, SentBroadcast) = getCDBInDirLocked(Path);
+            Result.PI.SourceRoot = Path;
+            return Result.CDB != nullptr;
+          });
+    }
+
+    if (!Result.CDB)
+      return llvm::None;
+
+    // Mark CDB as broadcasted to make sure discovery is performed once.
+    if (Request.ShouldBroadcast && !SentBroadcast)
+      CompilationDatabases[Result.PI.SourceRoot].SentBroadcast = true;
+
+  // FIXME: Maybe make the following part async, since this can block retrieval
+  // of compile commands.
+  if (Request.ShouldBroadcast && !SentBroadcast)
+    broadcastCDB(Result);
+  return Result;
+
+void DirectoryBasedGlobalCompilationDatabase::broadcastCDB(
+    CDBLookupResult Result) const {
+  assert(Result.CDB && "Trying to broadcast an invalid CDB!");
+
+  std::vector<std::string> AllFiles = Result.CDB->getAllFiles();
+  // We assume CDB in CompileCommandsDir owns all of its entries, since we don't
+  // perform any search in parent paths whenever it is set.
+  if (CompileCommandsDir) {
+    assert(*CompileCommandsDir == Result.PI.SourceRoot &&
+           "Trying to broadcast a CDB outside of CompileCommandsDir!");
+    OnCommandChanged.broadcast(std::move(AllFiles));
+    return;
+  }
+
+  llvm::StringMap<bool> DirectoryHasCDB;
+  {
+    std::lock_guard<std::mutex> Lock(Mutex);
+    // Uniquify all parent directories of all files.
+    for (llvm::StringRef File : AllFiles) {
+      actOnAllParentDirectories(File, [&](PathRef Path) {
+        auto It = DirectoryHasCDB.try_emplace(Path);
+        // Already seen this path, and all of its parents.
+        if (!It.second)
+          return true;
+
+        auto Res = getCDBInDirLocked(Path);
+        It.first->second = Res.first != nullptr;
+        return Path == Result.PI.SourceRoot;
+      });
+    }
+  }
+
+  std::vector<std::string> GovernedFiles;
+  for (llvm::StringRef File : AllFiles) {
+    // A file is governed by this CDB if lookup for the file would find it.
+    // Independent of whether it has an entry for that file or not.
+    actOnAllParentDirectories(File, [&](PathRef Path) {
+      if (DirectoryHasCDB.lookup(Path)) {
+        if (Path == Result.PI.SourceRoot)
+          GovernedFiles.push_back(File);
+        // Stop as soon as we hit a CDB.
+        return true;
+      }
+      return false;
+    });
+  }
+
+  OnCommandChanged.broadcast(std::move(GovernedFiles));
+}
+
+llvm::Optional<ProjectInfo>
+DirectoryBasedGlobalCompilationDatabase::getProjectInfo(PathRef File) const {
+  CDBLookupRequest Req;
+  Req.FileName = File;
+  Req.ShouldBroadcast = false;
+  auto Res = lookupCDB(Req);
+  if (!Res)
+    return llvm::None;
+  return Res->PI;
 }
 
 OverlayCDB::OverlayCDB(const GlobalCompilationDatabase *Base,
                        std::vector<std::string> FallbackFlags,
                        llvm::Optional<std::string> ResourceDir)
     : Base(Base), ResourceDir(ResourceDir ? std::move(*ResourceDir)
                                           : getStandardResourceDir()),
       FallbackFlags(std::move(FallbackFlags)) {
   if (Base)
     BaseChanged = Base->watch([this](const std::vector<std::string> Changes) {
       OnCommandChanged.broadcast(Changes);
     });
 }
 
 llvm::Optional<tooling::CompileCommand>
-OverlayCDB::getCompileCommand(PathRef File, ProjectInfo *Project) const {
+OverlayCDB::getCompileCommand(PathRef File) const {
   llvm::Optional<tooling::CompileCommand> Cmd;
   {
     std::lock_guard<std::mutex> Lock(Mutex);
     auto It = Commands.find(File);
-    if (It != Commands.end()) {
-      if (Project)
-        Project->SourceRoot = "";
+    if (It != Commands.end())
       Cmd = It->second;
-    }
-  }
   if (!Cmd && Base)
-    Cmd = Base->getCompileCommand(File, Project);
+    Cmd = Base->getCompileCommand(File);
   if (!Cmd)
     return llvm::None;
   adjustArguments(*Cmd, ResourceDir);
   return Cmd;
 }
 
 tooling::CompileCommand OverlayCDB::getFallbackCommand(PathRef File) const {
   auto Cmd = Base ? Base->getFallbackCommand(File)
void OverlayCDB::setCompileCommand(
     if (Cmd)
       Commands[File] = std::move(*Cmd);
     else
       Commands.erase(File);
   }
   OnCommandChanged.broadcast({File});
 }
 
+llvm::Optional<ProjectInfo> OverlayCDB::getProjectInfo(PathRef File) const {
+  {
+    std::lock_guard<std::mutex> Lock(Mutex);
+    auto It = Commands.find(File);
+    if (It != Commands.end())
+      return ProjectInfo{};
+  }
+  if (Base)
+    return Base->getProjectInfo(File);
+
+  return llvm::None;
+}
 } // namespace clangd
 } // namespace clang

clang-tools-extra/clangd/QueryDriverDatabase.cpp

QueryDriverDatabase(llvm::ArrayRef<std::string> QueryDriverGlobs,
     assert(this->Base);
     BaseChanged =
         this->Base->watch([this](const std::vector<std::string> &Changes) {
           OnCommandChanged.broadcast(Changes);
         });
   }
 
   llvm::Optional<tooling::CompileCommand>
-  getCompileCommand(PathRef File, ProjectInfo *PI = nullptr) const override {
-    auto Cmd = Base->getCompileCommand(File, PI);
+  getCompileCommand(PathRef File) const override {
+    auto Cmd = Base->getCompileCommand(File);
     if (!Cmd || Cmd->CommandLine.empty())
       return Cmd;
 
     llvm::SmallString<128> Driver(Cmd->CommandLine.front());
     llvm::sys::fs::make_absolute(Cmd->Directory, Driver);
     llvm::StringRef Ext = llvm::sys::path::extension(File).trim('.');
     auto Key = std::make_pair(Driver.str(), Ext);
 
     std::vector<std::string> SystemIncludes;
     {
       std::lock_guard<std::mutex> Lock(Mu);
 
       auto It = DriverToIncludesCache.find(Key);
       if (It != DriverToIncludesCache.end())
         SystemIncludes = It->second;
       else
         DriverToIncludesCache[Key] = SystemIncludes =
             extractSystemIncludes(Key.first, Key.second, QueryDriverRegex);
     }
 
     return addSystemIncludes(*Cmd, SystemIncludes);
   }
 
+  llvm::Optional<ProjectInfo> getProjectInfo(PathRef File) const override {
+    return Base->getProjectInfo(File);
+  }
+
 private:
   mutable std::mutex Mu;
   // Caches includes extracted from a driver.
   mutable std::map<std::pair<std::string, std::string>,
                    std::vector<std::string>>
       DriverToIncludesCache;
   mutable llvm::Regex QueryDriverRegex;
 

clang-tools-extra/clangd/index/Background.cpp

std::vector<std::pair<tooling::CompileCommand, BackgroundIndexStorage *>>
       NeedsReIndexing;
   // Keeps track of the files that will be reindexed, to make sure we won't
   // re-index same dependencies more than once. Keys are AbsolutePaths.
   llvm::StringSet<> FilesToIndex;
   // Keeps track of the loaded shards to make sure we don't perform redundant
   // disk IO. Keys are absolute paths.
   llvm::StringSet<> LoadedShards;
   for (const auto &File : ChangedFiles) {
-    ProjectInfo PI;
-    auto Cmd = CDB.getCompileCommand(File, &PI);
+    auto Cmd = CDB.getCompileCommand(File);
     if (!Cmd)
       continue;
-    BackgroundIndexStorage *IndexStorage = IndexStorageFactory(PI.SourceRoot);
+
+    std::string ProjectRoot;
+    if (auto PI = CDB.getProjectInfo(File))
+      ProjectRoot = std::move(PI->SourceRoot);
+
+    BackgroundIndexStorage *IndexStorage = IndexStorageFactory(ProjectRoot);
     auto Dependencies = loadShard(*Cmd, IndexStorage, LoadedShards);
     for (const auto &Dependency : Dependencies) {
       if (!Dependency.NeedsReIndexing || FilesToIndex.count(Dependency.Path))
         continue;
       // FIXME: Currently, we simply schedule indexing on a TU whenever any of
       // its dependencies needs re-indexing. We might do it smarter by figuring
       // out a minimal set of TUs that will cover all the stale dependencies.
       vlog("Enqueueing TU {0} because its dependency {1} needs re-indexing.",

clang-tools-extra/clangd/unittests/ClangdTests.cpp

 
 TEST_F(ClangdVFSTest, FallbackWhenPreambleIsNotReady) {
   MockFSProvider FS;
   ErrorCheckingDiagConsumer DiagConsumer;
   MockCompilationDatabase CDB;
   ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
 
   auto FooCpp = testPath("foo.cpp");
-   Annotations Code(R"cpp(
     namespace ns { int xyz; }
     using namespace ns;
     int main() {
        xy^
     })cpp");
   FS.Files[FooCpp] = FooCpp;
 
   auto Opts = clangd::CodeCompleteOptions();
TEST_F(ClangdVFSTest, FallbackWhenWaitingForCompileCommand) {
   ErrorCheckingDiagConsumer DiagConsumer;
   // Returns compile command only when notified.
   class DelayedCompilationDatabase : public GlobalCompilationDatabase {
   public:
     DelayedCompilationDatabase(Notification &CanReturnCommand)
         : CanReturnCommand(CanReturnCommand) {}
 
     llvm::Optional<tooling::CompileCommand>
-    getCompileCommand(PathRef File, ProjectInfo * = nullptr) const override {
+    getCompileCommand(PathRef File) const override {
       // FIXME: make this timeout and fail instead of waiting forever in case
       // something goes wrong.
       CanReturnCommand.wait();
       auto FileName = llvm::sys::path::filename(File);
       std::vector<std::string> CommandLine = {"clangd", "-ffreestanding", File};
       return {tooling::CompileCommand(llvm::sys::path::parent_path(File),
                                       FileName, std::move(CommandLine), "")};
     }

clang-tools-extra/clangd/unittests/GlobalCompilationDatabaseTests.cpp

 //===-- GlobalCompilationDatabaseTests.cpp ----------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "GlobalCompilationDatabase.h"
 
+#include "Path.h"
 #include "TestFS.h"
+#include "llvm/ADT/Optional.h"
+#include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/StringExtras.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/FormatVariadic.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/raw_ostream.h"
 #include "gmock/gmock.h"
 #include "gtest/gtest.h"
+#include <fstream>
+#include <string>
 
 namespace clang {
 namespace clangd {
 namespace {
 using ::testing::AllOf;
 using ::testing::Contains;
 using ::testing::ElementsAre;
 using ::testing::EndsWith;
+using ::testing::IsEmpty;
 using ::testing::Not;
 using ::testing::StartsWith;
+using ::testing::UnorderedElementsAre;
 
 TEST(GlobalCompilationDatabaseTest, FallbackCommand) {
   DirectoryBasedGlobalCompilationDatabase DB(None);
   auto Cmd = DB.getFallbackCommand(testPath("foo/bar.cc"));
   EXPECT_EQ(Cmd.Directory, testPath("foo"));
   EXPECT_THAT(Cmd.CommandLine,
               ElementsAre(EndsWith("clang"), testPath("foo/bar.cc")));
   EXPECT_EQ(Cmd.Output, "");

 static tooling::CompileCommand cmd(llvm::StringRef File, llvm::StringRef Arg) {
   return tooling::CompileCommand(testRoot(), File, {"clang", Arg, File}, "");
 }
 
 class OverlayCDBTest : public ::testing::Test {
   class BaseCDB : public GlobalCompilationDatabase {
   public:
     llvm::Optional<tooling::CompileCommand>
-    getCompileCommand(llvm::StringRef File,
-                      ProjectInfo *Project) const override {
-      if (File == testPath("foo.cc")) {
-        if (Project)
-          Project->SourceRoot = testRoot();
+    getCompileCommand(llvm::StringRef File) const override {
+      if (File == testPath("foo.cc"))
         return cmd(File, "-DA=1");
-      }
       return None;
     }
 
     tooling::CompileCommand
     getFallbackCommand(llvm::StringRef File) const override {
       return cmd(File, "-DA=2");
     }
+
+    llvm::Optional<ProjectInfo> getProjectInfo(PathRef File) const override {
+      return ProjectInfo{testRoot()};
+    }
   };
 
 protected:
   OverlayCDBTest() : Base(llvm::make_unique<BaseCDB>()) {}
   std::unique_ptr<GlobalCompilationDatabase> Base;
 };
 
 TEST_F(OverlayCDBTest, GetCompileCommand) {
TEST_F(OverlayCDBTest, Adjustments) {
   EXPECT_THAT(CDB.getCompileCommand(testPath("foo.cc"))->CommandLine,
               AllOf(Contains("-fsyntax-only"), Contains("-DA=5"),
                     Contains(testPath("foo.cc")), Not(Contains("-MF")),
                     Not(Contains("random-dependency")),
                     Not(Contains("-Xclang")), Not(Contains("-load")),
                     Not(Contains("random-plugin"))));
 }
 
+TEST(GlobalCompilationDatabaseTest, DiscoveryWithNestedCDBs) {
+  const char *const CDBOuter =
+      R"cdb(
+      [
+        {
+          "file": "a.cc",
+          "command": "",
+          "directory": "{0}",
+        },
+        {
+          "file": "build/gen.cc",
+          "command": "",
+          "directory": "{0}",
+        },
+        {
+          "file": "build/gen2.cc",
+          "command": "",
+          "directory": "{0}",
+        }
+      ]
+      )cdb";
+  const char *const CDBInner =
+      R"cdb(
+      [
+        {
+          "file": "gen.cc",
+          "command": "",
+          "directory": "{0}/build",
+        }
+      ]
+      )cdb";
+  class CleaningFS {
+  public:
+    llvm::SmallString<128> Root;
+
+    CleaningFS() {
+      EXPECT_FALSE(
+          llvm::sys::fs::createUniqueDirectory("clangd-cdb-test", Root))
+          << "Failed to create unique directory";
+    }
+
+    ~CleaningFS() {
+      EXPECT_FALSE(llvm::sys::fs::remove_directories(Root))
+          << "Failed to cleanup " << Root;
+    }
+
+    void registerFile(PathRef RelativePath, llvm::StringRef Contents) {
+      llvm::SmallString<128> AbsPath(Root);
+      llvm::sys::path::append(AbsPath, RelativePath);
+
+      EXPECT_FALSE(llvm::sys::fs::create_directories(
+          llvm::sys::path::parent_path(AbsPath)))
+          << "Failed to create directories for: " << AbsPath;
+
+      std::error_code EC;
+      llvm::raw_fd_ostream OS(AbsPath, EC);
+      EXPECT_FALSE(EC) << "Failed to open " << AbsPath << " for writing";
+      OS << llvm::formatv(Contents.data(), Root);
+      OS.close();
+
+      EXPECT_FALSE(OS.has_error());
+    }
+  };
+
+  CleaningFS FS;
+  FS.registerFile("compile_commands.json", CDBOuter);
+  FS.registerFile("build/compile_commands.json", CDBInner);
+
+  // Note that gen2.cc goes missing with our following model, not sure this
+  // happens in practice though.
+  {
+    DirectoryBasedGlobalCompilationDatabase DB(llvm::None);
+    std::vector<std::string> DiscoveredFiles;
+    auto Sub =
+        DB.watch([&DiscoveredFiles](const std::vector<std::string> Changes) {
+          DiscoveredFiles = Changes;
+        });
+    DB.getCompileCommand((FS.Root + "/a.cc").str());
+    EXPECT_THAT(DiscoveredFiles, UnorderedElementsAre(EndsWith("a.cc")));
+
+    DB.getCompileCommand((FS.Root + "/build/gen.cc").str());
+    EXPECT_THAT(DiscoveredFiles, UnorderedElementsAre(EndsWith("gen.cc")));
+  }
+
+  // With a custom compile commands dir.
+  {
+    DirectoryBasedGlobalCompilationDatabase DB(FS.Root.str().str());
+    std::vector<std::string> DiscoveredFiles;
+    auto Sub =
+        DB.watch([&DiscoveredFiles](const std::vector<std::string> Changes) {
+          DiscoveredFiles = Changes;
+        });
+    DB.getCompileCommand((FS.Root + "/a.cc").str());
+    EXPECT_THAT(DiscoveredFiles,
+                UnorderedElementsAre(EndsWith("a.cc"), EndsWith("gen.cc"),
+                                     EndsWith("gen2.cc")));
+
+    DiscoveredFiles.clear();
+    DB.getCompileCommand((FS.Root + "/build/gen.cc").str());
+    EXPECT_THAT(DiscoveredFiles, IsEmpty());
+  }
+}
+
 } // namespace
 } // namespace clangd
 } // namespace clang

clang-tools-extra/clangd/unittests/TestFS.h

 //===-- TestFS.h ------------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Allows setting up fake filesystem environments for tests.
 //
 //===----------------------------------------------------------------------===//
 #ifndef LLVM_CLANG_TOOLS_EXTRA_UNITTESTS_CLANGD_TESTFS_H
 #define LLVM_CLANG_TOOLS_EXTRA_UNITTESTS_CLANGD_TESTFS_H
 #include "ClangdServer.h"
+#include "GlobalCompilationDatabase.h"
+#include "Path.h"
 #include "llvm/ADT/IntrusiveRefCntPtr.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/VirtualFileSystem.h"
 
 namespace clang {
 namespace clangd {
 
 // Builds a VFS that provides access to the provided files, plus temporary
public:
   /// CompileCommand, and as project SourceRoot.
   ///
   /// If \p RelPathPrefix is not empty, use that as a prefix in front of the
   /// source file name, instead of using an absolute path.
   MockCompilationDatabase(StringRef Directory = StringRef(),
                           StringRef RelPathPrefix = StringRef());
 
   llvm::Optional<tooling::CompileCommand>
-  getCompileCommand(PathRef File, ProjectInfo * = nullptr) const override;
+  getCompileCommand(PathRef File) const override;
+
+  llvm::Optional<ProjectInfo> getProjectInfo(PathRef File) const override;
 
   std::vector<std::string> ExtraClangFlags;
 
 private:
   StringRef Directory;
   StringRef RelPathPrefix;
 };
 

clang-tools-extra/clangd/unittests/TestFS.cpp

 //===-- TestFS.cpp ----------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 #include "TestFS.h"
+#include "GlobalCompilationDatabase.h"
+#include "Path.h"
 #include "URI.h"
+#include "llvm/ADT/None.h"
+#include "llvm/ADT/Optional.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/Errc.h"
 #include "llvm/Support/Path.h"
 
 namespace clang {
 namespace clangd {
 
 llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem>

 
 MockCompilationDatabase::MockCompilationDatabase(llvm::StringRef Directory,
                                                  llvm::StringRef RelPathPrefix)
     : ExtraClangFlags({"-ffreestanding"}), Directory(Directory),
       RelPathPrefix(RelPathPrefix) {
   // -ffreestanding avoids implicit stdc-predef.h.
 }
 
+llvm::Optional<ProjectInfo>
+MockCompilationDatabase::getProjectInfo(PathRef File) const {
+  return ProjectInfo{Directory};
+};
+
 llvm::Optional<tooling::CompileCommand>
-MockCompilationDatabase::getCompileCommand(PathRef File,
-                                           ProjectInfo *Project) const {
+MockCompilationDatabase::getCompileCommand(PathRef File) const {
   if (ExtraClangFlags.empty())
     return None;
 
   auto FileName = llvm::sys::path::filename(File);
 
   // Build the compile command.
   auto CommandLine = ExtraClangFlags;
   CommandLine.insert(CommandLine.begin(), "clang");
   if (RelPathPrefix.empty()) {
     // Use the absolute path in the compile command.
     CommandLine.push_back(File);
   } else {
     // Build a relative path using RelPathPrefix.
     llvm::SmallString<32> RelativeFilePath(RelPathPrefix);
     llvm::sys::path::append(RelativeFilePath, FileName);
     CommandLine.push_back(RelativeFilePath.str());
   }
 
-  if (Project)
-    Project->SourceRoot = Directory;
   return {tooling::CompileCommand(Directory != llvm::StringRef()
                                       ? Directory
                                       : llvm::sys::path::parent_path(File),
                                   FileName, std::move(CommandLine), "")};
 }
 
 const char *testRoot() {
 #ifdef _WIN32