[clangd] Allow configuration database to be specified in config.

[sammccall]

Summary

This allows more flexibility than --compile-commands-dir flag or ancestor-based discovery.

See https://github.com/clangd/clangd/issues/116

[sammccall]

Rebase, implement yaml bits (no tests yet)

[sammccall]

Add tests and fix bugs

[kadircet]

I believe we should log some warning at startup if user has provided --compile-commands-dir. Saying that "CDB search customizations through config is disabled". (only emitting the warning when we hit a config with CDB search customization would be nicer, but plumbing and making ConfigCompiler aware of such things sounds terrifying :/)

mostly LG, didn't get a change to look at the tests yet (will do so soon)

clang-tools-extra/clangd/ConfigCompile.cpp

267

i hope no one tries to put their CDBs in a directory named Ancestors/None :)))

(it is definitely better than breaking any project with a top-level directory named Ancestors/None)

278

+1 i think it should.

clang-tools-extra/clangd/GlobalCompilationDatabase.cpp

617

missing a Child = &Info; ?

651

nit: DirValues[I]->State = DirCaches[I] == ThisCache ? DirInfo::TargetCDB : DirInfo::Unknown;, to reduce branching.

681

s/P.getInt()/1

as we'll bail out otherwise.

718

s/Recursive/Recursive=/

[kadircet]

clang-tools-extra/clangd/unittests/ConfigCompileTests.cpp

140

maybe also try with an absolute path in CompilationDatabase ?

[sammccall]

address comments

[sammccall]

In D95057#2518903, @kadircet wrote:

I believe we should log some warning at startup if user has provided --compile-commands-dir. Saying that "CDB search customizations through config is disabled".

So I considered this, but it depends on what we want the eventual interaction of flags and config to be.

A) if we're going to eliminate any flags that overlap with config, the message should say the flag is deprecated
B) if config is going to override flags, the message should say that the flag overrides config for now but this will change
C) if flags are going to override config, I don't think we should log a message when any such flag is used...

Personally I lean toward C and so don't want to log. I realize that there's legacy users of this flag that may not know the alternative, but equally there are future users of this flag that will be not-helped by such a log message.

clang-tools-extra/clangd/ConfigCompile.cpp

267

Haha, yes... you could always write ./Ancestors then, I suppose.

(Obviously we can work around this with a more complicated structure... but I'm not sure it's a good trade)

278

Right... the reason I didn't make the change in this patch is that it affected MountPoint of indexes, and there were tests of that, and code using starts_with in a way that suggested it might be important.

So we should clean this up somehow, but I didn't want to bite it off here.

clang-tools-extra/clangd/GlobalCompilationDatabase.cpp

617

Nice catch, I guess I didn't have enough levels in my tests...

651

This seems less clear to me, the branch is fairly predictable, and this function is pretty cold - I'd rather keep init as is for readability.

[kadircet]

thanks, lgtm!

clang-tools-extra/clangd/ConfigCompile.cpp

278

SG. As a note to future self, I don't remember the tests but in theory the startswith should be fine with and without the trailing slash.

clang-tools-extra/clangd/ClangdLSPServer.cpp

void ClangdLSPServer::onInitialize(const InitializeParams &Params,
   if (Server)
     return Reply(llvm::make_error<LSPError>("server already initialized",
                                             ErrorCode::InvalidRequest));
   if (const auto &Dir = Params.initializationOptions.compilationDatabasePath)
     Opts.CompileCommandsDir = Dir;
   if (Opts.UseDirBasedCDB) {
     DirectoryBasedGlobalCompilationDatabase::Options CDBOpts(TFS);
     CDBOpts.CompileCommandsDir = Opts.CompileCommandsDir;
+    CDBOpts.ContextProvider = Opts.ContextProvider;
     BaseCDB =
         std::make_unique<DirectoryBasedGlobalCompilationDatabase>(CDBOpts);
     BaseCDB = getQueryDriverDatabase(llvm::makeArrayRef(Opts.QueryDriverGlobs),
                                      std::move(BaseCDB));
   }
   auto Mangler = CommandMangler::detect();
   if (Opts.ResourceDir)
     Mangler.ResourceDir = *Opts.ResourceDir;

clang-tools-extra/clangd/Config.h

struct Config {
   static clangd::Key<Config> Key;
 
   Config() = default;
   Config(const Config &) = delete;
   Config &operator=(const Config &) = delete;
   Config(Config &&) = default;
   Config &operator=(Config &&) = default;
 
+  struct CDBSearchSpec {
+    enum { Ancestors, FixedDir, NoCDBSearch } Policy = Ancestors;
+    // Absolute, native slashes, no trailing slash.
+    llvm::Optional<std::string> FixedCDBPath;
+  };
+
   /// Controls how the compile command for the current file is determined.
   struct {
-    // Edits to apply to the compile command, in sequence.
+    /// Edits to apply to the compile command, in sequence.
     std::vector<llvm::unique_function<void(std::vector<std::string> &) const>>
         Edits;
+    /// Where to search for compilation databases for this file's flags.
+    CDBSearchSpec CDBSearch = {CDBSearchSpec::Ancestors, llvm::None};
   } CompileFlags;
 
   enum class BackgroundPolicy { Build, Skip };
   /// Describes an external index configuration.
   struct ExternalIndexSpec {
     enum { File, Server } Kind;
     /// This is one of:
     /// - Address of a clangd-index-server, in the form of "ip:port".

clang-tools-extra/clangd/ConfigCompile.cpp

if (!F.Add.empty()) {
       for (auto &A : F.Add)
         Add.push_back(std::move(*A));
       Out.Apply.push_back([Add(std::move(Add))](const Params &, Config &C) {
         C.CompileFlags.Edits.push_back([Add](std::vector<std::string> &Args) {
           Args.insert(Args.end(), Add.begin(), Add.end());
         });
       });
     }
+
+    if (F.CompilationDatabase) {
+      llvm::Optional<Config::CDBSearchSpec> Spec;
+      if (**F.CompilationDatabase == "Ancestors") {
+        Spec.emplace();
+        Spec->Policy = Config::CDBSearchSpec::Ancestors;
+      } else if (**F.CompilationDatabase == "None") {
+        Spec.emplace();
+        Spec->Policy = Config::CDBSearchSpec::NoCDBSearch;
+      } else {
+        if (auto Path =
+                makeAbsolute(*F.CompilationDatabase, "CompilationDatabase",
+                             llvm::sys::path::Style::native)) {
+          // Drop trailing slash to put the path in canonical form.
+          // Should makeAbsolute do this?
+          llvm::StringRef Rel = llvm::sys::path::relative_path(*Path);
+          if (!Rel.empty() && llvm::sys::path::is_separator(Rel.back()))
+            Path->pop_back();
+
+          Spec.emplace();
+          Spec->Policy = Config::CDBSearchSpec::FixedDir;
+          Spec->FixedCDBPath = std::move(Path);
+        }
+      }
+      if (Spec)
+        Out.Apply.push_back(
+            [Spec(std::move(*Spec))](const Params &, Config &C) {
+              C.CompileFlags.CDBSearch = Spec;
+            });
+    }
   }
 
   void compile(Fragment::IndexBlock &&F) {
     if (F.Background) {
       if (auto Val = compileEnum<Config::BackgroundPolicy>("Background",
                                                            **F.Background)
                          .map("Build", Config::BackgroundPolicy::Build)
                          .map("Skip", Config::BackgroundPolicy::Skip)

clang-tools-extra/clangd/ConfigFragment.h

struct CompileFlagsBlock {
     ///
     /// Example:
     ///   Command: clang++ --include-directory=/usr/include -DFOO=42 foo.cc
     ///   Remove: [-I, -DFOO=*]
     ///   Result: clang++ foo.cc
     ///
     /// Flags added by the same CompileFlags entry will not be removed.
     std::vector<Located<std::string>> Remove;
+
+    /// Directory to search for compilation database (compile_comands.json etc).
+    /// Valid values are:
+    /// - A single path to a directory (absolute, or relative to the fragment)
+    /// - Ancestors: search all parent directories (the default)
+    /// - None: do not use a compilation database, just default flags.
+    llvm::Optional<Located<std::string>> CompilationDatabase;
   };
   CompileFlagsBlock CompileFlags;
 
   /// Controls how clangd understands code outside the current file.
   /// clangd's indexes provide information about symbols that isn't available
   /// to clang's parser, such as incoming references.
   struct IndexBlock {
     /// Whether files are built in the background to produce a project index.

clang-tools-extra/clangd/ConfigYAML.cpp

void parse(Fragment::CompileFlagsBlock &F, Node &N) {
     Dict.handle("Add", [&](Node &N) {
       if (auto Values = scalarValues(N))
         F.Add = std::move(*Values);
     });
     Dict.handle("Remove", [&](Node &N) {
       if (auto Values = scalarValues(N))
         F.Remove = std::move(*Values);
     });
+    Dict.handle("CompilationDatabase", [&](Node &N) {
+      F.CompilationDatabase = scalarValue(N, "CompilationDatabase");
+    });
     Dict.parse(N);
   }
 
   void parse(Fragment::StyleBlock &F, Node &N) {
     DictParser Dict("Style", this);
     Dict.handle("FullyQualifiedNamespaces", [&](Node &N) {
       if (auto Values = scalarValues(N))
         F.FullyQualifiedNamespaces = std::move(*Values);

clang-tools-extra/clangd/GlobalCompilationDatabase.h

struct Options {
     const ThreadsafeFS &TFS;
     // Frequency to check whether e.g. compile_commands.json has changed.
     std::chrono::steady_clock::duration RevalidateAfter =
         std::chrono::seconds(5);
     // Frequency to check whether e.g. compile_commands.json has been created.
     // (This is more expensive to check frequently, as we check many locations).
     std::chrono::steady_clock::duration RevalidateMissingAfter =
         std::chrono::seconds(30);
+    // Used to provide per-file configuration.
+    std::function<Context(llvm::StringRef)> ContextProvider;
     // Only look for a compilation database in this one fixed directory.
+    // FIXME: fold this into config/context mechanism.
     llvm::Optional<Path> CompileCommandsDir;
   };
 
   DirectoryBasedGlobalCompilationDatabase(const Options &Opts);
   ~DirectoryBasedGlobalCompilationDatabase() override;
 
   /// Scans File's parents looking for compilation databases.
   /// Any extra flags will be added.
   /// Might trigger OnCommandChanged, if CDB wasn't broadcasted yet.
   llvm::Optional<tooling::CompileCommand>
   getCompileCommand(PathRef File) const override;
 
   /// Returns the path to first directory containing a compilation database in
   /// \p File's parents.
   llvm::Optional<ProjectInfo> getProjectInfo(PathRef File) const override;
 
   bool blockUntilIdle(Deadline Timeout) const override;
 
 private:
   Options Opts;
 
   class DirectoryCache;
-  // If there's an explicit CompileCommandsDir, cache of the CDB found there.
-  mutable std::unique_ptr<DirectoryCache> OnlyDirCache;
-
   // Keyed by possibly-case-folded directory path.
   // We can hand out pointers as they're stable and entries are never removed.
-  // Empty if CompileCommandsDir is given (OnlyDirCache is used instead).
   mutable llvm::StringMap<DirectoryCache> DirCaches;
-  // DirCaches access must be locked (unlike OnlyDirCache, which is threadsafe).
   mutable std::mutex DirCachesMutex;
 
   std::vector<DirectoryCache *>
   getDirectoryCaches(llvm::ArrayRef<llvm::StringRef> Dirs) const;
 
   struct CDBLookupRequest {
     PathRef FileName;
     // Whether this lookup should trigger discovery of the CDB found.

clang-tools-extra/clangd/GlobalCompilationDatabase.cpp

 //===--- GlobalCompilationDatabase.cpp ---------------------------*- C++-*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "GlobalCompilationDatabase.h"
+#include "Config.h"
 #include "FS.h"
 #include "SourceCode.h"
 #include "support/Logger.h"
 #include "support/Path.h"
 #include "support/Threading.h"
 #include "support/ThreadsafeFS.h"
 #include "clang/Frontend/CompilerInvocation.h"
 #include "clang/Tooling/ArgumentsAdjusters.h"
 #include "clang/Tooling/CompilationDatabase.h"
 #include "clang/Tooling/CompilationDatabasePluginRegistry.h"
 #include "clang/Tooling/JSONCompilationDatabase.h"
 #include "llvm/ADT/None.h"
 #include "llvm/ADT/Optional.h"
+#include "llvm/ADT/PointerIntPair.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/ScopeExit.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/StringMap.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/FileUtilities.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/Program.h"
bool DirectoryBasedGlobalCompilationDatabase::DirectoryCache::load(
   }
   dlog("No compilation database at {0}", Path);
   return true;
 }
 
 DirectoryBasedGlobalCompilationDatabase::
     DirectoryBasedGlobalCompilationDatabase(const Options &Opts)
     : Opts(Opts), Broadcaster(std::make_unique<BroadcastThread>(*this)) {
-  if (Opts.CompileCommandsDir)
-    OnlyDirCache = std::make_unique<DirectoryCache>(*Opts.CompileCommandsDir);
+  if (!this->Opts.ContextProvider)
+    this->Opts.ContextProvider = [](llvm::StringRef) {
+      return Context::current().clone();
+    };
 }
 
 DirectoryBasedGlobalCompilationDatabase::
     ~DirectoryBasedGlobalCompilationDatabase() = default;
 
 llvm::Optional<tooling::CompileCommand>
 DirectoryBasedGlobalCompilationDatabase::getCompileCommand(PathRef File) const {
   CDBLookupRequest Req;

 static std::string maybeCaseFoldPath(PathRef Path) {
 #if defined(_WIN32) || defined(__APPLE__)
   return Path.lower();
 #else
   return std::string(Path);
 #endif
 }
 
-static bool pathEqual(PathRef A, PathRef B) {
-#if defined(_WIN32) || defined(__APPLE__)
-  return A.equals_lower(B);
-#else
-  return A == B;
-#endif
-}
-
 std::vector<DirectoryBasedGlobalCompilationDatabase::DirectoryCache *>
 DirectoryBasedGlobalCompilationDatabase::getDirectoryCaches(
     llvm::ArrayRef<llvm::StringRef> Dirs) const {
   std::vector<std::string> FoldedDirs;
   FoldedDirs.reserve(Dirs.size());
   for (const auto &Dir : Dirs) {
 #ifndef NDEBUG
     if (!llvm::sys::path::is_absolute(Dir))

 }
 
 llvm::Optional<DirectoryBasedGlobalCompilationDatabase::CDBLookupResult>
 DirectoryBasedGlobalCompilationDatabase::lookupCDB(
     CDBLookupRequest Request) const {
   assert(llvm::sys::path::is_absolute(Request.FileName) &&
          "path must be absolute");
 
-  bool ShouldBroadcast = false;
-  DirectoryCache *DirCache = nullptr;
-  std::shared_ptr<const tooling::CompilationDatabase> CDB = nullptr;
-  if (OnlyDirCache) {
-    DirCache = OnlyDirCache.get();
-    ShouldBroadcast = Request.ShouldBroadcast;
-    CDB = DirCache->get(Opts.TFS, ShouldBroadcast, Request.FreshTime,
-                        Request.FreshTimeMissing);
-  } else {
-    // Traverse the canonical version to prevent false positives. i.e.:
-    // src/build/../a.cc can detect a CDB in /src/build if not canonicalized.
-    std::string CanonicalPath = removeDots(Request.FileName);
-    std::vector<llvm::StringRef> SearchDirs;
-    actOnAllParentDirectories(CanonicalPath, [&](PathRef Path) {
-      SearchDirs.push_back(Path);
+  std::string Storage;
+  if (Opts.CompileCommandsDir) // FIXME: unify this case with config.
+    SearchDirs = {Opts.CompileCommandsDir.getValue()};
+  else {
+    WithContext WithProvidedContext(Opts.ContextProvider(Request.FileName));
+    const auto &Spec = Config::current().CompileFlags.CDBSearch;
+    switch (Spec.Policy) {
+    case Config::CDBSearchSpec::NoCDBSearch:
+      return llvm::None;
+    case Config::CDBSearchSpec::FixedDir:
+      Storage = Spec.FixedCDBPath.getValue();
+      SearchDirs = {Storage};
+      break;
+    case Config::CDBSearchSpec::Ancestors:
+      // Traverse the canonical version to prevent false positives. i.e.:
+      // src/build/../a.cc can detect a CDB in /src/build if not
+      // canonicalized.
+      Storage = removeDots(Request.FileName);
+      actOnAllParentDirectories(Storage, [&](llvm::StringRef Dir) {
+        SearchDirs.push_back(Dir);
         return false;
       });
-    for (DirectoryCache *Candidate : getDirectoryCaches(SearchDirs)) {
-      bool CandidateShouldBroadcast = Request.ShouldBroadcast;
-      if ((CDB = Candidate->get(Opts.TFS, CandidateShouldBroadcast,
-                                Request.FreshTime, Request.FreshTimeMissing))) {
-        DirCache = Candidate;
-        ShouldBroadcast = CandidateShouldBroadcast;
-        break;
-      }
-    }
-  }
+    }
+  }
+
+  std::shared_ptr<const tooling::CompilationDatabase> CDB = nullptr;
+  bool ShouldBroadcast = false;
+  DirectoryCache *DirCache = nullptr;
 
   if (!CDB)
     return llvm::None;
 
   CDBLookupResult Result;
   Result.CDB = std::move(CDB);
   Result.PI.SourceRoot = DirCache->Path;
 
public:
 
   ~BroadcastThread() {
     ShouldStop.store(true, std::memory_order_release);
     CV.notify_all();
     Thread.join();
   }
 };
 
-void DirectoryBasedGlobalCompilationDatabase::BroadcastThread::process(
-    const CDBLookupResult &T) {
-  vlog("Broadcasting compilation database from {0}", T.PI.SourceRoot);
+// The DirBasedCDB associates each file with a specific CDB.
+// When a CDB is discovered, it may claim to describe files that we associate
+// with a different CDB. We do not want to broadcast discovery of these, and
+// trigger background indexing of them.
+//
+// We must filter the list, and check whether they are associated with this CDB.
+// This class attempts to do so efficiently.
+//
+// Roughly, it:
+//  - loads the config for each file, and determines the relevant search path
+//  - gathers all directories that are part of any search path
+//  - (lazily) checks for a CDB in each such directory at most once
+//  - walks the search path for each file and determines whether to include it.
+class DirectoryBasedGlobalCompilationDatabase::BroadcastThread::Filter {
+  llvm::StringRef ThisDir;
+  DirectoryBasedGlobalCompilationDatabase &Parent;
 
-  std::vector<std::string> AllFiles = T.CDB->getAllFiles();
-  // We assume CDB in CompileCommandsDir owns all of its entries, since we don't
-  // perform any search in parent paths whenever it is set.
-  if (Parent.OnlyDirCache) {
-    assert(Parent.OnlyDirCache->Path == T.PI.SourceRoot &&
-           "Trying to broadcast a CDB outside of CompileCommandsDir!");
-    Parent.OnCommandChanged.broadcast(std::move(AllFiles));
-    return;
-  }
+  // Keep track of all directories we might check for CDBs.
+  struct DirInfo {
+    DirectoryCache *Cache = nullptr;
+    enum { Unknown, Missing, TargetCDB, OtherCDB } State = Unknown;
+    DirInfo *Parent = nullptr;
+  };
+  llvm::StringMap<DirInfo> Dirs;
 
-  // Uniquify all parent directories of all files.
-  llvm::StringMap<bool> DirectoryHasCDB;
-  std::vector<llvm::StringRef> FileAncestors;
-  for (llvm::StringRef File : AllFiles) {
-    actOnAllParentDirectories(File, [&](PathRef Path) {
-      auto It = DirectoryHasCDB.try_emplace(Path);
-      // Already seen this path, and all of its parents.
-      if (!It.second)
-        return true;
+  // A search path starts at a directory, and either includes ancestors or not.
+  using SearchPath = llvm::PointerIntPair<DirInfo *, 1>;
 
-      FileAncestors.push_back(It.first->getKey());
-      return pathEqual(Path, T.PI.SourceRoot);
+  // Add all ancestor directories of FilePath to the tracked set.
+  // Returns the immediate parent of the file.
+  DirInfo *addParents(llvm::StringRef FilePath) {
+    DirInfo *Leaf = nullptr;
+    DirInfo *Child = nullptr;
+    actOnAllParentDirectories(FilePath, [&](llvm::StringRef Dir) {
+      auto &Info = Dirs[Dir];
+      // If this is the first iteration, then this node is the overall result.
+      if (!Leaf)
+        Leaf = &Info;
+      // Fill in the parent link from the previous iteration to this parent.
+      if (Child)
+        Child->Parent = &Info;
+      // Keep walking, whether we inserted or not, if parent link is missing.
+      // (If it's present, parent links must be present up to the root, so stop)
+      Child = &Info;
+      return Info.Parent != nullptr;
     });
+    return Leaf;
   }
-  // Work out which ones have CDBs in them.
-  // Given that we know that CDBs have been moved/generated, don't trust caches.
-  // (This should be rare, so it's OK to add a little latency).
-  constexpr auto IgnoreCache = std::chrono::steady_clock::time_point::max();
-  auto DirectoryCaches = Parent.getDirectoryCaches(FileAncestors);
-  assert(DirectoryCaches.size() == FileAncestors.size());
-  for (unsigned I = 0; I < DirectoryCaches.size(); ++I) {
-    bool ShouldBroadcast = false;
-    if (ShouldStop.load(std::memory_order_acquire)) {
-      log("Giving up on broadcasting CDB, as we're shutting down");
+
+  // Populates DirInfo::Cache (and State, if it is TargetCDB).
+  void grabCaches() {
+    // Fast path out if there were no files, or CDB loading is off.
+    if (Dirs.empty())
       return;
+
+    std::vector<llvm::StringRef> DirKeys;
+    std::vector<DirInfo *> DirValues;
+    DirKeys.reserve(Dirs.size() + 1);
+    DirValues.reserve(Dirs.size());
+    for (auto &E : Dirs) {
+      DirKeys.push_back(E.first());
+      DirValues.push_back(&E.second);
+    }
+
+    // Also look up the cache entry for the CDB we're broadcasting.
+    // Comparing DirectoryCache pointers is more robust than checking string
+    // equality, e.g. reuses the case-sensitivity handling.
+    DirKeys.push_back(ThisDir);
+    auto DirCaches = Parent.getDirectoryCaches(DirKeys);
+    const DirectoryCache *ThisCache = DirCaches.back();
+    DirCaches.pop_back();
+    DirKeys.pop_back();
+
+    for (unsigned I = 0; I < DirKeys.size(); ++I) {
+      DirValues[I]->Cache = DirCaches[I];
+      if (DirCaches[I] == ThisCache)
+        DirValues[I]->State = DirInfo::TargetCDB;
     }
-    if (DirectoryCaches[I]->get(Parent.Opts.TFS, ShouldBroadcast,
+  }
+
+  // Should we include a file from this search path?
+  bool shouldInclude(SearchPath P) {
+    DirInfo *Info = P.getPointer();
+    if (!Info)
+      return false;
+    if (Info->State == DirInfo::Unknown) {
+      assert(Info->Cache && "grabCaches() should have filled this");
+      // Given that we know that CDBs have been moved/generated, don't trust
+      // caches. (This should be rare, so it's OK to add a little latency).
+      constexpr auto IgnoreCache = std::chrono::steady_clock::time_point::max();
+      // Don't broadcast CDBs discovered while broadcasting!
+      bool ShouldBroadcast = false;
+      bool Exists =
+          nullptr != Info->Cache->get(Parent.Opts.TFS, ShouldBroadcast,
                                       /*FreshTime=*/IgnoreCache,
-                                /*FreshTimeMissing=*/IgnoreCache))
-      DirectoryHasCDB.find(FileAncestors[I])->setValue(true);
+                                      /*FreshTimeMissing=*/IgnoreCache);
+      Info->State = Exists ? DirInfo::OtherCDB : DirInfo::Missing;
+    }
+    // If we have a CDB, include the file if it's the target CDB only.
+    if (Info->State != DirInfo::Missing)
+      return Info->State == DirInfo::TargetCDB;
+    // If we have no CDB and no relevant parent, don't include the file.
+    if (!P.getInt() || !Info->Parent)
+      return false;
+    // Walk up to the next parent.
+    return shouldInclude(SearchPath(Info->Parent, 1));
   }
 
-  std::vector<std::string> GovernedFiles;
-  for (llvm::StringRef File : AllFiles) {
-    // A file is governed by this CDB if lookup for the file would find it.
-    // Independent of whether it has an entry for that file or not.
-    actOnAllParentDirectories(File, [&](PathRef Path) {
-      if (DirectoryHasCDB.lookup(Path)) {
-        if (pathEqual(Path, T.PI.SourceRoot))
-          // Make sure listeners always get a canonical path for the file.
-          GovernedFiles.push_back(removeDots(File));
-        // Stop as soon as we hit a CDB.
+public:
+  Filter(llvm::StringRef ThisDir,
+         DirectoryBasedGlobalCompilationDatabase &Parent)
+      : ThisDir(ThisDir), Parent(Parent) {}
+
+  std::vector<std::string> filter(std::vector<std::string> AllFiles,
+                                  std::atomic<bool> &ShouldStop) {
+    std::vector<std::string> Filtered;
+    // Allow for clean early-exit of the slow parts.
+    auto ExitEarly = [&] {
+      if (ShouldStop.load(std::memory_order_acquire)) {
+        log("Giving up on broadcasting CDB, as we're shutting down");
+        Filtered.clear();
         return true;
       }
       return false;
-    });
+    };
+    // Compute search path for each file.
+    std::vector<SearchPath> SearchPaths(AllFiles.size());
+    for (unsigned I = 0; I < AllFiles.size(); ++I) {
+      if (Parent.Opts.CompileCommandsDir) { // FIXME: unify with config
+        SearchPaths[I].setPointer(
+            &Dirs[Parent.Opts.CompileCommandsDir.getValue()]);
+        continue;
+      }
+      if (ExitEarly()) // loading config may be slow
+        return Filtered;
+      WithContext WithProvidedContent(Parent.Opts.ContextProvider(AllFiles[I]));
+      const Config::CDBSearchSpec &Spec =
+          Config::current().CompileFlags.CDBSearch;
+      switch (Spec.Policy) {
+      case Config::CDBSearchSpec::NoCDBSearch:
+        break;
+      case Config::CDBSearchSpec::Ancestors:
+        SearchPaths[I].setInt(/*Recursive=*/1);
+        SearchPaths[I].setPointer(addParents(AllFiles[I]));
+        break;
+      case Config::CDBSearchSpec::FixedDir:
+        SearchPaths[I].setPointer(&Dirs[Spec.FixedCDBPath.getValue()]);
+        break;
       }
+    }
+    // Get the CDB cache for each dir on the search path, but don't load yet.
+    grabCaches();
+    // Now work out which files we want to keep, loading CDBs where needed.
+    for (unsigned I = 0; I < AllFiles.size(); ++I) {
+      if (ExitEarly()) // loading CDBs may be slow
+        return Filtered;
+      if (shouldInclude(SearchPaths[I]))
+        Filtered.push_back(std::move(AllFiles[I]));
+    }
+    return Filtered;
+  }
+};
 
+void DirectoryBasedGlobalCompilationDatabase::BroadcastThread::process(
+    const CDBLookupResult &T) {
+  vlog("Broadcasting compilation database from {0}", T.PI.SourceRoot);
+  std::vector<std::string> GovernedFiles =
+      Filter(T.PI.SourceRoot, Parent).filter(T.CDB->getAllFiles(), ShouldStop);
+  if (!GovernedFiles.empty())
     Parent.OnCommandChanged.broadcast(std::move(GovernedFiles));
 }
 
 void DirectoryBasedGlobalCompilationDatabase::broadcastCDB(
     CDBLookupResult Result) const {
   assert(Result.CDB && "Trying to broadcast an invalid CDB!");
   Broadcaster->enqueue(Result);
 }
 

clang-tools-extra/clangd/unittests/ClangdLSPServerTests.cpp

auto Calls = Client
                          llvm::json::Object{{"item", FirstItem}})
                    .takeValue();
   auto FirstCall = *(*Calls.getAsArray())[0].getAsObject();
   EXPECT_EQ(FirstCall["fromRanges"], llvm::json::Value{Code.range()});
   auto From = *FirstCall["from"].getAsObject();
   EXPECT_EQ(From["name"], "caller1");
 }
 
+TEST_F(LSPTest, CDBConfigIntegration) {
+  auto CfgProvider =
+      config::Provider::fromAncestorRelativeYAMLFiles(".clangd", FS);
+  Opts.ConfigProvider = CfgProvider.get();
+
+  // Map bar.cpp to a different compilation database which defines FOO->BAR.
+  FS.Files[".clangd"] = R"yaml(
+If:
+  PathMatch: bar.cpp
+CompileFlags:
+  CompilationDatabase: bar
+)yaml";
+  FS.Files["bar/compile_flags.txt"] = "-DFOO=BAR";
+
+  auto &Client = start();
+  // foo.cpp gets parsed as normal.
+  Client.didOpen("foo.cpp", "int x = FOO;");
+  EXPECT_THAT(Client.diagnostics("foo.cpp"),
+              llvm::ValueIs(testing::ElementsAre(
+                  DiagMessage("Use of undeclared identifier 'FOO'"))));
+  // bar.cpp shows the configured compile command.
+  Client.didOpen("bar.cpp", "int x = FOO;");
+  EXPECT_THAT(Client.diagnostics("bar.cpp"),
+              llvm::ValueIs(testing::ElementsAre(
+                  DiagMessage("Use of undeclared identifier 'BAR'"))));
+}
+
 } // namespace
 } // namespace clangd
 } // namespace clang

clang-tools-extra/clangd/unittests/ConfigCompileTests.cpp

TEST_F(ConfigCompileTests, CompileCommands) {
   std::vector<std::string> Argv = {"clang", "-I", "bar/", "a.cc"};
   EXPECT_TRUE(compileAndApply());
   EXPECT_THAT(Conf.CompileFlags.Edits, SizeIs(2));
   for (auto &Edit : Conf.CompileFlags.Edits)
     Edit(Argv);
   EXPECT_THAT(Argv, ElementsAre("clang", "a.cc", "-foo"));
 }
 
+TEST_F(ConfigCompileTests, CompilationDatabase) {
+  Frag.CompileFlags.CompilationDatabase.emplace("None");
+  EXPECT_TRUE(compileAndApply());
+  EXPECT_EQ(Conf.CompileFlags.CDBSearch.Policy,
+            Config::CDBSearchSpec::NoCDBSearch);
+
+  Frag.CompileFlags.CompilationDatabase.emplace("Ancestors");
+  EXPECT_TRUE(compileAndApply());
+  EXPECT_EQ(Conf.CompileFlags.CDBSearch.Policy,
+            Config::CDBSearchSpec::Ancestors);
+
+  // Relative path not allowed without directory set.
+  Frag.CompileFlags.CompilationDatabase.emplace("Something");
+  EXPECT_TRUE(compileAndApply());
+  EXPECT_EQ(Conf.CompileFlags.CDBSearch.Policy,
+            Config::CDBSearchSpec::Ancestors)
+      << "default value";
+  EXPECT_THAT(Diags.Diagnostics,
+              ElementsAre(DiagMessage(
+                  "CompilationDatabase must be an absolute path, because this "
+                  "fragment is not associated with any directory.")));
+
+  // Relative path allowed if directory is set.
+  Frag.Source.Directory = testRoot();
+  EXPECT_TRUE(compileAndApply());
+  EXPECT_EQ(Conf.CompileFlags.CDBSearch.Policy,
+            Config::CDBSearchSpec::FixedDir);
+  EXPECT_EQ(Conf.CompileFlags.CDBSearch.FixedCDBPath, testPath("Something"));
+  EXPECT_THAT(Diags.Diagnostics, IsEmpty());
+
+  // Absolute path allowed.
+  Frag.Source.Directory.clear();
+  Frag.CompileFlags.CompilationDatabase.emplace(testPath("Something2"));
+  EXPECT_TRUE(compileAndApply());
+  EXPECT_EQ(Conf.CompileFlags.CDBSearch.Policy,
+            Config::CDBSearchSpec::FixedDir);
+  EXPECT_EQ(Conf.CompileFlags.CDBSearch.FixedCDBPath, testPath("Something2"));
+  EXPECT_THAT(Diags.Diagnostics, IsEmpty());
+}
+
 TEST_F(ConfigCompileTests, Index) {
   Frag.Index.Background.emplace("Skip");
   EXPECT_TRUE(compileAndApply());
   EXPECT_EQ(Conf.Index.Background, Config::BackgroundPolicy::Skip);
 
   Frag = {};
   Frag.Index.Background.emplace("Foo");
   EXPECT_TRUE(compileAndApply());

clang-tools-extra/clangd/unittests/GlobalCompilationDatabaseTests.cpp

 //===-- GlobalCompilationDatabaseTests.cpp ----------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "GlobalCompilationDatabase.h"
 
+#include "Config.h"
 #include "Matchers.h"
 #include "TestFS.h"
 #include "support/Path.h"
 #include "support/ThreadsafeFS.h"
 #include "clang/Tooling/CompilationDatabase.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallString.h"
const char *const CDBInner =
         }
       ]
       )cdb";
   MockFS FS;
   FS.Files[testPath("compile_commands.json")] =
       llvm::formatv(CDBOuter, llvm::sys::path::convert_to_slash(testRoot()));
   FS.Files[testPath("build/compile_commands.json")] =
       llvm::formatv(CDBInner, llvm::sys::path::convert_to_slash(testRoot()));
+  FS.Files[testPath("foo/compile_flags.txt")] = "-DFOO";
 
   // Note that gen2.cc goes missing with our following model, not sure this
   // happens in practice though.
   {
+    SCOPED_TRACE("Default ancestor scanning");
     DirectoryBasedGlobalCompilationDatabase DB(FS);
     std::vector<std::string> DiscoveredFiles;
     auto Sub =
         DB.watch([&DiscoveredFiles](const std::vector<std::string> Changes) {
           DiscoveredFiles = Changes;
         });
 
     DB.getCompileCommand(testPath("build/../a.cc"));
     ASSERT_TRUE(DB.blockUntilIdle(timeoutSeconds(10)));
     EXPECT_THAT(DiscoveredFiles, UnorderedElementsAre(AllOf(
                                      EndsWith("a.cc"), Not(HasSubstr("..")))));
     DiscoveredFiles.clear();
 
     DB.getCompileCommand(testPath("build/gen.cc"));
     ASSERT_TRUE(DB.blockUntilIdle(timeoutSeconds(10)));
     EXPECT_THAT(DiscoveredFiles, UnorderedElementsAre(EndsWith("gen.cc")));
   }
 
-  // With a custom compile commands dir.
   {
+    SCOPED_TRACE("With config");
+    DirectoryBasedGlobalCompilationDatabase::Options Opts(FS);
+    Opts.ContextProvider = [&](llvm::StringRef Path) {
+      Config Cfg;
+      if (Path.endswith("a.cc")) {
+        // a.cc uses another directory's CDB, so it won't be discovered.
+        Cfg.CompileFlags.CDBSearch.Policy = Config::CDBSearchSpec::FixedDir;
+        Cfg.CompileFlags.CDBSearch.FixedCDBPath = testPath("foo");
+      } else if (Path.endswith("gen.cc")) {
+        // gen.cc has CDB search disabled, so it won't be discovered.
+        Cfg.CompileFlags.CDBSearch.Policy = Config::CDBSearchSpec::NoCDBSearch;
+      } else if (Path.endswith("gen2.cc")) {
+        // gen2.cc explicitly lists this directory, so it will be discovered.
+        Cfg.CompileFlags.CDBSearch.Policy = Config::CDBSearchSpec::FixedDir;
+        Cfg.CompileFlags.CDBSearch.FixedCDBPath = testRoot();
+      }
+      return Context::current().derive(Config::Key, std::move(Cfg));
+    };
+    DirectoryBasedGlobalCompilationDatabase DB(Opts);
+    std::vector<std::string> DiscoveredFiles;
+    auto Sub =
+        DB.watch([&DiscoveredFiles](const std::vector<std::string> Changes) {
+          DiscoveredFiles = Changes;
+        });
+
+    // Does not use the root CDB, so no broadcast.
+    auto Cmd = DB.getCompileCommand(testPath("build/../a.cc"));
+    ASSERT_TRUE(Cmd.hasValue());
+    EXPECT_THAT(Cmd->CommandLine, Contains("-DFOO")) << "a.cc uses foo/ CDB";
+    ASSERT_TRUE(DB.blockUntilIdle(timeoutSeconds(10)));
+    EXPECT_THAT(DiscoveredFiles, IsEmpty()) << "Root CDB not discovered yet";
+
+    // No special config for b.cc, so we trigger broadcast of the root CDB.
+    DB.getCompileCommand(testPath("b.cc"));
+    ASSERT_TRUE(DB.blockUntilIdle(timeoutSeconds(10)));
+    EXPECT_THAT(DiscoveredFiles, ElementsAre(testPath("build/gen2.cc")));
+    DiscoveredFiles.clear();
+
+    // No CDB search so no discovery/broadcast triggered for build/ CDB.
+    DB.getCompileCommand(testPath("build/gen.cc"));
+    ASSERT_TRUE(DB.blockUntilIdle(timeoutSeconds(10)));
+    EXPECT_THAT(DiscoveredFiles, IsEmpty());
+  }
+
+  {
+    SCOPED_TRACE("With custom compile commands dir");
     DirectoryBasedGlobalCompilationDatabase::Options Opts(FS);
     Opts.CompileCommandsDir = testRoot();
     DirectoryBasedGlobalCompilationDatabase DB(Opts);
     std::vector<std::string> DiscoveredFiles;
     auto Sub =
         DB.watch([&DiscoveredFiles](const std::vector<std::string> Changes) {
           DiscoveredFiles = Changes;
         });
TEST(GlobalCompilationDatabaseTest, CompileFlagsDirectory) {
   DirectoryBasedGlobalCompilationDatabase CDB(FS);
   auto Commands = CDB.getCompileCommand(testPath("x/y.cpp"));
   ASSERT_TRUE(Commands.hasValue());
   EXPECT_THAT(Commands.getValue().CommandLine, Contains("-DFOO"));
   // Make sure we pick the right working directory.
   EXPECT_EQ(testPath("x"), Commands.getValue().Directory);
 }
 
+MATCHER_P(hasArg, Flag, "") {
+  if (!arg.hasValue()) {
+    *result_listener << "command is null";
+    return false;
+  }
+  if (!llvm::is_contained(arg->CommandLine, Flag)) {
+    *result_listener << "flags are " << llvm::join(arg->CommandLine, " ");
+    return false;
+  }
+  return true;
+}
+
+TEST(GlobalCompilationDatabaseTest, Config) {
+  MockFS FS;
+  FS.Files[testPath("x/compile_flags.txt")] = "-DX";
+  FS.Files[testPath("x/y/z/compile_flags.txt")] = "-DZ";
+
+  Config::CDBSearchSpec Spec;
+  DirectoryBasedGlobalCompilationDatabase::Options Opts(FS);
+  Opts.ContextProvider = [&](llvm::StringRef Path) {
+    Config C;
+    C.CompileFlags.CDBSearch = Spec;
+    return Context::current().derive(Config::Key, std::move(C));
+  };
+  DirectoryBasedGlobalCompilationDatabase CDB(Opts);
+
+  // Default ancestor behavior.
+  EXPECT_FALSE(CDB.getCompileCommand(testPath("foo.cc")));
+  EXPECT_THAT(CDB.getCompileCommand(testPath("x/foo.cc")), hasArg("-DX"));
+  EXPECT_THAT(CDB.getCompileCommand(testPath("x/y/foo.cc")), hasArg("-DX"));
+  EXPECT_THAT(CDB.getCompileCommand(testPath("x/y/z/foo.cc")), hasArg("-DZ"));
+
+  Spec.Policy = Config::CDBSearchSpec::NoCDBSearch;
+  EXPECT_FALSE(CDB.getCompileCommand(testPath("foo.cc")));
+  EXPECT_FALSE(CDB.getCompileCommand(testPath("x/foo.cc")));
+  EXPECT_FALSE(CDB.getCompileCommand(testPath("x/y/foo.cc")));
+  EXPECT_FALSE(CDB.getCompileCommand(testPath("x/y/z/foo.cc")));
+
+  Spec.Policy = Config::CDBSearchSpec::FixedDir;
+  Spec.FixedCDBPath = testPath("w"); // doesn't exist
+  EXPECT_FALSE(CDB.getCompileCommand(testPath("foo.cc")));
+  EXPECT_FALSE(CDB.getCompileCommand(testPath("x/foo.cc")));
+  EXPECT_FALSE(CDB.getCompileCommand(testPath("x/y/foo.cc")));
+  EXPECT_FALSE(CDB.getCompileCommand(testPath("x/y/z/foo.cc")));
+
+  Spec.FixedCDBPath = testPath("x/y/z");
+  EXPECT_THAT(CDB.getCompileCommand(testPath("foo.cc")), hasArg("-DZ"));
+  EXPECT_THAT(CDB.getCompileCommand(testPath("x/foo.cc")), hasArg("-DZ"));
+  EXPECT_THAT(CDB.getCompileCommand(testPath("x/y/foo.cc")), hasArg("-DZ"));
+  EXPECT_THAT(CDB.getCompileCommand(testPath("x/y/z/foo.cc")), hasArg("-DZ"));
+}
+
 TEST(GlobalCompilationDatabaseTest, NonCanonicalFilenames) {
   OverlayCDB DB(nullptr);
   std::vector<std::string> DiscoveredFiles;
   auto Sub =
       DB.watch([&DiscoveredFiles](const std::vector<std::string> Changes) {
         DiscoveredFiles = Changes;
       });