[clangd] Config: compute config in TUScheduler and BackgroundIndex

[sammccall]

Summary

ClangdServer owns the question of exactly which config to create, but
TUScheduler/BackgroundIndex control threads and so decide at which point
to inject it.

[kadircet]

LG, mostly nits apart from a question about moving the context generation logic into a different place.

clang-tools-extra/clangd/ClangdServer.cpp

748

why not make this a free function in ConfigProvider.h ?

that way we could just keep passing ConfigProvider around rather than a derived lambda.
It makes testing more cumbersome, but enables us to move the logic out of ClangdServer

756

also provide posix style instead of defaulted native

761

why not elog

clang-tools-extra/clangd/ClangdServer.h

340

nit: = nullptr;

clang-tools-extra/clangd/index/Background.h

142

doesn't need to be in this patch, bu I think it is time we have an opts struct in here.

clang-tools-extra/clangd/unittests/TUSchedulerTests.cpp

879–880

nit: maybe extract "somepath" to a constant

[sammccall]

address comments

clang-tools-extra/clangd/ClangdServer.cpp

748

Oh, I thought hiding this logic in ClangdServer was a feature!

For instance, if we want to report configuration errors as LSP diagnostics or as notifications, we need to have access to the ClangdServer to do that.

clang-tools-extra/clangd/index/Background.h

142

Yeah, makes sense. (But indeed would rather not do it here...)

[sammccall]

fix -Wreorder

[kadircet]

clang-tools-extra/clangd/ClangdServer.cpp

748

hmm, I thought we would rather make DiagnosticHandler an input parameter in such a scenario, (we even have the alias config::DiagnosticCallback 😅) with a nullptr just logging the errors and warnings (as in current implementation).

I don't see any other interaction but surfacing diagnostics, and I believe it would look nicer if we provided a callback for diags when creating context with configs. But this one also gets the job done, so up to you.

761

this is still logging though :D (not elog)

[kadircet]

thanks, lgtm!

[sammccall]

clang-tools-extra/clangd/ClangdServer.cpp

748

If i'm understanding what you mean (pass a callback in clangdserver::options?) I don't think it's the right level of abstraction. E.g. for diagnostics, we do have a lifecycle and data model around diagnostics that clangdserver is responsible for producing, and delivering raw diagnostics one at a time doesn't seem enough.
Similarly if we want warning/error notifications that seems like a higher level thing to add to clangdserver::callbacks.
(If it's the ClangdServer builder's responsibility to handle the errors, then the ConfigProvider interface isn't quite the right one in ClangdServer::Options - why expose the errors to clangdserver if they're just going to be passed back again?)

Anyway, we probably can't know for sure until we come to do richer error reporting, which isn't a top priority at the moment. This is my best guess for where we should handle errors (or at least translate them) but I don't think it's terribly hard to move later. So I'll leave as-is for now, to unblock the more critical parts of this patch.

clang-tools-extra/clangd/ClangdServer.h

 //===--- ClangdServer.h - Main clangd server code ----------------*- C++-*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_CLANGDSERVER_H
 #define LLVM_CLANG_TOOLS_EXTRA_CLANGD_CLANGDSERVER_H
 
 #include "../clang-tidy/ClangTidyOptions.h"
 #include "CodeComplete.h"
+#include "ConfigProvider.h"
 #include "GlobalCompilationDatabase.h"
 #include "Hover.h"
 #include "Protocol.h"
 #include "SemanticHighlighting.h"
 #include "TUScheduler.h"
 #include "XRefs.h"
 #include "index/Background.h"
 #include "index/FileIndex.h"
struct Options {
     bool HeavyweightDynamicSymbolIndex = true;
     /// If true, ClangdServer automatically indexes files in the current project
     /// on background threads. The index is stored in the project root.
     bool BackgroundIndex = false;
 
     /// If set, use this index to augment code completion results.
     SymbolIndex *StaticIndex = nullptr;
 
+    /// If set, queried to obtain the configuration to handle each request.
+    config::Provider *ConfigProvider = nullptr;
+
     /// If set, enable clang-tidy in clangd and use to it get clang-tidy
     /// configurations for a particular file.
     /// Clangd supports only a small subset of ClangTidyOptions, these options
     /// (Checks, CheckOptions) are about which clang-tidy checks will be
     /// enabled.
     ClangTidyOptionsBuilder GetClangTidyOptions;
 
     /// If true, turn on the `-frecovery-ast` clang flag.
public:
   LLVM_NODISCARD bool
   blockUntilIdleForTest(llvm::Optional<double> TimeoutSeconds = 10);
 
 private:
   void formatCode(PathRef File, llvm::StringRef Code,
                   ArrayRef<tooling::Range> Ranges,
                   Callback<tooling::Replacements> CB);
 
+  /// Derives a context for a task processing the specified source file.
+  /// This includes the current configuration (see Options::ConfigProvider).
+  /// The empty string means no particular file is the target.
+  /// Rather than called by each feature, this is exposed to the components
+  /// that control worker threads, like TUScheduler and BackgroundIndex.
+  /// This means it's OK to do some IO here, and it cuts across all features.
+  Context createProcessingContext(PathRef) const;
+  config::Provider *ConfigProvider = nullptr;
+
   const ThreadsafeFS &TFS;
 
   Path ResourceDir;
   // The index used to look up symbols. This could be:
   //   - null (all index functionality is optional)
   //   - the dynamic index owned by ClangdServer (DynamicIdx)
   //   - the static index passed to the constructor
   //   - a merged view of a static and dynamic index (MergedIndex)

clang-tools-extra/clangd/ClangdServer.cpp

 //===--- ClangdServer.cpp - Main clangd server code --------------*- C++-*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===-------------------------------------------------------------------===//
 
 #include "ClangdServer.h"
 #include "CodeComplete.h"
+#include "Config.h"
 #include "FindSymbols.h"
 #include "Format.h"
 #include "HeaderSourceSwitch.h"
 #include "Headers.h"
 #include "ParsedAST.h"
 #include "Preamble.h"
 #include "Protocol.h"
 #include "SemanticHighlighting.h"
ClangdServer::Options::operator TUScheduler::Options() const {
   Opts.UpdateDebounce = UpdateDebounce;
   Opts.AsyncPreambleBuilds = AsyncPreambleBuilds;
   return Opts;
 }
 
 ClangdServer::ClangdServer(const GlobalCompilationDatabase &CDB,
                            const ThreadsafeFS &TFS, const Options &Opts,
                            Callbacks *Callbacks)
-    : TFS(TFS),
+    : ConfigProvider(Opts.ConfigProvider), TFS(TFS),
       DynamicIdx(Opts.BuildDynamicSymbolIndex
                      ? new FileIndex(Opts.HeavyweightDynamicSymbolIndex)
                      : nullptr),
       GetClangTidyOptions(Opts.GetClangTidyOptions),
       SuggestMissingIncludes(Opts.SuggestMissingIncludes),
       BuildRecoveryAST(Opts.BuildRecoveryAST),
       PreserveRecoveryASTType(Opts.PreserveRecoveryASTType),
       TweakFilter(Opts.TweakFilter), WorkspaceRoot(Opts.WorkspaceRoot),
       // Pass a callback into `WorkScheduler` to extract symbols from a newly
       // parsed file and rebuild the file index synchronously each time an AST
       // is parsed.
       // FIXME(ioeric): this can be slow and we may be able to index on less
       // critical paths.
       WorkScheduler(
-          CDB, TUScheduler::Options(Opts),
+          CDB,
+          [&, this] {
+            TUScheduler::Options O(Opts);
+            O.ContextProvider = [this](PathRef P) {
+              return createProcessingContext(P);
+            };
+            return O;
+          }(),
           std::make_unique<UpdateIndexCallbacks>(
               DynamicIdx.get(), Callbacks, Opts.TheiaSemanticHighlighting)) {
   // Adds an index to the stack, at higher priority than existing indexes.
   auto AddIndex = [&](SymbolIndex *Idx) {
     if (this->Index != nullptr) {
       MergedIdx.push_back(std::make_unique<MergedIndex>(Idx, this->Index));
       this->Index = MergedIdx.back().get();
     } else {
       this->Index = Idx;
     }
   };
   if (Opts.StaticIndex)
     AddIndex(Opts.StaticIndex);
   if (Opts.BackgroundIndex) {
     BackgroundIdx = std::make_unique<BackgroundIndex>(
         Context::current().clone(), TFS, CDB,
         BackgroundIndexStorage::createDiskBackedStorageFactory(
             [&CDB](llvm::StringRef File) { return CDB.getProjectInfo(File); }),
         std::max(Opts.AsyncThreadsCount, 1u),
         [Callbacks](BackgroundQueue::Stats S) {
           if (Callbacks)
             Callbacks->onBackgroundIndexProgress(S);
-        });
+        },
+        [this](PathRef P) { return createProcessingContext(P); });
     AddIndex(BackgroundIdx.get());
   }
   if (DynamicIdx)
     AddIndex(DynamicIdx.get());
 }
 
 void ClangdServer::addDocument(PathRef File, llvm::StringRef Contents,
                                llvm::StringRef Version,
if (!Style)
       return CB(Style.takeError());
 
     std::vector<TextEdit> Result;
     for (const tooling::Replacement &R :
          formatIncremental(Code, CursorPos, TriggerText, *Style))
       Result.push_back(replacementToEdit(Code, R));
     return CB(Result);
   };
-  WorkScheduler.run("FormatOnType", std::move(Action));
+  WorkScheduler.run("FormatOnType", File, std::move(Action));
 }
 
 void ClangdServer::prepareRename(PathRef File, Position Pos,
                                  const RenameOptions &RenameOpts,
                                  Callback<llvm::Optional<Range>> CB) {
   auto Action = [Pos, File = File.str(), CB = std::move(CB), RenameOpts,
                  this](llvm::Expected<InputsAndAST> InpAST) mutable {
     if (!InpAST)
auto Action = [File = File.str(), Code = Code.str(), Ranges = Ranges.vec(),
     if (!Changed)
       return CB(Changed.takeError());
 
     CB(IncludeReplaces.merge(format::reformat(
         Style, *Changed,
         tooling::calculateRangesAfterReplacements(IncludeReplaces, Ranges),
         File)));
   };
-  WorkScheduler.run("Format", std::move(Action));
+  WorkScheduler.run("Format", File, std::move(Action));
 }
 
 void ClangdServer::findDocumentHighlights(
     PathRef File, Position Pos, Callback<std::vector<DocumentHighlight>> CB) {
   auto Action =
       [Pos, CB = std::move(CB)](llvm::Expected<InputsAndAST> InpAST) mutable {
         if (!InpAST)
           return CB(InpAST.takeError());
void ClangdServer::onFileEvent(const DidChangeWatchedFilesParams &Params) {
   // FIXME: Do nothing for now. This will be used for indexing and potentially
   // invalidating other caches.
 }
 
 void ClangdServer::workspaceSymbols(
     llvm::StringRef Query, int Limit,
     Callback<std::vector<SymbolInformation>> CB) {
   WorkScheduler.run(
-      "getWorkspaceSymbols",
+      "getWorkspaceSymbols", /*Path=*/"",
       [Query = Query.str(), Limit, CB = std::move(CB), this]() mutable {
         CB(clangd::getWorkspaceSymbols(Query, Limit, Index,
                                        WorkspaceRoot.getValueOr("")));
       });
 }
 
 void ClangdServer::documentSymbols(llvm::StringRef File,
                                    Callback<std::vector<DocumentSymbol>> CB) {
void ClangdServer::semanticHighlights(
   WorkScheduler.runWithAST("SemanticHighlights", File, std::move(Action),
                            TUScheduler::InvalidateOnUpdate);
 }
 
 llvm::StringMap<TUScheduler::FileStats> ClangdServer::fileStats() const {
   return WorkScheduler.fileStats();
 }
 
+Context ClangdServer::createProcessingContext(PathRef File) const {
+  if (!ConfigProvider)
+    return Context::current().clone();
+
+  config::Params Params;
+  if (!File.empty()) {
+    assert(llvm::sys::path::is_absolute(File));
+    llvm::SmallString<256> PosixPath = File;
+    llvm::sys::path::native(PosixPath, llvm::sys::path::Style::posix);
+    Params.Path = PosixPath.str();
+  }
+
+  auto DiagnosticHandler = [](const llvm::SMDiagnostic &Diag) {
+    if (Diag.getKind() == llvm::SourceMgr::DK_Error) {
+      log("config error at {0}:{1}:{2}: {3}", Diag.getFilename(),
+          Diag.getLineNo(), Diag.getColumnNo(), Diag.getMessage());
+    } else {
+      log("config warning at {0}:{1}:{2}: {3}", Diag.getFilename(),
+          Diag.getLineNo(), Diag.getColumnNo(), Diag.getMessage());
+    }
+  };
+  Config C = ConfigProvider->getConfig(Params, DiagnosticHandler);
+  return Context::current().derive(Config::Key, std::move(C));
+}
+
 LLVM_NODISCARD bool
 ClangdServer::blockUntilIdleForTest(llvm::Optional<double> TimeoutSeconds) {
   return WorkScheduler.blockUntilIdle(timeoutSeconds(TimeoutSeconds)) &&
          (!BackgroundIdx ||
           BackgroundIdx->blockUntilIdleForTest(TimeoutSeconds));
 }
 
 } // namespace clangd
 } // namespace clang

clang-tools-extra/clangd/TUScheduler.h

struct Options {
     DebouncePolicy UpdateDebounce;
 
     /// Determines when to keep idle ASTs in memory for future use.
     ASTRetentionPolicy RetentionPolicy;
 
     /// Whether to run PreamblePeer asynchronously.
     /// No-op if AsyncThreadsCount is 0.
     bool AsyncPreambleBuilds = false;
+
+    /// Used to create a context that wraps each single operation.
+    /// Typically to inject per-file configuration.
+    /// If the path is empty, context sholud be "generic".
+    std::function<Context(PathRef)> ContextProvider;
   };
 
   TUScheduler(const GlobalCompilationDatabase &CDB, const Options &Opts,
               std::unique_ptr<ParsingCallbacks> ASTCallbacks = nullptr);
   ~TUScheduler();
 
   struct FileStats {
     std::size_t UsedBytes = 0;
public:
   /// resources. Pending diagnostics for closed files may not be delivered, even
   /// if requested with WantDiags::Auto or WantDiags::Yes.
   void remove(PathRef File);
 
   /// Returns a snapshot of all file buffer contents, per last update().
   llvm::StringMap<std::string> getAllFileContents() const;
 
   /// Schedule an async task with no dependencies.
-  void run(llvm::StringRef Name, llvm::unique_function<void()> Action);
+  /// Path may be empty (it is used only to set the Context).
+  void run(llvm::StringRef Name, llvm::StringRef Path,
+           llvm::unique_function<void()> Action);
 
   /// Defines how a runWithAST action is implicitly cancelled by other actions.
   enum ASTActionInvalidation {
     /// The request will run unless explicitly cancelled.
     NoInvalidation,
     /// The request will be implicitly cancelled by a subsequent update().
     /// (Only if the request was not yet cancelled).
     /// Useful for requests that are generated by clients, without any explicit
public:
   /// an LRU cache.
   class ASTCache;
 
   // The file being built/processed in the current thread. This is a hack in
   // order to get the file name into the index implementations. Do not depend on
   // this inside clangd.
   // FIXME: remove this when there is proper index support via build system
   // integration.
+  // FIXME: move to ClangdServer via createProcessingContext.
   static llvm::Optional<llvm::StringRef> getFileBeingProcessedInContext();
 
 private:
   const GlobalCompilationDatabase &CDB;
   const Options Opts;
   std::unique_ptr<ParsingCallbacks> Callbacks; // not nullptr
   Semaphore Barrier;
   llvm::StringMap<std::unique_ptr<FileData>> Files;

clang-tools-extra/clangd/TUScheduler.cpp

while (true) {
         if (Done)
           break;
         CurrentReq = std::move(*NextReq);
         NextReq.reset();
       }
 
       {
         WithContext Guard(std::move(CurrentReq->Ctx));
+        // Note that we don't make use of the ContextProvider here.
+        // Preamble tasks are always scheduled by ASTWorker tasks, and we
+        // reuse the context/config that was created at that level.
+
         // Build the preamble and let the waiters know about it.
         build(std::move(*CurrentReq));
       }
       bool IsEmpty = false;
       {
         std::lock_guard<std::mutex> Lock(Mutex);
         CurrentReq.reset();
         IsEmpty = !NextReq.hasValue();
private:
 
   /// Handles retention of ASTs.
   TUScheduler::ASTCache &IdleASTs;
   const bool RunSync;
   /// Time to wait after an update to see whether another update obsoletes it.
   const DebouncePolicy UpdateDebounce;
   /// File that ASTWorker is responsible for.
   const Path FileName;
+  /// Callback to create processing contexts for tasks.
+  const std::function<Context(llvm::StringRef)> ContextProvider;
   const GlobalCompilationDatabase &CDB;
   /// Callback invoked when preamble or main file AST is built.
   ParsingCallbacks &Callbacks;
 
   Semaphore &Barrier;
   /// Whether the 'onMainAST' callback ran for the current FileInputs.
   bool RanASTCallback = false;
   /// Guards members used by both TUScheduler and the worker thread.
ASTWorkerHandle ASTWorker::create(PathRef FileName,
   return ASTWorkerHandle(std::move(Worker));
 }
 
 ASTWorker::ASTWorker(PathRef FileName, const GlobalCompilationDatabase &CDB,
                      TUScheduler::ASTCache &LRUCache, Semaphore &Barrier,
                      bool RunSync, const TUScheduler::Options &Opts,
                      ParsingCallbacks &Callbacks)
     : IdleASTs(LRUCache), RunSync(RunSync), UpdateDebounce(Opts.UpdateDebounce),
-      FileName(FileName), CDB(CDB), Callbacks(Callbacks), Barrier(Barrier),
-      Done(false), Status(FileName, Callbacks),
+      FileName(FileName), ContextProvider(Opts.ContextProvider), CDB(CDB),
+      Callbacks(Callbacks), Barrier(Barrier), Done(false),
+      Status(FileName, Callbacks),
       PreamblePeer(FileName, Callbacks, Opts.StorePreamblesInMemory,
                    RunSync || !Opts.AsyncPreambleBuilds, Status, *this) {
   // Set a fallback command because compile command can be accessed before
   // `Inputs` is initialized. Other fields are only used after initialization
   // from client inputs.
   FileInputs.CompileCommand = CDB.getFallbackCommand(FileName);
 }
 
// only writer is also this thread.
         Lock.lock();
       }
       WithContext Guard(std::move(CurrentRequest->Ctx));
       trace::Span Tracer(CurrentRequest->Name);
       Status.update([&](TUStatus &Status) {
         Status.ASTActivity.K = ASTAction::RunningAction;
         Status.ASTActivity.Name = CurrentRequest->Name;
       });
+      llvm::Optional<WithContext> WithProvidedContext;
+      if (ContextProvider)
+        WithProvidedContext.emplace(ContextProvider(FileName));
       CurrentRequest->Action();
     }
 
     bool IsEmpty = false;
     {
       std::lock_guard<std::mutex> Lock(Mutex);
       CurrentRequest.reset();
       IsEmpty = Requests.empty() && PreambleRequests.empty();

 
 llvm::StringMap<std::string> TUScheduler::getAllFileContents() const {
   llvm::StringMap<std::string> Results;
   for (auto &It : Files)
     Results.try_emplace(It.getKey(), It.getValue()->Contents);
   return Results;
 }
 
-void TUScheduler::run(llvm::StringRef Name,
+void TUScheduler::run(llvm::StringRef Name, llvm::StringRef Path,
                       llvm::unique_function<void()> Action) {
   if (!PreambleTasks)
     return Action();
   PreambleTasks->runAsync(Name, [this, Ctx = Context::current().clone(),
+                                 Path(Path.str()),
                                  Action = std::move(Action)]() mutable {
     std::lock_guard<Semaphore> BarrierLock(Barrier);
     WithContext WC(std::move(Ctx));
+    llvm::Optional<WithContext> WithProvidedContext;
+    if (Opts.ContextProvider)
+      WithProvidedContext.emplace(Opts.ContextProvider(Path));
     Action();
   });
 }
 
 void TUScheduler::runWithAST(
     llvm::StringRef Name, PathRef File,
     llvm::unique_function<void(llvm::Expected<InputsAndAST>)> Action,
     TUScheduler::ASTActionInvalidation Invalidation) {
auto Task =
           Worker->waitForFirstPreamble();
         }
         Preamble = Worker->getPossiblyStalePreamble();
 
         std::lock_guard<Semaphore> BarrierLock(Barrier);
         WithContext Guard(std::move(Ctx));
         trace::Span Tracer(Name);
         SPAN_ATTACH(Tracer, "file", File);
+        llvm::Optional<WithContext> WithProvidedContext;
+        if (Opts.ContextProvider)
+          WithProvidedContext.emplace(Opts.ContextProvider(File));
         Action(InputsAndPreamble{Contents, Command, Preamble.get()});
       };
 
   PreambleTasks->runAsync("task:" + llvm::sys::path::filename(File),
                           std::move(Task));
 }
 
 llvm::StringMap<TUScheduler::FileStats> TUScheduler::fileStats() const {

clang-tools-extra/clangd/index/Background.h

public:
   /// rebuilt for each indexed file.
   BackgroundIndex(
       Context BackgroundContext, const ThreadsafeFS &,
       const GlobalCompilationDatabase &CDB,
       BackgroundIndexStorage::Factory IndexStorageFactory,
       // Arbitrary value to ensure some concurrency in tests.
       // In production an explicit value is passed.
       size_t ThreadPoolSize = 4,
-      std::function<void(BackgroundQueue::Stats)> OnProgress = nullptr);
+      std::function<void(BackgroundQueue::Stats)> OnProgress = nullptr,
+      std::function<Context(PathRef)> ContextProvider = nullptr);
   ~BackgroundIndex(); // Blocks while the current task finishes.
 
   // Enqueue translation units for indexing.
   // The indexing happens in a background thread, so the symbols will be
   // available sometime later.
   void enqueue(const std::vector<std::string> &ChangedFiles) {
     Queue.push(changedFilesTask(ChangedFiles));
   }
private:
   void update(llvm::StringRef MainFile, IndexFileIn Index,
               const llvm::StringMap<ShardVersion> &ShardVersionsSnapshot,
               bool HadErrors);
 
   // configuration
   const ThreadsafeFS &TFS;
   const GlobalCompilationDatabase &CDB;
   Context BackgroundContext;
+  std::function<Context(PathRef)> ContextProvider;
 
   llvm::Error index(tooling::CompileCommand);
 
   FileSymbols IndexedSymbols;
   BackgroundIndexRebuilder Rebuilder;
   llvm::StringMap<ShardVersion> ShardVersions; // Key is absolute file path.
   std::mutex ShardVersionsMu;
 
   BackgroundIndexStorage::Factory IndexStorageFactory;
   // Tries to load shards for the MainFiles and their dependencies.
-  std::vector<tooling::CompileCommand>
-  loadProject(std::vector<std::string> MainFiles);
+  std::vector<std::string> loadProject(std::vector<std::string> MainFiles);
 
   BackgroundQueue::Task
   changedFilesTask(const std::vector<std::string> &ChangedFiles);
-  BackgroundQueue::Task indexFileTask(tooling::CompileCommand Cmd);
+  BackgroundQueue::Task indexFileTask(std::string Path);
 
   // from lowest to highest priority
   enum QueuePriority {
     IndexFile,
     IndexBoostedFile,
     LoadShards,
   };
   BackgroundQueue Queue;
   AsyncTaskRunner ThreadPool;
   GlobalCompilationDatabase::CommandChanged::Subscription CommandsChanged;
 };
 
 } // namespace clangd
 } // namespace clang
 
 #endif

clang-tools-extra/clangd/index/Background.cpp

 }
 
 } // namespace
 
 BackgroundIndex::BackgroundIndex(
     Context BackgroundContext, const ThreadsafeFS &TFS,
     const GlobalCompilationDatabase &CDB,
     BackgroundIndexStorage::Factory IndexStorageFactory, size_t ThreadPoolSize,
-    std::function<void(BackgroundQueue::Stats)> OnProgress)
+    std::function<void(BackgroundQueue::Stats)> OnProgress,
+    std::function<Context(PathRef)> ContextProvider)
     : SwapIndex(std::make_unique<MemIndex>()), TFS(TFS), CDB(CDB),
       BackgroundContext(std::move(BackgroundContext)),
+      ContextProvider(std::move(ContextProvider)),
       Rebuilder(this, &IndexedSymbols, ThreadPoolSize),
       IndexStorageFactory(std::move(IndexStorageFactory)),
       Queue(std::move(OnProgress)),
       CommandsChanged(
           CDB.watch([&](const std::vector<std::string> &ChangedFiles) {
             enqueue(ChangedFiles);
           })) {
   assert(ThreadPoolSize > 0 && "Thread pool size can't be zero.");
BackgroundIndex::~BackgroundIndex() {
   stop();
   ThreadPool.wait();
 }
 
 BackgroundQueue::Task BackgroundIndex::changedFilesTask(
     const std::vector<std::string> &ChangedFiles) {
   BackgroundQueue::Task T([this, ChangedFiles] {
     trace::Span Tracer("BackgroundIndexEnqueue");
+
+    llvm::Optional<WithContext> WithProvidedContext;
+    if (ContextProvider)
+      WithProvidedContext.emplace(ContextProvider(/*Path=*/""));
+
     // We're doing this asynchronously, because we'll read shards here too.
     log("Enqueueing {0} commands for indexing", ChangedFiles.size());
     SPAN_ATTACH(Tracer, "files", int64_t(ChangedFiles.size()));
 
     auto NeedsReIndexing = loadProject(std::move(ChangedFiles));
     // Run indexing for files that need to be updated.
     std::shuffle(NeedsReIndexing.begin(), NeedsReIndexing.end(),
                  std::mt19937(std::random_device{}()));
BackgroundQueue::Task BackgroundIndex::changedFilesTask(
   return T;
 }
 
 static llvm::StringRef filenameWithoutExtension(llvm::StringRef Path) {
   Path = llvm::sys::path::filename(Path);
   return Path.drop_back(llvm::sys::path::extension(Path).size());
 }
 
-BackgroundQueue::Task
-BackgroundIndex::indexFileTask(tooling::CompileCommand Cmd) {
-  BackgroundQueue::Task T([this, Cmd] {
-    // We can't use llvm::StringRef here since we are going to
-    // move from Cmd during the call below.
-    const std::string FileName = Cmd.Filename;
-    if (auto Error = index(std::move(Cmd)))
-      elog("Indexing {0} failed: {1}", FileName, std::move(Error));
+BackgroundQueue::Task BackgroundIndex::indexFileTask(std::string Path) {
+  std::string Tag = filenameWithoutExtension(Path).str();
+  BackgroundQueue::Task T([this, Path(std::move(Path))] {
+    llvm::Optional<WithContext> WithProvidedContext;
+    if (ContextProvider)
+      WithProvidedContext.emplace(ContextProvider(Path));
+    auto Cmd = CDB.getCompileCommand(Path);
+    if (!Cmd)
+      return;
+    if (auto Error = index(std::move(*Cmd)))
+      elog("Indexing {0} failed: {1}", Path, std::move(Error));
   });
   T.QueuePri = IndexFile;
-  T.Tag = std::string(filenameWithoutExtension(Cmd.Filename));
+  T.Tag = std::move(Tag);
   return T;
 }
 
 void BackgroundIndex::boostRelated(llvm::StringRef Path) {
   if (isHeaderFile(Path))
     Queue.boost(filenameWithoutExtension(Path), IndexBoostedFile);
 }
 
llvm::Error BackgroundIndex::index(tooling::CompileCommand Cmd) {
 
   Rebuilder.indexedTU();
   return llvm::Error::success();
 }
 
 // Restores shards for \p MainFiles from index storage. Then checks staleness of
 // those shards and returns a list of TUs that needs to be indexed to update
 // staleness.
-std::vector<tooling::CompileCommand>
+std::vector<std::string>
 BackgroundIndex::loadProject(std::vector<std::string> MainFiles) {
-  std::vector<tooling::CompileCommand> NeedsReIndexing;
-
   Rebuilder.startLoading();
   // Load shards for all of the mainfiles.
   const std::vector<LoadedShard> Result =
       loadIndexShards(MainFiles, IndexStorageFactory, CDB);
   size_t LoadedShards = 0;
   {
     // Update in-memory state.
     std::lock_guard<std::mutex> Lock(ShardVersionsMu);
for (auto &LS : Result) {
     // FIXME: Currently, we simply schedule indexing on a TU whenever any of
     // its dependencies needs re-indexing. We might do it smarter by figuring
     // out a minimal set of TUs that will cover all the stale dependencies.
     // FIXME: Try looking at other TUs if no compile commands are available
     // for this TU, i.e TU was deleted after we performed indexing.
     TUsToIndex.insert(TUForFile);
   }
 
-  for (PathRef TU : TUsToIndex) {
-    auto Cmd = CDB.getCompileCommand(TU);
-    if (!Cmd)
-      continue;
-    NeedsReIndexing.emplace_back(std::move(*Cmd));
-  }
-
-  return NeedsReIndexing;
+  return {TUsToIndex.begin(), TUsToIndex.end()};
 }
 
 } // namespace clangd
 } // namespace clang

clang-tools-extra/clangd/unittests/BackgroundIndexTests.cpp

+#include "CompileCommands.h"
+#include "Config.h"
 #include "Headers.h"
 #include "SyncAPI.h"
 #include "TestFS.h"
 #include "TestIndex.h"
 #include "TestTU.h"
 #include "index/Background.h"
 #include "index/BackgroundRebuild.h"
+#include "clang/Tooling/ArgumentsAdjusters.h"
 #include "clang/Tooling/CompilationDatabase.h"
 #include "llvm/Support/ScopedPrinter.h"
 #include "llvm/Support/Threading.h"
 #include "gmock/gmock.h"
 #include "gtest/gtest.h"
 #include <deque>
 #include <thread>
 
 using ::testing::_;
 using ::testing::AllOf;
 using ::testing::Contains;
 using ::testing::ElementsAre;
 using ::testing::Not;
 using ::testing::UnorderedElementsAre;
 
 namespace clang {
 namespace clangd {
 
 MATCHER_P(Named, N, "") { return arg.Name == N; }
+MATCHER_P(QName, N, "") { return (arg.Scope + arg.Name).str() == N; }
 MATCHER(Declared, "") {
   return !StringRef(arg.CanonicalDeclaration.FileURI).empty();
 }
 MATCHER(Defined, "") { return !StringRef(arg.Definition.FileURI).empty(); }
 MATCHER_P(FileURI, F, "") { return StringRef(arg.Location.FileURI) == F; }
 ::testing::Matcher<const RefSlab &>
 RefsAre(std::vector<::testing::Matcher<Ref>> Matchers) {
   return ElementsAre(::testing::Pair(_, UnorderedElementsAreArray(Matchers)));
TEST_F(BackgroundIndexTest, NoCrashOnErrorFile) {
   Cmd.Filename = testPath("root/A.cc");
   Cmd.Directory = testPath("root");
   Cmd.CommandLine = {"clang++", "-DA=1", testPath("root/A.cc")};
   CDB.setCompileCommand(testPath("root/A.cc"), Cmd);
 
   ASSERT_TRUE(Idx.blockUntilIdleForTest());
 }
 
+TEST_F(BackgroundIndexTest, Config) {
+  MockFS FS;
+  // Set up two identical TUs, foo and bar.
+  // They define foo::one and bar::one.
+  std::vector<tooling::CompileCommand> Cmds;
+  for (std::string Name : {"foo", "bar"}) {
+    std::string Filename = Name + ".cpp";
+    std::string Header = Name + ".h";
+    FS.Files[Filename] = "#include \"" + Header + "\"";
+    FS.Files[Header] = "namespace " + Name + " { int one; }";
+    tooling::CompileCommand Cmd;
+    Cmd.Filename = Filename;
+    Cmd.Directory = testRoot();
+    Cmd.CommandLine = {"clang++", Filename};
+    Cmds.push_back(std::move(Cmd));
+  }
+  // Context provider that installs a configuration mutating foo's command.
+  // This causes it to define foo::two instead of foo::one.
+  auto ContextProvider = [](PathRef P) {
+    Config C;
+    if (P.endswith("foo.cpp"))
+      C.CompileFlags.Edits.push_back(
+          [](std::vector<std::string> &Argv) { Argv.push_back("-Done=two"); });
+    return Context::current().derive(Config::Key, std::move(C));
+  };
+  // Create the background index.
+  llvm::StringMap<std::string> Storage;
+  size_t CacheHits = 0;
+  MemoryShardStorage MSS(Storage, CacheHits);
+  // We need the CommandMangler, because that applies the config we're testing.
+  OverlayCDB CDB(/*Base=*/nullptr, /*FallbackFlags=*/{},
+                 tooling::ArgumentsAdjuster(CommandMangler::forTests()));
+  BackgroundIndex Idx(
+      Context::empty(), FS, CDB, [&](llvm::StringRef) { return &MSS; },
+      /*ThreadPoolSize=*/4, /*OnProgress=*/nullptr, std::move(ContextProvider));
+  // Index the two files.
+  for (auto &Cmd : Cmds)
+    CDB.setCompileCommand(testPath(Cmd.Filename), std::move(Cmd));
+  // Wait for both files to be indexed.
+  ASSERT_TRUE(Idx.blockUntilIdleForTest());
+  EXPECT_THAT(runFuzzyFind(Idx, ""),
+              UnorderedElementsAre(QName("foo"), QName("foo::two"),
+                                   QName("bar"), QName("bar::one")));
+}
+
 TEST_F(BackgroundIndexTest, IndexTwoFiles) {
   MockFS FS;
   // a.h yields different symbols when included by A.cc vs B.cc.
   FS.Files[testPath("root/A.h")] = R"cpp(
       void common();
       void f_b();
       #if A
         class A_CC {};

clang-tools-extra/clangd/unittests/ClangdTests.cpp

 //===-- ClangdTests.cpp - Clangd unit tests ---------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "Annotations.h"
 #include "ClangdLSPServer.h"
 #include "ClangdServer.h"
 #include "CodeComplete.h"
+#include "ConfigFragment.h"
 #include "GlobalCompilationDatabase.h"
 #include "Matchers.h"
 #include "SyncAPI.h"
 #include "TestFS.h"
 #include "URI.h"
 #include "support/Path.h"
 #include "support/Threading.h"
 #include "clang/Config/config.h"
 #include "clang/Sema/CodeCompleteConsumer.h"
+#include "clang/Tooling/ArgumentsAdjusters.h"
 #include "llvm/ADT/None.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringMap.h"
 #include "llvm/Support/Errc.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/Regex.h"
 #include "gmock/gmock.h"

 namespace {
 
 using ::testing::AllOf;
 using ::testing::ElementsAre;
 using ::testing::Field;
 using ::testing::Gt;
 using ::testing::IsEmpty;
 using ::testing::Pair;
+using ::testing::SizeIs;
 using ::testing::UnorderedElementsAre;
 
 MATCHER_P2(DeclAt, File, Range, "") {
   return arg.PreferredDeclaration ==
          Location{URIForFile::canonicalize(File, testRoot()), Range};
 }
 
 bool diagsContainErrors(const std::vector<Diag> &Diagnostics) {
ClangdServer Server(CDB, FS, ClangdServer::optsForTest(), &Callbacks);
     WithContextValue Entrypoint(Secret, 42);
     Server.addDocument(testPath("foo.cpp"), "void main(){}");
   }
   ASSERT_TRUE(Server.blockUntilIdleForTest());
   EXPECT_EQ(FS.Got, 42);
   EXPECT_EQ(Callbacks.Got, 42);
 }
 
+TEST(ClangdServerTest, RespectsConfig) {
+  // Go-to-definition will resolve as marked if FOO is defined.
+  Annotations Example(R"cpp(
+  #ifdef FOO
+  int [[x]];
+  #else
+  int x;
+  #endif
+  int y = ^x;
+  )cpp");
+  // Provide conditional config that defines FOO for foo.cc.
+  class ConfigProvider : public config::Provider {
+    std::vector<config::CompiledFragment>
+    getFragments(const config::Params &,
+                 config::DiagnosticCallback DC) const override {
+      config::Fragment F;
+      F.If.PathMatch.emplace_back(".*foo.cc");
+      F.CompileFlags.Add.emplace_back("-DFOO=1");
+      return {std::move(F).compile(DC)};
+    }
+  } CfgProvider;
+
+  auto Opts = ClangdServer::optsForTest();
+  Opts.ConfigProvider = &CfgProvider;
+  OverlayCDB CDB(/*Base=*/nullptr, /*FallbackFlags=*/{},
+                 tooling::ArgumentsAdjuster(CommandMangler::forTests()));
+  MockFS FS;
+  ClangdServer Server(CDB, FS, Opts);
+  // foo.cc sees the expected definition, as FOO is defined.
+  Server.addDocument(testPath("foo.cc"), Example.code());
+  auto Result = runLocateSymbolAt(Server, testPath("foo.cc"), Example.point());
+  ASSERT_TRUE(bool(Result)) << Result.takeError();
+  ASSERT_THAT(*Result, SizeIs(1));
+  EXPECT_EQ(Result->front().PreferredDeclaration.range, Example.range());
+  // bar.cc gets a different result, as FOO is not defined.
+  Server.addDocument(testPath("bar.cc"), Example.code());
+  Result = runLocateSymbolAt(Server, testPath("bar.cc"), Example.point());
+  ASSERT_TRUE(bool(Result)) << Result.takeError();
+  ASSERT_THAT(*Result, SizeIs(1));
+  EXPECT_NE(Result->front().PreferredDeclaration.range, Example.range());
+}
+
 TEST_F(ClangdVFSTest, PropagatesVersion) {
   MockCompilationDatabase CDB;
   MockFS FS;
   struct Callbacks : public ClangdServer::Callbacks {
     void onDiagnosticsReady(PathRef File, llvm::StringRef Version,
                             std::vector<Diag> Diagnostics) override {
       Got = Version.str();
     }

clang-tools-extra/clangd/unittests/TUSchedulerTests.cpp

MATCHER_P2(TUState, PreambleActivity, ASTActivity, "") {
   }
   if (arg.ASTActivity.K != ASTActivity) {
     *result_listener << "aststate is " << arg.ASTActivity.K;
     return false;
   }
   return true;
 }
 
+// Dummy ContextProvider to verify the provider is invoked & contexts are used.
+static Key<std::string> BoundPath;
+Context bindPath(PathRef F) {
+  return Context::current().derive(BoundPath, F.str());
+}
+llvm::StringRef boundPath() {
+  const std::string *V = Context::current().get(BoundPath);
+  return V ? *V : llvm::StringRef("");
+}
+
 TUScheduler::Options optsForTest() {
-  return TUScheduler::Options(ClangdServer::optsForTest());
+  TUScheduler::Options Opts(ClangdServer::optsForTest());
+  Opts.ContextProvider = bindPath;
+  return Opts;
 }
 
 class TUSchedulerTests : public ::testing::Test {
 protected:
   ParseInputs getInputs(PathRef File, std::string Contents) {
     ParseInputs Inputs;
     Inputs.CompileCommand = *CDB.getCompileCommand(File);
     Inputs.TFS = &FS;
for (int FileI = 0; FileI < FilesCount; ++FileI) {
         {
           WithContextValue WithNonce(NonceKey, ++Nonce);
           Inputs.Version = std::to_string(UpdateI);
           updateWithDiags(
               S, File, Inputs, WantDiagnostics::Auto,
               [File, Nonce, Version(Inputs.Version), &Mut, &TotalUpdates,
                &LatestDiagVersion](std::vector<Diag>) {
                 EXPECT_THAT(Context::current().get(NonceKey), Pointee(Nonce));
+                EXPECT_EQ(File, boundPath());
 
                 std::lock_guard<std::mutex> Lock(Mut);
                 ++TotalUpdates;
                 EXPECT_EQ(File, *TUScheduler::getFileBeingProcessedInContext());
                 // Make sure Diags are for a newer version.
                 auto It = LatestDiagVersion.try_emplace(File, -1);
                 const int PrevVersion = It.first->second;
                 int CurVersion;
                 ASSERT_TRUE(llvm::to_integer(Version, CurVersion, 10));
                 EXPECT_LT(PrevVersion, CurVersion);
                 It.first->getValue() = CurVersion;
               });
         }
         {
           WithContextValue WithNonce(NonceKey, ++Nonce);
           S.runWithAST(
               "CheckAST", File,
               [File, Inputs, Nonce, &Mut,
                &TotalASTReads](Expected<InputsAndAST> AST) {
                 EXPECT_THAT(Context::current().get(NonceKey), Pointee(Nonce));
+                EXPECT_EQ(File, boundPath());
 
                 ASSERT_TRUE((bool)AST);
                 EXPECT_EQ(AST->Inputs.Contents, Inputs.Contents);
                 EXPECT_EQ(AST->Inputs.Version, Inputs.Version);
                 EXPECT_EQ(AST->AST.version(), Inputs.Version);
 
                 std::lock_guard<std::mutex> Lock(Mut);
                 ++TotalASTReads;
                 EXPECT_EQ(File, *TUScheduler::getFileBeingProcessedInContext());
               });
         }
 
         {
           WithContextValue WithNonce(NonceKey, ++Nonce);
           S.runWithPreamble(
               "CheckPreamble", File, TUScheduler::Stale,
               [File, Inputs, Nonce, &Mut,
                &TotalPreambleReads](Expected<InputsAndPreamble> Preamble) {
                 EXPECT_THAT(Context::current().get(NonceKey), Pointee(Nonce));
+                EXPECT_EQ(File, boundPath());
 
                 ASSERT_TRUE((bool)Preamble);
                 EXPECT_EQ(Preamble->Contents, Inputs.Contents);
 
                 std::lock_guard<std::mutex> Lock(Mut);
                 ++TotalPreambleReads;
                 EXPECT_EQ(File, *TUScheduler::getFileBeingProcessedInContext());
               });
TEST_F(TUSchedulerTests, NoChangeDiags) {
   // diags have previously been reported and the inputs didn't change.
   updateWithDiags(
       S, FooCpp, Contents, WantDiagnostics::Auto,
       [&](std::vector<Diag>) { ADD_FAILURE() << "Should not be called."; });
   ASSERT_TRUE(S.blockUntilIdle(timeoutSeconds(10)));
 }
 
 TEST_F(TUSchedulerTests, Run) {
-  TUScheduler S(CDB, optsForTest());
+  auto Opts = optsForTest();
+  Opts.ContextProvider = bindPath;
+  TUScheduler S(CDB, Opts);
   std::atomic<int> Counter(0);
-  S.run("add 1", [&] { ++Counter; });
-  S.run("add 2", [&] { Counter += 2; });
+  S.run("add 1", /*Path=*/"", [&] { ++Counter; });
+  S.run("add 2", /*Path=*/"", [&] { Counter += 2; });
   ASSERT_TRUE(S.blockUntilIdle(timeoutSeconds(10)));
   EXPECT_EQ(Counter.load(), 3);
 
   Notification TaskRun;
   Key<int> TestKey;
   WithContextValue CtxWithKey(TestKey, 10);
-  S.run("props context", [&] {
+  const char *Path = "somepath";
+  S.run("props context", Path, [&] {
     EXPECT_EQ(Context::current().getExisting(TestKey), 10);
+    EXPECT_EQ(Path, boundPath());
     TaskRun.notify();
   });
   TaskRun.wait();
 }
 
 TEST_F(TUSchedulerTests, TUStatus) {
   class CaptureTUStatus : public ClangdServer::Callbacks {
   public: