[clangd] Introduce ASTHooks to FeatureModules

[kadircet]

Summary

These can be invoked at different stages while building an AST to let
FeatureModules implement features on top of it. The patch also
introduces a sawDiagnostic hook, which can mutate the final clangd::Diag
while reading a clang::Diagnostic.

[kadircet]

• Add tests

[sammccall]

I'm getting a little nervous about the amount of stuff we're packing into modules without in-tree examples.
I should split out some of the "standard" features into modules as that's possible already.

My model for modules using this diagnostic stuff (apart from the build-system stuff which sadly can't be meaningfully upstreamed) are IncludeFixer, ClangTidy, and IWYU - worth thinking about how we'd build those on top of this model. (Doesn't mean we need to add a hook to emit diagnostics, but it probably means we should know where it would go)

clang-tools-extra/clangd/FeatureModule.h

104

naming: giving this a plural name and having a collection of them is a little confusing (is Hooks the hooks from one module, or from all of them?). What about ASTListener?
(Listener suggests passive but listeners that "participate" is a common enough idiom I think)

112

This comment hints at what I *thought* was the idea: astHooks() is called each time we parse a file, the returned object has methods called on it while the file is being parsed, and is then destroyed.

But the code suggests we call once globally and it has effectively the same lifetime as the module.
This seems much less useful, e.g. if we want to observe several diagnostics, examine the preamble, and emit new diagnostics, then we have to plumb around some notion of "AST identity" rather than just tying it to the identity of the ParseASTHooks object itself.

(Lots of natural extensions here like providing ParseInputs to astHooks(), but YAGNI for now)

[kadircet]

In D98499#2626502, @sammccall wrote:

My model for modules using this diagnostic stuff (apart from the build-system stuff which sadly can't be meaningfully upstreamed) are IncludeFixer, ClangTidy, and IWYU - worth thinking about how we'd build those on top of this model. (Doesn't mean we need to add a hook to emit diagnostics, but it probably means we should know where it would go)

Agreed. I believe they all would need extra end points to ASTHooks though.

ClangTidy:

• needs extra hooks to register PP callbacks, and take in a diagnostics engine
• needs a new endpoint to traverse ast and *emit* diags
• CTContext needs to be alive until the parsing is done: so we can:
  • make ASTHooks own it and instantiate a new one on every parse (i think the cleanest and most explicit)
  • make the module own them and control the lifetime with entry/exit calls on the asthooks. (there's more burden on the modules now, and they'll need extra synchronisation on enter/exit calls)
  • return some other object on parsing entry hook that'll be kept alive by the caller (needs design around semantics of that object).

IncludeFixer:

• needs a new hook to act as an external sema source, so that it can fix unresolved names.
• similar to CTContext issue, it references per-tu state like HeaderSearchInfo, so will need to incorporate these into entry hook, and somehow disambiguate hooks for different TUs (all the options proposed for tidy).

IWYU:

• I'd expect this to make use of ~same API as IncludeFixer.

PreamblePatching:

• can mutate compiler instance via a hook
• drop diagnostics from ParsedAST on exit (requires some mutation on parsedast though)

clang-tools-extra/clangd/FeatureModule.h

112

This comment hints at what I *thought* was the idea: astHooks() is called each time we parse a file, the returned object has methods called on it while the file is being parsed, and is then destroyed.

This was the intent actually (to enable modularization of other features like clangtidy, includefixer, etc. as mentioned above), but looks like i got confused while integrating this into clangdserver :/

While trying to resolve my confusion i actually noticed that we cannot uphold the contract of "hooks being called synchronously", because we actually have both a preamblethread and astworker that can invoke hooks (embarrassing of me to forget that 😓).

So we can:

• Give up on that promise and make life slightly complicated for module implementers
• Don't invoke hooks from preamblethread, (at the cost of limiting functionality, we can still have downstream users that act on PPCallbacks, but no direct integration with compiler, and that's where layering violations are generated :/)
• Handle the synchronization ourselves, only complicates TUScheduler more, rather than all the module implementers.
• Propogate FeatureModuleSet into TUScheduler and create 2 set of hooks on each thread :)

I am leaning towards 4, but unsure (mostly hesitant about dependency schema, as featuremodules also depend on TUScheduler..). WDYT?

[kadircet]

• Rename ParsedASTHooks to Listeners.
• Generate list of hooks on each parse.

[sammccall]

In D98499#2644313, @kadircet wrote:

In D98499#2626502, @sammccall wrote:

My model for modules using this diagnostic stuff (apart from the build-system stuff which sadly can't be meaningfully upstreamed) are IncludeFixer, ClangTidy, and IWYU - worth thinking about how we'd build those on top of this model. (Doesn't mean we need to add a hook to emit diagnostics, but it probably means we should know where it would go)

Agreed. I believe they all would need extra end points to ASTHooks though.

Yup. Let's not bite off more for now.

• make ASTHooks own it and instantiate a new one on every parse (i think the cleanest and most explicit)

Agreed. (And this only one that overlaps this patch a lot)

clang-tools-extra/clangd/FeatureModule.h

112

we actually have both a preamblethread and astworker that can invoke hooks (embarrassing of me to forget that 😓)

Ha! And yes, in our motivating case of fixing build system rules, the diagnostics are mostly going to be in the preamble.

The options that seem most tempting to me are:

• don't attempt to create/run astHooks while building preambles, but *do* feed the preamble's Diags into the main-file's AST hooks every time it's used. (you won't have a clang::Diagnostic, so that param would have to be gone/optional, and we'd scrape the messages instead of extracting args). This is kind of thematic, remember how we replay PPCallbacks for clang-tidy :-). This is the smallest tweak to your #2 that actually works for us, I think.
• create one astHooks for the preamble, and another for the AST build, and try to make the interface suitable for both. This is cute but there may be too much tension between the two cases. (Is this what you mean by #4?)
• or have separate ASTHooks & PreambleHooks interfaces and support all this crap for both. (Or is *this* what you mean by #4?) Hybrid is also possible, give the interfaces an inheritance relationship, or have one interface but pass boolean parameters to indicate which version we're doing, or...
• preambles and ASTs are a tree, so... give modules a PreambleHooks factory, and the factory function for ASTHooks is PreambleHooks::astHooks(). Holy overengineering batman...

These are roughly in order of complexity so we should probably start toward the top of the list somewhere. Up to you.

(I don't like the #1 or #3 in your list above much at all.)

[kadircet]

clang-tools-extra/clangd/FeatureModule.h

112

create one astHooks for the preamble, and another for the AST build, and try to make the interface suitable for both. This is cute but there may be too much tension between the two cases. (Is this what you mean by #4?)

yes this is what i meant by #4.

These are roughly in order of complexity so we should probably start toward the top of the list somewhere. Up to you.

I'd lean towards the second option(creating separate ASTHooks with the same interface for preamble and astworker). As discussed before first one (i.e. scraping diag message) is a limited solution that's likely to get us into a corner in the future. let me know if you have any concerns around this approach.

[kadircet]

• Have 2 separate hooks for preamble and mainfile ASTs, as they are produced async.

[sammccall]

clang-tools-extra/clangd/ClangdServer.cpp

237

Now we're creating two ASTListeners for each version of the file seen, but most won't be parsed twice (and some won't even be parsed once).

This seems like it's going to be a weird wart for e.g. hooks that expect to do something interesting when a file finishes, or those that allocate something heavyweight.

I'd suggest sinking the ModuleSet into ParsedAST::build etc, or at least moving the astHooks() calls into TUScheduler. (And documenting that astHooks() should be threadsafe)

Otherwise, need to document that sometimes hooks are created and then never used.

clang-tools-extra/clangd/Compiler.h

60

I think having these in ParseInputs is conceptually confusing and shared_ptr is risky.
For example we copy inputs into ASTWorker::FileInputs, which extends the lifetime of the listeners in a way that seems undesirable.

I think these listeners should be clearly either:

• bound to the lifetime of the ParsedAST (and therefore owned by it)
• bound to the *creation* of the ParsedAST (and therefore strictly scoped within ParsedAST::build)

You could consider the ModuleSet to be part of ParseInputs, though...

clang-tools-extra/clangd/Diagnostics.h

147

ArrayRef<shared_ptr<ASTListener>> seems like we're leaking a lot. And even mentioning AST hooks seems dubious layering-wise.
Can we use a std::function like LevelAdjuster?

clang-tools-extra/clangd/FeatureModule.h

101

nit: this describes the interaction more but not what it's for.
Maybe

/// Extension point that allows modules to observe and modify an AST build.
/// One instance is created each time clangd produces a ParsedAST or PrecompiledPreamble.
/// For a given instance, lifecycle methods are always called on a single thread.

113

the "hooks" name is still used here and throughout

[kadircet]

• Pass FeatureModuleSet rather than astListeners in ParseInputs.
• Create listeners only when building ASTs.
• Use a callback function in StoreDiags to notify about diagnostics.

[sammccall]

clang-tools-extra/clangd/FeatureModule.h

106

comment: listeners are destroyed once the AST is built.

180

this seems like a slightly weird thing to have in FeatureModule.h, I'd be tempted to inline it in the callsites instead, but up to you.

[kadircet]

• Inline helper to call-sites
• Add comments about destruction of ast listeners.

clang-tools-extra/clangd/ClangdServer.h

 #include "support/ThreadsafeFS.h"
 #include "clang/Tooling/CompilationDatabase.h"
 #include "clang/Tooling/Core/Replacement.h"
 #include "llvm/ADT/FunctionExtras.h"
 #include "llvm/ADT/IntrusiveRefCntPtr.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/StringRef.h"
 #include <functional>
+#include <memory>
 #include <string>
 #include <type_traits>
 #include <utility>
 #include <vector>
 
 namespace clang {
 namespace clangd {
 /// Manages a collection of source files and derived data (ASTs, indexes),

clang-tools-extra/clangd/ClangdServer.cpp

void ClangdServer::addDocument(PathRef File, llvm::StringRef Contents,
   ParseInputs Inputs;
   Inputs.TFS = &TFS;
   Inputs.Contents = std::string(Contents);
   Inputs.Version = std::move(ActualVersion);
   Inputs.ForceRebuild = ForceRebuild;
   Inputs.Opts = std::move(Opts);
   Inputs.Index = Index;
   Inputs.ClangTidyProvider = ClangTidyProvider;
+  if (FeatureModules) {
+    for (auto &Mod : *FeatureModules)
+      Inputs.ASTHooks.emplace_back(Mod.astHooks());
+  }
   bool NewFile = WorkScheduler->update(File, Inputs, WantDiags);
   // If we loaded Foo.h, we want to make sure Foo.cpp is indexed.
   if (NewFile && BackgroundIdx)
     BackgroundIdx->boostRelated(File);
 }
 
 void ClangdServer::reparseOpenFilesIfNeeded(
     llvm::function_ref<bool(llvm::StringRef File)> Filter) {

clang-tools-extra/clangd/Compiler.h

 // Most callers will use this through Preamble/ParsedAST, but some features like
 // CodeComplete run their own compile actions that share these low-level pieces.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_COMPILER_H
 #define LLVM_CLANG_TOOLS_EXTRA_CLANGD_COMPILER_H
 
+#include "FeatureModule.h"
 #include "GlobalCompilationDatabase.h"
 #include "TidyProvider.h"
 #include "index/Index.h"
 #include "support/ThreadsafeFS.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/PrecompiledPreamble.h"
 #include "clang/Tooling/CompilationDatabase.h"
+#include <memory>
+#include <vector>
 
 namespace clang {
 namespace clangd {
 
 class IgnoreDiagnostics : public DiagnosticConsumer {
 public:
   static void log(DiagnosticsEngine::Level DiagLevel,
                   const clang::Diagnostic &Info);
struct ParseInputs {
   std::string Version = "null";
   // Prevent reuse of the cached preamble/AST. Slow! Useful to workaround
   // clangd's assumption that missing header files will stay missing.
   bool ForceRebuild = false;
   // Used to recover from diagnostics (e.g. find missing includes for symbol).
   const SymbolIndex *Index = nullptr;
   ParseOptions Opts = ParseOptions();
   TidyProviderRef ClangTidyProvider = {};
+  std::vector<std::shared_ptr<FeatureModule::ASTListener>> ASTHooks = {};
 };
 
 /// Builds compiler invocation that could be used to build AST or preamble.
 std::unique_ptr<CompilerInvocation>
 buildCompilerInvocation(const ParseInputs &Inputs, clang::DiagnosticConsumer &D,
                         std::vector<std::string> *CC1Args = nullptr);
 
 /// Creates a compiler instance, configured so that:

clang-tools-extra/clangd/Diagnostics.h

 //===--- Diagnostics.h -------------------------------------------*- C++-*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_DIAGNOSTICS_H
 #define LLVM_CLANG_TOOLS_EXTRA_CLANGD_DIAGNOSTICS_H
 
+#include "FeatureModule.h"
 #include "Protocol.h"
 #include "support/Path.h"
 #include "clang/Basic/Diagnostic.h"
 #include "clang/Basic/LangOptions.h"
 #include "clang/Basic/SourceLocation.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/DenseSet.h"
 #include "llvm/ADT/None.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/StringSet.h"
 #include "llvm/Support/SourceMgr.h"
 #include <cassert>
+#include <memory>
 #include <string>
 
 namespace clang {
 namespace tidy {
 class ClangTidyContext;
 } // namespace tidy
 namespace clangd {
 
public:
   using LevelAdjuster = std::function<DiagnosticsEngine::Level(
       DiagnosticsEngine::Level, const clang::Diagnostic &)>;
   /// If set, possibly adds fixes for diagnostics using \p Fixer.
   void contributeFixes(DiagFixer Fixer) { this->Fixer = Fixer; }
   /// If set, this allows the client of this class to adjust the level of
   /// diagnostics, such as promoting warnings to errors, or ignoring
   /// diagnostics.
   void setLevelAdjuster(LevelAdjuster Adjuster) { this->Adjuster = Adjuster; }
+  void setASTHooks(
+      llvm::ArrayRef<std::shared_ptr<FeatureModule::ASTListener>> Hooks) {
+    this->Hooks = Hooks;
+  }
 
 private:
   void flushLastDiag();
 
   DiagFixer Fixer = nullptr;
   LevelAdjuster Adjuster = nullptr;
   std::vector<Diag> Output;
   llvm::Optional<LangOptions> LangOpts;
   llvm::Optional<Diag> LastDiag;
   llvm::Optional<FullSourceLoc> LastDiagLoc; // Valid only when LastDiag is set.
   bool LastDiagOriginallyError = false;      // Valid only when LastDiag is set.
   SourceManager *OrigSrcMgr = nullptr;
 
   llvm::DenseSet<std::pair<unsigned, unsigned>> IncludedErrorLocations;
   bool LastPrimaryDiagnosticWasSuppressed = false;
+  llvm::ArrayRef<std::shared_ptr<FeatureModule::ASTListener>> Hooks;
 };
 
 /// Determine whether a (non-clang-tidy) diagnostic is suppressed by config.
 bool isBuiltinDiagnosticSuppressed(unsigned ID,
                                    const llvm::StringSet<> &Suppressed);
 /// Take a user-specified diagnostic code, and convert it to a normalized form
 /// stored in the config and consumed by isBuiltinDiagnosticsSuppressed.
 ///
 /// (This strips err_ and -W prefix so we can match with or without them.)
 llvm::StringRef normalizeSuppressedCode(llvm::StringRef);
 
 } // namespace clangd
 } // namespace clang
 
 #endif // LLVM_CLANG_TOOLS_EXTRA_CLANGD_DIAGNOSTICS_H

clang-tools-extra/clangd/Diagnostics.cpp

if (!isNote(DiagLevel)) {
 
     if (!Info.getFixItHints().empty())
       AddFix(true /* try to invent a message instead of repeating the diag */);
     if (Fixer) {
       auto ExtraFixes = Fixer(DiagLevel, Info);
       LastDiag->Fixes.insert(LastDiag->Fixes.end(), ExtraFixes.begin(),
                              ExtraFixes.end());
     }
+    for (auto &Hook : Hooks)
+      Hook->sawDiagnostic(Info, *LastDiag);
   } else {
     // Handle a note to an existing diagnostic.
 
     // If a diagnostic was suppressed due to the suppression filter,
     // also suppress notes associated with it.
     if (LastPrimaryDiagnosticWasSuppressed) {
       return;
     }

clang-tools-extra/clangd/FeatureModule.h

 //===--- FeatureModule.h - Plugging features into clangd ----------*-C++-*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_FEATUREMODULE_H
 #define LLVM_CLANG_TOOLS_EXTRA_CLANGD_FEATUREMODULE_H
 
 #include "support/Function.h"
 #include "support/Threading.h"
+#include "clang/Basic/Diagnostic.h"
 #include "llvm/ADT/FunctionExtras.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/JSON.h"
 #include <memory>
 #include <type_traits>
 #include <vector>
 
 namespace clang {
 namespace clangd {
+struct Diag;
 class LSPBinder;
 class SymbolIndex;
 class ThreadsafeFS;
 class TUScheduler;
 class Tweak;
 
 /// A FeatureModule contributes a vertical feature to clangd.
 ///
public:
   /// FeatureModules should go idle quickly if stop() has been called.
   /// Called by the server when shutting down, and also by tests.
   virtual bool blockUntilIdle(Deadline) { return true; }
 
   /// Tweaks implemented by this module. Can be called asynchronously when
   /// enumerating or applying code actions.
   virtual void contributeTweaks(std::vector<std::unique_ptr<Tweak>> &Out) {}
 
+  /// Various hooks that can be invoked by ASTWorker thread while building an
+  /// AST. These hooks are always called from a single thread, so
+  /// implementations don't need to be thread-safe.
+  struct ASTListener {
+    virtual ~ASTListener() = default;
+
+    /// Called everytime a diagnostic is encountered. Modules can use this
+    /// modify the final diagnostic, or store some information to surface code
+    /// actions later on.
+    virtual void sawDiagnostic(const clang::Diagnostic &, clangd::Diag &) {}
+  };
+  /// Can be called asynchronously before building an AST.
+  virtual std::unique_ptr<ASTListener> astHooks() { return nullptr; }
+
 protected:
   /// Accessors for modules to access shared server facilities they depend on.
   Facilities &facilities();
   /// The scheduler is used to run tasks on worker threads and access ASTs.
   TUScheduler &scheduler() { return facilities().Scheduler; }
   /// The index is used to get information about the whole codebase.
   const SymbolIndex *index() { return facilities().Index; }
   /// The filesystem is used to read source files on disk.
public:
   }
   template <typename Mod> const Mod *get() const {
     return const_cast<FeatureModuleSet *>(this)->get<Mod>();
   }
 };
 
 template <typename Mod> int FeatureModuleSet::ID<Mod>::Key;
 
 } // namespace clangd
 } // namespace clang
 #endif

clang-tools-extra/clangd/ParsedAST.cpp

ParsedAST::build(llvm::StringRef Filename, const ParseInputs &Inputs,
   const PrecompiledPreamble *PreamblePCH =
       Preamble ? &Preamble->Preamble : nullptr;
 
   // This is on-by-default in windows to allow parsing SDK headers, but it
   // breaks many features. Disable it for the main-file (not preamble).
   CI->getLangOpts()->DelayedTemplateParsing = false;
 
   StoreDiags ASTDiags;
+  ASTDiags.setASTHooks(Inputs.ASTHooks);
 
   llvm::Optional<PreamblePatch> Patch;
   bool PreserveDiags = true;
   if (Preamble) {
     Patch = PreamblePatch::create(Filename, Inputs, *Preamble);
     Patch->apply(*CI);
     PreserveDiags = Patch->preserveDiagnostics();
   }
llvm::erase_if(CTChecks, [&](const auto &Check) {
       return !Check->isLanguageVersionSupported(CTContext->getLangOpts());
     });
     Preprocessor *PP = &Clang->getPreprocessor();
     for (const auto &Check : CTChecks) {
       Check->registerPPCallbacks(Clang->getSourceManager(), PP, PP);
       Check->registerMatchers(&CTFinder);
     }
 
-    const Config& Cfg = Config::current();
+    const Config &Cfg = Config::current();
     ASTDiags.setLevelAdjuster([&](DiagnosticsEngine::Level DiagLevel,
                                   const clang::Diagnostic &Info) {
       if (Cfg.Diagnostics.SuppressAll ||
           isBuiltinDiagnosticSuppressed(Info.getID(), Cfg.Diagnostics.Suppress))
         return DiagnosticsEngine::Ignored;
       if (!CTChecks.empty()) {
         std::string CheckName = CTContext->getCheckName(Info.getID());
         bool IsClangTidyDiag = !CheckName.empty();

clang-tools-extra/clangd/Preamble.cpp

buildPreamble(PathRef FileName, CompilerInvocation CI,
   // without those.
   auto ContentsBuffer =
       llvm::MemoryBuffer::getMemBuffer(Inputs.Contents, FileName);
   auto Bounds = ComputePreambleBounds(*CI.getLangOpts(), *ContentsBuffer, 0);
 
   trace::Span Tracer("BuildPreamble");
   SPAN_ATTACH(Tracer, "File", FileName);
   StoreDiags PreambleDiagnostics;
+  PreambleDiagnostics.setASTHooks(Inputs.ASTHooks);
   llvm::IntrusiveRefCntPtr<DiagnosticsEngine> PreambleDiagsEngine =
       CompilerInstance::createDiagnostics(&CI.getDiagnosticOpts(),
                                           &PreambleDiagnostics, false);
 
   // Skip function bodies when building the preamble to speed up building
   // the preamble and make it smaller.
   assert(!CI.getFrontendOpts().SkipFunctionBodies);
   CI.getFrontendOpts().SkipFunctionBodies = true;

clang-tools-extra/clangd/TUScheduler.cpp

auto Task = [=]() mutable {
     }
 
     log("ASTWorker building file {0} version {1} with command {2}\n[{3}]\n{4}",
         FileName, Inputs.Version, Inputs.CompileCommand.Heuristic,
         Inputs.CompileCommand.Directory,
         printArgv(Inputs.CompileCommand.CommandLine));
 
     StoreDiags CompilerInvocationDiagConsumer;
+    CompilerInvocationDiagConsumer.setASTHooks(Inputs.ASTHooks);
     std::vector<std::string> CC1Args;
     std::unique_ptr<CompilerInvocation> Invocation = buildCompilerInvocation(
         Inputs, CompilerInvocationDiagConsumer, &CC1Args);
     // Log cc1 args even (especially!) if creating invocation failed.
     if (!CC1Args.empty())
       vlog("Driver produced command: cc1 {0}", printArgv(CC1Args));
     std::vector<Diag> CompilerInvocationDiags =
         CompilerInvocationDiagConsumer.take();
static constexpr trace::Metric ASTAccessForRead(
       "ast_access_read", trace::Metric::Counter, "result");
   auto Task = [=, Action = std::move(Action)]() mutable {
     if (auto Reason = isCancelled())
       return Action(llvm::make_error<CancelledError>(Reason));
     llvm::Optional<std::unique_ptr<ParsedAST>> AST =
         IdleASTs.take(this, &ASTAccessForRead);
     if (!AST) {
       StoreDiags CompilerInvocationDiagConsumer;
+      CompilerInvocationDiagConsumer.setASTHooks(FileInputs.ASTHooks);
       std::unique_ptr<CompilerInvocation> Invocation =
           buildCompilerInvocation(FileInputs, CompilerInvocationDiagConsumer);
       // Try rebuilding the AST.
       vlog("ASTWorker rebuilding evicted AST to run {0}: {1} version {2}", Name,
            FileName, FileInputs.Version);
       // FIXME: We might need to build a patched ast once preamble thread starts
       // running async. Currently getPossiblyStalePreamble below will always
       // return a compatible preamble as ASTWorker::update blocks.

clang-tools-extra/clangd/tool/Check.cpp

 #include "clang/Basic/DiagnosticIDs.h"
 #include "clang/Format/Format.h"
 #include "clang/Frontend/CompilerInvocation.h"
 #include "clang/Tooling/CompilationDatabase.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/Support/Path.h"
+#include <memory>
 
 namespace clang {
 namespace clangd {
 namespace {
 
 // Print (and count) the error-level diagnostics (warnings are ignored).
 unsigned showErrors(llvm::ArrayRef<Diag> Diags) {
   unsigned ErrCount = 0;
if (auto TrueCmd = CDB->getCompileCommand(File)) {
       Cmd = CDB->getFallbackCommand(File);
       log("Generic fallback command is: {0}", printArgv(Cmd.CommandLine));
     }
 
     return true;
   }
 
   // Prepare inputs and build CompilerInvocation (parsed compile command).
-  bool buildInvocation(const ThreadsafeFS &TFS,
-                       llvm::Optional<std::string> Contents) {
-    StoreDiags CaptureInvocationDiags;
+  bool buildInvocation(
+      const ThreadsafeFS &TFS, llvm::Optional<std::string> Contents,
+      llvm::ArrayRef<std::shared_ptr<FeatureModule::ASTListener>> Hooks) {
     std::vector<std::string> CC1Args;
     Inputs.CompileCommand = Cmd;
     Inputs.TFS = &TFS;
     Inputs.ClangTidyProvider = Opts.ClangTidyProvider;
+    Inputs.ASTHooks = Hooks;
+
+    StoreDiags CaptureInvocationDiags;
+    CaptureInvocationDiags.setASTHooks(Hooks);
     if (Contents.hasValue()) {
       Inputs.Contents = *Contents;
       log("Imaginary source file contents:\n{0}", Inputs.Contents);
     } else {
       if (auto Contents = TFS.view(llvm::None)->getBufferForFile(File)) {
         Inputs.Contents = Contents->get()->getBuffer().str();
       } else {
         elog("Couldn't read {0}: {1}", File, Contents.getError().message());
if (File.empty()) {
     )cpp";
   }
   log("Testing on source file {0}", File);
 
   auto ContextProvider = ClangdServer::createConfiguredContextProvider(
       Opts.ConfigProvider, nullptr);
   WithContext Ctx(ContextProvider(""));
   Checker C(File, Opts);
-  if (!C.buildCommand(TFS) || !C.buildInvocation(TFS, Contents) ||
+  std::vector<std::shared_ptr<FeatureModule::ASTListener>> Hooks;
+  if (Opts.FeatureModules) {
+    for (auto &M : *Opts.FeatureModules) {
+      if (auto H = M.astHooks())
+        Hooks.push_back(std::move(H));
+    }
+  }
+
+  if (!C.buildCommand(TFS) || !C.buildInvocation(TFS, Contents, Hooks) ||
       !C.buildAST())
     return false;
   C.testLocationFeatures();
 
   log("All checks completed, {0} errors", C.ErrCount);
   return C.ErrCount == 0;
 }
 
 } // namespace clangd
 } // namespace clang

clang-tools-extra/clangd/unittests/ClangdLSPServerTests.cpp

TEST_F(LSPTest, FeatureModulesThreadingTest) {
   EXPECT_EQ(3, FeatureModules.get<AsyncCounter>()->getSync());
   // Throw some work on the queue to make sure shutdown blocks on it.
   Client.notify("increment", nullptr);
   Client.notify("increment", nullptr);
   Client.notify("increment", nullptr);
   // And immediately shut down. FeatureModule destructor verifies we blocked.
 }
 
+TEST_F(LSPTest, DiagModuleTest) {
+  static constexpr llvm::StringLiteral DiagMsg = "DiagMsg";
+  class DiagModule final : public FeatureModule {
+    struct DiagHooks : public ASTListener {
+      void sawDiagnostic(const clang::Diagnostic &, clangd::Diag &D) override {
+        D.Message = DiagMsg.str();
+      }
+    };
+
+  public:
+    std::unique_ptr<ASTListener> astHooks() override {
+      return std::make_unique<DiagHooks>();
+    }
+  };
+  FeatureModules.add(std::make_unique<DiagModule>());
+
+  auto &Client = start();
+  Client.didOpen("foo.cpp", "test;");
+  EXPECT_THAT(Client.diagnostics("foo.cpp"),
+              llvm::ValueIs(testing::ElementsAre(DiagMessage(DiagMsg))));
+}
 } // namespace
 } // namespace clangd
 } // namespace clang

clang-tools-extra/clangd/unittests/DiagnosticsTests.cpp

TEST(ToLSPDiag, RangeIsInMain) {
   D.InsideMainFile = false;
   N.InsideMainFile = true;
   toLSPDiags(D, {}, Opts,
              [&](clangd::Diagnostic LSPDiag, ArrayRef<clangd::Fix>) {
                EXPECT_EQ(LSPDiag.range, N.Range);
              });
 }
 
+TEST(ParsedASTTest, ModuleSawDiag) {
+  static constexpr const llvm::StringLiteral KDiagMsg = "StampedDiag";
+  struct Hooks : public FeatureModule::ASTListener {
+    void sawDiagnostic(const clang::Diagnostic &Info,
+                       clangd::Diag &Diag) override {
+      Diag.Message = KDiagMsg.str();
+    }
+  };
+
+  Annotations Code("[[test]]; /* error-ok */");
+  TestTU TU;
+  TU.Code = Code.code().str();
+  TU.Hooks.emplace_back(new Hooks);
+
+  auto AST = TU.build();
+  EXPECT_THAT(*AST.getDiagnostics(),
+              testing::Contains(Diag(Code.range(), KDiagMsg.str())));
+}
 } // namespace
 } // namespace clangd
 } // namespace clang

clang-tools-extra/clangd/unittests/TestTU.h

 #include "../TidyProvider.h"
 #include "Compiler.h"
 #include "ParsedAST.h"
 #include "TestFS.h"
 #include "index/Index.h"
 #include "support/Path.h"
 #include "llvm/ADT/StringMap.h"
 #include "gtest/gtest.h"
+#include <memory>
 #include <string>
 #include <utility>
 #include <vector>
 
 namespace clang {
 namespace clangd {
 
 struct TestTU {
struct TestTU {
   // disk and later read back.
   // FIXME: Change the way reading/writing modules work to allow us to keep them
   // in memory across multiple clang invocations, at least in tests, to
   // eliminate the need for real file system here.
   // Please avoid using this for things other than implicit modules. The plan is
   // to eliminate this option some day.
   bool OverlayRealFileSystemForModules = false;
 
+  std::vector<std::shared_ptr<FeatureModule::ASTListener>> Hooks;
+
   // By default, build() will report Error diagnostics as GTest errors.
   // Suppress this behavior by adding an 'error-ok' comment to the code.
   // The result will always have getDiagnostics() populated.
   ParsedAST build() const;
   std::shared_ptr<const PreambleData>
   preamble(PreambleParsedCallback PreambleCallback = nullptr) const;
   ParseInputs inputs(MockFS &FS) const;
   SymbolSlab headerSymbols() const;

clang-tools-extra/clangd/unittests/TestTU.cpp

ParseInputs TestTU::inputs(MockFS &FS) const {
   std::string ThunkContents = "#import \"" + FullHeaderName + "\"\n";
 
   FS.Files = AdditionalFiles;
   FS.Files[FullFilename] = Code;
   FS.Files[FullHeaderName] = HeaderCode;
   FS.Files[ImportThunk] = ThunkContents;
 
   ParseInputs Inputs;
+  Inputs.ASTHooks = Hooks;
   auto &Argv = Inputs.CompileCommand.CommandLine;
   Argv = {"clang"};
   // FIXME: this shouldn't need to be conditional, but it breaks a
   // GoToDefinition test for some reason (getMacroArgExpandedLocation fails).
   if (!HeaderCode.empty()) {
     Argv.push_back("-include");
     Argv.push_back(ImplicitHeaderGuard ? ImportThunk : FullHeaderName);
     // ms-compatibility changes the meaning of #import.
TestTU::preamble(PreambleParsedCallback PreambleCallback) const {
   return clang::clangd::buildPreamble(testPath(Filename), *CI, Inputs,
                                       /*StoreInMemory=*/true, PreambleCallback);
 }
 
 ParsedAST TestTU::build() const {
   MockFS FS;
   auto Inputs = inputs(FS);
   StoreDiags Diags;
+  Diags.setASTHooks(Inputs.ASTHooks);
   auto CI = buildCompilerInvocation(Inputs, Diags);
   assert(CI && "Failed to build compilation invocation.");
   if (OverlayRealFileSystemForModules)
     initializeModuleCache(*CI);
   auto ModuleCacheDeleter = llvm::make_scope_exit(
       std::bind(deleteModuleCache, CI->getHeaderSearchOpts().ModuleCachePath));
 
   auto Preamble = clang::clangd::buildPreamble(testPath(Filename), *CI, Inputs,