[clangd] Interactive AST matchers with #pragma clang query

[sammccall]

Summary

The idea: you can type in AST matchers, and clangd shows you matches
within the document.

Syntax is like:

#pragma clang query varDecl().bind("var")

Features:

• matches are shown within the document as diagnostics
• find-references on the pragma will produce a list of matches
• errors within matchers are shown as diagnostics
• code completion is available within matchers

Demo: https://asciinema.org/a/O63vj8af7c8RcgcCqKvhH2TRn

There are some rough edges here, e.g. the diagnostics are warnings and
have no LSP diagnostic code, the diagnostic text for matcher errors
isn't ideal etc.
I think this is a good start as-is and we can work out if we want to
improve these cases later, if this feature ends up being useful.

[sammccall]

Add demo link

[sammccall]

@kadircet this is a slightly silly feature that I put together on vacation

I'd like your call on whether this is something we should have (take a look at the demo). If so, feel free to hand off the review.

It's definitely a power-user feature and nigh undiscoverable. I'd plan to write some docs, but even so I'm not sure how much use it will get.

[Endill]

This is an interesting feature!
There's a related #pragma clang __debug dump, that dumps name lookup result or a bit of AST for expression you pass to it (https://clang.llvm.org/docs/LanguageExtensions.html#debugging-the-compiler). I think clangd might be interested to support that pragma as well.

clang-tools-extra/clangd/ASTMatchers.h

+//===--- ASTMatchers.h - Query the current file with matchers ----*- C++-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// `#pragma clang query` lets users run AST matchers against the current file.
+// It is named after the clang-query tool and serves a similar purpose.
+//
+// Examples:
+//   // Marks all function parameters
+//   #pragma clang query parmVarDecl()
+//
+//   // Marks the types of function parameters
+//   #pragma clang query parmVarDecl(hasTypeLoc().bind("type"))
+//
+// The results are exposed in two ways:
+//  - matches are shown as warning diagnostics
+//  - find-references on the pragma will show the matches
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_ASTMATCHERS_H
+#define LLVM_CLANG_TOOLS_EXTRA_CLANGD_ASTMATCHERS_H
+
+#include "clang/AST/ASTContext.h"
+#include "clang/ASTMatchers/Dynamic/VariantValue.h"
+#include "clang/Basic/Diagnostic.h"
+#include "clang/Basic/SourceLocation.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/StringRef.h"
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+namespace clang::clangd {
+
+/// A parsed `#pragma clang query <matcher>` directive.
+/// These define AST matchers which can be evaluated against the document.
+struct MatcherPragma {
+  /// The compiled matcher.
+  /// Always valid: in case of errors we produce no MatcherPragma at all.
+  ast_matchers::dynamic::DynTypedMatcher Matcher;
+  /// Closed interval of (1-based) line numbers covered by the pragma.
+  std::pair<unsigned, unsigned> LineRange;
+
+  /// Find ranges of code in the matcher matches, along with their bound names.
+  std::vector<std::pair<llvm::StringRef, SourceRange>>
+  match(ASTContext &) const;
+
+  // Registers a handler to parse `#pragma clang query` directives.
+  //
+  // The AST matchers are written into *Out, if provided.
+  // Problems with the matcher specs are reported as diagnostics.
+  static void parse(std::vector<MatcherPragma> *Out, CompilerInstance &);
+
+  // Issues diagnostics marking the locations of all matchers in the list.
+  static void diagnose(llvm::ArrayRef<MatcherPragma>, ASTContext &,
+                       DiagnosticsEngine &);
+};
+
+} // namespace clang::clangd
+
+#endif

clang-tools-extra/clangd/ASTMatchers.cpp

+//===--- ASTMatchers.cpp --------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "ASTMatchers.h"
+#include "SourceCode.h"
+#include "clang/AST/ASTContext.h"
+#include "clang/ASTMatchers/ASTMatchFinder.h"
+#include "clang/ASTMatchers/ASTMatchersInternal.h"
+#include "clang/ASTMatchers/Dynamic/Diagnostics.h"
+#include "clang/ASTMatchers/Dynamic/Parser.h"
+#include "clang/Basic/SourceLocation.h"
+#include "clang/Basic/TokenKinds.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Lex/Pragma.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Sema/CodeCompleteConsumer.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/raw_ostream.h"
+#include <algorithm>
+#include <iterator>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+namespace clang::clangd {
+namespace {
+
+// The matcher parser reads from a simple buffer (StringRef).
+// But we want to report errors in terms of SourceLocation of the original file.
+// This struct gathers the text, but retains enough information to translate
+// offsets into it back into SourceLocations.
+struct TextFromPragmaTokens {
+  std::string Text;
+  std::vector<std::pair</*Offset*/ unsigned, SourceLocation>> Mapping;
+
+  TextFromPragmaTokens(Preprocessor &PP, SourceLocation Initial) {
+    Mapping.emplace_back(0, Initial);
+    clang::Token Tok;
+    while (1) {
+      PP.LexUnexpandedToken(Tok);
+      if (Tok.is(tok::eod))
+        break;
+      if (Tok.is(tok::code_completion)) {
+        PP.setCodeCompletionReached();
+        break;
+      }
+      Text.push_back(' ');
+      Mapping.emplace_back(Text.size(), Tok.getLocation());
+      Text.append(PP.getSourceManager().getCharacterData(Tok.getLocation()),
+                  Tok.getLength());
+    }
+  }
+
+  // Returns the location corresponding to an offset in the buffer.
+  SourceLocation loc(unsigned Offset) const {
+    auto Result = prevTokenImpl(Offset);
+    return Result.first.getLocWithOffset(Result.second);
+  }
+
+  // Returns the location of the last token <= Offset.
+  SourceLocation prevToken(unsigned Offset) const {
+    return prevTokenImpl(Offset).first;
+  }
+
+  /// Returns the location of the last token.
+  SourceLocation end() const { return Mapping.back().second; }
+
+private:
+  // Returns the token that Offset is part of, and the position with it.
+  std::pair<SourceLocation, unsigned> prevTokenImpl(unsigned Offset) const {
+    // Text:   " tok1 tok2"
+    // Markers: ^^    ^
+    if (Offset == 0)
+      return {Mapping.begin()->second, 0};
+    auto Entry = std::prev(
+        std::partition_point(Mapping.begin(), Mapping.end(), [&](auto &Entry) {
+          return Entry.first <= Offset;
+        }));
+    return {Entry->second, Offset - Entry->first};
+  }
+};
+
+class ASTMatcherParser : public PragmaHandler {
+  std::vector<MatcherPragma> *Out;
+  CompilerInstance &Clang;
+  unsigned DiagMatcherProblem;
+
+public:
+  ASTMatcherParser(std::vector<MatcherPragma> *Out, CompilerInstance &Clang)
+      : PragmaHandler("query"), Out(Out), Clang(Clang),
+        DiagMatcherProblem(Clang.getDiagnostics().getCustomDiagID(
+            DiagnosticsEngine::Error, "Bad matcher: %0")) {}
+
+  void HandlePragma(Preprocessor &PP, PragmaIntroducer PI,
+                    Token &FirstToken) override {
+    auto &SM = PP.getSourceManager();
+    // clangd only respects main-file pragmas (completion+preamble+main parse)
+    if (!isInsideMainFile(PI.Loc, SM))
+      return;
+    TextFromPragmaTokens Text(PP, FirstToken.getLocation());
+    if (PP.isCodeCompletionReached())
+      return handleCompletion(Text.Text);
+    if (!Out)
+      return;
+
+    ast_matchers::dynamic::Diagnostics ParserDiags;
+    llvm::StringRef MatcherText = Text.Text;
+    bool HasBinds = MatcherText.contains(". bind (");
+    if (auto Matcher = ast_matchers::dynamic::Parser::parseMatcherExpression(
+            MatcherText, &ParserDiags)) {
+      // Add "root" binding if there are no binds.
+      if (!HasBinds) {
+        std::string ID =
+            "query:" + std::to_string(SM.getSpellingLineNumber(PI.Loc));
+        if (auto Bound = Matcher->tryBind(ID))
+          Matcher = std::move(Bound);
+      }
+      Out->push_back(
+          MatcherPragma{std::move(*Matcher),
+                        std::make_pair(SM.getSpellingLineNumber(PI.Loc),
+                                       SM.getSpellingLineNumber(Text.end()))});
+    }
+    for (const auto &ParserError : ParserDiags.errors()) {
+      for (const auto &Message : ParserError.Messages) {
+        std::string Msg;
+        {
+          llvm::raw_string_ostream OS(Msg);
+          // FIXME: Only show the message of diag in question!
+          ParserDiags.printToStream(OS);
+        }
+        SourceRange Range(Text.loc(Message.Range.Start.Column - 1),
+                          Text.prevToken(Message.Range.End.Column - 1));
+        Clang.getDiagnostics().Report(Range.getBegin(), DiagMatcherProblem)
+            << Range << Msg;
+      }
+    }
+  }
+
+  void handleCompletion(llvm::StringRef Text) {
+    std::vector<CodeCompletionResult> Results;
+    auto Copy = [&](llvm::StringRef Text) {
+      return Clang.getCodeCompletionConsumer().getAllocator().CopyString(Text);
+    };
+
+    for (auto &Completion :
+         ast_matchers::dynamic::Parser::completeExpression(Text, Text.size())) {
+      clang::CodeCompletionBuilder Builder(
+          Clang.getCodeCompletionConsumer().getAllocator(),
+          Clang.getCodeCompletionConsumer().getCodeCompletionTUInfo());
+      auto [Return, Signature] =
+          llvm::StringRef(Completion.MatcherDecl).split(' ');
+      // The plain Completion.TypedText is a prefix string like "varDecl(".
+      // This is not ideal for LSP, parse the "MatcherDecl" string instead.
+
+      // Matcher<Decl> varDecl(Matcher<VarDecl>...)
+      if (Return.starts_with("Matcher<") && Return.ends_with(">") &&
+          !Signature.empty()) {
+        Builder.AddResultTypeChunk(Copy(Return));
+        if (Signature.ends_with(")")) {
+          auto [Name, Args] = Signature.drop_back().split("(");
+          Builder.AddTypedTextChunk(Copy(Name));
+          Builder.AddChunk(CodeCompletionString::CK_LeftParen, "(");
+          // FIXME: we could add individual arg placeholders.
+          Builder.AddPlaceholderChunk(Copy(Args));
+          Builder.AddChunk(CodeCompletionString::CK_RightParen, ")");
+        } else {
+          Builder.AddTypedTextChunk(Copy(Completion.TypedText));
+        }
+      } else if (Completion.MatcherDecl == "bind") {
+        Builder.AddTypedTextChunk("bind");
+        Builder.AddChunk(CodeCompletionString::CK_LeftParen, "(");
+        Builder.AddChunk(CodeCompletionString::CK_Text, "\"");
+        Builder.AddPlaceholderChunk("name");
+        Builder.AddChunk(CodeCompletionString::CK_Text, "\"");
+        Builder.AddChunk(CodeCompletionString::CK_RightParen, ")");
+      } else {
+        // Generic fallback, we don't expect to get here much.
+        Builder.AddTypedTextChunk(Copy(Completion.TypedText));
+      }
+      Results.push_back(Builder.TakeString());
+    }
+
+    Clang.getCodeCompletionConsumer().ProcessCodeCompleteResults(
+        Clang.getSema(), CodeCompletionContext::CCC_Other, Results.data(),
+        Results.size());
+  }
+};
+
+} // namespace
+
+void MatcherPragma::parse(std::vector<MatcherPragma> *Out,
+                          CompilerInstance &Clang) {
+  Clang.getPreprocessor().AddPragmaHandler("clang",
+                                           new ASTMatcherParser(Out, Clang));
+}
+
+std::vector<std::pair<llvm::StringRef, SourceRange>>
+MatcherPragma::match(ASTContext &Ctx) const {
+  std::vector<std::pair<llvm::StringRef, SourceRange>> Result;
+  for (const auto &Match : ast_matchers::matchDynamic(Matcher, Ctx))
+    for (const auto &KV : Match.getMap())
+      Result.emplace_back(KV.first, KV.second.getSourceRange());
+  return Result;
+}
+
+void MatcherPragma::diagnose(llvm::ArrayRef<MatcherPragma> Matchers,
+                             ASTContext &AST, DiagnosticsEngine &Diags) {
+  ast_matchers::MatchFinder Finder;
+  class ReportMatch : public ast_matchers::MatchFinder::MatchCallback {
+    DiagnosticsEngine &Diags;
+    unsigned DiagBindsHere;
+
+  public:
+    ReportMatch(DiagnosticsEngine &Diags)
+        : Diags(Diags),
+          // FIXME: somehow propagate a sensible LSP severity/source/code.
+          // For now, this is a warning, as we expect notes to have parents.
+          DiagBindsHere(Diags.getCustomDiagID(DiagnosticsEngine::Warning,
+                                              "%0 matches")) {}
+
+    void run(const ast_matchers::MatchFinder::MatchResult &Result) override {
+      for (const auto &E : Result.Nodes.getMap()) {
+        E.second.dump(llvm::errs() << E.first << ": ", *Result.Context);
+        Diags.Report(E.second.getSourceRange().getBegin(), DiagBindsHere)
+            << E.second.getSourceRange() << E.first;
+      }
+    }
+  } Reporter(Diags);
+
+  for (const auto &M : Matchers)
+    Finder.addDynamicMatcher(M.Matcher, &Reporter);
+  Finder.matchAST(AST);
+}
+
+} // namespace clang::clangd

clang-tools-extra/clangd/CMakeLists.txt

  set_source_files_properties(CompileCommands.cpp PROPERTIES COMPILE_FLAGS -wd4130) # disables C4130: logical operation on address of string constant
 endif()
 
 include_directories(BEFORE "${CMAKE_CURRENT_BINARY_DIR}/../clang-tidy")
 include_directories(BEFORE "${CMAKE_CURRENT_SOURCE_DIR}/../include-cleaner/include")
 
 add_clang_library(clangDaemon
   AST.cpp
+  ASTMatchers.cpp
   ASTSignals.cpp
   ClangdLSPServer.cpp
   ClangdServer.cpp
   CodeComplete.cpp
   CodeCompletionStrings.cpp
   CollectMacros.cpp
   CompileCommands.cpp
   Compiler.cpp

 target_include_directories(clangDaemon PUBLIC
   $<BUILD_INTERFACE:${CMAKE_CURRENT_BINARY_DIR}>
 )
 
 clang_target_link_libraries(clangDaemon
   PRIVATE
   clangAST
   clangASTMatchers
+  clangDynamicASTMatchers
   clangBasic
   clangDriver
   clangFormat
   clangFrontend
   clangIndex
   clangLex
   clangSema
   clangSerialization

clang-tools-extra/clangd/CodeComplete.cpp

 #include "index/SymbolOrigin.h"
 #include "support/Logger.h"
 #include "support/Markup.h"
 #include "support/Threading.h"
 #include "support/ThreadsafeFS.h"
 #include "support/Trace.h"
 #include "clang/AST/Decl.h"
 #include "clang/AST/DeclBase.h"
+#include "clang/ASTMatchers/Dynamic/Parser.h"
 #include "clang/Basic/CharInfo.h"
 #include "clang/Basic/LangOptions.h"
 #include "clang/Basic/SourceLocation.h"
 #include "clang/Basic/TokenKinds.h"
 #include "clang/Format/Format.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/FrontendActions.h"
 #include "clang/Lex/ExternalPreprocessorSource.h"
bool semaCodeComplete(std::unique_ptr<CodeCompleteConsumer> Consumer,
   Clang->setCodeCompletionConsumer(Consumer.release());
 
   SyntaxOnlyAction Action;
   if (!Action.BeginSourceFile(*Clang, Clang->getFrontendOpts().Inputs[0])) {
     log("BeginSourceFile() failed when running codeComplete for {0}",
         Input.FileName);
     return false;
   }
+  MatcherPragma::parse(/*Matchers=*/nullptr, *Clang);
   // Macros can be defined within the preamble region of the main file.
   // They don't fall nicely into our index/Sema dichotomy:
   //  - they're not indexed for completion (they're not available across files)
   //  - but Sema code complete won't see them: as part of the preamble, they're
   //    deserialized only when mentioned.
   // Force them to be deserialized so SemaCodeComplete sees them.
   loadMainFilePreambleMacros(Clang->getPreprocessor(), Input.Preamble);
   if (Includes)

clang-tools-extra/clangd/Compiler.cpp

 //===--- Compiler.cpp --------------------------------------------*- C++-*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "Compiler.h"
+#include "ASTMatchers.h"
 #include "support/Logger.h"
 #include "clang/Basic/TargetInfo.h"
 #include "clang/Frontend/CompilerInvocation.h"
 #include "clang/Lex/PreprocessorOptions.h"
 #include "clang/Serialization/PCHContainerOperations.h"
 #include "llvm/ADT/StringRef.h"
 
 namespace clang {

clang-tools-extra/clangd/Diagnostics.cpp

if (const char *ClangDiag = getDiagnosticCode(Diag.ID)) {
             Msg.resize(Rest.size());
         };
         CleanMessage(Diag.Message);
         for (auto &Note : Diag.Notes)
           CleanMessage(Note.Message);
         for (auto &Fix : Diag.Fixes)
           CleanMessage(Fix.Message);
       }
+    } else {
+      Diag.Source = Diag::Clangd;
     }
     setTags(Diag);
   }
   // Deduplicate clang-tidy diagnostics -- some clang-tidy checks may emit
   // duplicated messages due to various reasons (e.g. the check doesn't handle
   // template instantiations well; clang-tidy alias checks).
   std::set<std::pair<Range, std::string>> SeenDiags;
   llvm::erase_if(Output, [&](const Diag &D) {

clang-tools-extra/clangd/ParsedAST.h

 //  - capturing diagnostics for later access
 //  - running clang-tidy checks
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_PARSEDAST_H
 #define LLVM_CLANG_TOOLS_EXTRA_CLANGD_PARSEDAST_H
 
+#include "ASTMatchers.h"
 #include "CollectMacros.h"
 #include "Compiler.h"
 #include "Diagnostics.h"
 #include "Headers.h"
 #include "Preamble.h"
 #include "clang-include-cleaner/Record.h"
 #include "index/CanonicalIncludes.h"
 #include "support/Path.h"
public:
   const IncludeStructure &getIncludeStructure() const;
   const CanonicalIncludes &getCanonicalIncludes() const;
 
   /// Gets all macro references (definition, expansions) present in the main
   /// file, including those in the preamble region.
   const MainFileMacros &getMacros() const;
   /// Gets all pragma marks in the main file.
   const std::vector<PragmaMark> &getMarks() const;
+  /// Gets all AST matcher pragmas in the main file.
+  llvm::ArrayRef<MatcherPragma> getMatchers() const { return Matchers; }
   /// Tokens recorded while parsing the main file.
   /// (!) does not have tokens from the preamble.
   const syntax::TokenBuffer &getTokens() const { return Tokens; }
   /// Returns the PramaIncludes from the preamble.
   /// Might be null if AST is built without a preamble.
   const include_cleaner::PragmaIncludes *getPragmaIncludes() const;
 
   /// Returns the version of the ParseInputs this AST was built from.
public:
   }
 
 private:
   ParsedAST(PathRef TUPath, llvm::StringRef Version,
             std::shared_ptr<const PreambleData> Preamble,
             std::unique_ptr<CompilerInstance> Clang,
             std::unique_ptr<FrontendAction> Action, syntax::TokenBuffer Tokens,
             MainFileMacros Macros, std::vector<PragmaMark> Marks,
+            std::vector<MatcherPragma> Matchers,
             std::vector<Decl *> LocalTopLevelDecls,
             std::optional<std::vector<Diag>> Diags, IncludeStructure Includes,
             CanonicalIncludes CanonIncludes);
 
   Path TUPath;
   std::string Version;
   // In-memory preambles must outlive the AST, it is important that this member
   // goes before Clang and Action.
private:
   ///   - Includes expanded tokens produced **after** preamble.
   ///   - Does not have spelled or expanded tokens for files from preamble.
   syntax::TokenBuffer Tokens;
 
   /// All macro definitions and expansions in the main file.
   MainFileMacros Macros;
   // Pragma marks in the main file.
   std::vector<PragmaMark> Marks;
+  // #pragma clang query directives in the main file.
+  std::vector<MatcherPragma> Matchers;
   // Data, stored after parsing. std::nullopt if AST was built with a stale
   // preamble.
   std::optional<std::vector<Diag>> Diags;
   // Top-level decls inside the current file. Not that this does not include
   // top-level decls from the preamble.
   std::vector<Decl *> LocalTopLevelDecls;
   IncludeStructure Includes;
   CanonicalIncludes CanonIncludes;
   std::unique_ptr<HeuristicResolver> Resolver;
 };
 
 } // namespace clangd
 } // namespace clang
 
 #endif // LLVM_CLANG_TOOLS_EXTRA_CLANGD_PARSEDAST_H

clang-tools-extra/clangd/ParsedAST.cpp

 //===--- ParsedAST.cpp -------------------------------------------*- C++-*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "ParsedAST.h"
 #include "../clang-tidy/ClangTidyCheck.h"
 #include "../clang-tidy/ClangTidyDiagnosticConsumer.h"
 #include "../clang-tidy/ClangTidyModule.h"
 #include "../clang-tidy/ClangTidyModuleRegistry.h"
 #include "AST.h"
+#include "ASTMatchers.h"
 #include "Compiler.h"
 #include "Config.h"
 #include "Diagnostics.h"
 #include "Feature.h"
 #include "FeatureModule.h"
 #include "Headers.h"
 #include "HeuristicResolver.h"
 #include "IncludeCleaner.h"

 #include "clang/Lex/Preprocessor.h"
 #include "clang/Serialization/ASTWriter.h"
 #include "clang/Tooling/CompilationDatabase.h"
 #include "clang/Tooling/Syntax/Tokens.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringRef.h"
+#include <iterator>
 #include <memory>
 #include <optional>
+#include <utility>
 #include <vector>
 
 // Force the linker to link in Clang-tidy modules.
 // clangd doesn't support the static analyzer.
 #if CLANGD_TIDY_CHECKS
 #define CLANG_TIDY_DISABLE_STATIC_ANALYZER_CHECKS
 #include "../clang-tidy/ClangTidyForceLinker.h"
 #endif
ASTDiags.setDiagCallback(
       });
 
   std::optional<PreamblePatch> Patch;
   bool PreserveDiags = true;
   // We might use an ignoring diagnostic consumer if they are going to be
   // dropped later on to not pay for extra latency by processing them.
   DiagnosticConsumer *DiagConsumer = &ASTDiags;
   IgnoreDiagnostics DropDiags;
+  std::vector<MatcherPragma> Matchers;
   if (Preamble) {
     Patch = PreamblePatch::createFullPatch(Filename, Inputs, *Preamble);
     Patch->apply(*CI);
     PreserveDiags = Patch->preserveDiagnostics();
     if (!PreserveDiags)
       DiagConsumer = &DropDiags;
+    Matchers = Preamble->Matchers;
   }
   auto Clang = prepareCompilerInstance(
       std::move(CI), PreamblePCH,
       llvm::MemoryBuffer::getMemBufferCopy(Inputs.Contents, Filename), VFS,
       *DiagConsumer);
   if (!Clang) {
     // The last diagnostic contains information about the reason of this
     // failure.
ParsedAST::build(llvm::StringRef Filename, const ParseInputs &Inputs,
   // This information is part of the HeaderFileInfo but is not loaded from the
   // preamble as the file's size is part of its identity and may have changed.
   // (The rest of HeaderFileInfo is not relevant for our purposes).
   if (Preamble && Preamble->MainIsIncludeGuarded) {
     const SourceManager &SM = Clang->getSourceManager();
     const FileEntry *MainFE = SM.getFileEntryForID(SM.getMainFileID());
     Clang->getPreprocessor().getHeaderSearchInfo().MarkFileIncludeOnce(MainFE);
   }
+  MatcherPragma::parse(&Matchers, *Clang);
 
   // Set up ClangTidy. Must happen after BeginSourceFile() so ASTContext exists.
   // Clang-tidy has some limitations to ensure reasonable performance:
   //  - checks don't see all preprocessor events in the preamble
   //  - matchers run only over the main-file top-level decls (and can't see
   //    ancestors outside this scope).
   // In practice almost all checks work well without modifications.
   std::vector<std::unique_ptr<tidy::ClangTidyCheck>> CTChecks;
ParsedAST::build(llvm::StringRef Filename, const ParseInputs &Inputs,
   // AST traversals should exclude the preamble, to avoid performance cliffs.
   Clang->getASTContext().setTraversalScope(ParsedDecls);
   if (!CTChecks.empty()) {
     // Run the AST-dependent part of the clang-tidy checks.
     // (The preprocessor part ran already, via PPCallbacks).
     trace::Span Tracer("ClangTidyMatch");
     CTFinder.matchAST(Clang->getASTContext());
   }
+  MatcherPragma::diagnose(Matchers, Clang->getASTContext(),
+                          Clang->getDiagnostics());
 
   // XXX: This is messy: clang-tidy checks flush some diagnostics at EOF.
   // However Action->EndSourceFile() would destroy the ASTContext!
   // So just inform the preprocessor of EOF, while keeping everything alive.
   PP.EndSourceFile();
   // UnitDiagsConsumer is local, we can not store it in CompilerInstance that
   // has a longer lifetime.
   Clang->getDiagnostics().setClient(new IgnoreDiagnostics);
if (PreserveDiags) {
     // Finally, add diagnostics coming from the AST.
     {
       std::vector<Diag> D = ASTDiags.take(&*CTContext);
       Diags->insert(Diags->end(), D.begin(), D.end());
     }
   }
   ParsedAST Result(Filename, Inputs.Version, std::move(Preamble),
                    std::move(Clang), std::move(Action), std::move(Tokens),
-                   std::move(Macros), std::move(Marks), std::move(ParsedDecls),
-                   std::move(Diags), std::move(Includes),
-                   std::move(CanonIncludes));
+                   std::move(Macros), std::move(Marks), std::move(Matchers),
+                   std::move(ParsedDecls), std::move(Diags),
+                   std::move(Includes), std::move(CanonIncludes));
   if (Result.Diags)
     llvm::move(issueIncludeCleanerDiagnostics(Result, Inputs.Contents),
                std::back_inserter(*Result.Diags));
   return std::move(Result);
 }
 
 ParsedAST::ParsedAST(ParsedAST &&Other) = default;
 

 }
 
 ParsedAST::ParsedAST(PathRef TUPath, llvm::StringRef Version,
                      std::shared_ptr<const PreambleData> Preamble,
                      std::unique_ptr<CompilerInstance> Clang,
                      std::unique_ptr<FrontendAction> Action,
                      syntax::TokenBuffer Tokens, MainFileMacros Macros,
                      std::vector<PragmaMark> Marks,
+                     std::vector<MatcherPragma> Matchers,
                      std::vector<Decl *> LocalTopLevelDecls,
                      std::optional<std::vector<Diag>> Diags,
                      IncludeStructure Includes, CanonicalIncludes CanonIncludes)
     : TUPath(TUPath), Version(Version), Preamble(std::move(Preamble)),
       Clang(std::move(Clang)), Action(std::move(Action)),
       Tokens(std::move(Tokens)), Macros(std::move(Macros)),
-      Marks(std::move(Marks)), Diags(std::move(Diags)),
+      Marks(std::move(Marks)), Matchers(std::move(Matchers)),
+      Diags(std::move(Diags)),
       LocalTopLevelDecls(std::move(LocalTopLevelDecls)),
       Includes(std::move(Includes)), CanonIncludes(std::move(CanonIncludes)) {
   Resolver = std::make_unique<HeuristicResolver>(getASTContext());
   assert(this->Clang);
   assert(this->Action);
 }
 
 const include_cleaner::PragmaIncludes *ParsedAST::getPragmaIncludes() const {

clang-tools-extra/clangd/Preamble.h

 // code-completion to have sub-second latency. It is supported by the
 // PrecompiledPreamble functionality in clang, which wraps the techniques used
 // by PCH files, modules etc into a convenient interface.
 //
 //===----------------------------------------------------------------------===//
 #ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_PREAMBLE_H
 #define LLVM_CLANG_TOOLS_EXTRA_CLANGD_PREAMBLE_H
 
+#include "ASTMatchers.h"
 #include "CollectMacros.h"
 #include "Compiler.h"
 #include "Diagnostics.h"
 #include "FS.h"
 #include "Headers.h"
 #include "clang-include-cleaner/Record.h"
 #include "index/CanonicalIncludes.h"
 #include "support/Path.h"
struct PreambleData {
   // Captures #include-mapping information in #included headers.
   include_cleaner::PragmaIncludes Pragmas;
   // Macros defined in the preamble section of the main file.
   // Users care about headers vs main-file, not preamble vs non-preamble.
   // These should be treated as main-file entities e.g. for code completion.
   MainFileMacros Macros;
   // Pragma marks defined in the preamble section of the main file.
   std::vector<PragmaMark> Marks;
+  // Matcher pragmas defined in the preamble section of the main file.
+  std::vector<MatcherPragma> Matchers;
   // Cache of FS operations performed when building the preamble.
   // When reusing a preamble, this cache can be consumed to save IO.
   std::unique_ptr<PreambleFileStatusCache> StatCache;
   CanonicalIncludes CanonIncludes;
   // Whether there was a (possibly-incomplete) include-guard on the main file.
   // We need to propagate this information "by hand" to subsequent parses.
   bool MainIsIncludeGuarded = false;
 };

clang-tools-extra/clangd/Preamble.cpp

 //===--- Preamble.cpp - Reusing expensive parts of the AST ----------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "Preamble.h"
+#include "ASTMatchers.h"
 #include "CollectMacros.h"
 #include "Compiler.h"
 #include "Config.h"
 #include "Diagnostics.h"
 #include "Headers.h"
 #include "Protocol.h"
 #include "SourceCode.h"
 #include "clang-include-cleaner/Record.h"
CppFilePreambleCallbacks(
         ParseForwardingFunctions(ParseForwardingFunctions),
         BeforeExecuteCallback(std::move(BeforeExecuteCallback)) {}
 
   IncludeStructure takeIncludes() { return std::move(Includes); }
 
   MainFileMacros takeMacros() { return std::move(Macros); }
 
   std::vector<PragmaMark> takeMarks() { return std::move(Marks); }
+  std::vector<MatcherPragma> takeMatchers() { return std::move(Matchers); }
 
   include_cleaner::PragmaIncludes takePragmaIncludes() {
     return std::move(Pragmas);
   }
   CanonicalIncludes takeCanonicalIncludes() { return std::move(CanonIncludes); }
 
   bool isMainFileIncludeGuarded() const { return IsMainFileIncludeGuarded; }
 
public:
 
   void BeforeExecute(CompilerInstance &CI) override {
     CanonIncludes.addSystemHeadersMapping(CI.getLangOpts());
     LangOpts = &CI.getLangOpts();
     SourceMgr = &CI.getSourceManager();
     PP = &CI.getPreprocessor();
     Includes.collect(CI);
     Pragmas.record(CI);
+    MatcherPragma::parse(&Matchers, CI);
     if (BeforeExecuteCallback)
       BeforeExecuteCallback(CI);
   }
 
   std::unique_ptr<PPCallbacks> createPPCallbacks() override {
     assert(SourceMgr && LangOpts && PP &&
            "SourceMgr, LangOpts and PP must be set at this point");
 

 private:
   PathRef File;
   PreambleParsedCallback ParsedCallback;
   IncludeStructure Includes;
   CanonicalIncludes CanonIncludes;
   include_cleaner::PragmaIncludes Pragmas;
   MainFileMacros Macros;
   std::vector<PragmaMark> Marks;
+  std::vector<MatcherPragma> Matchers;
   bool IsMainFileIncludeGuarded = false;
   std::unique_ptr<CommentHandler> IWYUHandler = nullptr;
   const clang::LangOptions *LangOpts = nullptr;
   const SourceManager *SourceMgr = nullptr;
   const Preprocessor *PP = nullptr;
   PreambleBuildStats *Stats;
   bool ParseForwardingFunctions;
   std::function<void(CompilerInstance &)> BeforeExecuteCallback;
if (BuiltPreamble) {
     auto Result = std::make_shared<PreambleData>(std::move(*BuiltPreamble));
     Result->Version = Inputs.Version;
     Result->CompileCommand = Inputs.CompileCommand;
     Result->Diags = std::move(Diags);
     Result->Includes = CapturedInfo.takeIncludes();
     Result->Pragmas = CapturedInfo.takePragmaIncludes();
     Result->Macros = CapturedInfo.takeMacros();
     Result->Marks = CapturedInfo.takeMarks();
+    Result->Matchers = CapturedInfo.takeMatchers();
     Result->CanonIncludes = CapturedInfo.takeCanonicalIncludes();
     Result->StatCache = std::move(StatCache);
     Result->MainIsIncludeGuarded = CapturedInfo.isMainFileIncludeGuarded();
     return Result;
   }
 
   elog("Could not build a preamble for file {0} version {1}: {2}", FileName,
        Inputs.Version, BuiltPreamble.getError().message());

clang-tools-extra/clangd/XRefs.cpp

 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/raw_ostream.h"
 #include <optional>
 #include <string>
+#include <utility>
 #include <vector>
 
 namespace clang {
 namespace clangd {
 namespace {
 
 // Returns the single definition of the entity declared by D, if visible.
 // In particular:
Result.Loc.range = clangd::Range{Position{Inc.HashLine, 0},
                                    Position{Inc.HashLine, (int)IncludeLen}};
   Result.Loc.uri = URIMainFile;
   Results.References.push_back(std::move(Result));
 
   if (Results.References.empty())
     return std::nullopt;
   return Results;
 }
+
+std::optional<ReferencesResult>
+maybeFindMatcherReferences(ParsedAST &AST, Position Pos,
+                           URIForFile URIMainFile) {
+  const auto &Matchers = AST.getMatchers();
+  auto MatcherOnLine = llvm::find_if(Matchers, [&Pos](const MatcherPragma &M) {
+    return M.LineRange.first <= unsigned(Pos.line + 1) &&
+           unsigned(Pos.line + 1) <= M.LineRange.second;
+  });
+  if (MatcherOnLine == Matchers.end())
+    return std::nullopt;
+
+  ReferencesResult Results;
+  for (const auto &Match : MatcherOnLine->match(AST.getASTContext())) {
+    auto Range = AST.getTokens().spelledForExpanded(
+        AST.getTokens().expandedTokens(Match.second));
+    if (!Range || Range->empty())
+      continue;
+    ReferencesResult::Reference Ref;
+    Ref.Loc.uri = URIMainFile;
+    Ref.Loc.range.start =
+        sourceLocToPosition(AST.getSourceManager(), Range->front().location());
+    Ref.Loc.range.end = sourceLocToPosition(AST.getSourceManager(),
+                                            Range->back().endLocation());
+    Results.References.push_back(std::move(Ref));
+  }
+  return Results;
+}
+
 } // namespace
 
 ReferencesResult findReferences(ParsedAST &AST, Position Pos, uint32_t Limit,
                                 const SymbolIndex *Index, bool AddContext) {
   ReferencesResult Results;
   const SourceManager &SM = AST.getSourceManager();
   auto MainFilePath = AST.tuPath();
   auto URIMainFile = URIForFile::canonicalize(MainFilePath, MainFilePath);
   auto CurLoc = sourceLocationInMainFile(SM, Pos);
   if (!CurLoc) {
     llvm::consumeError(CurLoc.takeError());
     return {};
   }
 
-  const auto IncludeReferences =
-      maybeFindIncludeReferences(AST, Pos, URIMainFile);
-  if (IncludeReferences)
+  if (auto IncludeReferences =
+          maybeFindIncludeReferences(AST, Pos, URIMainFile))
     return *IncludeReferences;
+  if (auto MatcherReferences =
+          maybeFindMatcherReferences(AST, Pos, URIMainFile))
+    return *MatcherReferences;
 
   llvm::DenseSet<SymbolID> IDsToQuery, OverriddenMethods;
 
   const auto *IdentifierAtCursor =
       syntax::spelledIdentifierTouching(*CurLoc, AST.getTokens());
   std::optional<DefinedMacro> Macro;
   if (IdentifierAtCursor)
     Macro = locateMacroAt(*IdentifierAtCursor, AST.getPreprocessor());

clang-tools-extra/clangd/unittests/CodeCompleteTests.cpp

TEST(CompletionTest, Concepts) {
 
   for (auto P : Code.points("other")) {
     EXPECT_THAT(completions(TU, P, Syms).Completions,
                 AllOf(Contains(named("A")), Contains(named("same_as"))))
         << "Completing 'requires' expression at position " << P;
   }
 }
 
+TEST(CompletionTest, MatcherPragma) {
+  Annotations Code(R"cpp(
+    #pragma clang query parmVarDecl(hasType^)
+  )cpp");
+  auto TU = TestTU::withCode(Code.code());
+
+  ASSERT_THAT(completions(TU, Code.point()).Completions,
+              ElementsAre(named("hasType"), named("hasTypeLoc")));
+}
+
 TEST(SignatureHelp, DocFormat) {
   Annotations Code(R"cpp(
     // Comment `with` markup.
     void foo(int);
     void bar() { foo(^); }
   )cpp");
   for (auto DocumentationFormat :
        {MarkupKind::PlainText, MarkupKind::Markdown}) {

clang-tools-extra/clangd/unittests/DiagnosticsTests.cpp

TEST(DiagnosticsTest, ClangTidyEOF) {
   TU.ClangTidyProvider = addTidyChecks("llvm-include-order");
   EXPECT_THAT(
       *TU.build().getDiagnostics(),
       ifTidyChecks(Contains(
           AllOf(Diag(Test.range(), "#includes are not sorted properly"),
                 diagSource(Diag::ClangTidy), diagName("llvm-include-order")))));
 }
 
+TEST(DiagnosticsTest, Matchers) {
+  Annotations Test(R"cpp(
+    ;
+    #pragma clang query parmVarDecl()
+    void foo([[int x]], [[int y]]);
+  )cpp");
+  auto TU = TestTU::withCode(Test.code());
+  EXPECT_THAT(*TU.build().getDiagnostics(),
+              ElementsAre(Diag(Test.ranges()[0], "query:3 matches"),
+                          Diag(Test.ranges()[1], "query:3 matches")));
+}
+
+TEST(DiagnosticsTest, MatchersPreamble) {
+  Annotations Test(R"cpp(
+    #pragma clang query parmVarDecl().bind("parm")
+    void foo([[int x]]);
+  )cpp");
+  auto TU = TestTU::withCode(Test.code());
+  EXPECT_THAT(*TU.build().getDiagnostics(),
+              ElementsAre(Diag(Test.range(), "parm matches")));
+}
+
+TEST(DiagnosticsTest, MatchersError) {
+  Annotations Test(R"cpp(
+    #pragma clang query [[pmVrDcl]]() // error-ok
+    void foo(int x);
+  )cpp");
+  auto TU = TestTU::withCode(Test.code());
+  EXPECT_THAT(
+      *TU.build().getDiagnostics(),
+      ElementsAre(
+          Diag(Test.range(), "Bad matcher: 1:2: Matcher not found: pmVrDcl")));
+}
+
 TEST(DiagnosticTest, TemplatesInHeaders) {
   // Diagnostics from templates defined in headers are placed at the expansion.
   Annotations Main(R"cpp(
     Derived<int> [[y]]; // error-ok
   )cpp");
   Annotations Header(R"cpp(
     template <typename T>
     struct Derived : [[T]] {};

clang-tools-extra/clangd/unittests/XRefsTests.cpp

for (const char *Test : Tests) {
     for (const auto &P : T.points()) 
       EXPECT_THAT(findReferences(AST, P, 0).References,
                   UnorderedElementsAreArray(ExpectedLocations))
           << "Failed for Refs at " << P << "\n"
           << Test;
   }
 }
 
+TEST(FindReferences, Matchers) {
+  checkFindRefs(R"cpp(
+    #pragma clang query \
+          ^parmVarDecl()           
+    void foo([[int x]], [[int y]]);
+  )cpp");
+}
+
 TEST(FindReferences, NeedsIndexForSymbols) {
   const char *Header = "int foo();";
   Annotations Main("int main() { [[f^oo]](); }");
   TestTU TU;
   TU.Code = std::string(Main.code());
   TU.HeaderCode = Header;
   auto AST = TU.build();