[clangd] Implement configs to stop clangd produce a certain semantic tokens

[daiyousei-qz]

Summary

This patch introduces the following configurations to .clangd:

SemanticTokens:
    DisabledKinds: [ ... ]
    DisabledModifiers: [ ... ]

Based on the config, clangd would stop producing a certain type of semantic tokens from the source file.

[daiyousei-qz]

Please hold off from reviewing this. I have been figuring out how arc tools work and the unittest of the current version is broken.

[daiyousei-qz]

Fix unittest

[daiyousei-qz]

Add a test in SemanticHighlightingTests.cpp

[daiyousei-qz]

Fix an issue where the previous change is lost.

clang-tools-extra/clangd/SemanticHighlighting.cpp

1279

I'm not sure if we should parse the enums here or in parsing configs using EnumSwitch. The problem is that I don't want to introduce dependencies to "SemanticHighlighting.h" in "ConfigCompile.cpp".

[daiyousei-qz]

Add some comments to the config headers. Resubmit to retrigger build test since I cannot reproduce the failure.

[kadircet]

before you dive any deeper into the patch, could you give some reasoning about why this is needed/useful?

[nridge]

In D148489#4276634, @kadircet wrote:

before you dive any deeper into the patch, could you give some reasoning about why this is needed/useful?

https://github.com/clangd/clangd/discussions/1598 contains some context/motivation

[daiyousei-qz]

In D148489#4276634, @kadircet wrote:

before you dive any deeper into the patch, could you give some reasoning about why this is needed/useful?

The discussion is at https://github.com/clangd/clangd/discussions/1598.

The original motivation is that in the latest clangd, we start to assign semantic token "operator" to all operators in the source code. Since the semantic tokens take priority over the textmate rules in vscode, now we cannot theme different operators differently anymore. For example, we cannot color "new" in blue while have "+" in white.

To address the issue, this patch adds configurable filter to the semantic token to filter out unwanted kinds and modifiers. We should have the following benefits with such filter in place.

1. Address the issue mentioned above about unwanted token kinds and modifiers so textmate rules could fallback.
2. Improve the performance of semantic tokens (if anyone is taking advantage of the config OFC) since we could strip away unwanted tokens
3. Not directly supported by this patch, but the filter could also be used to support adding semantic tokens that are disabled by default. Say if we want to support augmentsSyntaxTokens in the future.

[nridge]

Thanks for the patch!

I think this is a nice and general solution, which rather than solving just a specific problem (e.g. highlighting of new as an operator vs. a keyword), gives users a broader ability to fine-tune clangd's highlightings to suit their needs.

Could I kindly ask you to make a PR that updates https://github.com/llvm/clangd-www/blob/main/config.md to cover the new config keys, as well?

clang-tools-extra/clangd/SemanticHighlighting.cpp

400

For good measure, let's static_assert(HighlightingModifier::LastModifier < 32)

587

This needs to be stored by value, otherwise it will dangle

[daiyousei-qz]

Sorry, I'm a little occupied lately and don't have time to fix the test failure. I'll try to fix that in this week.

clang-tools-extra/clangd/SemanticHighlighting.cpp

400

IIRC, we already have such static_assert after the definition of HighlightingModifier in SemanticHighlight.h

[nridge]

clang-tools-extra/clangd/SemanticHighlighting.cpp

400

Good point, I overlooked that.

[daiyousei-qz]

Pass filter by value.

[daiyousei-qz]

Redo arc as some changes are missing from previous update.

[daiyousei-qz]

Fix format issue

[nridge]

Thanks! Let me know if you need me to commit.

clang-tools-extra/clangd/SemanticHighlighting.cpp

406

The filter can be passed by const reference in the constructor (and it's better to do so to avoid making an additional copy), it just needs to be stored by value as a class member

[daiyousei-qz]

Yes please.

[nridge]

(Do you plan to address the last comment before I merge?)

[daiyousei-qz]

In D148489#4362990, @nridge wrote:

(Do you plan to address the last comment before I merge?)

Oh, okay. Let me address that. Didn't see that.

[daiyousei-qz]

• Address review comment and remove a unnecessary flag

[daiyousei-qz]

Sorry for the inactivity. I have replace the parameter with const ref. By the way, I also removed the existing IncludeInactiveRegionTokens flag and uses this filter instead. Please help review the change. Thanks!

[daiyousei-qz]

• Fix reversed logic of including inactive region token

[nridge]

Thanks, LGTM!

[nridge]

I went ahead and merged this.

When you get a chance, could you update https://github.com/llvm/clangd-www/pull/85 as well so we can merge that too?

[daiyousei-qz]

Thank you! Since the documentation shouldn't be available until LLVM 17, I'm slowly working on it. Let me update the PR in this weekend :)

clang-tools-extra/clangd/Config.h

struct {
 
     // Whether specific categories of hints are enabled.
     bool Parameters = true;
     bool DeducedTypes = true;
     bool Designators = true;
     // Limit the length of type names in inlay hints. (0 means no limit)
     uint32_t TypeNameLimit = 32;
   } InlayHints;
+
+  struct {
+    /// Controls highlighting kinds that are disabled.
+    std::vector<std::string> DisabledKinds;
+    /// Controls highlighting modifiers that are disabled.
+    std::vector<std::string> DisabledModifiers;
+  } SemanticTokens;
 };
 
 } // namespace clangd
 } // namespace clang
 
 namespace llvm {
 template <> struct DenseMapInfo<clang::clangd::Config::ExternalIndexSpec> {
   using ExternalIndexSpec = clang::clangd::Config::ExternalIndexSpec;

clang-tools-extra/clangd/ConfigCompile.cpp

void compile(Fragment &&F) {
     }
     compile(std::move(F.If));
     compile(std::move(F.CompileFlags));
     compile(std::move(F.Index));
     compile(std::move(F.Diagnostics));
     compile(std::move(F.Completion));
     compile(std::move(F.Hover));
     compile(std::move(F.InlayHints));
+    compile(std::move(F.SemanticTokens));
     compile(std::move(F.Style));
   }
 
   void compile(Fragment::IfBlock &&F) {
     if (F.HasUnrecognizedCondition)
       Out.Conditions.push_back([&](const Params &) { return false; });
 
 #ifdef CLANGD_PATH_CASE_INSENSITIVE
if (F.Designators)
       });
     if (F.TypeNameLimit)
       Out.Apply.push_back(
           [Value(**F.TypeNameLimit)](const Params &, Config &C) {
             C.InlayHints.TypeNameLimit = Value;
           });
   }
 
+  void compile(Fragment::SemanticTokensBlock &&F) {
+    if (!F.DisabledKinds.empty()) {
+      std::vector<std::string> DisabledKinds;
+      for (auto &Kind : F.DisabledKinds)
+        DisabledKinds.push_back(std::move(*Kind));
+
+      Out.Apply.push_back(
+          [DisabledKinds(std::move(DisabledKinds))](const Params &, Config &C) {
+            for (auto &Kind : DisabledKinds) {
+              auto It = llvm::find(C.SemanticTokens.DisabledKinds, Kind);
+              if (It == C.SemanticTokens.DisabledKinds.end())
+                C.SemanticTokens.DisabledKinds.push_back(std::move(Kind));
+            }
+          });
+    }
+    if (!F.DisabledModifiers.empty()) {
+      std::vector<std::string> DisabledModifiers;
+      for (auto &Kind : F.DisabledModifiers)
+        DisabledModifiers.push_back(std::move(*Kind));
+
+      Out.Apply.push_back([DisabledModifiers(std::move(DisabledModifiers))](
+                              const Params &, Config &C) {
+        for (auto &Kind : DisabledModifiers) {
+          auto It = llvm::find(C.SemanticTokens.DisabledModifiers, Kind);
+          if (It == C.SemanticTokens.DisabledModifiers.end())
+            C.SemanticTokens.DisabledModifiers.push_back(std::move(Kind));
+        }
+      });
+    }
+  }
+
   constexpr static llvm::SourceMgr::DiagKind Error = llvm::SourceMgr::DK_Error;
   constexpr static llvm::SourceMgr::DiagKind Warning =
       llvm::SourceMgr::DK_Warning;
   void diag(llvm::SourceMgr::DiagKind Kind, llvm::StringRef Message,
             llvm::SMRange Range) {
     if (Range.isValid() && SourceMgr != nullptr)
       Diagnostic(SourceMgr->GetMessage(Range.Start, Kind, Message, Range));
     else

clang-tools-extra/clangd/ConfigFragment.h

struct DiagnosticsBlock {
     /// transitively includes headers that are used. This should be fixed by
     /// including those headers directly.
     ///
     /// Valid values are:
     /// - Strict
     /// - std::nullopt
     std::optional<Located<std::string>> UnusedIncludes;
 
-
     /// Enable emitting diagnostics using stale preambles.
     std::optional<Located<bool>> AllowStalePreamble;
 
     /// Controls if clangd should analyze missing #include directives.
     /// clangd will warn if no header providing a symbol is `#include`d
     /// (missing) directly, and suggest adding it.
     ///
     /// Strict means a header providing a symbol is missing if it is not
struct InlayHintsBlock {
     /// Show deduced types for `auto`.
     std::optional<Located<bool>> DeducedTypes;
     /// Show designators in aggregate initialization.
     std::optional<Located<bool>> Designators;
     /// Limit the length of type name hints. (0 means no limit)
     std::optional<Located<uint32_t>> TypeNameLimit;
   };
   InlayHintsBlock InlayHints;
+
+  /// Configures semantic tokens that are produced by clangd.
+  struct SemanticTokensBlock {
+    /// Disables clangd to produce semantic tokens for the given kinds.
+    std::vector<Located<std::string>> DisabledKinds;
+    /// Disables clangd to assign semantic tokens with the given modifiers.
+    std::vector<Located<std::string>> DisabledModifiers;
+  };
+  SemanticTokensBlock SemanticTokens;
 };
 
 } // namespace config
 } // namespace clangd
 } // namespace clang
 
 #endif

clang-tools-extra/clangd/ConfigYAML.cpp

bool parse(Fragment &F, Node &N) {
     Dict.handle("If", [&](Node &N) { parse(F.If, N); });
     Dict.handle("CompileFlags", [&](Node &N) { parse(F.CompileFlags, N); });
     Dict.handle("Index", [&](Node &N) { parse(F.Index, N); });
     Dict.handle("Style", [&](Node &N) { parse(F.Style, N); });
     Dict.handle("Diagnostics", [&](Node &N) { parse(F.Diagnostics, N); });
     Dict.handle("Completion", [&](Node &N) { parse(F.Completion, N); });
     Dict.handle("Hover", [&](Node &N) { parse(F.Hover, N); });
     Dict.handle("InlayHints", [&](Node &N) { parse(F.InlayHints, N); });
+    Dict.handle("SemanticTokens", [&](Node &N) { parse(F.SemanticTokens, N); });
     Dict.parse(N);
     return !(N.failed() || HadError);
   }
 
 private:
   void parse(Fragment::IfBlock &F, Node &N) {
     DictParser Dict("If", this);
     Dict.unrecognized([&](Located<std::string>, Node &) {
void parse(Fragment::InlayHintsBlock &F, Node &N) {
     });
     Dict.handle("TypeNameLimit", [&](Node &N) {
       if (auto Value = uint32Value(N, "TypeNameLimit"))
         F.TypeNameLimit = *Value;
     });
     Dict.parse(N);
   }
 
+  void parse(Fragment::SemanticTokensBlock &F, Node &N) {
+    DictParser Dict("SemanticTokens", this);
+    Dict.handle("DisabledKinds", [&](Node &N) {
+      if (auto Values = scalarValues(N))
+        F.DisabledKinds = std::move(*Values);
+    });
+    Dict.handle("DisabledModifiers", [&](Node &N) {
+      if (auto Values = scalarValues(N))
+        F.DisabledModifiers = std::move(*Values);
+    });
+    Dict.parse(N);
+  }
+
   // Helper for parsing mapping nodes (dictionaries).
   // We don't use YamlIO as we want to control over unknown keys.
   class DictParser {
     llvm::StringRef Description;
     std::vector<std::pair<llvm::StringRef, std::function<void(Node &)>>> Keys;
     std::function<bool(Located<std::string>, Node &)> UnknownHandler;
     Parser *Outer;
 

clang-tools-extra/clangd/SemanticHighlighting.h

enum class HighlightingKind {
   // This one is different from the other kinds as it's a line style
   // rather than a token style.
   InactiveCode,
 
   LastKind = InactiveCode
 };
 
 llvm::raw_ostream &operator<<(llvm::raw_ostream &OS, HighlightingKind K);
+std::optional<HighlightingKind>
+highlightingKindFromString(llvm::StringRef Name);
 
 enum class HighlightingModifier {
   Declaration,
   Definition,
   Deprecated,
   Deduced,
   Readonly,
   Static,
enum class HighlightingModifier {
   FileScope,
   GlobalScope,
 
   LastModifier = GlobalScope
 };
 static_assert(static_cast<unsigned>(HighlightingModifier::LastModifier) < 32,
               "Increase width of modifiers bitfield!");
 llvm::raw_ostream &operator<<(llvm::raw_ostream &OS, HighlightingModifier K);
+std::optional<HighlightingModifier>
+highlightingModifierFromString(llvm::StringRef Name);
 
 // Contains all information needed for the highlighting a token.
 struct HighlightingToken {
   HighlightingKind Kind;
   uint32_t Modifiers = 0;
   Range R;
 
   HighlightingToken &addModifier(HighlightingModifier M) {

clang-tools-extra/clangd/SemanticHighlighting.cpp

 //===--- SemanticHighlighting.cpp - ------------------------- ---*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "SemanticHighlighting.h"
+#include "Config.h"
 #include "FindTarget.h"
 #include "HeuristicResolver.h"
 #include "ParsedAST.h"
 #include "Protocol.h"
 #include "SourceCode.h"
 #include "support/Logger.h"
 #include "clang/AST/ASTContext.h"
 #include "clang/AST/Decl.h"
resolveConflict(ArrayRef<HighlightingToken> Tokens) {
   assert(Tokens.size() >= 2);
   std::optional<HighlightingToken> Winner =
       resolveConflict(Tokens[0], Tokens[1]);
   for (size_t I = 2; Winner && I < Tokens.size(); ++I)
     Winner = resolveConflict(*Winner, Tokens[I]);
   return Winner;
 }
 
+/// Filter to remove particular kinds of highlighting tokens and modifiers from
+/// the output.
+class HighlightingFilter {
+public:
+  HighlightingFilter() {
+    for (auto &Active : ActiveKindLookup)
+      Active = true;
+
+    ActiveModifiersMask = ~0;
+  }
+
+  void disableKind(HighlightingKind Kind) {
+    ActiveKindLookup[static_cast<size_t>(Kind)] = false;
+  }
+
+  void disableModifier(HighlightingModifier Modifier) {
+    ActiveModifiersMask &= ~(1 << static_cast<uint32_t>(Modifier));
+  }
+
+  bool isHighlightKindActive(HighlightingKind Kind) const {
+    return ActiveKindLookup[static_cast<size_t>(Kind)];
+  }
+
+  uint32_t maskModifiers(uint32_t Modifiers) const {
+    return Modifiers & ActiveModifiersMask;
+  }
+
+  static HighlightingFilter fromCurrentConfig() {
+    const Config &C = Config::current();
+    HighlightingFilter Filter;
+    for (const auto &Kind : C.SemanticTokens.DisabledKinds)
+      if (auto K = highlightingKindFromString(Kind))
+        Filter.disableKind(*K);
+    for (const auto &Modifier : C.SemanticTokens.DisabledModifiers)
+      if (auto M = highlightingModifierFromString(Modifier))
+        Filter.disableModifier(*M);
+
+    return Filter;
+  }
+
+private:
+  bool ActiveKindLookup[static_cast<size_t>(HighlightingKind::LastKind) + 1];
+  uint32_t ActiveModifiersMask;
+};
+
 /// Consumes source locations and maps them to text ranges for highlightings.
 class HighlightingsBuilder {
 public:
-  HighlightingsBuilder(const ParsedAST &AST, bool IncludeInactiveRegionTokens)
+  HighlightingsBuilder(const ParsedAST &AST, const HighlightingFilter &Filter)
       : TB(AST.getTokens()), SourceMgr(AST.getSourceManager()),
-        LangOpts(AST.getLangOpts()),
-        IncludeInactiveRegionTokens(IncludeInactiveRegionTokens) {}
+        LangOpts(AST.getLangOpts()), Filter(Filter) {}
 
   HighlightingToken &addToken(SourceLocation Loc, HighlightingKind Kind) {
     auto Range = getRangeForSourceLocation(Loc);
     if (!Range)
       return InvalidHighlightingToken;
 
     return addToken(*Range, Kind);
   }
if (RTok->kind() == tok::greater) {
         addToken(*LRange, HighlightingKind::Bracket);
         addToken(*RRange, HighlightingKind::Bracket);
       }
       return;
     }
   }
 
   HighlightingToken &addToken(Range R, HighlightingKind Kind) {
+    if (!Filter.isHighlightKindActive(Kind))
+      return InvalidHighlightingToken;
+
     HighlightingToken HT;
     HT.R = std::move(R);
     HT.Kind = Kind;
     Tokens.push_back(std::move(HT));
     return Tokens.back();
   }
 
   void addExtraModifier(SourceLocation Loc, HighlightingModifier Modifier) {
for (ArrayRef<HighlightingToken> TokRef = Tokens; !TokRef.empty();) {
         // Apply extra collected highlighting modifiers
         auto Modifiers = ExtraModifiers.find(Resolved->R);
         if (Modifiers != ExtraModifiers.end()) {
           for (HighlightingModifier Mod : Modifiers->second) {
             Resolved->addModifier(Mod);
           }
         }
 
+        Resolved->Modifiers = Filter.maskModifiers(Resolved->Modifiers);
         NonConflicting.push_back(*Resolved);
       }
       // TokRef[Conflicting.size()] is the next token with a different range (or
       // the end of the Tokens).
       TokRef = TokRef.drop_front(Conflicting.size());
     }
 
-    if (!IncludeInactiveRegionTokens)
+    if (!Filter.isHighlightKindActive(HighlightingKind::InactiveCode))
       return NonConflicting;
 
     const auto &SM = AST.getSourceManager();
     StringRef MainCode = SM.getBufferOrFake(SM.getMainFileID()).getBuffer();
 
     // Merge token stream with "inactive line" markers.
     std::vector<HighlightingToken> WithInactiveLines;
     auto SortedSkippedRanges = AST.getMacros().SkippedRanges;
if (!Tok)
       return std::nullopt;
     return halfOpenToRange(SourceMgr,
                            Tok->range(SourceMgr).toCharRange(SourceMgr));
   }
 
   const syntax::TokenBuffer &TB;
   const SourceManager &SourceMgr;
   const LangOptions &LangOpts;
-  bool IncludeInactiveRegionTokens;
+  HighlightingFilter Filter;
   std::vector<HighlightingToken> Tokens;
   std::map<Range, llvm::SmallVector<HighlightingModifier, 1>> ExtraModifiers;
   const HeuristicResolver *Resolver = nullptr;
   // returned from addToken(InvalidLoc)
   HighlightingToken InvalidHighlightingToken;
 };
 
 std::optional<HighlightingModifier> scopeModifier(const NamedDecl *D) {

 private:
   HighlightingsBuilder &H;
 };
 } // namespace
 
 std::vector<HighlightingToken>
 getSemanticHighlightings(ParsedAST &AST, bool IncludeInactiveRegionTokens) {
   auto &C = AST.getASTContext();
+  HighlightingFilter Filter = HighlightingFilter::fromCurrentConfig();
+  if (!IncludeInactiveRegionTokens)
+    Filter.disableKind(HighlightingKind::InactiveCode);
   // Add highlightings for AST nodes.
-  HighlightingsBuilder Builder(AST, IncludeInactiveRegionTokens);
+  HighlightingsBuilder Builder(AST, Filter);
   // Highlight 'decltype' and 'auto' as their underlying types.
   CollectExtraHighlightings(Builder).TraverseAST(C);
   // Highlight all decls and references coming from the AST.
   findExplicitReferences(
       C,
       [&](ReferenceLoc R) {
         for (const NamedDecl *Decl : R.Targets) {
           if (!canHighlightName(Decl->getDeclName()))
case HighlightingKind::Operator:
     return OS << "Operator";
   case HighlightingKind::Bracket:
     return OS << "Bracket";
   case HighlightingKind::InactiveCode:
     return OS << "InactiveCode";
   }
   llvm_unreachable("invalid HighlightingKind");
 }
+std::optional<HighlightingKind>
+highlightingKindFromString(llvm::StringRef Name) {
+  static llvm::StringMap<HighlightingKind> Lookup = {
+      {"Variable", HighlightingKind::Variable},
+      {"LocalVariable", HighlightingKind::LocalVariable},
+      {"Parameter", HighlightingKind::Parameter},
+      {"Function", HighlightingKind::Function},
+      {"Method", HighlightingKind::Method},
+      {"StaticMethod", HighlightingKind::StaticMethod},
+      {"Field", HighlightingKind::Field},
+      {"StaticField", HighlightingKind::StaticField},
+      {"Class", HighlightingKind::Class},
+      {"Interface", HighlightingKind::Interface},
+      {"Enum", HighlightingKind::Enum},
+      {"EnumConstant", HighlightingKind::EnumConstant},
+      {"Typedef", HighlightingKind::Typedef},
+      {"Type", HighlightingKind::Type},
+      {"Unknown", HighlightingKind::Unknown},
+      {"Namespace", HighlightingKind::Namespace},
+      {"TemplateParameter", HighlightingKind::TemplateParameter},
+      {"Concept", HighlightingKind::Concept},
+      {"Primitive", HighlightingKind::Primitive},
+      {"Macro", HighlightingKind::Macro},
+      {"Modifier", HighlightingKind::Modifier},
+      {"Operator", HighlightingKind::Operator},
+      {"Bracket", HighlightingKind::Bracket},
+      {"InactiveCode", HighlightingKind::InactiveCode},
+  };
+
+  auto It = Lookup.find(Name);
+  return It != Lookup.end() ? std::make_optional(It->getValue()) : std::nullopt;
+}
 llvm::raw_ostream &operator<<(llvm::raw_ostream &OS, HighlightingModifier K) {
   switch (K) {
   case HighlightingModifier::Declaration:
     return OS << "decl"; // abbreviation for common case
   case HighlightingModifier::Definition:
     return OS << "def"; // abbrevation for common case
   case HighlightingModifier::ConstructorOrDestructor:
     return OS << "constrDestr";
   default:
     return OS << toSemanticTokenModifier(K);
   }
 }
+std::optional<HighlightingModifier>
+highlightingModifierFromString(llvm::StringRef Name) {
+  static llvm::StringMap<HighlightingModifier> Lookup = {
+      {"Declaration", HighlightingModifier::Declaration},
+      {"Definition", HighlightingModifier::Definition},
+      {"Deprecated", HighlightingModifier::Deprecated},
+      {"Deduced", HighlightingModifier::Deduced},
+      {"Readonly", HighlightingModifier::Readonly},
+      {"Static", HighlightingModifier::Static},
+      {"Abstract", HighlightingModifier::Abstract},
+      {"Virtual", HighlightingModifier::Virtual},
+      {"DependentName", HighlightingModifier::DependentName},
+      {"DefaultLibrary", HighlightingModifier::DefaultLibrary},
+      {"UsedAsMutableReference", HighlightingModifier::UsedAsMutableReference},
+      {"UsedAsMutablePointer", HighlightingModifier::UsedAsMutablePointer},
+      {"ConstructorOrDestructor",
+       HighlightingModifier::ConstructorOrDestructor},
+      {"UserDefined", HighlightingModifier::UserDefined},
+      {"FunctionScope", HighlightingModifier::FunctionScope},
+      {"ClassScope", HighlightingModifier::ClassScope},
+      {"FileScope", HighlightingModifier::FileScope},
+      {"GlobalScope", HighlightingModifier::GlobalScope},
+  };
+
+  auto It = Lookup.find(Name);
+  return It != Lookup.end() ? std::make_optional(It->getValue()) : std::nullopt;
+}
 
 bool operator==(const HighlightingToken &L, const HighlightingToken &R) {
   return std::tie(L.R, L.Kind, L.Modifiers) ==
          std::tie(R.R, R.Kind, R.Modifiers);
 }
 bool operator<(const HighlightingToken &L, const HighlightingToken &R) {
   return std::tie(L.R, L.Kind, L.Modifiers) <
          std::tie(R.R, R.Kind, R.Modifiers);

clang-tools-extra/clangd/unittests/ConfigYAMLTests.cpp

auto Results =
       Fragment::parseYAML(YAML.code(), "config.yaml", Diags.callback());
   ASSERT_THAT(Diags.Diagnostics, IsEmpty());
   ASSERT_EQ(Results.size(), 1u);
   EXPECT_THAT(Results[0].InlayHints.Enabled, llvm::ValueIs(val(false)));
   EXPECT_THAT(Results[0].InlayHints.ParameterNames, llvm::ValueIs(val(true)));
   EXPECT_EQ(Results[0].InlayHints.DeducedTypes, std::nullopt);
 }
 
+TEST(ParseYAML, SemanticTokens) {
+  CapturedDiags Diags;
+  Annotations YAML(R"yaml(
+SemanticTokens:
+  DisabledKinds: [ Operator, InactiveCode]
+  DisabledModifiers: Readonly
+  )yaml");
+  auto Results =
+      Fragment::parseYAML(YAML.code(), "config.yaml", Diags.callback());
+  ASSERT_THAT(Diags.Diagnostics, IsEmpty());
+  ASSERT_EQ(Results.size(), 1u);
+  EXPECT_THAT(Results[0].SemanticTokens.DisabledKinds,
+              ElementsAre(val("Operator"), val("InactiveCode")));
+  EXPECT_THAT(Results[0].SemanticTokens.DisabledModifiers,
+              ElementsAre(val("Readonly")));
+}
+
 TEST(ParseYAML, IncludesIgnoreHeader) {
   CapturedDiags Diags;
   Annotations YAML(R"yaml(
 Diagnostics:
   Includes:
     IgnoreHeader: [foo, bar]
   )yaml");
   auto Results =

clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp

 //==- SemanticHighlightingTests.cpp - SemanticHighlighting tests-*- C++ -* -==//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "Annotations.h"
+#include "Config.h"
 #include "Protocol.h"
 #include "SemanticHighlighting.h"
 #include "SourceCode.h"
 #include "TestTU.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/Error.h"
o]] [[bar]])cpp";
   EXPECT_EQ(Toks[2].deltaStart, 0u);
   EXPECT_EQ(Toks[2].length, 1u);
 
   // bar
   EXPECT_EQ(Toks[3].deltaLine, 0u);
   EXPECT_EQ(Toks[3].deltaStart, 2u);
   EXPECT_EQ(Toks[3].length, 3u);
 }
+
+TEST(SemanticHighlighting, WithHighlightingFilter) {
+  llvm::StringRef AnnotatedCode = R"cpp(
+int *$Variable[[x]] = new int;
+)cpp";
+  Config Cfg;
+  Cfg.SemanticTokens.DisabledKinds = {"Operator"};
+  Cfg.SemanticTokens.DisabledModifiers = {"Declaration", "Definition"};
+  WithContextValue WithCfg(Config::Key, std::move(Cfg));
+  checkHighlightings(AnnotatedCode, {}, ~ScopeModifierMask);
+}
 } // namespace
 } // namespace clangd
 } // namespace clang