[clangd] WIP/RFC: Prototype for semantic highlighting proposal

[nik]

Summary

TODO: Is the visitor the right approach?
TODO: Add tests
TODO: Test that only highlightings for main file are returned, but not headers.

...as specified at

https://github.com/microsoft/vscode-languageserver-node/pull/367

See also

https://github.com/microsoft/vscode-languageserver-node/issues/368

So far, this is an *incomplete* implementation providing highlighting
for identifiers of VarDecl/DeclRefExpr.

[nik]

Any interest in having the *proposal* implemented?

While there is a client implementation available in the theia ide (https://github.com/theia-ide/theia/pull/2332), I have only tested this against Qt Creator's client implementation (also work in progress).

[hokein]

Yeah, we definitely have interest in this feature, and our intern @jvikstrom will work on this feature this summer.

You probably missed the discussion on clangd-dev mailing list, @nridge also has an unfinished prototype for semantic highlighting.

To avoid multiple people working on the same feature, I think the plan is that @jvikstrom will pick up the current stuff, and continue working/improving it.

[nridge]

In D63331#1543441, @hokein wrote:

To avoid multiple people working on the same feature, I think the plan is that @jvikstrom will pick up the current stuff, and continue working/improving it.

Sounds good! I'm excited to see this moving forward :)

I've had a look at this patch, and it's very similar to my prototype implementation in D61842. Mine handles some more cases in the RecursiveASTVIsitor, especially around dependent names inside templates, and has corresponding test cases that you may want to reuse @jvikstrom.

[nik]

In D63331#1543441, @hokein wrote:

Yeah, we definitely have interest in this feature, and our intern @jvikstrom will work on this feature this summer.

"this summer" depends on the location, so can you concretize this? :)

You probably missed the discussion on clangd-dev mailing list, @nridge also has an unfinished prototype for semantic highlighting.

Yes, I wasn't aware that there is a clangd-dev mailing list and thus was not subscribed to it. Subscribed now.

I'm glad that I haven't put more work into this ;)

To avoid multiple people working on the same feature, I think the plan is that @jvikstrom will pick up the current stuff, and continue working/improving it.

Great. The patch from @nridge is more advanced so I'll abandon this one.

[hokein]

In D63331#1548020, @nik wrote:

In D63331#1543441, @hokein wrote:

Yeah, we definitely have interest in this feature, and our intern @jvikstrom will work on this feature this summer.

"this summer" depends on the location, so can you concretize this? :)

oops, sorry about that. I meant summer in Germany (June ~ the end of August).

You probably missed the discussion on clangd-dev mailing list, @nridge also has an unfinished prototype for semantic highlighting.

Yes, I wasn't aware that there is a clangd-dev mailing list and thus was not subscribed to it. Subscribed now.

I'm glad that I haven't put more work into this ;)

To avoid multiple people working on the same feature, I think the plan is that @jvikstrom will pick up the current stuff, and continue working/improving it.

Great. The patch from @nridge is more advanced so I'll abandon this one.

clang-tools-extra/clangd/CMakeLists.txt

add_clang_library(clangDaemon
   Headers.cpp
   IncludeFixer.cpp
   JSONTransport.cpp
   Logger.cpp
   Protocol.cpp
   Quality.cpp
   RIFF.cpp
   Selection.cpp
+  SemanticHighlighting.cpp
   SourceCode.cpp
   Threading.cpp
   Trace.cpp
   TUScheduler.cpp
   URI.cpp
   XRefs.cpp
 
   index/Background.cpp
add_clang_library(clangDaemon
   index/dex/Dex.cpp
   index/dex/Iterator.cpp
   index/dex/PostingList.cpp
   index/dex/Trigram.cpp
 
   refactor/Rename.cpp
   refactor/Tweak.cpp
 
+  thirdparty/base64/base64.cpp
+
   LINK_LIBS
   clangAST
   clangASTMatchers
   clangBasic
   clangDriver
   clangFormat
   clangFrontend
   clangIndex

clang-tools-extra/clangd/ClangdLSPServer.h

 
 class SymbolIndex;
 
 /// This class exposes ClangdServer's capabilities via Language Server Protocol.
 ///
 /// MessageHandler binds the implemented LSP methods (e.g. onInitialize) to
 /// corresponding JSON-RPC methods ("initialize").
 /// The server also supports $/cancelRequest (MessageHandler provides this).
-class ClangdLSPServer : private DiagnosticsConsumer {
+class ClangdLSPServer : private DiagnosticsConsumer,
+                        private HighlightingsConsumer {
 public:
   /// If \p CompileCommandsDir has a value, compile_commands.json will be
   /// loaded only from \p CompileCommandsDir. Otherwise, clangd will look
   /// for compile_commands.json in all parent directories of each file.
   /// If UseDirBasedCDB is false, compile commands are not read from disk.
   // FIXME: Clean up signature around CDBs.
   ClangdLSPServer(Transport &Transp, const FileSystemProvider &FSProvider,
                   const clangd::CodeCompleteOptions &CCOpts,
                   llvm::Optional<Path> CompileCommandsDir, bool UseDirBasedCDB,
                   llvm::Optional<OffsetEncoding> ForcedOffsetEncoding,
                   const ClangdServer::Options &Opts);
   ~ClangdLSPServer();
 
   /// Run LSP server loop, communicating with the Transport provided in the
   /// constructor. This method must not be executed more than once.
   ///
   /// \return Whether we shut down cleanly with a 'shutdown' -> 'exit' sequence.
   bool run();
 
 private:
   // Implement DiagnosticsConsumer.
   void onDiagnosticsReady(PathRef File, std::vector<Diag> Diagnostics) override;
   void onFileUpdated(PathRef File, const TUStatus &Status) override;
 
+  // Implement HighlightingsConsumer
+  void onHighlightingsReady(PathRef File,
+                            HighlightingsByLine Highlightings) override;
+
   // LSP methods. Notifications have signature void(const Params&).
   // Calls have signature void(const Params&, Callback<Response>).
   void onInitialize(const InitializeParams &, Callback<llvm::json::Value>);
   void onShutdown(const ShutdownParams &, Callback<std::nullptr_t>);
   void onSync(const NoParams &, Callback<std::nullptr_t>);
   void onDocumentDidOpen(const DidOpenTextDocumentParams &);
   void onDocumentDidChange(const DidChangeTextDocumentParams &);
   void onDocumentDidClose(const DidCloseTextDocumentParams &);

clang-tools-extra/clangd/ClangdLSPServer.cpp

void ClangdLSPServer::onInitialize(const InitializeParams &Params,
   if (const auto &Dir = Params.initializationOptions.compilationDatabasePath)
     CompileCommandsDir = Dir;
   if (UseDirBasedCDB)
     BaseCDB = llvm::make_unique<DirectoryBasedGlobalCompilationDatabase>(
         CompileCommandsDir);
   CDB.emplace(BaseCDB.get(), Params.initializationOptions.fallbackFlags,
               ClangdServerOpts.ResourceDir);
   Server.emplace(*CDB, FSProvider, static_cast<DiagnosticsConsumer &>(*this),
-                 ClangdServerOpts);
+                 static_cast<HighlightingsConsumer &>(*this), ClangdServerOpts);
   applyConfiguration(Params.initializationOptions.ConfigSettings);
 
   CCOpts.EnableSnippets = Params.capabilities.CompletionSnippets;
   DiagOpts.EmbedFixesInDiagnostics = Params.capabilities.DiagnosticFixes;
   DiagOpts.SendDiagnosticCategory = Params.capabilities.DiagnosticCategory;
   DiagOpts.EmitRelatedLocations =
       Params.capabilities.DiagnosticRelatedInformation;
   if (Params.capabilities.WorkspaceSymbolKinds)
llvm::json::Object Result{
             {"referencesProvider", true},
             {"executeCommandProvider",
              llvm::json::Object{
                  {"commands",
                   {ExecuteCommandParams::CLANGD_APPLY_FIX_COMMAND,
                    ExecuteCommandParams::CLANGD_APPLY_TWEAK}},
              }},
             {"typeHierarchyProvider", true},
+            {"semanticHighlighting",
+             llvm::json::Object{
+                 {"scopes", {{"variable.other.cpp"}}},
+             }},
         }}}};
   if (NegotiatedOffsetEncoding)
     Result["offsetEncoding"] = *NegotiatedOffsetEncoding;
   Reply(std::move(Result));
 }
 
 void ClangdLSPServer::onShutdown(const ShutdownParams &Params,
                                  Callback<std::nullptr_t> Reply) {
void ClangdLSPServer::onFileUpdated(PathRef File, const TUStatus &Status) {
   // changing, and doesn't provide much value. We may want to emit status at a
   // reasonable time interval (e.g. 0.5s).
   if (Status.Action.S == TUAction::BuildingFile ||
       Status.Action.S == TUAction::RunningAction)
     return;
   notify("textDocument/clangd.fileStatus", Status.render(File));
 }
 
+void ClangdLSPServer::onHighlightingsReady(PathRef File,
+                                           HighlightingsByLine Highlightings) {
+  auto URI = URIForFile::canonicalize(File, /*TUPath=*/File);
+
+  std::vector<SemanticHighlightingInformation> LSPHighlightings;
+  for (auto &H : Highlightings) {
+    SemanticHighlightingInformation I;
+    I.line = H.first;
+    if (!H.second.empty())
+      I.tokens = toTokensString(H.second);
+    LSPHighlightings.emplace_back(I);
+  }
+
+  notify("textDocument/semanticHighlighting",
+         llvm::json::Object{
+             {"textDocument", llvm::json::Object{{"version", 1}, {"uri", URI}}},
+             {"lines", std::move(LSPHighlightings)},
+         });
+}
+
 void ClangdLSPServer::reparseOpenedFiles() {
   for (const Path &FilePath : DraftMgr.getActiveFiles())
     Server->addDocument(FilePath, *DraftMgr.getDraft(FilePath),
                         WantDiagnostics::Auto);
 }
 
 } // namespace clangd
 } // namespace clang

clang-tools-extra/clangd/ClangdServer.h

 #include <future>
 #include <string>
 #include <type_traits>
 #include <utility>
 
 namespace clang {
 namespace clangd {
 
+// TODO: Merge with DiagnosticsConsumer to "DocumentAnnotationsConsumer"?
+class HighlightingsConsumer {
+public:
+  virtual ~HighlightingsConsumer() = default;
+  virtual void onHighlightingsReady(PathRef File, HighlightingsByLine Infos){};
+};
+
 // FIXME: find a better name.
 class DiagnosticsConsumer {
 public:
   virtual ~DiagnosticsConsumer() = default;
 
   /// Called by ClangdServer when \p Diagnostics for \p File are ready.
   virtual void onDiagnosticsReady(PathRef File,
                                   std::vector<Diag> Diagnostics) = 0;
public:
   /// if compilation arguments changed on calls to forceReparse().
   ///
   /// After each parsing request finishes, ClangdServer reports diagnostics to
   /// \p DiagConsumer. Note that a callback to \p DiagConsumer happens on a
   /// worker thread. Therefore, instances of \p DiagConsumer must properly
   /// synchronize access to shared state.
   ClangdServer(const GlobalCompilationDatabase &CDB,
                const FileSystemProvider &FSProvider,
-               DiagnosticsConsumer &DiagConsumer, const Options &Opts);
+               DiagnosticsConsumer &DiagConsumer,
+               HighlightingsConsumer &HighlightConsumer, const Options &Opts);
 
   /// Add a \p File to the list of tracked C++ files or update the contents if
   /// \p File is already tracked. Also schedules parsing of the AST for it on a
   /// separate thread. When the parsing is complete, DiagConsumer passed in
   /// constructor will receive onDiagnosticsReady callback.
   void addDocument(PathRef File, StringRef Contents,
                    WantDiagnostics WD = WantDiagnostics::Auto);
 

clang-tools-extra/clangd/ClangdServer.cpp

 #include <mutex>
 
 namespace clang {
 namespace clangd {
 namespace {
 
 // Update the FileIndex with new ASTs and plumb the diagnostics responses.
 struct UpdateIndexCallbacks : public ParsingCallbacks {
-  UpdateIndexCallbacks(FileIndex *FIndex, DiagnosticsConsumer &DiagConsumer)
-      : FIndex(FIndex), DiagConsumer(DiagConsumer) {}
+  UpdateIndexCallbacks(FileIndex *FIndex, DiagnosticsConsumer &DiagConsumer,
+                       HighlightingsConsumer &HighlightConsumer)
+      : FIndex(FIndex), DiagConsumer(DiagConsumer),
+        HighlightConsumer(HighlightConsumer) {}
 
   void onPreambleAST(PathRef Path, ASTContext &Ctx,
                      std::shared_ptr<clang::Preprocessor> PP,
                      const CanonicalIncludes &CanonIncludes) override {
     if (FIndex)
       FIndex->updatePreamble(Path, Ctx, std::move(PP), CanonIncludes);
   }
 
   void onMainAST(PathRef Path, ParsedAST &AST) override {
     if (FIndex)
       FIndex->updateMain(Path, AST);
   }
 
   void onDiagnostics(PathRef File, std::vector<Diag> Diags) override {
     DiagConsumer.onDiagnosticsReady(File, std::move(Diags));
   }
 
+  void onSemanticHighlight(PathRef File,
+                           HighlightingsByLine Highlightings) override {
+    HighlightConsumer.onHighlightingsReady(File, std::move(Highlightings));
+  }
+
   void onFileUpdated(PathRef File, const TUStatus &Status) override {
     DiagConsumer.onFileUpdated(File, Status);
   }
 
 private:
   FileIndex *FIndex;
   DiagnosticsConsumer &DiagConsumer;
+  HighlightingsConsumer &HighlightConsumer;
 };
 } // namespace
 
 ClangdServer::Options ClangdServer::optsForTest() {
   ClangdServer::Options Opts;
   Opts.UpdateDebounce = std::chrono::steady_clock::duration::zero(); // Faster!
   Opts.StorePreamblesInMemory = true;
   Opts.AsyncThreadsCount = 4; // Consistent!
   return Opts;
 }
 
 ClangdServer::ClangdServer(const GlobalCompilationDatabase &CDB,
                            const FileSystemProvider &FSProvider,
                            DiagnosticsConsumer &DiagConsumer,
+                           HighlightingsConsumer &HighlightConsumer,
                            const Options &Opts)
     : FSProvider(FSProvider),
       DynamicIdx(Opts.BuildDynamicSymbolIndex
                      ? new FileIndex(Opts.HeavyweightDynamicSymbolIndex)
                      : nullptr),
       GetClangTidyOptions(Opts.GetClangTidyOptions),
       SuggestMissingIncludes(Opts.SuggestMissingIncludes),
       WorkspaceRoot(Opts.WorkspaceRoot),
       // Pass a callback into `WorkScheduler` to extract symbols from a newly
       // parsed file and rebuild the file index synchronously each time an AST
       // is parsed.
       // FIXME(ioeric): this can be slow and we may be able to index on less
       // critical paths.
       WorkScheduler(CDB, Opts.AsyncThreadsCount, Opts.StorePreamblesInMemory,
-                    llvm::make_unique<UpdateIndexCallbacks>(DynamicIdx.get(),
-                                                            DiagConsumer),
+                    llvm::make_unique<UpdateIndexCallbacks>(
+                        DynamicIdx.get(), DiagConsumer, HighlightConsumer),
                     Opts.UpdateDebounce, Opts.RetentionPolicy) {
   // Adds an index to the stack, at higher priority than existing indexes.
   auto AddIndex = [&](SymbolIndex *Idx) {
     if (this->Index != nullptr) {
       MergedIdx.push_back(llvm::make_unique<MergedIndex>(Idx, this->Index));
       this->Index = MergedIdx.back().get();
     } else {
       this->Index = Idx;

clang-tools-extra/clangd/ClangdUnit.h

 
 #include "Compiler.h"
 #include "Diagnostics.h"
 #include "FS.h"
 #include "Function.h"
 #include "Headers.h"
 #include "Path.h"
 #include "Protocol.h"
+#include "SemanticHighlighting.h"
 #include "index/CanonicalIncludes.h"
 #include "index/Index.h"
 #include "clang/Frontend/FrontendAction.h"
 #include "clang/Frontend/PrecompiledPreamble.h"
 #include "clang/Lex/Preprocessor.h"
 #include "clang/Serialization/ASTBitCodes.h"
 #include "clang/Tooling/CompilationDatabase.h"
 #include "clang/Tooling/Core/Replacement.h"
public:
   }
 
   /// This function returns top-level decls present in the main file of the AST.
   /// The result does not include the decls that come from the preamble.
   /// (These should be const, but RecursiveASTVisitor requires Decl*).
   ArrayRef<Decl *> getLocalTopLevelDecls();
 
   const std::vector<Diag> &getDiagnostics() const;
+  const HighlightingsByLine &getHighlightings() const;
 
   /// Returns the esitmated size of the AST and the accessory structures, in
   /// bytes. Does not include the size of the preamble.
   std::size_t getUsedBytes() const;
   const IncludeStructure &getIncludeStructure() const;
   const CanonicalIncludes &getCanonicalIncludes() const;
 
 private:
   ParsedAST(std::shared_ptr<const PreambleData> Preamble,
             std::unique_ptr<CompilerInstance> Clang,
             std::unique_ptr<FrontendAction> Action,
             std::vector<Decl *> LocalTopLevelDecls, std::vector<Diag> Diags,
-            IncludeStructure Includes, CanonicalIncludes CanonIncludes);
+            HighlightingsByLine Highlightings, IncludeStructure Includes,
+            CanonicalIncludes CanonIncludes);
 
   // In-memory preambles must outlive the AST, it is important that this member
   // goes before Clang and Action.
   std::shared_ptr<const PreambleData> Preamble;
   // We store an "incomplete" FrontendAction (i.e. no EndSourceFile was called
   // on it) and CompilerInstance used to run it. That way we don't have to do
   // complex memory management of all Clang structures on our own. (They are
   // stored in CompilerInstance and cleaned up by
   // FrontendAction.EndSourceFile).
   std::unique_ptr<CompilerInstance> Clang;
   std::unique_ptr<FrontendAction> Action;
 
   // Data, stored after parsing.
   std::vector<Diag> Diags;
+  HighlightingsByLine Highlightings;
   // Top-level decls inside the current file. Not that this does not include
   // top-level decls from the preamble.
   std::vector<Decl *> LocalTopLevelDecls;
   IncludeStructure Includes;
   CanonicalIncludes CanonIncludes;
 };
 
 using PreambleParsedCallback =

clang-tools-extra/clangd/ClangdUnit.cpp

 #include "Headers.h"
 #include "IncludeFixer.h"
 #include "Logger.h"
 #include "SourceCode.h"
 #include "Trace.h"
 #include "index/CanonicalIncludes.h"
 #include "index/Index.h"
 #include "clang/AST/ASTContext.h"
+#include "clang/AST/RecursiveASTVisitor.h"
 #include "clang/Basic/LangOptions.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Basic/TokenKinds.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/CompilerInvocation.h"
 #include "clang/Frontend/FrontendActions.h"
 #include "clang/Frontend/Utils.h"
 #include "clang/Index/IndexDataConsumer.h"
private:
 
   const IncludeStructure &Includes;
   PPCallbacks *Delegate;
   const SourceManager &SM;
   Preprocessor &PP;
   const LangOptions &LangOpts;
 };
 
+class FindHighlightingsVisitor
+    : public RecursiveASTVisitor<FindHighlightingsVisitor> {
+public:
+  FindHighlightingsVisitor(SourceManager &SourceManager) : SM(SourceManager) {}
+
+  HighlightingsByLine find(ASTContext &Context) {
+    TraverseAST(Context);
+    return std::move(Lines);
+  }
+
+  bool VisitVarDecl(VarDecl *D) {
+    if (IdentifierInfo *Identifier = D->getIdentifier()) {
+      Highlighting H;
+      H.character = SM.getPresumedColumnNumber(D->getLocation(), nullptr) - 1;
+      H.length = Identifier->getLength();
+      H.scope = 0;
+
+      const auto Line = SM.getPresumedLineNumber(D->getLocation(), nullptr) - 1;
+      Lines[Line].emplace_back(H);
+    }
+
+    return true;
+  }
+
+  bool VisitDeclRefExpr(clang::DeclRefExpr *E) {
+    // TODO: De-dup
+    if (IdentifierInfo *Identifier = E->getDecl()->getIdentifier()) {
+      Highlighting H;
+      H.character = SM.getPresumedColumnNumber(E->getLocation(), nullptr) - 1;
+      H.length = Identifier->getLength();
+      H.scope = 0;
+
+      const auto Line = SM.getPresumedLineNumber(E->getLocation(), nullptr) - 1;
+      Lines[Line].emplace_back(H);
+    }
+
+    return true;
+  }
+
+private:
+  SourceManager &SM;
+  HighlightingsByLine Lines;
+};
+
 } // namespace
 
 void dumpAST(ParsedAST &AST, llvm::raw_ostream &OS) {
   AST.getASTContext().getTranslationUnitDecl()->dump(OS, true);
 }
 
 llvm::Optional<ParsedAST>
 ParsedAST::build(std::unique_ptr<CompilerInvocation> CI,
ParsedAST::build(std::unique_ptr<CompilerInvocation> CI,
   // However Action->EndSourceFile() would destroy the ASTContext!
   // So just inform the preprocessor of EOF, while keeping everything alive.
   Clang->getPreprocessor().EndSourceFile();
 
   std::vector<Diag> Diags = ASTDiags.take(CTContext.getPointer());
   // Add diagnostics from the preamble, if any.
   if (Preamble)
     Diags.insert(Diags.begin(), Preamble->Diags.begin(), Preamble->Diags.end());
+
+  HighlightingsByLine Highlightings;
+  {
+    Highlightings = FindHighlightingsVisitor(Clang->getSourceManager())
+                        .find(Clang->getASTContext());
+  }
+
   return ParsedAST(std::move(Preamble), std::move(Clang), std::move(Action),
                    std::move(ParsedDecls), std::move(Diags),
-                   std::move(Includes), std::move(CanonIncludes));
+                   std::move(Highlightings), std::move(Includes),
+                   std::move(CanonIncludes));
 }
 
 ParsedAST::ParsedAST(ParsedAST &&Other) = default;
 
 ParsedAST &ParsedAST::operator=(ParsedAST &&Other) = default;
 
 ParsedAST::~ParsedAST() {
   if (Action) {

 }
 
 llvm::ArrayRef<Decl *> ParsedAST::getLocalTopLevelDecls() {
   return LocalTopLevelDecls;
 }
 
 const std::vector<Diag> &ParsedAST::getDiagnostics() const { return Diags; }
 
+const HighlightingsByLine &ParsedAST::getHighlightings() const {
+  return Highlightings;
+}
+
 std::size_t ParsedAST::getUsedBytes() const {
   auto &AST = getASTContext();
   // FIXME(ibiryukov): we do not account for the dynamically allocated part of
   // Message and Fixes inside each diagnostic.
   std::size_t Total =
       clangd::getUsedBytes(LocalTopLevelDecls) + clangd::getUsedBytes(Diags);
 
   // FIXME: the rest of the function is almost a direct copy-paste from
: Preamble(std::move(Preamble)), Diags(std::move(Diags)),
       Includes(std::move(Includes)), MainFileMacros(std::move(MainFileMacros)),
       StatCache(std::move(StatCache)), CanonIncludes(std::move(CanonIncludes)) {
 }
 
 ParsedAST::ParsedAST(std::shared_ptr<const PreambleData> Preamble,
                      std::unique_ptr<CompilerInstance> Clang,
                      std::unique_ptr<FrontendAction> Action,
                      std::vector<Decl *> LocalTopLevelDecls,
-                     std::vector<Diag> Diags, IncludeStructure Includes,
-                     CanonicalIncludes CanonIncludes)
+                     std::vector<Diag> Diags, HighlightingsByLine Highlightings,
+                     IncludeStructure Includes, CanonicalIncludes CanonIncludes)
     : Preamble(std::move(Preamble)), Clang(std::move(Clang)),
       Action(std::move(Action)), Diags(std::move(Diags)),
+      Highlightings(std::move(Highlightings)),
       LocalTopLevelDecls(std::move(LocalTopLevelDecls)),
       Includes(std::move(Includes)), CanonIncludes(std::move(CanonIncludes)) {
   assert(this->Clang);
   assert(this->Action);
 }
 
 std::shared_ptr<const PreambleData>
 buildPreamble(PathRef FileName, CompilerInvocation &CI,

clang-tools-extra/clangd/Protocol.h

 struct DiagnosticRelatedInformation {
   /// The location of this related diagnostic information.
   Location location;
   /// The message of this related diagnostic information.
   std::string message;
 };
 llvm::json::Value toJSON(const DiagnosticRelatedInformation &);
 
+struct SemanticHighlightingInformation {
+    // The zero-based line position in the text document.
+    int line;
+
+    // A base64 encoded string representing every single highlighted characters
+    // with its start position, length and the "lookup table" index of of the
+    // semantic highlighting [TextMate
+    // scopes](https://manual.macromates.com/en/language_grammars). If the
+    // `tokens` is empty or not defined, then no highlighted positions are
+    // available for the line.
+    llvm::Optional<std::string> tokens;
+};
+llvm::json::Value toJSON(const SemanticHighlightingInformation &);
+
 struct CodeAction;
 struct Diagnostic {
   /// The range at which the message applies.
   Range range;
 
   /// The diagnostic's severity. Can be omitted. If omitted it is up to the
   /// client to interpret diagnostics as error, warning, info or hint.
   int severity = 0;

clang-tools-extra/clangd/Protocol.cpp

OE = llvm::StringSwitch<OffsetEncoding>(*Str)
            .Case("utf-32", OffsetEncoding::UTF32)
            .Default(OffsetEncoding::UnsupportedEncoding);
   return true;
 }
 llvm::raw_ostream &operator<<(llvm::raw_ostream &OS, OffsetEncoding Enc) {
   return OS << toString(Enc);
 }
 
+llvm::json::Value toJSON(const SemanticHighlightingInformation &I) {
+  llvm::json::Object O{
+      {"line", I.line},
+  };
+
+  if (I.tokens)
+    O["tokens"] = I.tokens;
+
+  return std::move(O);
+}
+
 } // namespace clangd
 } // namespace clang

clang-tools-extra/clangd/SemanticHighlighting.h

+//===--- SemanticHighlighting.h ----------------------------------*- C++-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_SEMANTIC_HIGHLIGHTING_H
+#define LLVM_CLANG_TOOLS_EXTRA_CLANGD_SEMANTIC_HIGHLIGHTING_H
+
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/Support/raw_ostream.h"
+#include <string>
+#include <vector>
+
+namespace clang {
+namespace clangd {
+
+struct Highlighting {
+  uint32_t character = 0; // zero-based
+  uint16_t length = 0;
+  uint16_t scope = 0;
+};
+using HighlightingsByLine = llvm::DenseMap<unsigned, std::vector<Highlighting>>;
+
+std::string toTokensString(const std::vector<Highlighting> &Highlightings);
+llvm::raw_ostream &operator<<(llvm::raw_ostream &OS, const Highlighting &I);
+
+} // namespace clangd
+} // namespace clang
+
+#endif // LLVM_CLANG_TOOLS_EXTRA_CLANGD_SEMANTIC_HIGHLIGHTING_H

clang-tools-extra/clangd/SemanticHighlighting.cpp

+//===--- SemanticHighlighting.cpp --------------------------------*- C++-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "SemanticHighlighting.h"
+#include "thirdparty/base64/base64.h"
+#include "llvm/ADT/STLExtras.h"
+
+namespace clang {
+namespace clangd {
+
+llvm::raw_ostream &operator<<(llvm::raw_ostream &OS, const Highlighting &I) {
+  return OS << "Highlighting{" << I.character << ',' << I.length << ','
+            << I.scope << '}';
+}
+
+// _Tokens_:
+//
+// Tokens are encoded in a memory friendly way straight from the wire. The
+// `tokens` string encapsulates multiple tokens as a `base64` encoded string. A
+// single semantic highlighting token can be interpreted as a range with
+// additional TextMate scopes information. The following properties can be
+// inferred from a single token: `character` is the zero-based offset where the
+// range starts. It is represented as a 32-bit unsigned integer. The `length`
+// property is the length of the range a semantic highlighting token. And
+// finally, it also carries the TextMate `scope` information as an integer
+// between zero and 2<sup>16</sup>-1 (inclusive) values. Clients must reuse the
+// `scopes` "lookup table" from the `initialize` request if they want to map the
+// `scope` index value to the actual TextMate scopes represented as a string.
+//
+// _Encoding the Tokens_:
+//
+// Following example shows how three individual tokens are encoded into its
+// `base64` form. Let assume, there is a series of token information (`[12, 15,
+// 1, 2, 5, 0, 7, 1000, 1]`) that can be interpreted as the following.
+//
+// ```json
+// [
+//  {
+//    "character": 12,
+//    "length": 15,
+//    "scope": 1
+//  },
+//  {
+//    "character": 2,
+//    "length": 5,
+//    "scope": 0
+//  },
+//  {
+//    "character": 7,
+//    "length": 1000,
+//    "scope": 1
+//  }
+// ]
+// ```
+//
+// The `character` (`12` )property will be stored as is but the `length` (`15`)
+// and the `scope` (`1`) will be stored as a single 32-bit unsigned integer. The
+// initial value of this 32-bit unsigned integer is zero. First, we set the
+// value of the `length`, then we make some room (2<sup>16</sup>) for the
+// `scope` by shifting the `length` 16 times to the left and applying a bitwise
+// OR with the value of the `scope`.
+//
+// ```
+//
+// 00000000000000000000000000000000 // initial
+// 00000000000000000000000000001111 // set the `length` value (15)
+// 00000000000011110000000000000000 // shift [<< 0x0000010] the `length` and
+//                                     make some space for the scope
+// 00000000000011110000000000000001 // bitwise OR the `scope` value (1)
+// ```
+std::string toTokensString(const std::vector<Highlighting> &Highlightings) {
+  const unsigned int BufferSize = Highlightings.size() * 8;
+  const auto Buffer = llvm::make_unique<unsigned char[]>(BufferSize);
+
+  // Convert Highlightings to byte stream...
+  unsigned Index = 0;
+  for (const auto &H : Highlightings) {
+    Buffer[Index++] = (H.character & 0xFF000000) >> 24;
+    Buffer[Index++] = (H.character & 0x00FF0000) >> 16;
+    Buffer[Index++] = (H.character & 0x0000FF00) >> 8;
+    Buffer[Index++] = (H.character & 0x000000FF);
+
+    Buffer[Index++] = (H.length & 0x0000FF00) >> 8;
+    Buffer[Index++] = (H.length & 0x000000FF);
+
+    Buffer[Index++] = (H.scope & 0x0000FF00) >> 8;
+    Buffer[Index++] = (H.scope & 0x000000FF);
+  }
+
+  // ...in order to convert it to a base64 string.
+  return base64_encode(Buffer.get(), BufferSize);
+}
+
+} // namespace clangd
+} // namespace clang

clang-tools-extra/clangd/TUScheduler.h

 //===--- TUScheduler.h -------------------------------------------*-C++-*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_TUSCHEDULER_H
 #define LLVM_CLANG_TOOLS_EXTRA_CLANGD_TUSCHEDULER_H
 
 #include "ClangdUnit.h"
 #include "Function.h"
 #include "GlobalCompilationDatabase.h"
+#include "SemanticHighlighting.h"
 #include "Threading.h"
 #include "index/CanonicalIncludes.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/StringMap.h"
 #include "llvm/ADT/StringRef.h"
 #include <future>
 
 namespace clang {
public:
   /// etc. Clients are expected to process only the AST nodes from the main file
   /// in this callback (obtained via ParsedAST::getLocalTopLevelDecls) to obtain
   /// optimal performance.
   virtual void onMainAST(PathRef Path, ParsedAST &AST) {}
 
   /// Called whenever the diagnostics for \p File are produced.
   virtual void onDiagnostics(PathRef File, std::vector<Diag> Diags) {}
 
+  virtual void onSemanticHighlight(PathRef File,
+                                   HighlightingsByLine Infos) {}
+
   /// Called whenever the TU status is updated.
   virtual void onFileUpdated(PathRef File, const TUStatus &Status) {}
 };
 
 /// Handles running tasks for ClangdServer and managing the resources (e.g.,
 /// preambles and ASTs) for opened files.
 /// TUScheduler is not thread-safe, only one thread should be providing updates
 /// and scheduling tasks.

clang-tools-extra/clangd/TUScheduler.cpp

auto Task = [=]() mutable {
     }
     // We want to report the diagnostics even if this update was cancelled.
     // It seems more useful than making the clients wait indefinitely if they
     // spam us with updates.
     // Note *AST can still be null if buildAST fails.
     if (*AST) {
       {
         std::lock_guard<std::mutex> Lock(DiagsMu);
-        if (ReportDiagnostics)
+        if (ReportDiagnostics) {
           Callbacks.onDiagnostics(FileName, (*AST)->getDiagnostics());
+          Callbacks.onSemanticHighlight(FileName, (*AST)->getHighlightings());
+        }
       }
       trace::Span Span("Running main AST callback");
       Callbacks.onMainAST(FileName, **AST);
       DiagsWereReported = true;
     }
     // Stash the AST in the cache for further use.
     IdleASTs.put(this, std::move(*AST));
   };

clang-tools-extra/clangd/test/initialize-params.test

 # CHECK-NEXT:        "commands": [
 # CHECK-NEXT:          "clangd.applyFix",
 # CHECK-NEXT:          "clangd.applyTweak"
 # CHECK-NEXT:        ]
 # CHECK-NEXT:      },
 # CHECK-NEXT:      "hoverProvider": true,
 # CHECK-NEXT:      "referencesProvider": true,
 # CHECK-NEXT:      "renameProvider": true,
+# CHECK-NEXT:      "semanticHighlighting": {
+# CHECK-NEXT:        "scopes": [
+# CHECK-NEXT:          [
+# CHECK-NEXT:            "variable.other.cpp"
+# CHECK-NEXT:          ]
+# CHECK-NEXT:        ]
+# CHECK-NEXT:      },
 # CHECK-NEXT:      "signatureHelpProvider": {
 # CHECK-NEXT:        "triggerCharacters": [
 # CHECK-NEXT:          "(",
 # CHECK-NEXT:          ","
 # CHECK-NEXT:        ]
 # CHECK-NEXT:      },
 # CHECK-NEXT:      "textDocumentSync": 2,
 # CHECK-NEXT:      "typeHierarchyProvider": true

clang-tools-extra/clangd/thirdparty/base64/base64.h

+//
+//  base64 encoding and decoding with C++.
+//  Version: 1.01.00
+//
+
+#ifndef BASE64_H_C0CE2A47_D10E_42C9_A27C_C883944E704A
+#define BASE64_H_C0CE2A47_D10E_42C9_A27C_C883944E704A
+
+#include <string>
+
+std::string base64_encode(unsigned char const* , unsigned int len);
+std::string base64_decode(std::string const& s);
+
+#endif /* BASE64_H_C0CE2A47_D10E_42C9_A27C_C883944E704A */

clang-tools-extra/clangd/thirdparty/base64/base64.cpp

+/* 
+   base64.cpp and base64.h
+
+   base64 encoding and decoding with C++.
+
+   Version: 1.01.00
+
+   Copyright (C) 2004-2017 René Nyffenegger
+
+   This source code is provided 'as-is', without any express or implied
+   warranty. In no event will the author be held liable for any damages
+   arising from the use of this software.
+
+   Permission is granted to anyone to use this software for any purpose,
+   including commercial applications, and to alter it and redistribute it
+   freely, subject to the following restrictions:
+
+   1. The origin of this source code must not be misrepresented; you must not
+      claim that you wrote the original source code. If you use this source code
+      in a product, an acknowledgment in the product documentation would be
+      appreciated but is not required.
+
+   2. Altered source versions must be plainly marked as such, and must not be
+      misrepresented as being the original source code.
+
+   3. This notice may not be removed or altered from any source distribution.
+
+   René Nyffenegger [e-mail]
+
+*/
+
+#include "base64.h"
+#include <iostream>
+
+static const std::string base64_chars = 
+             "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
+             "abcdefghijklmnopqrstuvwxyz"
+             "0123456789+/";
+
+
+static inline bool is_base64(unsigned char c) {
+  return (isalnum(c) || (c == '+') || (c == '/'));
+}
+
+std::string base64_encode(unsigned char const* bytes_to_encode, unsigned int in_len) {
+  std::string ret;
+  int i = 0;
+  int j = 0;
+  unsigned char char_array_3[3];
+  unsigned char char_array_4[4];
+
+  while (in_len--) {
+    char_array_3[i++] = *(bytes_to_encode++);
+    if (i == 3) {
+      char_array_4[0] = (char_array_3[0] & 0xfc) >> 2;
+      char_array_4[1] = ((char_array_3[0] & 0x03) << 4) + ((char_array_3[1] & 0xf0) >> 4);
+      char_array_4[2] = ((char_array_3[1] & 0x0f) << 2) + ((char_array_3[2] & 0xc0) >> 6);
+      char_array_4[3] = char_array_3[2] & 0x3f;
+
+      for(i = 0; (i <4) ; i++)
+        ret += base64_chars[char_array_4[i]];
+      i = 0;
+    }
+  }
+
+  if (i)
+  {
+    for(j = i; j < 3; j++)
+      char_array_3[j] = '\0';
+
+    char_array_4[0] = ( char_array_3[0] & 0xfc) >> 2;
+    char_array_4[1] = ((char_array_3[0] & 0x03) << 4) + ((char_array_3[1] & 0xf0) >> 4);
+    char_array_4[2] = ((char_array_3[1] & 0x0f) << 2) + ((char_array_3[2] & 0xc0) >> 6);
+
+    for (j = 0; (j < i + 1); j++)
+      ret += base64_chars[char_array_4[j]];
+
+    while((i++ < 3))
+      ret += '=';
+
+  }
+
+  return ret;
+
+}
+
+std::string base64_decode(std::string const& encoded_string) {
+  int in_len = encoded_string.size();
+  int i = 0;
+  int j = 0;
+  int in_ = 0;
+  unsigned char char_array_4[4], char_array_3[3];
+  std::string ret;
+
+  while (in_len-- && ( encoded_string[in_] != '=') && is_base64(encoded_string[in_])) {
+    char_array_4[i++] = encoded_string[in_]; in_++;
+    if (i ==4) {
+      for (i = 0; i <4; i++)
+        char_array_4[i] = base64_chars.find(char_array_4[i]);
+
+      char_array_3[0] = ( char_array_4[0] << 2       ) + ((char_array_4[1] & 0x30) >> 4);
+      char_array_3[1] = ((char_array_4[1] & 0xf) << 4) + ((char_array_4[2] & 0x3c) >> 2);
+      char_array_3[2] = ((char_array_4[2] & 0x3) << 6) +   char_array_4[3];
+
+      for (i = 0; (i < 3); i++)
+        ret += char_array_3[i];
+      i = 0;
+    }
+  }
+
+  if (i) {
+    for (j = 0; j < i; j++)
+      char_array_4[j] = base64_chars.find(char_array_4[j]);
+
+    char_array_3[0] = (char_array_4[0] << 2) + ((char_array_4[1] & 0x30) >> 4);
+    char_array_3[1] = ((char_array_4[1] & 0xf) << 4) + ((char_array_4[2] & 0x3c) >> 2);
+
+    for (j = 0; (j < i - 1); j++) ret += char_array_3[j];
+  }
+
+  return ret;
+}

clang-tools-extra/clangd/unittests/ClangdTests.cpp

 
 class ClangdVFSTest : public ::testing::Test {
 protected:
   std::string parseSourceAndDumpAST(
       PathRef SourceFileRelPath, llvm::StringRef SourceContents,
       std::vector<std::pair<PathRef, llvm::StringRef>> ExtraFiles = {},
       bool ExpectErrors = false) {
     MockFSProvider FS;
+    HighlightingsConsumer HighlightConsumer;
     ErrorCheckingDiagConsumer DiagConsumer;
     MockCompilationDatabase CDB;
-    ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+    ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                        ClangdServer::optsForTest());
     for (const auto &FileWithContents : ExtraFiles)
       FS.Files[testPath(FileWithContents.first)] = FileWithContents.second;
 
     auto SourceFilename = testPath(SourceFileRelPath);
     Server.addDocument(SourceFilename, SourceContents);
     auto Result = dumpASTWithoutMemoryLocs(Server, SourceFilename);
     EXPECT_TRUE(Server.blockUntilIdleForTest()) << "Waiting for diagnostics";
     EXPECT_EQ(ExpectErrors, DiagConsumer.hadErrorInLastDiags());
     return Result;
   }
+
+  HighlightingsConsumer HighlightConsumer;
 };
 
 TEST_F(ClangdVFSTest, Parse) {
   // FIXME: figure out a stable format for AST dumps, so that we can check the
   // output of the dump itself is equal to the expected one, not just that it's
   // different.
   auto Empty = parseSourceAndDumpAST("foo.cpp", "", {});
   auto OneDecl = parseSourceAndDumpAST("foo.cpp", "int a;", {});
)cpp";
   parseSourceAndDumpAST("foo.cpp", SourceContents, {{"foo.h", "int a;"}},
                         /*ExpectErrors=*/false);
 }
 
 TEST_F(ClangdVFSTest, Reparse) {
   MockFSProvider FS;
   ErrorCheckingDiagConsumer DiagConsumer;
   MockCompilationDatabase CDB;
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
 
   const auto SourceContents = R"cpp(
 #include "foo.h"
 int b = a;
 )cpp";
 
   auto FooCpp = testPath("foo.cpp");
 
)cpp";
   EXPECT_EQ(DumpParse1, DumpParse2);
   EXPECT_NE(DumpParse1, DumpParseEmpty);
 }
 
 TEST_F(ClangdVFSTest, ReparseOnHeaderChange) {
   MockFSProvider FS;
   ErrorCheckingDiagConsumer DiagConsumer;
   MockCompilationDatabase CDB;
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
 
   const auto SourceContents = R"cpp(
 #include "foo.h"
 int b = a;
 )cpp";
 
   auto FooCpp = testPath("foo.cpp");
   auto FooH = testPath("foo.h");
void onDiagnosticsReady(PathRef File,
                             std::vector<Diag> Diagnostics) override {
       Got = Context::current().getExisting(Secret);
     }
     int Got;
   } DiagConsumer;
   MockCompilationDatabase CDB;
 
   // Verify that the context is plumbed to the FS provider and diagnostics.
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
   {
     WithContextValue Entrypoint(Secret, 42);
     Server.addDocument(testPath("foo.cpp"), "void main(){}");
   }
   ASSERT_TRUE(Server.blockUntilIdleForTest());
   EXPECT_EQ(FS.Got, 42);
   EXPECT_EQ(DiagConsumer.Got, 42);
 }
 
 // Only enable this test on Unix
 #ifdef LLVM_ON_UNIX
 TEST_F(ClangdVFSTest, SearchLibDir) {
   // Checks that searches for GCC installation is done through vfs.
   MockFSProvider FS;
   ErrorCheckingDiagConsumer DiagConsumer;
   MockCompilationDatabase CDB;
   CDB.ExtraClangFlags.insert(CDB.ExtraClangFlags.end(),
                              {"-xc++", "-target", "x86_64-linux-unknown",
                               "-m64", "--gcc-toolchain=/randomusr",
                               "-stdlib=libstdc++"});
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
 
   // Just a random gcc version string
   SmallString<8> Version("4.9.3");
 
   // A lib dir for gcc installation
   SmallString<64> LibDir("/randomusr/lib/gcc/x86_64-linux-gnu");
   llvm::sys::path::append(LibDir, Version);
 
)cpp";
   EXPECT_TRUE(DiagConsumer.hadErrorInLastDiags());
 }
 #endif // LLVM_ON_UNIX
 
 TEST_F(ClangdVFSTest, ForceReparseCompileCommand) {
   MockFSProvider FS;
   ErrorCheckingDiagConsumer DiagConsumer;
   MockCompilationDatabase CDB;
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
 
   auto FooCpp = testPath("foo.cpp");
   const auto SourceContents1 = R"cpp(
 template <class T>
 struct foo { T x; };
 )cpp";
   const auto SourceContents2 = R"cpp(
 template <class T>
)cpp";
   runAddDocument(Server, FooCpp, SourceContents2);
   EXPECT_FALSE(DiagConsumer.hadErrorInLastDiags());
 }
 
 TEST_F(ClangdVFSTest, ForceReparseCompileCommandDefines) {
   MockFSProvider FS;
   ErrorCheckingDiagConsumer DiagConsumer;
   MockCompilationDatabase CDB;
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
 
   auto FooCpp = testPath("foo.cpp");
   const auto SourceContents = R"cpp(
 #ifdef WITH_ERROR
 this
 #endif
 
 int main() { return 0; }
)cpp");
 
   Annotations BazSource(R"cpp(
 int hello;
 )cpp");
 
   MockFSProvider FS;
   MockCompilationDatabase CDB;
   MultipleErrorCheckingDiagConsumer DiagConsumer;
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
 
   auto FooCpp = testPath("foo.cpp");
   auto BarCpp = testPath("bar.cpp");
   auto BazCpp = testPath("baz.cpp");
 
   FS.Files[FooCpp] = "";
   FS.Files[BarCpp] = "";
   FS.Files[BazCpp] = "";
)cpp");
   EXPECT_TRUE(bool(Locations));
   EXPECT_THAT(*Locations, ElementsAre(DeclAt(FooCpp, FooSource.range("two"))));
 }
 
 TEST_F(ClangdVFSTest, MemoryUsage) {
   MockFSProvider FS;
   ErrorCheckingDiagConsumer DiagConsumer;
   MockCompilationDatabase CDB;
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer, ClangdServer::optsForTest());
 
   Path FooCpp = testPath("foo.cpp");
   const auto SourceContents = R"cpp(
 struct Something {
   int method();
 };
 )cpp";
   Path BarCpp = testPath("bar.cpp");
)cpp";
   EXPECT_THAT(Server.getUsedBytesPerFile(), IsEmpty());
 }
 
 TEST_F(ClangdVFSTest, InvalidCompileCommand) {
   MockFSProvider FS;
   ErrorCheckingDiagConsumer DiagConsumer;
   MockCompilationDatabase CDB;
 
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
 
   auto FooCpp = testPath("foo.cpp");
   // clang cannot create CompilerInvocation if we pass two files in the
   // CompileCommand. We pass the file in ExtraFlags once and CDB adds another
   // one in getCompileCommand().
   CDB.ExtraClangFlags.push_back(FooCpp);
 
   // Clang can't parse command args in that case, but we shouldn't crash.
)cpp";
   std::vector<RequestStats> ReqStats;
   ReqStats.reserve(FilesCount);
   for (unsigned FileIndex = 0; FileIndex < FilesCount; ++FileIndex)
     ReqStats.emplace_back();
 
   TestDiagConsumer DiagConsumer;
   {
     MockCompilationDatabase CDB;
-    ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+    ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                        ClangdServer::optsForTest());
 
     // Prepare some random distributions for the test.
     std::random_device RandGen;
 
     std::uniform_int_distribution<unsigned> FileIndexDist(0, FilesCount - 1);
     // Pass a text that contains compiler errors to addDocument in about 20% of
     // all requests.
     std::bernoulli_distribution ShouldHaveErrorsDist(0.2);
for (unsigned I = 0; I < FilesCount; ++I) {
     ASSERT_LE(Stats[I].HitsWithoutErrors, ReqStats[I].RequestsWithoutErrors);
   }
 }
 
 TEST_F(ClangdVFSTest, CheckSourceHeaderSwitch) {
   MockFSProvider FS;
   ErrorCheckingDiagConsumer DiagConsumer;
   MockCompilationDatabase CDB;
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
 
   auto SourceContents = R"cpp(
   #include "foo.h"
   int b = a;
   )cpp";
 
   auto FooCpp = testPath("foo.cpp");
   auto FooH = testPath("foo.h");
)cpp";
   MockFSProvider FS;
   FS.Files[FooCpp] = "";
 
   std::promise<void> StartSecondPromise;
   std::future<void> StartSecond = StartSecondPromise.get_future();
 
   NoConcurrentAccessDiagConsumer DiagConsumer(std::move(StartSecondPromise));
   MockCompilationDatabase CDB;
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
   Server.addDocument(FooCpp, SourceContentsWithErrors);
   StartSecond.wait();
   Server.addDocument(FooCpp, SourceContentsWithoutErrors);
   ASSERT_TRUE(Server.blockUntilIdleForTest()) << "Waiting for diagnostics";
   ASSERT_EQ(DiagConsumer.Count, 2); // Sanity check - we actually ran both?
 }
 
 TEST_F(ClangdVFSTest, FormatCode) {
   MockFSProvider FS;
   ErrorCheckingDiagConsumer DiagConsumer;
   MockCompilationDatabase CDB;
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
 
   auto Path = testPath("foo.cpp");
   std::string Code = R"cpp(
 #include "x.h"
 #include "y.h"
 
 void f(  )  {}
 )cpp";
)cpp";
   EXPECT_TRUE(static_cast<bool>(Changed));
   EXPECT_EQ(Expected, *Changed);
 }
 
 TEST_F(ClangdVFSTest, ChangedHeaderFromISystem) {
   MockFSProvider FS;
   ErrorCheckingDiagConsumer DiagConsumer;
   MockCompilationDatabase CDB;
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
 
   auto SourcePath = testPath("source/foo.cpp");
   auto HeaderPath = testPath("headers/foo.h");
   FS.Files[HeaderPath] = "struct X { int bar; };";
   Annotations Code(R"cpp(
     #include "foo.h"
 
     int main() {
public:
 
     // If relative paths are used, they are resolved with testPath().
     llvm::StringMap<std::string> Files;
     llvm::StringMap<unsigned> &CountStats;
   };
 
   llvm::StringMap<unsigned> CountStats;
   ListenStatsFSProvider FS(CountStats);
+  HighlightingsConsumer HighlightConsumer;
   ErrorCheckingDiagConsumer DiagConsumer;
   MockCompilationDatabase CDB;
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
 
   auto SourcePath = testPath("foo.cpp");
   auto HeaderPath = testPath("foo.h");
   FS.Files[HeaderPath] = "struct TestSym {};";
   Annotations Code(R"cpp(
     #include "foo.h"
 
     int main() {
TEST_F(ClangdVFSTest, FlagsWithPlugins) {
   MockCompilationDatabase CDB;
   CDB.ExtraClangFlags = {
       "-Xclang",
       "-add-plugin",
       "-Xclang",
       "random-plugin",
   };
   OverlayCDB OCDB(&CDB);
-  ClangdServer Server(OCDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  ClangdServer Server(OCDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
 
   auto FooCpp = testPath("foo.cpp");
   const auto SourceContents = "int main() { return 0; }";
   FS.Files[FooCpp] = FooCpp;
   Server.addDocument(FooCpp, SourceContents);
   auto Result = dumpASTWithoutMemoryLocs(Server, FooCpp);
   EXPECT_TRUE(Server.blockUntilIdleForTest()) << "Waiting for diagnostics";
   EXPECT_NE(Result, "<no-ast>");
 }
 
 TEST_F(ClangdVFSTest, FallbackWhenPreambleIsNotReady) {
   MockFSProvider FS;
   ErrorCheckingDiagConsumer DiagConsumer;
   MockCompilationDatabase CDB;
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
 
   auto FooCpp = testPath("foo.cpp");
    Annotations Code(R"cpp(
     namespace ns { int xyz; }
     using namespace ns;
     int main() {
        xy^
     })cpp");
public:
     std::vector<std::string> ExtraClangFlags;
 
   private:
     Notification &CanReturnCommand;
   };
 
   Notification CanReturnCommand;
   DelayedCompilationDatabase CDB(CanReturnCommand);
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
 
   auto FooCpp = testPath("foo.cpp");
   Annotations Code(R"cpp(
     namespace ns { int xyz; }
     using namespace ns;
     int main() {
        xy^
     })cpp");

clang-tools-extra/clangd/unittests/CodeCompleteTests.cpp

 // If IndexSymbols is non-empty, an index will be built and passed to opts.
 CodeCompleteResult completions(llvm::StringRef Text,
                                std::vector<Symbol> IndexSymbols = {},
                                clangd::CodeCompleteOptions Opts = {},
                                PathRef FilePath = "foo.cpp") {
   MockFSProvider FS;
   MockCompilationDatabase CDB;
   IgnoreDiagnostics DiagConsumer;
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  HighlightingsConsumer HighlightConsumer;
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
   return completions(Server, Text, std::move(IndexSymbols), std::move(Opts),
                      FilePath);
 }
 
 // Builds a server and runs code completion.
 // If IndexSymbols is non-empty, an index will be built and passed to opts.
 CodeCompleteResult completionsNoCompile(llvm::StringRef Text,
                                         std::vector<Symbol> IndexSymbols = {},
TEST(CompletionTest, IncludeInsertionPreprocessorIntegrationTests) {
   MockCompilationDatabase CDB;
   std::string Subdir = testPath("sub");
   std::string SearchDirArg = (Twine("-I") + Subdir).str();
   CDB.ExtraClangFlags = {SearchDirArg.c_str()};
   std::string BarHeader = testPath("sub/bar.h");
   FS.Files[BarHeader] = "";
 
   IgnoreDiagnostics DiagConsumer;
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  HighlightingsConsumer HighlightConsumer;
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
   auto BarURI = URI::create(BarHeader).toString();
   Symbol Sym = cls("ns::X");
   Sym.CanonicalDeclaration.FileURI = BarURI.c_str();
   Sym.IncludeHeaders.emplace_back(BarURI, 1);
   // Shoten include path based on search dirctory and insert.
   auto Results = completions(Server,
                              R"cpp(
           int main() { ns::^ }
EXPECT_THAT(Results.Completions, ElementsAre(AllOf(Named("X"), Labeled("X"),
                                                      Not(InsertInclude()))));
 }
 
 TEST(CompletionTest, NoIncludeInsertionWhenDeclFoundInFile) {
   MockFSProvider FS;
   MockCompilationDatabase CDB;
 
   IgnoreDiagnostics DiagConsumer;
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  HighlightingsConsumer HighlightConsumer;
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
   Symbol SymX = cls("ns::X");
   Symbol SymY = cls("ns::Y");
   std::string BarHeader = testPath("bar.h");
   auto BarURI = URI::create(BarHeader).toString();
   SymX.CanonicalDeclaration.FileURI = BarURI.c_str();
   SymY.CanonicalDeclaration.FileURI = BarURI.c_str();
   SymX.IncludeHeaders.emplace_back("<bar>", 1);
   SymY.IncludeHeaders.emplace_back("<bar>", 1);
EXPECT_THAT(Results.Completions,
               ElementsAre(AllOf(Named("X"), Not(InsertInclude())),
                           AllOf(Named("Y"), Not(InsertInclude()))));
 }
 
 TEST(CompletionTest, IndexSuppressesPreambleCompletions) {
   MockFSProvider FS;
   MockCompilationDatabase CDB;
   IgnoreDiagnostics DiagConsumer;
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  HighlightingsConsumer HighlightConsumer;
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
 
   FS.Files[testPath("bar.h")] =
       R"cpp(namespace ns { struct preamble { int member; }; })cpp";
   auto File = testPath("foo.cpp");
   Annotations Test(R"cpp(
       #include "bar.h"
       namespace ns { int local; }
       void f() { ns::^; }
)cpp")
                      .Completions;
   EXPECT_THAT(Results, ElementsAre(Named("ifndef")));
 }
 
 TEST(CompletionTest, DynamicIndexIncludeInsertion) {
   MockFSProvider FS;
   MockCompilationDatabase CDB;
   IgnoreDiagnostics DiagConsumer;
+  HighlightingsConsumer HighlightConsumer;
   ClangdServer::Options Opts = ClangdServer::optsForTest();
   Opts.BuildDynamicSymbolIndex = true;
-  ClangdServer Server(CDB, FS, DiagConsumer, Opts);
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer, Opts);
 
   FS.Files[testPath("foo_header.h")] = R"cpp(
     #pragma once
     struct Foo {
        // Member doc
        int foo();
     };
   )cpp";
EXPECT_THAT(completions(Server, "Foo^ foo;").Completions,
                                            '"'),
                                 InsertInclude())));
 }
 
 TEST(CompletionTest, DynamicIndexMultiFile) {
   MockFSProvider FS;
   MockCompilationDatabase CDB;
   IgnoreDiagnostics DiagConsumer;
+  HighlightingsConsumer HighlightConsumer;
   auto Opts = ClangdServer::optsForTest();
   Opts.BuildDynamicSymbolIndex = true;
-  ClangdServer Server(CDB, FS, DiagConsumer, Opts);
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer, Opts);
 
   FS.Files[testPath("foo.h")] = R"cpp(
       namespace ns { class XYZ {}; void foo(int x) {} }
   )cpp";
   runAddDocument(Server, testPath("foo.cpp"), R"cpp(
       #include "foo.h"
   )cpp");
 
SignatureHelp signatures(llvm::StringRef Text, Position Point,
                          std::vector<Symbol> IndexSymbols = {}) {
   std::unique_ptr<SymbolIndex> Index;
   if (!IndexSymbols.empty())
     Index = memIndex(IndexSymbols);
 
   MockFSProvider FS;
   MockCompilationDatabase CDB;
   IgnoreDiagnostics DiagConsumer;
+  HighlightingsConsumer HighlightConsumer;
   ClangdServer::Options Opts = ClangdServer::optsForTest();
   Opts.StaticIndex = Index.get();
 
-  ClangdServer Server(CDB, FS, DiagConsumer, Opts);
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer, Opts);
   auto File = testPath("foo.cpp");
   runAddDocument(Server, File, Text);
   return llvm::cantFail(runSignatureHelp(Server, File, Point));
 }
 
 SignatureHelp signatures(llvm::StringRef Text,
                          std::vector<Symbol> IndexSymbols = {}) {
   Annotations Test(Text);
TEST(CompletionTest, DocumentationFromChangedFileCrash) {
   FS.Files[FooH] = R"cpp(
     // this is my documentation comment.
     int func();
   )cpp";
   FS.Files[FooCpp] = "";
 
   MockCompilationDatabase CDB;
   IgnoreDiagnostics DiagConsumer;
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  HighlightingsConsumer HighlightConsumer;
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
 
   Annotations Source(R"cpp(
     #include "foo.h"
     int func() {
       // This makes sure we have func from header in the AST.
     }
     int a = fun^
   )cpp");

 
 TEST(CompletionTest, NonDocComments) {
   MockFSProvider FS;
   auto FooCpp = testPath("foo.cpp");
   FS.Files[FooCpp] = "";
 
   MockCompilationDatabase CDB;
   IgnoreDiagnostics DiagConsumer;
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  HighlightingsConsumer HighlightConsumer;
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
 
   Annotations Source(R"cpp(
     // We ignore namespace comments, for rationale see CodeCompletionStrings.h.
     namespace comments_ns {
     }
 
     // ------------------
     int comments_foo();
EXPECT_THAT(
                            AllOf(IsDocumented(), Named("comments_qux"))));
 }
 
 TEST(CompletionTest, CompleteOnInvalidLine) {
   auto FooCpp = testPath("foo.cpp");
 
   MockCompilationDatabase CDB;
   IgnoreDiagnostics DiagConsumer;
+  HighlightingsConsumer HighlightConsumer;
   MockFSProvider FS;
   FS.Files[FooCpp] = "// empty file";
 
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
   // Run completion outside the file range.
   Position Pos;
   Pos.line = 100;
   Pos.character = 0;
   EXPECT_THAT_EXPECTED(
       runCodeComplete(Server, FooCpp, Pos, clangd::CodeCompleteOptions()),
       Failed());
 }
TEST(CompletionTest, CodeCompletionContext) {
 
   EXPECT_THAT(Results.Context, CodeCompletionContext::CCC_DotMemberAccess);
 }
 
 TEST(CompletionTest, FixItForArrowToDot) {
   MockFSProvider FS;
   MockCompilationDatabase CDB;
   IgnoreDiagnostics DiagConsumer;
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  HighlightingsConsumer HighlightConsumer;
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
 
   CodeCompleteOptions Opts;
   Opts.IncludeFixIts = true;
   Annotations TestCode(
       R"cpp(
         class Auxilary {
          public:
           void AuxFunction();
for (const auto &C : Results.Completions) {
     }
   }
 }
 
 TEST(CompletionTest, FixItForDotToArrow) {
   MockFSProvider FS;
   MockCompilationDatabase CDB;
   IgnoreDiagnostics DiagConsumer;
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  HighlightingsConsumer HighlightConsumer;
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
 
   CodeCompleteOptions Opts;
   Opts.IncludeFixIts = true;
   Annotations TestCode(
       R"cpp(
         class Auxilary {
          public:
           void AuxFunction();
TEST(CompletionTest, RenderWithFixItNonMerged) {
   EXPECT_EQ(R.textEdit->newText, "Foo::x");
   EXPECT_THAT(R.additionalTextEdits, UnorderedElementsAre(FixIt));
 }
 
 TEST(CompletionTest, CompletionTokenRange) {
   MockFSProvider FS;
   MockCompilationDatabase CDB;
   IgnoreDiagnostics DiagConsumer;
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  HighlightingsConsumer HighlightConsumer;
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
 
   constexpr const char *TestCodes[] = {
       R"cpp(
         class Auxilary {
          public:
           void AuxFunction();
         };
         void f() {
EXPECT_THAT(
           AllOf(Sig("foo([[int]]) -> int"), SigDoc("Overriden doc from sema")),
           AllOf(Sig("foo([[int]], [[int]]) -> int"), SigDoc("Doc from sema"))));
 }
 
 TEST(SignatureHelpTest, DynamicIndexDocumentation) {
   MockFSProvider FS;
   MockCompilationDatabase CDB;
   IgnoreDiagnostics DiagConsumer;
+  HighlightingsConsumer HighlightConsumer;
   ClangdServer::Options Opts = ClangdServer::optsForTest();
   Opts.BuildDynamicSymbolIndex = true;
-  ClangdServer Server(CDB, FS, DiagConsumer, Opts);
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer, Opts);
 
   FS.Files[testPath("foo.h")] = R"cpp(
     struct Foo {
        // Member doc
        int foo();
     };
   )cpp";
   Annotations FileContent(R"cpp(
for (llvm::StringRef Case : {
     EXPECT_EQ(WantName.begin(), Prefix.Name.begin()) << Case;
   }
 }
 
 TEST(CompletionTest, EnableSpeculativeIndexRequest) {
   MockFSProvider FS;
   MockCompilationDatabase CDB;
   IgnoreDiagnostics DiagConsumer;
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  HighlightingsConsumer HighlightConsumer;
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
 
   auto File = testPath("foo.cpp");
   Annotations Test(R"cpp(
       namespace ns1 { int abc; }
       namespace ns2 { int abc; }
       void f() { ns1::ab$1^; ns1::ab$2^; }
       void f2() { ns2::ab$3^; }
   )cpp");

 TEST(CompletionTest, NoInsertIncludeIfOnePresent) {
   MockFSProvider FS;
   MockCompilationDatabase CDB;
 
   std::string FooHeader = testPath("foo.h");
   FS.Files[FooHeader] = "";
 
   IgnoreDiagnostics DiagConsumer;
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  HighlightingsConsumer HighlightConsumer;
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
 
   std::string DeclFile = URI::create(testPath("foo")).toString();
   Symbol sym = func("Func");
   sym.CanonicalDeclaration.FileURI = DeclFile.c_str();
   sym.IncludeHeaders.emplace_back("\"foo.h\"", 2);
   sym.IncludeHeaders.emplace_back("\"bar.h\"", 1000);
 
   EXPECT_THAT(

 }
 
 TEST(CompletionTest, MacroFromPreamble) {
   MockFSProvider FS;
   MockCompilationDatabase CDB;
   std::string FooHeader = testPath("foo.h");
   FS.Files[FooHeader] = "#define CLANGD_PREAMBLE_HEADER x\n";
   IgnoreDiagnostics DiagConsumer;
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  HighlightingsConsumer HighlightConsumer;
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
   auto Results = completions(
       R"cpp(#include "foo.h"
           #define CLANGD_PREAMBLE_MAIN x
 
           int x = 0;
           #define CLANGD_MAIN x
           void f() { CLANGD_^ }
       )cpp",
TEST(CompletionTest, IncludedCompletionKinds) {
   MockFSProvider FS;
   MockCompilationDatabase CDB;
   std::string Subdir = testPath("sub");
   std::string SearchDirArg = (Twine("-I") + Subdir).str();
   CDB.ExtraClangFlags = {SearchDirArg.c_str()};
   std::string BarHeader = testPath("sub/bar.h");
   FS.Files[BarHeader] = "";
   IgnoreDiagnostics DiagConsumer;
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  HighlightingsConsumer HighlightConsumer;
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
   auto Results = completions(Server,
                              R"cpp(
         #include "^"
       )cpp");
   EXPECT_THAT(Results.Completions,
               AllOf(Has("sub/", CompletionItemKind::Folder),
                     Has("bar.h\"", CompletionItemKind::File)));
 }

clang-tools-extra/clangd/unittests/FindSymbolsTests.cpp

ClangdServer::Options optsForTests() {
   ServerOpts.WorkspaceRoot = testRoot();
   ServerOpts.BuildDynamicSymbolIndex = true;
   return ServerOpts;
 }
 
 class WorkspaceSymbolsTest : public ::testing::Test {
 public:
   WorkspaceSymbolsTest()
-      : Server(CDB, FSProvider, DiagConsumer, optsForTests()) {
+      : Server(CDB, FSProvider, DiagConsumer, HighlightConsumer,
+               optsForTests()) {
     // Make sure the test root directory is created.
     FSProvider.Files[testPath("unused")] = "";
     CDB.ExtraClangFlags = {"-xc++"};
   }
 
 protected:
   MockFSProvider FSProvider;
   MockCompilationDatabase CDB;
   IgnoreDiagnostics DiagConsumer;
+  HighlightingsConsumer HighlightConsumer;
   ClangdServer Server;
   int Limit = 0;
 
   std::vector<SymbolInformation> getSymbols(llvm::StringRef Query) {
     EXPECT_TRUE(Server.blockUntilIdleForTest()) << "Waiting for preamble";
     auto SymbolInfos = runWorkspaceSymbols(Server, Query, Limit);
     EXPECT_TRUE(bool(SymbolInfos)) << "workspaceSymbols returned an error";
     return *SymbolInfos;
EXPECT_THAT(
           AllOf(QName("Foo<bool, int>"), WithKind(SymbolKind::Class)),
           AllOf(QName("Foo<bool, int, 3>"), WithKind(SymbolKind::Class))));
 }
 
 namespace {
 class DocumentSymbolsTest : public ::testing::Test {
 public:
   DocumentSymbolsTest()
-      : Server(CDB, FSProvider, DiagConsumer, optsForTests()) {}
+      : Server(CDB, FSProvider, DiagConsumer, HighlightConsumer,
+               optsForTests()) {}
 
 protected:
   MockFSProvider FSProvider;
   MockCompilationDatabase CDB;
   IgnoreDiagnostics DiagConsumer;
+  HighlightingsConsumer HighlightConsumer;
   ClangdServer Server;
 
   std::vector<DocumentSymbol> getSymbols(PathRef File) {
     EXPECT_TRUE(Server.blockUntilIdleForTest()) << "Waiting for preamble";
     auto SymbolInfos = runDocumentSymbols(Server, File);
     EXPECT_TRUE(bool(SymbolInfos)) << "documentSymbols returned an error";
     return *SymbolInfos;
   }

clang-tools-extra/clangd/unittests/TUSchedulerTests.cpp

public:
     }
 
   private:
     std::mutex Mutex;
     std::vector<TUStatus> AllStatus;
   } CaptureTUStatus;
   MockFSProvider FS;
   MockCompilationDatabase CDB;
-  ClangdServer Server(CDB, FS, CaptureTUStatus, ClangdServer::optsForTest());
+  HighlightingsConsumer HighlightConsumer;
+  ClangdServer Server(CDB, FS, CaptureTUStatus, HighlightConsumer,
+                      ClangdServer::optsForTest());
   Annotations Code("int m^ain () {}");
 
   // We schedule the following tasks in the queue:
   //   [Update] [GoToDefinition]
   Server.addDocument(testPath("foo.cpp"), Code.code(), WantDiagnostics::Yes);
   Server.locateSymbolAt(testPath("foo.cpp"), Code.point(),
                         [](Expected<std::vector<LocatedSymbol>> Result) {
                           ASSERT_TRUE((bool)Result);

clang-tools-extra/clangd/unittests/XRefsTests.cpp

)cpp");
   // Make the compilation paths appear as ../src/foo.cpp in the compile
   // commands.
   SmallString<32> RelPathPrefix("..");
   llvm::sys::path::append(RelPathPrefix, "src");
   std::string BuildDir = testPath("build");
   MockCompilationDatabase CDB(BuildDir, RelPathPrefix);
 
   IgnoreDiagnostics DiagConsumer;
+  HighlightingsConsumer HighlightConsumer;
   MockFSProvider FS;
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
 
   // Fill the filesystem.
   auto FooCpp = testPath("src/foo.cpp");
   FS.Files[FooCpp] = "";
   auto HeaderInPreambleH = testPath("src/header_in_preamble.h");
   FS.Files[HeaderInPreambleH] = HeaderInPreambleAnnotations.code();
   auto HeaderNotInPreambleH = testPath("src/header_not_in_preamble.h");
   FS.Files[HeaderNotInPreambleH] = HeaderNotInPreambleAnnotations.code();
for (const OneTest &Test : Tests) {
     } else
       EXPECT_EQ("", Test.ExpectedHover.str()) << Test.Input;
   }
 }
 
 TEST(GoToInclude, All) {
   MockFSProvider FS;
   IgnoreDiagnostics DiagConsumer;
+  HighlightingsConsumer HighlightConsumer;
   MockCompilationDatabase CDB;
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
 
   auto FooCpp = testPath("foo.cpp");
   const char *SourceContents = R"cpp(
   #include ^"$2^foo.h$3^"
   #include "$4^invalid.h"
   int b = a;
   // test
   int foo;
TEST(GoToInclude, All) {
   EXPECT_THAT(*Locations, ElementsAre(Sym("foo.h", HeaderAnnotations.range())));
 }
 
 TEST(LocateSymbol, WithPreamble) {
   // Test stragety: AST should always use the latest preamble instead of last
   // good preamble.
   MockFSProvider FS;
   IgnoreDiagnostics DiagConsumer;
+  HighlightingsConsumer HighlightConsumer;
   MockCompilationDatabase CDB;
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  ClangdServer Server(CDB, FS, DiagConsumer, HighlightConsumer,
+                      ClangdServer::optsForTest());
 
   auto FooCpp = testPath("foo.cpp");
   // The trigger locations must be the same.
   Annotations FooWithHeader(R"cpp(#include "fo^o.h")cpp");
   Annotations FooWithoutHeader(R"cpp(double    [[fo^o]]();)cpp");
 
   FS.Files[FooCpp] = FooWithHeader.code();