[clangd] Remove support for pre-standard semanticHighlighting notification

[sammccall]

Summary

This is obsoleted by the standard semanticTokens request family.
As well as the protocol details, this allows us to remove a bunch of plumbing
around pushing highlights to clients.

This should not land until the new protocol has feature parity, see D77702.

[kadircet]

Thanks, LGTM!

there are also some comments on ASTWorker::PublishMu talking about semantic highlights publishing.

clang-tools-extra/clangd/SemanticHighlighting.h

15

not sure if this sentence will help anyone. if you choose to drop this one, i suppose the former can also be rephrased as Protocol details can be found at ...

[kadircet]

oh also in addition to This should not land until the new protocol has feature parity, see D77702., it might be better to hold on to it a little bit longer, since it is touching lots of pieces and might make cherry-picking some fixes that touch the same files harder to the release branch.

[sammccall]

In D95576#2527407, @kadircet wrote:

oh also in addition to This should not land until the new protocol has feature parity, see D77702., it might be better to hold on to it a little bit longer, since it is touching lots of pieces and might make cherry-picking some fixes that touch the same files harder to the release branch.

Yeah I don't want to land this for a couple of weeks in any case.
On the other hand, the release branch has sometimes been long-lived (months) and I don't think we should try to avoid invasive changes until the release has landed necessarily, it could eat a good fraction of the release cycle.

[sammccall]

Remove some comments

[sammccall]

Rebase, now the feature parity stuff is in

clang-tools-extra/clangd/ClangdLSPServer.h

public:
   /// Profiles resource-usage.
   void profile(MemoryTree &MT) const;
 
 private:
   // Implement ClangdServer::Callbacks.
   void onDiagnosticsReady(PathRef File, llvm::StringRef Version,
                           std::vector<Diag> Diagnostics) override;
   void onFileUpdated(PathRef File, const TUStatus &Status) override;
-  void
-  onHighlightingsReady(PathRef File, llvm::StringRef Version,
-                       std::vector<HighlightingToken> Highlightings) override;
   void onBackgroundIndexProgress(const BackgroundQueue::Stats &Stats) override;
 
   // LSP methods. Notifications have signature void(const Params&).
   // Calls have signature void(const Params&, Callback<Response>).
   void onInitialize(const InitializeParams &, Callback<llvm::json::Value>);
   void onInitialized(const InitializedParams &);
   void onShutdown(Callback<std::nullptr_t>);
   void onSync(Callback<std::nullptr_t>);
private:
   /// Requests a reparse of currently opened files using their latest source.
   /// This will typically only rebuild if something other than the source has
   /// changed (e.g. the CDB yields different flags, or files included in the
   /// preamble have been modified).
   void reparseOpenFilesIfNeeded(
       llvm::function_ref<bool(llvm::StringRef File)> Filter);
   void applyConfiguration(const ConfigurationSettings &Settings);
 
-  /// Sends a "publishSemanticHighlighting" notification to the LSP client.
-  void
-  publishTheiaSemanticHighlighting(const TheiaSemanticHighlightingParams &);
-
   /// Sends a "publishDiagnostics" notification to the LSP client.
   void publishDiagnostics(const PublishDiagnosticsParams &);
 
   /// Runs profiling and exports memory usage metrics if tracing is enabled and
   /// profiling hasn't happened recently.
   void maybeExportMemoryProfile();
   PeriodicThrottler ShouldProfile;
 
private:
   /// Used to indicate the ClangdLSPServer is being destroyed.
   std::atomic<bool> IsBeingDestroyed = {false};
 
   std::mutex FixItsMutex;
   typedef std::map<clangd::Diagnostic, std::vector<Fix>, LSPDiagnosticCompare>
       DiagnosticToReplacementMap;
   /// Caches FixIts per file and diagnostics
   llvm::StringMap<DiagnosticToReplacementMap> FixItsMap;
-  std::mutex HighlightingsMutex;
-  llvm::StringMap<std::vector<HighlightingToken>> FileToHighlightings;
   // Last semantic-tokens response, for incremental requests.
   std::mutex SemanticTokensMutex;
   llvm::StringMap<SemanticTokens> LastSemanticTokens;
 
   // Most code should not deal with Transport directly.
   // MessageHandler deals with incoming messages, use call() etc for outgoing.
   clangd::Transport &Transp;
   class MessageHandler;

clang-tools-extra/clangd/ClangdLSPServer.cpp

 CompletionItemKindBitset defaultCompletionItemKinds() {
   CompletionItemKindBitset Defaults;
   for (size_t I = CompletionItemKindMin;
        I <= static_cast<size_t>(CompletionItemKind::Reference); ++I)
     Defaults.set(I);
   return Defaults;
 }
 
-// Build a lookup table (HighlightingKind => {TextMate Scopes}), which is sent
-// to the LSP client.
-std::vector<std::vector<std::string>> buildHighlightScopeLookupTable() {
-  std::vector<std::vector<std::string>> LookupTable;
-  // HighlightingKind is using as the index.
-  for (int KindValue = 0; KindValue <= (int)HighlightingKind::LastKind;
-       ++KindValue)
-    LookupTable.push_back(
-        {std::string(toTextMateScope((HighlightingKind)(KindValue)))});
-  return LookupTable;
-}
-
 // Makes sure edits in \p FE are applicable to latest file contents reported by
 // editor. If not generates an error message containing information about files
 // that needs to be saved.
 llvm::Error validateEdits(const DraftStore &DraftMgr, const FileEdits &FE) {
   size_t InvalidFileCount = 0;
   llvm::StringRef LastInvalidFile;
   for (const auto &It : FE) {
     if (auto Draft = DraftMgr.getDraft(It.first())) {
if (Params.capabilities.offsetEncoding && !Opts.Encoding) {
     Opts.Encoding = OffsetEncoding::UTF16; // fallback
     for (OffsetEncoding Supported : *Params.capabilities.offsetEncoding)
       if (Supported != OffsetEncoding::UnsupportedEncoding) {
         Opts.Encoding = Supported;
         break;
       }
   }
 
-  Opts.TheiaSemanticHighlighting =
-      Params.capabilities.TheiaSemanticHighlighting;
   if (Params.capabilities.TheiaSemanticHighlighting &&
-      Params.capabilities.SemanticTokens) {
-    log("Client supports legacy semanticHighlights notification and standard "
-        "semanticTokens request, choosing the latter (no notifications).");
-    Opts.TheiaSemanticHighlighting = false;
-  }
-  if (Opts.TheiaSemanticHighlighting) {
-    log("Using legacy semanticHighlights notification, which will be removed "
-        "in clangd 13. Clients should use the standard semanticTokens "
-        "request instead.");
+      !Params.capabilities.SemanticTokens) {
+    elog("Client requested legacy semanticHighlights notification, which is "
+         "no longer supported. Migrate to standard semanticTokens request");
   }
 
   if (Params.rootUri && *Params.rootUri)
     Opts.WorkspaceRoot = std::string(Params.rootUri->file());
   else if (Params.rootPath && !Params.rootPath->empty())
     Opts.WorkspaceRoot = *Params.rootPath;
   if (Server)
     return Reply(llvm::make_error<LSPError>("server already initialized",
llvm::json::Object Result{
             {"typeHierarchyProvider", true},
             {"memoryUsageProvider", true}, // clangd extension
             {"compilationDatabase",        // clangd extension
              llvm::json::Object{{"automaticReload", true}}},
             {"callHierarchyProvider", true},
         }}}};
   if (Opts.Encoding)
     Result["offsetEncoding"] = *Opts.Encoding;
-  if (Opts.TheiaSemanticHighlighting)
-    Result.getObject("capabilities")
-        ->insert(
-            {"semanticHighlighting",
-             llvm::json::Object{{"scopes", buildHighlightScopeLookupTable()}}});
   if (Opts.FoldingRanges)
     Result.getObject("capabilities")->insert({"foldingRangeProvider", true});
   Reply(std::move(Result));
 }
 
 void ClangdLSPServer::onInitialized(const InitializedParams &Params) {}
 
 void ClangdLSPServer::onShutdown(Callback<std::nullptr_t> Reply) {
void ClangdLSPServer::onDocumentDidClose(
   DraftMgr.removeDraft(File);
   Server->removeDocument(File);
 
   {
     std::lock_guard<std::mutex> Lock(FixItsMutex);
     FixItsMap.erase(File);
   }
   {
-    std::lock_guard<std::mutex> HLock(HighlightingsMutex);
-    FileToHighlightings.erase(File);
-  }
-  {
     std::lock_guard<std::mutex> HLock(SemanticTokensMutex);
     LastSemanticTokens.erase(File);
   }
   // clangd will not send updates for this file anymore, so we empty out the
   // list of diagnostics shown on the client (e.g. in the "Problems" pane of
   // VSCode). Note that this cannot race with actual diagnostics responses
   // because removeDocument() guarantees no diagnostic callbacks will be
   // executed after it returns.
if (Old != New) {
       ModifiedFiles.insert(File);
     }
   }
 
   reparseOpenFilesIfNeeded(
       [&](llvm::StringRef File) { return ModifiedFiles.count(File) != 0; });
 }
 
-void ClangdLSPServer::publishTheiaSemanticHighlighting(
-    const TheiaSemanticHighlightingParams &Params) {
-  notify("textDocument/semanticHighlighting", Params);
-}
-
 void ClangdLSPServer::publishDiagnostics(
     const PublishDiagnosticsParams &Params) {
   notify("textDocument/publishDiagnostics", Params);
 }
 
 void ClangdLSPServer::maybeExportMemoryProfile() {
   if (!trace::enabled() || !ShouldProfile())
     return;
bool ClangdLSPServer::shouldRunCompletion(
   if (!Offset) {
     vlog("could not convert position '{0}' to offset for file '{1}'",
          Params.position, Params.textDocument.uri.file());
     return true;
   }
   return allowImplicitCompletion(Code->Contents, *Offset);
 }
 
-void ClangdLSPServer::onHighlightingsReady(
-    PathRef File, llvm::StringRef Version,
-    std::vector<HighlightingToken> Highlightings) {
-  std::vector<HighlightingToken> Old;
-  std::vector<HighlightingToken> HighlightingsCopy = Highlightings;
-  {
-    std::lock_guard<std::mutex> Lock(HighlightingsMutex);
-    Old = std::move(FileToHighlightings[File]);
-    FileToHighlightings[File] = std::move(HighlightingsCopy);
-  }
-  // LSP allows us to send incremental edits of highlightings. Also need to diff
-  // to remove highlightings from tokens that should no longer have them.
-  std::vector<LineHighlightings> Diffed = diffHighlightings(Highlightings, Old);
-  TheiaSemanticHighlightingParams Notification;
-  Notification.TextDocument.uri =
-      URIForFile::canonicalize(File, /*TUPath=*/File);
-  Notification.TextDocument.version = decodeVersion(Version);
-  Notification.Lines = toTheiaSemanticHighlightingInformation(Diffed);
-  publishTheiaSemanticHighlighting(Notification);
-}
-
 void ClangdLSPServer::onDiagnosticsReady(PathRef File, llvm::StringRef Version,
                                          std::vector<Diag> Diagnostics) {
   PublishDiagnosticsParams Notification;
   Notification.version = decodeVersion(Version);
   Notification.uri = URIForFile::canonicalize(File, /*TUPath=*/File);
   DiagnosticToReplacementMap LocalFixIts; // Temporary storage
   for (auto &Diag : Diagnostics) {
     toLSPDiags(Diag, Notification.uri, DiagOpts,

clang-tools-extra/clangd/ClangdServer.h

public:
     /// Called by ClangdServer when \p Diagnostics for \p File are ready.
     /// May be called concurrently for separate files, not for a single file.
     virtual void onDiagnosticsReady(PathRef File, llvm::StringRef Version,
                                     std::vector<Diag> Diagnostics) {}
     /// Called whenever the file status is updated.
     /// May be called concurrently for separate files, not for a single file.
     virtual void onFileUpdated(PathRef File, const TUStatus &Status) {}
 
-    /// Called by ClangdServer when some \p Highlightings for \p File are ready.
-    /// May be called concurrently for separate files, not for a single file.
-    virtual void
-    onHighlightingsReady(PathRef File, llvm::StringRef Version,
-                         std::vector<HighlightingToken> Highlightings) {}
-
     /// Called when background indexing tasks are enqueued/started/completed.
     /// Not called concurrently.
     virtual void
     onBackgroundIndexProgress(const BackgroundQueue::Stats &Stats) {}
   };
   /// Creates a context provider that loads and installs config.
   /// Errors in loading config are reported as diagnostics via Callbacks.
   /// (This is typically used as ClangdServer::Options::ContextProvider).
DebouncePolicy UpdateDebounce = DebouncePolicy{
         /*Max=*/std::chrono::milliseconds(500),
         /*RebuildRatio=*/1,
     };
 
     /// Clangd will execute compiler drivers matching one of these globs to
     /// fetch system include path.
     std::vector<std::string> QueryDriverGlobs;
 
-    /// Enable notification-based semantic highlighting.
-    bool TheiaSemanticHighlighting = false;
-
     /// Enable preview of FoldingRanges feature.
     bool FoldingRanges = false;
 
     explicit operator TUScheduler::Options() const;
   };
   // Sensible default options for use in tests.
   // Features like indexing must be enabled if desired.
   static Options optsForTest();

clang-tools-extra/clangd/ClangdServer.cpp

 
 namespace clang {
 namespace clangd {
 namespace {
 
 // Update the FileIndex with new ASTs and plumb the diagnostics responses.
 struct UpdateIndexCallbacks : public ParsingCallbacks {
   UpdateIndexCallbacks(FileIndex *FIndex,
-                       ClangdServer::Callbacks *ServerCallbacks,
-                       bool TheiaSemanticHighlighting)
-      : FIndex(FIndex), ServerCallbacks(ServerCallbacks),
-        TheiaSemanticHighlighting(TheiaSemanticHighlighting) {}
+                       ClangdServer::Callbacks *ServerCallbacks)
+      : FIndex(FIndex), ServerCallbacks(ServerCallbacks) {}
 
   void onPreambleAST(PathRef Path, llvm::StringRef Version, ASTContext &Ctx,
                      std::shared_ptr<clang::Preprocessor> PP,
                      const CanonicalIncludes &CanonIncludes) override {
     if (FIndex)
       FIndex->updatePreamble(Path, Version, Ctx, std::move(PP), CanonIncludes);
   }
 
   void onMainAST(PathRef Path, ParsedAST &AST, PublishFn Publish) override {
     if (FIndex)
       FIndex->updateMain(Path, AST);
 
     std::vector<Diag> Diagnostics = AST.getDiagnostics();
-    std::vector<HighlightingToken> Highlightings;
-    if (TheiaSemanticHighlighting)
-      Highlightings = getSemanticHighlightings(AST);
-
     if (ServerCallbacks)
       Publish([&]() {
         ServerCallbacks->onDiagnosticsReady(Path, AST.version(),
                                             std::move(Diagnostics));
-        if (TheiaSemanticHighlighting)
-          ServerCallbacks->onHighlightingsReady(Path, AST.version(),
-                                                std::move(Highlightings));
       });
   }
 
   void onFailedAST(PathRef Path, llvm::StringRef Version,
                    std::vector<Diag> Diags, PublishFn Publish) override {
     if (ServerCallbacks)
       Publish(
           [&]() { ServerCallbacks->onDiagnosticsReady(Path, Version, Diags); });
   }
 
   void onFileUpdated(PathRef File, const TUStatus &Status) override {
     if (ServerCallbacks)
       ServerCallbacks->onFileUpdated(File, Status);
   }
 
 private:
   FileIndex *FIndex;
   ClangdServer::Callbacks *ServerCallbacks;
-  bool TheiaSemanticHighlighting;
 };
 
 } // namespace
 
 ClangdServer::Options ClangdServer::optsForTest() {
   ClangdServer::Options Opts;
   Opts.UpdateDebounce = DebouncePolicy::fixed(/*zero*/ {});
   Opts.StorePreamblesInMemory = true;
   Opts.AsyncThreadsCount = 4; // Consistent!
-  Opts.TheiaSemanticHighlighting = true;
   return Opts;
 }
 
 ClangdServer::Options::operator TUScheduler::Options() const {
   TUScheduler::Options Opts;
   Opts.AsyncThreadsCount = AsyncThreadsCount;
   Opts.RetentionPolicy = RetentionPolicy;
   Opts.StorePreamblesInMemory = StorePreamblesInMemory;
: CDB(CDB), TFS(TFS),
       WorkspaceRoot(Opts.WorkspaceRoot),
       // Pass a callback into `WorkScheduler` to extract symbols from a newly
       // parsed file and rebuild the file index synchronously each time an AST
       // is parsed.
       // FIXME(ioeric): this can be slow and we may be able to index on less
       // critical paths.
       WorkScheduler(
           CDB, TUScheduler::Options(Opts),
-          std::make_unique<UpdateIndexCallbacks>(
-              DynamicIdx.get(), Callbacks, Opts.TheiaSemanticHighlighting)) {
+          std::make_unique<UpdateIndexCallbacks>(DynamicIdx.get(), Callbacks)) {
   // Adds an index to the stack, at higher priority than existing indexes.
   auto AddIndex = [&](SymbolIndex *Idx) {
     if (this->Index != nullptr) {
       MergedIdx.push_back(std::make_unique<MergedIndex>(Idx, this->Index));
       this->Index = MergedIdx.back().get();
     } else {
       this->Index = Idx;
     }

clang-tools-extra/clangd/Protocol.h

struct ClientCapabilities {
   bool CodeActionStructure = false;
 
   /// Client advertises support for the semanticTokens feature.
   /// We support the textDocument/semanticTokens request in any case.
   /// textDocument.semanticTokens
   bool SemanticTokens = false;
   /// Client supports Theia semantic highlighting extension.
   /// https://github.com/microsoft/vscode-languageserver-node/pull/367
-  /// This will be ignored if the client also supports semanticTokens.
+  /// clangd no longer supports this, we detect it just to log a warning.
   /// textDocument.semanticHighlightingCapabilities.semanticHighlighting
-  /// FIXME: drop this support once clients support LSP 3.16 Semantic Tokens.
   bool TheiaSemanticHighlighting = false;
 
   /// Supported encodings for LSP character offsets. (clangd extension).
   llvm::Optional<std::vector<OffsetEncoding>> offsetEncoding;
 
   /// The content format that should be used for Hover requests.
   /// textDocument.hover.contentEncoding
   MarkupKind HoverContentFormat = MarkupKind::PlainText;
struct SemanticTokensOrDelta {
   std::string resultId;
   /// Set if we computed edits relative to a previous set of tokens.
   llvm::Optional<std::vector<SemanticTokensEdit>> edits;
   /// Set if we computed a fresh set of tokens.
   llvm::Optional<std::vector<SemanticToken>> tokens; // encoded as integer array
 };
 llvm::json::Value toJSON(const SemanticTokensOrDelta &);
 
-/// Represents a semantic highlighting information that has to be applied on a
-/// specific line of the text document.
-struct TheiaSemanticHighlightingInformation {
-  /// The line these highlightings belong to.
-  int Line = 0;
-  /// The base64 encoded string of highlighting tokens.
-  std::string Tokens;
-  /// Is the line in an inactive preprocessor branch?
-  /// This is a clangd extension.
-  /// An inactive line can still contain highlighting tokens as well;
-  /// clients should combine line style and token style if possible.
-  bool IsInactive = false;
-};
-bool operator==(const TheiaSemanticHighlightingInformation &Lhs,
-                const TheiaSemanticHighlightingInformation &Rhs);
-llvm::json::Value
-toJSON(const TheiaSemanticHighlightingInformation &Highlighting);
-
-/// Parameters for the semantic highlighting (server-side) push notification.
-struct TheiaSemanticHighlightingParams {
-  /// The textdocument these highlightings belong to.
-  VersionedTextDocumentIdentifier TextDocument;
-  /// The lines of highlightings that should be sent.
-  std::vector<TheiaSemanticHighlightingInformation> Lines;
-};
-llvm::json::Value toJSON(const TheiaSemanticHighlightingParams &Highlighting);
-
 struct SelectionRangeParams {
   /// The text document.
   TextDocumentIdentifier textDocument;
 
   /// The positions inside the text document.
   std::vector<Position> positions;
 };
 bool fromJSON(const llvm::json::Value &, SelectionRangeParams &,

clang-tools-extra/clangd/Protocol.cpp

OE = llvm::StringSwitch<OffsetEncoding>(*Str)
            .Case("utf-32", OffsetEncoding::UTF32)
            .Default(OffsetEncoding::UnsupportedEncoding);
   return true;
 }
 llvm::raw_ostream &operator<<(llvm::raw_ostream &OS, OffsetEncoding Enc) {
   return OS << toString(Enc);
 }
 
-bool operator==(const TheiaSemanticHighlightingInformation &Lhs,
-                const TheiaSemanticHighlightingInformation &Rhs) {
-  return Lhs.Line == Rhs.Line && Lhs.Tokens == Rhs.Tokens;
-}
-
-llvm::json::Value
-toJSON(const TheiaSemanticHighlightingInformation &Highlighting) {
-  return llvm::json::Object{{"line", Highlighting.Line},
-                            {"tokens", Highlighting.Tokens},
-                            {"isInactive", Highlighting.IsInactive}};
-}
-
-llvm::json::Value toJSON(const TheiaSemanticHighlightingParams &Highlighting) {
-  return llvm::json::Object{
-      {"textDocument", Highlighting.TextDocument},
-      {"lines", std::move(Highlighting.Lines)},
-  };
-}
-
 bool fromJSON(const llvm::json::Value &Params, SelectionRangeParams &S,
               llvm::json::Path P) {
   llvm::json::ObjectMapper O(Params, P);
   return O && O.map("textDocument", S.textDocument) &&
          O.map("positions", S.positions);
 }
 
 llvm::json::Value toJSON(const SelectionRange &Out) {

clang-tools-extra/clangd/SemanticHighlighting.h

 //==-- SemanticHighlighting.h - Generating highlights from the AST-- C++ -*-==//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file supports semantic highlighting: categorizing tokens in the file so
 // that the editor can color/style them differently.
-//
 // This is particularly valuable for C++: its complex and context-dependent
 // grammar is a challenge for simple syntax-highlighting techniques.
 //
-// We support two protocols for providing highlights to the client:
-// - the `textDocument/semanticTokens` request from LSP 3.16
-//   https://github.com/microsoft/vscode-languageserver-node/blob/release/protocol/3.16.0-next.1/protocol/src/protocol.semanticTokens.proposed.ts
-// - the earlier proposed `textDocument/semanticHighlighting` notification
-//   https://github.com/microsoft/vscode-languageserver-node/pull/367
-//   This is referred to as "Theia" semantic highlighting in the code.
-//   It was supported from clangd 9 but should be considered deprecated as of
-//   clangd 11 and eventually removed.
-//
 // Semantic highlightings are calculated for an AST by visiting every AST node
 // and classifying nodes that are interesting to highlight (variables/function
 // calls etc.).
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_SEMANTICHIGHLIGHTING_H
 #define LLVM_CLANG_TOOLS_EXTRA_CLANGD_SEMANTICHIGHLIGHTING_H
 
 #include "Protocol.h"
HighlightingToken &addModifier(HighlightingModifier M) {
     Modifiers |= 1 << static_cast<unsigned>(M);
     return *this;
   }
 };
 
 bool operator==(const HighlightingToken &L, const HighlightingToken &R);
 bool operator<(const HighlightingToken &L, const HighlightingToken &R);
 
-/// Contains all information about highlightings on a single line.
-struct LineHighlightings {
-  int Line;
-  std::vector<HighlightingToken> Tokens;
-  bool IsInactive;
-};
-
-bool operator==(const LineHighlightings &L, const LineHighlightings &R);
-
 // Returns all HighlightingTokens from an AST. Only generates highlights for the
 // main AST.
 std::vector<HighlightingToken> getSemanticHighlightings(ParsedAST &AST);
 
 std::vector<SemanticToken> toSemanticTokens(llvm::ArrayRef<HighlightingToken>);
 llvm::StringRef toSemanticTokenType(HighlightingKind Kind);
 llvm::StringRef toSemanticTokenModifier(HighlightingModifier Modifier);
 std::vector<SemanticTokensEdit> diffTokens(llvm::ArrayRef<SemanticToken> Before,
                                            llvm::ArrayRef<SemanticToken> After);
 
-/// Converts a HighlightingKind to a corresponding TextMate scope
-/// (https://manual.macromates.com/en/language_grammars).
-llvm::StringRef toTextMateScope(HighlightingKind Kind);
-
-/// Convert to LSP's semantic highlighting information.
-std::vector<TheiaSemanticHighlightingInformation>
-toTheiaSemanticHighlightingInformation(
-    llvm::ArrayRef<LineHighlightings> Tokens);
-
-/// Return a line-by-line diff between two highlightings.
-///  - if the tokens on a line are the same in both highlightings, this line is
-///  omitted.
-///  - if a line exists in New but not in Old, the tokens on this line are
-///  emitted.
-///  - if a line does not exist in New but exists in Old, an empty line is
-///  emitted (to tell client to clear the previous highlightings on this line).
-///
-/// REQUIRED: Old and New are sorted.
-std::vector<LineHighlightings>
-diffHighlightings(ArrayRef<HighlightingToken> New,
-                  ArrayRef<HighlightingToken> Old);
-
 } // namespace clangd
 } // namespace clang
 
 #endif

clang-tools-extra/clangd/SemanticHighlighting.cpp

if (NestedNameSpecifier *NNS = Q.getNestedNameSpecifier()) {
             .addModifier(HighlightingModifier::ClassScope);
     }
     return RecursiveASTVisitor::TraverseNestedNameSpecifierLoc(Q);
   }
 
 private:
   HighlightingsBuilder &H;
 };
-
-void write32be(uint32_t I, llvm::raw_ostream &OS) {
-  std::array<char, 4> Buf;
-  llvm::support::endian::write32be(Buf.data(), I);
-  OS.write(Buf.data(), Buf.size());
-}
-
-void write16be(uint16_t I, llvm::raw_ostream &OS) {
-  std::array<char, 2> Buf;
-  llvm::support::endian::write16be(Buf.data(), I);
-  OS.write(Buf.data(), Buf.size());
-}
-
-// Get the highlightings on \c Line where the first entry of line is at \c
-// StartLineIt. If it is not at \c StartLineIt an empty vector is returned.
-ArrayRef<HighlightingToken>
-takeLine(ArrayRef<HighlightingToken> AllTokens,
-         ArrayRef<HighlightingToken>::iterator StartLineIt, int Line) {
-  return ArrayRef<HighlightingToken>(StartLineIt, AllTokens.end())
-      .take_while([Line](const HighlightingToken &Token) {
-        return Token.R.start.line == Line;
-      });
-}
 } // namespace
 
 std::vector<HighlightingToken> getSemanticHighlightings(ParsedAST &AST) {
   auto &C = AST.getASTContext();
   // Add highlightings for AST nodes.
   HighlightingsBuilder Builder(AST);
   // Highlight 'decltype' and 'auto' as their underlying types.
   CollectExtraHighlightings(Builder).TraverseAST(C);
llvm::raw_ostream &operator<<(llvm::raw_ostream &OS, HighlightingModifier K) {
   switch (K) {
   case HighlightingModifier::Declaration:
     return OS << "decl"; // abbrevation for common case
   default:
     return OS << toSemanticTokenModifier(K);
   }
 }
 
-std::vector<LineHighlightings>
-diffHighlightings(ArrayRef<HighlightingToken> New,
-                  ArrayRef<HighlightingToken> Old) {
-  assert(std::is_sorted(New.begin(), New.end()) &&
-         "New must be a sorted vector");
-  assert(std::is_sorted(Old.begin(), Old.end()) &&
-         "Old must be a sorted vector");
-
-  // FIXME: There's an edge case when tokens span multiple lines. If the first
-  // token on the line started on a line above the current one and the rest of
-  // the line is the equal to the previous one than we will remove all
-  // highlights but the ones for the token spanning multiple lines. This means
-  // that when we get into the LSP layer the only highlights that will be
-  // visible are the ones for the token spanning multiple lines.
-  // Example:
-  // EndOfMultilineToken  Token Token Token
-  // If "Token Token Token" don't differ from previously the line is
-  // incorrectly removed. Suggestion to fix is to separate any multiline tokens
-  // into one token for every line it covers. This requires reading from the
-  // file buffer to figure out the length of each line though.
-  std::vector<LineHighlightings> DiffedLines;
-  // ArrayRefs to the current line in the highlightings.
-  ArrayRef<HighlightingToken> NewLine(New.begin(),
-                                      /*length*/ static_cast<size_t>(0));
-  ArrayRef<HighlightingToken> OldLine(Old.begin(),
-                                      /*length*/ static_cast<size_t>(0));
-  auto NewEnd = New.end();
-  auto OldEnd = Old.end();
-  auto NextLineNumber = [&]() {
-    int NextNew = NewLine.end() != NewEnd ? NewLine.end()->R.start.line
-                                          : std::numeric_limits<int>::max();
-    int NextOld = OldLine.end() != OldEnd ? OldLine.end()->R.start.line
-                                          : std::numeric_limits<int>::max();
-    return std::min(NextNew, NextOld);
-  };
-
-  for (int LineNumber = 0; NewLine.end() < NewEnd || OldLine.end() < OldEnd;
-       LineNumber = NextLineNumber()) {
-    NewLine = takeLine(New, NewLine.end(), LineNumber);
-    OldLine = takeLine(Old, OldLine.end(), LineNumber);
-    if (NewLine != OldLine) {
-      DiffedLines.push_back({LineNumber, NewLine, /*IsInactive=*/false});
-
-      // Turn a HighlightingKind::InactiveCode token into the IsInactive flag.
-      auto &AddedLine = DiffedLines.back();
-      llvm::erase_if(AddedLine.Tokens, [&](const HighlightingToken &T) {
-        if (T.Kind == HighlightingKind::InactiveCode) {
-          AddedLine.IsInactive = true;
-          return true;
-        }
-        return false;
-      });
-    }
-  }
-
-  return DiffedLines;
-}
-
 bool operator==(const HighlightingToken &L, const HighlightingToken &R) {
   return std::tie(L.R, L.Kind, L.Modifiers) ==
          std::tie(R.R, R.Kind, R.Modifiers);
 }
 bool operator<(const HighlightingToken &L, const HighlightingToken &R) {
   return std::tie(L.R, L.Kind, R.Modifiers) <
          std::tie(R.R, R.Kind, R.Modifiers);
 }
-bool operator==(const LineHighlightings &L, const LineHighlightings &R) {
-  return std::tie(L.Line, L.Tokens) == std::tie(R.Line, R.Tokens);
-}
 
 std::vector<SemanticToken>
 toSemanticTokens(llvm::ArrayRef<HighlightingToken> Tokens) {
   assert(std::is_sorted(Tokens.begin(), Tokens.end()));
   std::vector<SemanticToken> Result;
   const HighlightingToken *Last = nullptr;
   for (const HighlightingToken &Tok : Tokens) {
     Result.emplace_back();
llvm::StringRef toSemanticTokenModifier(HighlightingModifier Modifier) {
   case HighlightingModifier::FileScope:
     return "fileScope"; // nonstandard
   case HighlightingModifier::GlobalScope:
     return "globalScope"; // nonstandard
   }
   llvm_unreachable("unhandled HighlightingModifier");
 }
 
-std::vector<TheiaSemanticHighlightingInformation>
-toTheiaSemanticHighlightingInformation(
-    llvm::ArrayRef<LineHighlightings> Tokens) {
-  if (Tokens.size() == 0)
-    return {};
-
-  // FIXME: Tokens might be multiple lines long (block comments) in this case
-  // this needs to add multiple lines for those tokens.
-  std::vector<TheiaSemanticHighlightingInformation> Lines;
-  Lines.reserve(Tokens.size());
-  for (const auto &Line : Tokens) {
-    llvm::SmallVector<char> LineByteTokens;
-    llvm::raw_svector_ostream OS(LineByteTokens);
-    for (const auto &Token : Line.Tokens) {
-      // Writes the token to LineByteTokens in the byte format specified by the
-      // LSP proposal. Described below.
-      // |<---- 4 bytes ---->|<-- 2 bytes -->|<--- 2 bytes -->|
-      // |    character      |  length       |    index       |
-
-      write32be(Token.R.start.character, OS);
-      write16be(Token.R.end.character - Token.R.start.character, OS);
-      write16be(static_cast<int>(Token.Kind), OS);
-    }
-
-    Lines.push_back({Line.Line, encodeBase64(LineByteTokens), Line.IsInactive});
-  }
-
-  return Lines;
-}
-
-llvm::StringRef toTextMateScope(HighlightingKind Kind) {
-  // FIXME: Add scopes for C and Objective C.
-  switch (Kind) {
-  case HighlightingKind::Function:
-    return "entity.name.function.cpp";
-  case HighlightingKind::Method:
-    return "entity.name.function.method.cpp";
-  case HighlightingKind::StaticMethod:
-    return "entity.name.function.method.static.cpp";
-  case HighlightingKind::Variable:
-    return "variable.other.cpp";
-  case HighlightingKind::LocalVariable:
-    return "variable.other.local.cpp";
-  case HighlightingKind::Parameter:
-    return "variable.parameter.cpp";
-  case HighlightingKind::Field:
-    return "variable.other.field.cpp";
-  case HighlightingKind::StaticField:
-    return "variable.other.field.static.cpp";
-  case HighlightingKind::Class:
-    return "entity.name.type.class.cpp";
-  case HighlightingKind::Enum:
-    return "entity.name.type.enum.cpp";
-  case HighlightingKind::EnumConstant:
-    return "variable.other.enummember.cpp";
-  case HighlightingKind::Typedef:
-    return "entity.name.type.typedef.cpp";
-  case HighlightingKind::Type:
-    // Fragile: all paths emitting `Type` are dependent names for now.
-    // But toTextMateScope is going away soon.
-    return "entity.name.type.dependent.cpp";
-  case HighlightingKind::Unknown:
-    // Fragile: all paths emitting `Unknown` are dependent names for now.
-    // But toTextMateScope is going away soon.
-    return "entity.name.other.dependent.cpp";
-  case HighlightingKind::Namespace:
-    return "entity.name.namespace.cpp";
-  case HighlightingKind::TemplateParameter:
-    return "entity.name.type.template.cpp";
-  case HighlightingKind::Concept:
-    return "entity.name.type.concept.cpp";
-  case HighlightingKind::Primitive:
-    return "storage.type.primitive.cpp";
-  case HighlightingKind::Macro:
-    return "entity.name.function.preprocessor.cpp";
-  case HighlightingKind::InactiveCode:
-    return "meta.disabled";
-  }
-  llvm_unreachable("unhandled HighlightingKind");
-}
-
 std::vector<SemanticTokensEdit>
 diffTokens(llvm::ArrayRef<SemanticToken> Old,
            llvm::ArrayRef<SemanticToken> New) {
   // For now, just replace everything from the first-last modification.
   // FIXME: use a real diff instead, this is bad with include-insertion.
 
   unsigned Offset = 0;
   while (!Old.empty() && !New.empty() && Old.front() == New.front()) {

clang-tools-extra/clangd/TUScheduler.h

public:
   /// The \p AST always contains all AST nodes for the main file itself, and
   /// only a portion of the AST nodes deserialized from the preamble. Note that
   /// some nodes from the preamble may have been deserialized and may also be
   /// accessed from the main file AST, e.g. redecls of functions from preamble,
   /// etc. Clients are expected to process only the AST nodes from the main file
   /// in this callback (obtained via ParsedAST::getLocalTopLevelDecls) to obtain
   /// optimal performance.
   ///
-  /// When information about the file (diagnostics, syntax highlighting) is
+  /// When information about the file (e.g. diagnostics) is
   /// published to clients, this should be wrapped in Publish, e.g.
   ///   void onMainAST(...) {
-  ///     Highlights = computeHighlights();
-  ///     Publish([&] { notifyHighlights(Path, Highlights); });
+  ///     Diags = renderDiagnostics();
+  ///     Publish([&] { notifyDiagnostics(Path, Diags); });
   ///   }
   /// This guarantees that clients will see results in the correct sequence if
   /// the file is concurrently closed and/or reopened. (The lambda passed to
   /// Publish() may never run in this case).
   virtual void onMainAST(PathRef Path, ParsedAST &AST, PublishFn Publish) {}
 
   /// Called whenever the AST fails to build. \p Diags will have the diagnostics
   /// that led to failure.

clang-tools-extra/clangd/TUScheduler.cpp

private:
   /// None means no builds yet, null means there was an error while building.
   /// Only written by ASTWorker's thread.
   llvm::Optional<std::shared_ptr<const PreambleData>> LatestPreamble;
   std::deque<Request> PreambleRequests; /* GUARDED_BY(Mutex) */
   /// Signaled whenever LatestPreamble changes state or there's a new
   /// PreambleRequest.
   mutable std::condition_variable PreambleCV;
   /// Guards the callback that publishes results of AST-related computations
-  /// (diagnostics, highlightings) and file statuses.
+  /// (diagnostics) and file statuses.
   std::mutex PublishMu;
   // Used to prevent remove document + add document races that lead to
   // out-of-order callbacks for publishing results of onMainAST callback.
   //
   // The lifetime of the old/new ASTWorkers will overlap, but their handles
   // don't. When the old handle is destroyed, the old worker will stop reporting
   // any results to the user.
   bool CanPublishResults = true; /* GUARDED_BY(PublishMu) */

clang-tools-extra/clangd/test/semantic-highlighting.test

-# RUN: clangd -lit-test < %s | FileCheck -strict-whitespace %s
-{"jsonrpc":"2.0","id":0,"method":"initialize","params":{"processId":123,"rootPath":"clangd","capabilities":{"textDocument":{"semanticHighlightingCapabilities":{"semanticHighlighting":true}}},"trace":"off"}}
----
-#      CHECK:  "id": 0,
-# CHECK:      "semanticHighlighting": {
-# CHECK-NEXT:        "scopes": [
-# CHECK-NEXT:          [
-# CHECK-NEXT:            "variable.other.cpp"
-# CHECK-NEXT:          ],
-# CHECK-NEXT:          [
-# CHECK-NEXT:            "variable.other.local.cpp"
-# CHECK-NEXT:          ],
-# CHECK-NEXT:          [
-# CHECK-NEXT:            "variable.parameter.cpp"
-# CHECK-NEXT:          ],
-# CHECK-NEXT:          [
-# CHECK-NEXT:            "entity.name.function.cpp"
-# CHECK-NEXT:          ],
-# CHECK-NEXT:          [
-# CHECK-NEXT:            "entity.name.function.method.cpp"
-# CHECK-NEXT:          ],
-# CHECK-NEXT:          [
-# CHECK-NEXT:            "entity.name.function.method.static.cpp"
-# CHECK-NEXT:          ],
-# CHECK-NEXT:          [
-# CHECK-NEXT:            "variable.other.field.cpp"
-# CHECK-NEXT:          ],
-# CHECK-NEXT:          [
-# CHECK-NEXT:            "variable.other.field.static.cpp"
-# CHECK-NEXT:          ],
-# CHECK-NEXT:          [
-# CHECK-NEXT:            "entity.name.type.class.cpp"
-# CHECK-NEXT:          ],
-# CHECK-NEXT:          [
-# CHECK-NEXT:            "entity.name.type.enum.cpp"
-# CHECK-NEXT:          ],
-# CHECK-NEXT:          [
-# CHECK-NEXT:            "variable.other.enummember.cpp"
-# CHECK-NEXT:          ],
-# CHECK-NEXT:          [
-# CHECK-NEXT:            "entity.name.type.typedef.cpp"
-# CHECK-NEXT:          ],
-# CHECK-NEXT:          [
-# CHECK-NEXT:            "entity.name.type.dependent.cpp"
-# CHECK-NEXT:          ],
-# CHECK-NEXT:          [
-# CHECK-NEXT:            "entity.name.other.dependent.cpp"
-# CHECK-NEXT:          ],
-# CHECK-NEXT:          [
-# CHECK-NEXT:            "entity.name.namespace.cpp"
-# CHECK-NEXT:          ],
-# CHECK-NEXT:          [
-# CHECK-NEXT:            "entity.name.type.template.cpp"
-# CHECK-NEXT:          ],
-# CHECK-NEXT:          [
-# CHECK-NEXT:            "entity.name.type.concept.cpp"
-# CHECK-NEXT:          ],
-# CHECK-NEXT:          [
-# CHECK-NEXT:            "storage.type.primitive.cpp"
-# CHECK-NEXT:          ],
-# CHECK-NEXT:          [
-# CHECK-NEXT:            "entity.name.function.preprocessor.cpp"
-# CHECK-NEXT:          ],
-# CHECK-NEXT:          [
-# CHECK-NEXT:            "meta.disabled"
-# CHECK-NEXT:          ]
-# CHECK-NEXT:        ]
-# CHECK-NEXT:      },
----
-{"jsonrpc":"2.0","method":"textDocument/didOpen","params":{"textDocument":{"uri":"test:///foo.cpp","languageId":"cpp","text":"int x = 2;"}}}
-#      CHECK:  "method": "textDocument/semanticHighlighting",
-# CHECK-NEXT:  "params": {
-# CHECK-NEXT:    "lines": [
-# CHECK-NEXT:      {
-# CHECK-NEXT:        "isInactive": false,
-# CHECK-NEXT:        "line": 0,
-# CHECK-NEXT:        "tokens": "AAAABAABAAA="
-# CHECK-NEXT:      }
-# CHECK-NEXT:    ],
-# CHECK-NEXT:    "textDocument": {
-# CHECK-NEXT:      "uri": "file://{{.*}}/clangd-test/foo.cpp",
-# CHECK-NEXT:      "version": 0
-# CHECK-NEXT:    }
-# CHECK-NEXT:  }
-# CHECK-NEXT:}
----
-{"jsonrpc":"2.0","method":"textDocument/didOpen","params":{"textDocument":{"uri":"test:///foo2.cpp","languageId":"cpp","text":"int x = 2;\nint y = 2;"}}}
-#      CHECK:  "method": "textDocument/semanticHighlighting",
-# CHECK-NEXT:  "params": {
-# CHECK-NEXT:    "lines": [
-# CHECK-NEXT:      {
-# CHECK-NEXT:        "isInactive": false,
-# CHECK-NEXT:        "line": 0,
-# CHECK-NEXT:        "tokens": "AAAABAABAAA="
-# CHECK-NEXT:      }
-# CHECK-NEXT:      {
-# CHECK-NEXT:        "isInactive": false,
-# CHECK-NEXT:        "line": 1,
-# CHECK-NEXT:        "tokens": "AAAABAABAAA="
-# CHECK-NEXT:      }
-# CHECK-NEXT:    ],
-# CHECK-NEXT:    "textDocument": {
-# CHECK-NEXT:      "uri": "file://{{.*}}/clangd-test/foo2.cpp",
-# CHECK-NEXT:      "version": 0
-# CHECK-NEXT:    }
-# CHECK-NEXT:  }
-# CHECK-NEXT:}
----
-{"jsonrpc":"2.0","method":"textDocument/didChange","params":{"textDocument":{"uri":"test:///foo.cpp"},"contentChanges": [{"range":{"start": {"line": 0,"character": 10},"end": {"line": 0,"character": 10}},"rangeLength": 0,"text": "\nint y = 2;"}]}}
-#      CHECK:  "method": "textDocument/semanticHighlighting",
-# CHECK-NEXT:  "params": {
-# CHECK-NEXT:    "lines": [
-# CHECK-NEXT:      {
-# CHECK-NEXT:        "isInactive": false,
-# CHECK-NEXT:        "line": 1,
-# CHECK-NEXT:        "tokens": "AAAABAABAAA="
-# CHECK-NEXT:      }
-# CHECK-NEXT:   ],
-# CHECK-NEXT:    "textDocument": {
-# CHECK-NEXT:      "uri": "file://{{.*}}/clangd-test/foo.cpp",
-# CHECK-NEXT:      "version": 1
-# CHECK-NEXT:    }
-# CHECK-NEXT:  }
-# CHECK-NEXT:}
----
-{"jsonrpc":"2.0","method":"textDocument/didChange","params":{"textDocument":{"uri":"test:///foo.cpp"},"contentChanges": [{"range":{"start": {"line": 0,"character": 10},"end": {"line": 1,"character": 10}},"rangeLength": 11,"text": ""}]}}
-#      CHECK:  "method": "textDocument/semanticHighlighting",
-# CHECK-NEXT:  "params": {
-# CHECK-NEXT:    "lines": [
-# CHECK-NEXT:      {
-# CHECK-NEXT:        "isInactive": false,
-# CHECK-NEXT:        "line": 1,
-# CHECK-NEXT:        "tokens": ""
-# CHECK-NEXT:      }
-# CHECK-NEXT:   ],
-# CHECK-NEXT:    "textDocument": {
-# CHECK-NEXT:      "uri": "file://{{.*}}/clangd-test/foo.cpp",
-# CHECK-NEXT:      "version": 2
-# CHECK-NEXT:    }
-# CHECK-NEXT:  }
-# CHECK-NEXT:}
----
-{"jsonrpc":"2.0","id":3,"method":"shutdown"}
----
-{"jsonrpc":"2.0","method":"exit"}

clang-tools-extra/clangd/test/semantic-tokens.test

 # RUN: clangd -lit-test < %s | FileCheck -strict-whitespace %s -implicit-check-not=semanticHighlight
 # Send capabilities for both Theia semanticHighlight & standard semanticTokens.
-# clangd should not use/acknowledge the Theia protocol in this case.
+# clangd should not use/acknowledge the Theia protocol.
 {"jsonrpc":"2.0","id":0,"method":"initialize","params":{"capabilities":{"textDocument":{
   "semanticHighlightingCapabilities":{"semanticHighlighting":true},
   "semanticTokens":{"dynamicRegistration":true}
 }}}}
 ---
 {"jsonrpc":"2.0","method":"textDocument/didOpen","params":{"textDocument":{
   "uri": "test:///foo.cpp",
   "languageId": "cpp",

clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp

 
 namespace clang {
 namespace clangd {
 namespace {
 
 using testing::IsEmpty;
 using testing::SizeIs;
 
-MATCHER_P(LineNumber, L, "") { return arg.Line == L; }
-MATCHER(EmptyHighlightings, "") { return arg.Tokens.empty(); }
-
-std::vector<HighlightingToken>
-makeHighlightingTokens(llvm::ArrayRef<Range> Ranges, HighlightingKind Kind) {
-  std::vector<HighlightingToken> Tokens(Ranges.size());
-  for (int I = 0, End = Ranges.size(); I < End; ++I) {
-    Tokens[I].R = Ranges[I];
-    Tokens[I].Kind = Kind;
-  }
-
-  return Tokens;
-}
-
-std::vector<HighlightingToken> getExpectedTokens(Annotations &Test) {
-  static const std::map<HighlightingKind, std::string> KindToString{
-      {HighlightingKind::Variable, "Variable"},
-      {HighlightingKind::LocalVariable, "LocalVariable"},
-      {HighlightingKind::Parameter, "Parameter"},
-      {HighlightingKind::Function, "Function"},
-      {HighlightingKind::Class, "Class"},
-      {HighlightingKind::Enum, "Enum"},
-      {HighlightingKind::Namespace, "Namespace"},
-      {HighlightingKind::EnumConstant, "EnumConstant"},
-      {HighlightingKind::Field, "Field"},
-      {HighlightingKind::StaticField, "StaticField"},
-      {HighlightingKind::Method, "Method"},
-      {HighlightingKind::StaticMethod, "StaticMethod"},
-      {HighlightingKind::Typedef, "Typedef"},
-      {HighlightingKind::Type, "Type"},
-      {HighlightingKind::Unknown, "Unknown"},
-      {HighlightingKind::TemplateParameter, "TemplateParameter"},
-      {HighlightingKind::Concept, "Concept"},
-      {HighlightingKind::Primitive, "Primitive"},
-      {HighlightingKind::Macro, "Macro"}};
-  std::vector<HighlightingToken> ExpectedTokens;
-  for (const auto &KindString : KindToString) {
-    std::vector<HighlightingToken> Toks = makeHighlightingTokens(
-        Test.ranges(KindString.second), KindString.first);
-    ExpectedTokens.insert(ExpectedTokens.end(), Toks.begin(), Toks.end());
-  }
-  llvm::sort(ExpectedTokens);
-  return ExpectedTokens;
-}
-
 /// Annotates the input code with provided semantic highlightings. Results look
 /// something like:
 ///   class $Class[[X]] {
 ///     $Primitive[[int]] $Field[[a]] = 0;
 ///   };
 std::string annotate(llvm::StringRef Input,
                      llvm::ArrayRef<HighlightingToken> Tokens) {
   assert(std::is_sorted(
void checkHighlightings(llvm::StringRef Code,
   auto AST = TU.build();
   auto Actual = getSemanticHighlightings(AST);
   for (auto &Token : Actual)
     Token.Modifiers &= ModifierMask;
 
   EXPECT_EQ(Code, annotate(Test.code(), Actual));
 }
 
-// Any annotations in OldCode and NewCode are converted into their corresponding
-// HighlightingToken. The tokens are diffed against each other. Any lines where
-// the tokens should diff must be marked with a ^ somewhere on that line in
-// NewCode. If there are diffs that aren't marked with ^ the test fails. The
-// test also fails if there are lines marked with ^ that don't differ.
-void checkDiffedHighlights(llvm::StringRef OldCode, llvm::StringRef NewCode) {
-  Annotations OldTest(OldCode);
-  Annotations NewTest(NewCode);
-  std::vector<HighlightingToken> OldTokens = getExpectedTokens(OldTest);
-  std::vector<HighlightingToken> NewTokens = getExpectedTokens(NewTest);
-
-  llvm::DenseMap<int, std::vector<HighlightingToken>> ExpectedLines;
-  for (const Position &Point : NewTest.points()) {
-    ExpectedLines[Point.line]; // Default initialize to an empty line. Tokens
-                               // are inserted on these lines later.
-  }
-  std::vector<LineHighlightings> ExpectedLinePairHighlighting;
-  for (const HighlightingToken &Token : NewTokens) {
-    auto It = ExpectedLines.find(Token.R.start.line);
-    if (It != ExpectedLines.end())
-      It->second.push_back(Token);
-  }
-  for (auto &LineTokens : ExpectedLines)
-    ExpectedLinePairHighlighting.push_back(
-        {LineTokens.first, LineTokens.second, /*IsInactive = */ false});
-
-  std::vector<LineHighlightings> ActualDiffed =
-      diffHighlightings(NewTokens, OldTokens);
-  EXPECT_THAT(ActualDiffed,
-              testing::UnorderedElementsAreArray(ExpectedLinePairHighlighting))
-      << OldCode;
-}
-
 constexpr static uint32_t ScopeModifierMask =
     1 << unsigned(HighlightingModifier::FunctionScope) |
     1 << unsigned(HighlightingModifier::ClassScope) |
     1 << unsigned(HighlightingModifier::FileScope) |
     1 << unsigned(HighlightingModifier::GlobalScope);
 
 TEST(SemanticHighlighting, GetsCorrectTokens) {
   const char *TestCases[] = {
const char *TestCases[] = {
         int $Variable_globalScope[[Y]] = $Macro_globalScope[[X]];
       )cpp",
   };
 
   for (const char *Test : TestCases)
     checkHighlightings(Test, {}, ScopeModifierMask);
 }
 
-TEST(SemanticHighlighting, GeneratesHighlightsWhenFileChange) {
-  class HighlightingsCounter : public ClangdServer::Callbacks {
-  public:
-    std::atomic<int> Count = {0};
-
-    void onHighlightingsReady(
-        PathRef File, llvm::StringRef Version,
-        std::vector<HighlightingToken> Highlightings) override {
-      ++Count;
-    }
-  };
-
-  auto FooCpp = testPath("foo.cpp");
-  MockFS FS;
-  FS.Files[FooCpp] = "";
-
-  MockCompilationDatabase MCD;
-  HighlightingsCounter Counter;
-  ClangdServer Server(MCD, FS, ClangdServer::optsForTest(), &Counter);
-  Server.addDocument(FooCpp, "int a;");
-  ASSERT_TRUE(Server.blockUntilIdleForTest()) << "Waiting for server";
-  ASSERT_EQ(Counter.Count, 1);
-}
-
 // Ranges are highlighted as variables, unless highlighted as $Function etc.
 std::vector<HighlightingToken> tokens(llvm::StringRef MarkedText) {
   Annotations A(MarkedText);
   std::vector<HighlightingToken> Results;
   for (const Range& R : A.ranges())
     Results.push_back({HighlightingKind::Variable, 0, R});
   for (unsigned I = 0; I < static_cast<unsigned>(HighlightingKind::LastKind); ++I) {
     HighlightingKind Kind = static_cast<HighlightingKind>(I);
TEST(SemanticHighlighting, diffSemanticTokens) {
   EXPECT_EQ(6u, Diff.front().tokens[1].deltaStart);
   EXPECT_EQ(5u, Diff.front().tokens[1].length);
   // baz
   EXPECT_EQ(0u, Diff.front().tokens[2].deltaLine);
   EXPECT_EQ(6u, Diff.front().tokens[2].deltaStart);
   EXPECT_EQ(3u, Diff.front().tokens[2].length);
 }
 
-TEST(SemanticHighlighting, toTheiaSemanticHighlightingInformation) {
-  auto CreatePosition = [](int Line, int Character) -> Position {
-    Position Pos;
-    Pos.line = Line;
-    Pos.character = Character;
-    return Pos;
-  };
-
-  std::vector<LineHighlightings> Tokens{
-      {3,
-       {{HighlightingKind::Variable, 0,
-         Range{CreatePosition(3, 8), CreatePosition(3, 12)}},
-        {HighlightingKind::Function, 0,
-         Range{CreatePosition(3, 4), CreatePosition(3, 7)}}},
-       /* IsInactive = */ false},
-      {1,
-       {{HighlightingKind::Variable, 0,
-         Range{CreatePosition(1, 1), CreatePosition(1, 5)}}},
-       /* IsInactive = */ true}};
-  std::vector<TheiaSemanticHighlightingInformation> ActualResults =
-      toTheiaSemanticHighlightingInformation(Tokens);
-  std::vector<TheiaSemanticHighlightingInformation> ExpectedResults = {
-      {3, "AAAACAAEAAAAAAAEAAMAAw=="}, {1, "AAAAAQAEAAA="}};
-  EXPECT_EQ(ActualResults, ExpectedResults);
-}
-
-TEST(SemanticHighlighting, HighlightingDiffer) {
-  struct {
-    llvm::StringRef OldCode;
-    llvm::StringRef NewCode;
-  } TestCases[]{{
-                    R"(
-        $Variable[[A]]
-        $Class[[B]]
-        $Function[[C]]
-      )",
-                    R"(
-        $Variable[[A]]
-        $Class[[D]]
-        $Function[[C]]
-      )"},
-                {
-                    R"(
-        $Class[[C]]
-        $Field[[F]]
-        $Variable[[V]]
-        $Class[[C]] $Variable[[V]] $Field[[F]]
-      )",
-                    R"(
-        $Class[[C]]
-        $Field[[F]]
-       ^$Function[[F]]
-        $Class[[C]] $Variable[[V]] $Field[[F]]
-      )"},
-                {
-                    R"(
-
-        $Class[[A]]
-        $Variable[[A]]
-      )",
-                    R"(
-
-       ^
-       ^$Class[[A]]
-       ^$Variable[[A]]
-      )"},
-                {
-                    R"(
-        $Class[[C]]
-        $Field[[F]]
-        $Variable[[V]]
-        $Class[[C]] $Variable[[V]] $Field[[F]]
-      )",
-                    R"(
-        $Class[[C]]
-       ^
-       ^
-        $Class[[C]] $Variable[[V]] $Field[[F]]
-      )"},
-                {
-                    R"(
-        $Class[[A]]
-        $Variable[[A]]
-        $Variable[[A]]
-      )",
-                    R"(
-        $Class[[A]]
-       ^$Variable[[AA]]
-        $Variable[[A]]
-      )"},
-                {
-                    R"(
-        $Class[[A]]
-        $Variable[[A]]
-      )",
-                    R"(
-        $Class[[A]]
-        $Variable[[A]]
-       ^$Class[[A]]
-       ^$Variable[[A]]
-      )"},
-                {
-                    R"(
-        $Variable[[A]]
-        $Variable[[A]]
-        $Variable[[A]]
-      )",
-                    R"(
-       ^$Class[[A]]
-       ^$Class[[A]]
-       ^$Class[[A]]
-      )"}};
-
-  for (const auto &Test : TestCases)
-    checkDiffedHighlights(Test.OldCode, Test.NewCode);
-}
-
-TEST(SemanticHighlighting, DiffBeyondTheEndOfFile) {
-  llvm::StringRef OldCode =
-      R"(
-        $Class[[A]]
-        $Variable[[A]]
-        $Class[[A]]
-        $Variable[[A]]
-      )";
-  llvm::StringRef NewCode =
-      R"(
-        $Class[[A]] // line 1
-        $Variable[[A]] // line 2
-      )";
-
-  Annotations OldTest(OldCode);
-  Annotations NewTest(NewCode);
-  std::vector<HighlightingToken> OldTokens = getExpectedTokens(OldTest);
-  std::vector<HighlightingToken> NewTokens = getExpectedTokens(NewTest);
-
-  auto ActualDiff = diffHighlightings(NewTokens, OldTokens);
-  EXPECT_THAT(ActualDiff,
-              testing::UnorderedElementsAre(
-                  testing::AllOf(LineNumber(3), EmptyHighlightings()),
-                  testing::AllOf(LineNumber(4), EmptyHighlightings())));
-}
-
 } // namespace
 } // namespace clangd
 } // namespace clang