[clangd] Emit publishSemanticHighlighting in LSP if enabled

[jvikstrom]

Summary

Emit publishSemanticHighlighting in LSP if enabled

[hokein]

clang-tools-extra/clangd/SemanticHighlighting.h

31 ↗

(On Diff #207010)

The structures defined here are similar to the structures proposed in LSP SemanticHighlightingParams and SemanticHighlightingInformation, we could define these structures in the Protocol.h (you can find similar pattern there).

[jvikstrom]

Added SemanticHighlightingInformation and SemantigHighlightingParams.

[hokein]

clang-tools-extra/clangd/ClangdLSPServer.cpp

332 ↗

(On Diff #207031)

I'm a bit nervous -- we will turn on this feature when the client states that it supports semantic highlighting, client may disable its reg-based highlighting, and relies on clangd's, but the feature in clangd is in pretty early stage...

I think we'd need to add a new command-line flag to clangd (disabled by default).

clang-tools-extra/clangd/Protocol.h

1179 ↗

(On Diff #207031)

please use /// which is doxygen comment.

And /// Represents a semantic highlighting information that has to be applied on a specific line of the text document.

1182 ↗

(On Diff #207031)

nit: using int is fine here.

1189 ↗

(On Diff #207031)

Add a comment: /// Parameters for the semantic highlighting (server-side) push notification.

clang-tools-extra/clangd/SemanticHighlighting.cpp

120 ↗

(On Diff #207031)

add unittests for this function?

132 ↗

(On Diff #207031)

the implementation requires Tokens sorted, maybe consider using a map to the group-by?

152 ↗

(On Diff #207031)

thinking more about this, the .cpp suffix is language-specific, clangd also supports other C-family languages (e.g. C, ObjC), so we may need corresponding language suffix in the scopes (based on the language mode).

No need to do the change here, but please add a FIXME.

clang-tools-extra/clangd/SemanticHighlighting.h

8 ↗

(On Diff #207031)

The semantic highlighting is not in LSP yet, I think we need some documentations here to explain some more details about this feature in clangd, like the implementations are based on the proposal (https://github.com/microsoft/vscode-languageserver-node/pull/367).

19 ↗

(On Diff #207031)

as this kind is used as the lookup table index, let's add Variable = 0

37 ↗

(On Diff #207031)

I'd name it textMateScopeLookupTable

40 ↗

(On Diff #207031)

the assumption seems too strict here. IIUC, LSP doesn't require the order of the SemanticHighlightingInformation[], would be nice to return an ordered one (but that's implementation detail).

46 ↗

(On Diff #207031)

just toSemanticHighlightingInformation.

[jvikstrom]

Addressed comments.

[hokein]

clang-tools-extra/clangd/ClangdLSPServer.cpp

333 ↗

(On Diff #207241)

I'd put the logic here rather than in ClangdServer.cpp, just ClangdServerOpts.SemanticHighlighting = ClangdServerOpts.HiddenFeatures && Params.capabilities.SemanticHighlighting;.

clang-tools-extra/clangd/SemanticHighlighting.cpp

131 ↗

(On Diff #207241)

here we lookup the map twice, we could save one cost of lookup.

TokenLines[LindIdx].push_back(Token) should work.

143 ↗

(On Diff #207241)

could we have some comments explaining the details about encoding Token here?

144 ↗

(On Diff #207241)

if the token is across multiple line, we will emit an ill-formed results.

clang-tools-extra/clangd/SemanticHighlighting.h

44 ↗

(On Diff #207241)

just: convert to LSP's semantic highlighting information.

clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp

103 ↗

(On Diff #207241)

we could drop the explicit HighlightingToken, just

... Tokens = {
   {HighlightingKind::Variable, Range {...}},
  {....}
}

104 ↗

(On Diff #207241)

Range{ /*start*/{3, 8}, /*end*/{3, 12} } should be compilable.

112 ↗

(On Diff #207241)

here as well, just { {1, "...." } }.

[jvikstrom]

Address comments.

[jvikstrom]

clang-tools-extra/clangd/SemanticHighlighting.cpp

144 ↗

(On Diff #207241)

There's a FIXME above (which is where it should probably be handled). A bit unsure how to solve though. If a token is a block comment spanning multiple lines we would need to know the length of every line in the block comment. Probably something that can be solved with the ASTContext or SourceManager but that can't be accessed in this function.

clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp

104 ↗

(On Diff #207241)

Doesn't compile because of the default initialization of line and character in Position.

[jvikstrom]

Removed from hiding under hidden flag .

[jvikstrom]

Removed unused headers

[hokein]

clang-tools-extra/clangd/ClangdLSPServer.cpp

409 ↗

(On Diff #207563)

The proposal says If the client declares its capabilities with respect to the semantic highlighting feature, and if the server supports this feature too, the server should set all the available TextMate scopes as a "lookup table" during the initialize request..

now it seems that clangd always emit this lookup table to the client.

clang-tools-extra/clangd/SemanticHighlighting.cpp

144 ↗

(On Diff #207241)

oh, i missed that FIXME, the FIXME is a bit far away, maybe move it here (now we assume the token is always on the same line).

74 ↗

(On Diff #207563)

The name U doesn't provide much information, maybe Bytes?

138 ↗

(On Diff #207563)

The code is a bit tricky here, if I understand the code correctly, LineHighlights is the binary data of tokens (each char represents a byte).
Maybe LineEncodingTokens?

141 ↗

(On Diff #207563)

don't repeat what the code does in the comment. Here we are encoding a Token in a form specified in the proposal, I think we should have high-level comment.

clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp

109 ↗

(On Diff #207563)

nit: ActualResults?

111 ↗

(On Diff #207563)

nit: ExpectedResults?

114 ↗

(On Diff #207563)

nit: we should use EXPECT_* when comparing the expected results and actual results.

[jvikstrom]

Not sending TextMate scopes over LSP if semantic highlighting is not enabled.

[jvikstrom]

clang-tools-extra/clangd/SemanticHighlighting.cpp

144 ↗

(On Diff #207241)

I think that this should probably be handled above (so if a token covers N different lines it would be separated into N tokens in the TokenLines map.
I don't really see how it would be possible to handle that in this loop (without basically putting all the smallvectors for every line in another vector and writing to Lines in another loop after the loop is done. But it feels like that would be way more complex)

138 ↗

(On Diff #207563)

They aren't "fully" encoded yet though, they get encoded after the inner loop is done. How about LineByteTokens?

clang-tools-extra/clangd/SemanticHighlighting.h

8 ↗

(On Diff #207031)

Changed the initialize-params test to have semantic highlighting enabled as well.

[hokein]

mostly good, a few more nits.

clang-tools-extra/clangd/Protocol.h

1187 ↗

(On Diff #207758)

nit: remove the blank line to be consistent with the existing style in this file.

1199 ↗

(On Diff #207758)

nit: remove this blank line.

clang-tools-extra/clangd/SemanticHighlighting.cpp

121 ↗

(On Diff #207758)

nit: remove the "{}"

138 ↗

(On Diff #207758)

add

|<--- 4 bytes  --->|<-2 bytes->|<-2 bytes->|
|    character      |  length       |    index       |

to the comment, it helps reader to understand how the encode like.

156 ↗

(On Diff #207758)

nit: remove the {} for one-body statement.

138 ↗

(On Diff #207563)

sounds good.

clang-tools-extra/clangd/SemanticHighlighting.h

7 ↗

(On Diff #207758)

nit: this should in a new section:

//==-- SemanticHighlighting.h - Generating highlights from the AST-- C++ -*-==//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//===---------------------------------------------------------------------------------===//
//  
//  An implementation ....
// 
//===---------------------------------------------------------------------------------===//

clang-tools-extra/clangd/test/initialize-params.test

3 ↗

(On Diff #207758)

How about moving this to the new semantic-highlighting.test?

[jvikstrom]

Addressed comments.

[hokein]

thanks, looks good.

clang-tools-extra/clangd/ClangdLSPServer.h

59 ↗

(On Diff #207783)

nit: clang-format.

clang-tools-extra/clangd/test/semantic-highlighting.test

36 ↗

(On Diff #207783)

nit: we could just verify this field like:

# CHECK:  "id": 0,
# CHECK:  " "semanticHighlighting": {
# CHECK-NEXT:        "scopes": [
# CHECK-NEXT:          [
# CHECK-NEXT:            "variable.cpp"
# CHECK-NEXT:          ],
# CHECK-NEXT:          [
# CHECK-NEXT:            "entity.name.function.cpp"
# CHECK-NEXT:          ]
# CHECK-NEXT:        ]
# CHECK-NEXT:      },

clang-tools-extra/trunk/clangd/ClangdLSPServer.h

public:
   ///
   /// \return Whether we shut down cleanly with a 'shutdown' -> 'exit' sequence.
   bool run();
 
 private:
   // Implement DiagnosticsConsumer.
   void onDiagnosticsReady(PathRef File, std::vector<Diag> Diagnostics) override;
   void onFileUpdated(PathRef File, const TUStatus &Status) override;
+  void onHighlightingsReady(PathRef File,
+                         std::vector<HighlightingToken> Highlightings) override;
 
   // LSP methods. Notifications have signature void(const Params&).
   // Calls have signature void(const Params&, Callback<Response>).
   void onInitialize(const InitializeParams &, Callback<llvm::json::Value>);
   void onShutdown(const ShutdownParams &, Callback<std::nullptr_t>);
   void onSync(const NoParams &, Callback<std::nullptr_t>);
   void onDocumentDidOpen(const DidOpenTextDocumentParams &);
   void onDocumentDidChange(const DidChangeTextDocumentParams &);
private:
   bool shouldRunCompletion(const CompletionParams &Params) const;
 
   /// Forces a reparse of all currently opened files.  As a result, this method
   /// may be very expensive.  This method is normally called when the
   /// compilation database is changed.
   void reparseOpenedFiles();
   void applyConfiguration(const ConfigurationSettings &Settings);
 
+  /// Sends a "publishSemanticHighlighting" notification to the LSP client.
+  void publishSemanticHighlighting(SemanticHighlightingParams Params);
+
   /// Sends a "publishDiagnostics" notification to the LSP client.
   void publishDiagnostics(const URIForFile &File,
                           std::vector<clangd::Diagnostic> Diagnostics);
 
   /// Used to indicate that the 'shutdown' request was received from the
   /// Language Server client.
   bool ShutdownRequestReceived = false;
 

clang-tools-extra/trunk/clangd/ClangdLSPServer.cpp

 //===--- ClangdLSPServer.cpp - LSP server ------------------------*- C++-*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "ClangdLSPServer.h"
 #include "Diagnostics.h"
 #include "FormattedString.h"
 #include "GlobalCompilationDatabase.h"
 #include "Protocol.h"
+#include "SemanticHighlighting.h"
 #include "SourceCode.h"
 #include "Trace.h"
 #include "URI.h"
 #include "refactor/Tweak.h"
 #include "clang/Tooling/Core/Replacement.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/ScopeExit.h"
for (OffsetEncoding Supported : *Params.capabilities.offsetEncoding)
         break;
       }
   }
   llvm::Optional<WithContextValue> WithOffsetEncoding;
   if (NegotiatedOffsetEncoding)
     WithOffsetEncoding.emplace(kCurrentOffsetEncoding,
                                *NegotiatedOffsetEncoding);
 
+  ClangdServerOpts.SemanticHighlighting =
+      Params.capabilities.SemanticHighlighting;
   if (Params.rootUri && *Params.rootUri)
     ClangdServerOpts.WorkspaceRoot = Params.rootUri->file();
   else if (Params.rootPath && !Params.rootPath->empty())
     ClangdServerOpts.WorkspaceRoot = *Params.rootPath;
   if (Server)
     return Reply(llvm::make_error<LSPError>("server already initialized",
                                             ErrorCode::InvalidRequest));
   if (const auto &Dir = Params.initializationOptions.compilationDatabasePath)
llvm::json::Object Result{
                  {"commands",
                   {ExecuteCommandParams::CLANGD_APPLY_FIX_COMMAND,
                    ExecuteCommandParams::CLANGD_APPLY_TWEAK}},
              }},
             {"typeHierarchyProvider", true},
         }}}};
   if (NegotiatedOffsetEncoding)
     Result["offsetEncoding"] = *NegotiatedOffsetEncoding;
+  if (Params.capabilities.SemanticHighlighting)
+    Result.getObject("capabilities")
+        ->insert(
+            {"semanticHighlighting",
+             llvm::json::Object{{"scopes", getTextMateScopeLookupTable()}}});
   Reply(std::move(Result));
 }
 
 void ClangdLSPServer::onShutdown(const ShutdownParams &Params,
                                  Callback<std::nullptr_t> Reply) {
   // Do essentially nothing, just say we're ready to exit.
   ShutdownRequestReceived = true;
   Reply(nullptr);
if (Old != New) {
       CDB->setCompileCommand(File, std::move(New));
       ShouldReparseOpenFiles = true;
     }
   }
   if (ShouldReparseOpenFiles)
     reparseOpenedFiles();
 }
 
+void ClangdLSPServer::publishSemanticHighlighting(
+    SemanticHighlightingParams Params) {
+  notify("textDocument/semanticHighlighting", Params);
+}
+
 void ClangdLSPServer::publishDiagnostics(
     const URIForFile &File, std::vector<clangd::Diagnostic> Diagnostics) {
   // Publish diagnostics.
   notify("textDocument/publishDiagnostics",
          llvm::json::Object{
              {"uri", File},
              {"diagnostics", std::move(Diagnostics)},
          });
bool ClangdLSPServer::shouldRunCompletion(
   if (Trigger == ">")
     return (*Code)[*Offset - 2] == '-'; // trigger only on '->'.
   if (Trigger == ":")
     return (*Code)[*Offset - 2] == ':'; // trigger only on '::'.
   assert(false && "unhandled trigger character");
   return true;
 }
 
+void ClangdLSPServer::onHighlightingsReady(
+    PathRef File, std::vector<HighlightingToken> Highlightings) {
+  publishSemanticHighlighting(
+      {{URIForFile::canonicalize(File, /*TUPath=*/File)},
+       toSemanticHighlightingInformation(Highlightings)});
+}
+
 void ClangdLSPServer::onDiagnosticsReady(PathRef File,
                                          std::vector<Diag> Diagnostics) {
   auto URI = URIForFile::canonicalize(File, /*TUPath=*/File);
   std::vector<Diagnostic> LSPDiagnostics;
   DiagnosticToReplacementMap LocalFixIts; // Temporary storage
   for (auto &Diag : Diagnostics) {
     toLSPDiags(Diag, URI, DiagOpts,
                [&](clangd::Diagnostic Diag, llvm::ArrayRef<Fix> Fixes) {

clang-tools-extra/trunk/clangd/Protocol.h

struct ClientCapabilities {
   /// The supported set of CompletionItemKinds for textDocument/completion.
   /// textDocument.completion.completionItemKind.valueSet
   llvm::Optional<CompletionItemKindBitset> CompletionItemKinds;
 
   /// Client supports CodeAction return value for textDocument/codeAction.
   /// textDocument.codeAction.codeActionLiteralSupport.
   bool CodeActionStructure = false;
 
+  /// Client supports semantic highlighting.
+  bool SemanticHighlighting = false;
+
   /// Supported encodings for LSP character offsets. (clangd extension).
   llvm::Optional<std::vector<OffsetEncoding>> offsetEncoding;
 
   /// The content format that should be used for Hover requests.
   MarkupKind HoverContentFormat = MarkupKind::PlainText;
 };
 bool fromJSON(const llvm::json::Value &, ClientCapabilities &);
 
struct FileStatus {
   URIForFile uri;
   /// The human-readable string presents the current state of the file, can be
   /// shown in the UI (e.g. status bar).
   std::string state;
   // FIXME: add detail messages.
 };
 llvm::json::Value toJSON(const FileStatus &FStatus);
 
+/// Represents a semantic highlighting information that has to be applied on a
+/// specific line of the text document.
+struct SemanticHighlightingInformation {
+  /// The line these highlightings belong to.
+  int Line;
+  /// The base64 encoded string of highlighting tokens.
+  std::string Tokens;
+};
+bool operator==(const SemanticHighlightingInformation &Lhs,
+                const SemanticHighlightingInformation &Rhs);
+llvm::json::Value toJSON(const SemanticHighlightingInformation &Highlighting);
+
+/// Parameters for the semantic highlighting (server-side) push notification.
+struct SemanticHighlightingParams {
+  /// The textdocument these highlightings belong to.
+  TextDocumentIdentifier TextDocument;
+  /// The lines of highlightings that should be sent.
+  std::vector<SemanticHighlightingInformation> Lines;
+};
+llvm::json::Value toJSON(const SemanticHighlightingParams &Highlighting);
+
 } // namespace clangd
 } // namespace clang
 
 namespace llvm {
 template <> struct format_provider<clang::clangd::Position> {
   static void format(const clang::clangd::Position &Pos, raw_ostream &OS,
                      StringRef Style) {
     assert(Style.empty() && "style modifiers for this type are not supported");
     OS << Pos;
   }
 };
 } // namespace llvm
 
 #endif

clang-tools-extra/trunk/clangd/Protocol.cpp

SymbolKind indexSymbolKindToSymbolKind(index::SymbolKind Kind) {
   llvm_unreachable("invalid symbol kind");
 }
 
 bool fromJSON(const llvm::json::Value &Params, ClientCapabilities &R) {
   const llvm::json::Object *O = Params.getAsObject();
   if (!O)
     return false;
   if (auto *TextDocument = O->getObject("textDocument")) {
+    if (auto *SemanticHighlighting =
+            TextDocument->getObject("semanticHighlightingCapabilities")) {
+      if (auto SemanticHighlightingSupport =
+              SemanticHighlighting->getBoolean("semanticHighlighting"))
+        R.SemanticHighlighting = *SemanticHighlightingSupport;
+    }
     if (auto *Diagnostics = TextDocument->getObject("publishDiagnostics")) {
       if (auto CategorySupport = Diagnostics->getBoolean("categorySupport"))
         R.DiagnosticCategory = *CategorySupport;
       if (auto CodeActions = Diagnostics->getBoolean("codeActionsInline"))
         R.DiagnosticFixes = *CodeActions;
       if (auto RelatedInfo = Diagnostics->getBoolean("relatedInformation"))
         R.DiagnosticRelatedInformation = *RelatedInfo;
     }
OE = llvm::StringSwitch<OffsetEncoding>(*Str)
            .Case("utf-32", OffsetEncoding::UTF32)
            .Default(OffsetEncoding::UnsupportedEncoding);
   return true;
 }
 llvm::raw_ostream &operator<<(llvm::raw_ostream &OS, OffsetEncoding Enc) {
   return OS << toString(Enc);
 }
 
+bool operator==(const SemanticHighlightingInformation &Lhs,
+                const SemanticHighlightingInformation &Rhs) {
+  return Lhs.Line == Rhs.Line && Lhs.Tokens == Rhs.Tokens;
+}
+
+llvm::json::Value toJSON(const SemanticHighlightingInformation &Highlighting) {
+  return llvm::json::Object{{"line", Highlighting.Line},
+                            {"tokens", Highlighting.Tokens}};
+}
+
+llvm::json::Value toJSON(const SemanticHighlightingParams &Highlighting) {
+  return llvm::json::Object{
+      {"textDocument", Highlighting.TextDocument},
+      {"lines", std::move(Highlighting.Lines)},
+  };
+}
+
 } // namespace clangd
 } // namespace clang

clang-tools-extra/trunk/clangd/SemanticHighlighting.h

 //==-- SemanticHighlighting.h - Generating highlights from the AST-- C++ -*-==//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
+//
+// An implementation of semantic highlighting based on this proposal:
+// https://github.com/microsoft/vscode-languageserver-node/pull/367 in clangd.
+// Semantic highlightings are calculated for an AST by visiting every AST node
+// and classifying nodes that are interesting to highlight (variables/function
+// calls etc.).
+//
+//===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_SEMANTICHIGHLIGHTING_H
 #define LLVM_CLANG_TOOLS_EXTRA_CLANGD_SEMANTICHIGHLIGHTING_H
 
 #include "ClangdUnit.h"
+#include "Protocol.h"
 
 namespace clang {
 namespace clangd {
 
 enum class HighlightingKind {
-  Variable,
-  Function,
+  Variable = 0,
+  Function = 1,
 };
 
 // Contains all information needed for the highlighting a token.
 struct HighlightingToken {
   HighlightingKind Kind;
   Range R;
 };
 
 bool operator==(const HighlightingToken &Lhs, const HighlightingToken &Rhs);
 
 // Returns all HighlightingTokens from an AST. Only generates highlights for the
 // main AST.
 std::vector<HighlightingToken> getSemanticHighlightings(ParsedAST &AST);
 
+// Gets the TextMate scopes as a double nested array where the
+// SemanticHighlightKind indexes correctly into this vector.
+std::vector<std::vector<std::string>> getTextMateScopeLookupTable();
+
+// Convert to LSP's semantic highlighting information.
+std::vector<SemanticHighlightingInformation>
+toSemanticHighlightingInformation(llvm::ArrayRef<HighlightingToken> Tokens);
+
 } // namespace clangd
 } // namespace clang
 
 #endif

clang-tools-extra/trunk/clangd/SemanticHighlighting.cpp

 //===--- SemanticHighlighting.cpp - ------------------------- ---*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "SemanticHighlighting.h"
 #include "Logger.h"
+#include "Protocol.h"
 #include "SourceCode.h"
 #include "clang/AST/ASTContext.h"
 #include "clang/AST/RecursiveASTVisitor.h"
 
 namespace clang {
 namespace clangd {
 namespace {
 
if (!R) {
       elog("Tried to add semantic token with an invalid range");
       return;
     }
 
     Tokens.push_back({Kind, R.getValue()});
   }
 };
 
+// Encode binary data into base64.
+// This was copied from compiler-rt/lib/fuzzer/FuzzerUtil.cpp.
+// FIXME: Factor this out into llvm/Support?
+std::string encodeBase64(const llvm::SmallVectorImpl<char> &Bytes) {
+  static const char Table[] = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
+                              "abcdefghijklmnopqrstuvwxyz"
+                              "0123456789+/";
+  std::string Res;
+  size_t I;
+  for (I = 0; I + 2 < Bytes.size(); I += 3) {
+    uint32_t X = (Bytes[I] << 16) + (Bytes[I + 1] << 8) + Bytes[I + 2];
+    Res += Table[(X >> 18) & 63];
+    Res += Table[(X >> 12) & 63];
+    Res += Table[(X >> 6) & 63];
+    Res += Table[X & 63];
+  }
+  if (I + 1 == Bytes.size()) {
+    uint32_t X = (Bytes[I] << 16);
+    Res += Table[(X >> 18) & 63];
+    Res += Table[(X >> 12) & 63];
+    Res += "==";
+  } else if (I + 2 == Bytes.size()) {
+    uint32_t X = (Bytes[I] << 16) + (Bytes[I + 1] << 8);
+    Res += Table[(X >> 18) & 63];
+    Res += Table[(X >> 12) & 63];
+    Res += Table[(X >> 6) & 63];
+    Res += "=";
+  }
+  return Res;
+}
+
+void write32be(uint32_t I, llvm::raw_ostream &OS) {
+  std::array<char, 4> Buf;
+  llvm::support::endian::write32be(Buf.data(), I);
+  OS.write(Buf.data(), Buf.size());
+}
+
+void write16be(uint16_t I, llvm::raw_ostream &OS) {
+  std::array<char, 2> Buf;
+  llvm::support::endian::write16be(Buf.data(), I);
+  OS.write(Buf.data(), Buf.size());
+}
 } // namespace
 
 bool operator==(const HighlightingToken &Lhs, const HighlightingToken &Rhs) {
   return Lhs.Kind == Rhs.Kind && Lhs.R == Rhs.R;
 }
 
 std::vector<HighlightingToken> getSemanticHighlightings(ParsedAST &AST) {
   return HighlightingTokenCollector(AST).collectTokens();
 }
 
+std::vector<SemanticHighlightingInformation>
+toSemanticHighlightingInformation(llvm::ArrayRef<HighlightingToken> Tokens) {
+  if (Tokens.size() == 0)
+    return {};
+
+  // FIXME: Tokens might be multiple lines long (block comments) in this case
+  // this needs to add multiple lines for those tokens.
+  std::map<int, std::vector<HighlightingToken>> TokenLines;
+  for (const HighlightingToken &Token : Tokens)
+    TokenLines[Token.R.start.line].push_back(Token);
+
+  std::vector<SemanticHighlightingInformation> Lines;
+  Lines.reserve(TokenLines.size());
+  for (const auto &Line : TokenLines) {
+    llvm::SmallVector<char, 128> LineByteTokens;
+    llvm::raw_svector_ostream OS(LineByteTokens);
+    for (const auto &Token : Line.second) {
+      // Writes the token to LineByteTokens in the byte format specified by the
+      // LSP proposal. Described below.
+      // |<---- 4 bytes ---->|<-- 2 bytes -->|<--- 2 bytes -->|
+      // |    character      |  length       |    index       |
+
+      write32be(Token.R.start.character, OS);
+      write16be(Token.R.end.character - Token.R.start.character, OS);
+      write16be(static_cast<int>(Token.Kind), OS);
+    }
+
+    Lines.push_back({Line.first, encodeBase64(LineByteTokens)});
+  }
+
+  return Lines;
+}
+
+std::vector<std::vector<std::string>> getTextMateScopeLookupTable() {
+  // FIXME: Add scopes for C and Objective C.
+  std::map<HighlightingKind, std::vector<std::string>> Scopes = {
+      {HighlightingKind::Variable, {"variable.cpp"}},
+      {HighlightingKind::Function, {"entity.name.function.cpp"}}};
+  std::vector<std::vector<std::string>> NestedScopes(Scopes.size());
+  for (const auto &Scope : Scopes)
+    NestedScopes[static_cast<int>(Scope.first)] = Scope.second;
+
+  return NestedScopes;
+}
+
 } // namespace clangd
 } // namespace clang

clang-tools-extra/trunk/clangd/test/semantic-highlighting.test

+# RUN: clangd -lit-test < %s | FileCheck -strict-whitespace %s
+{"jsonrpc":"2.0","id":0,"method":"initialize","params":{"processId":123,"rootPath":"clangd","capabilities":{"textDocument":{"semanticHighlightingCapabilities":{"semanticHighlighting":true}}},"trace":"off"}}
+---
+#      CHECK:  "id": 0,
+# CHECK-NEXT:  "jsonrpc": "2.0",
+# CHECK-NEXT:  "result": {
+# CHECK-NEXT:    "capabilities": {
+# CHECK-NEXT:      "codeActionProvider": true,
+# CHECK-NEXT:      "completionProvider": {
+# CHECK-NEXT:        "resolveProvider": false,
+# CHECK-NEXT:        "triggerCharacters": [
+# CHECK-NEXT:          ".",
+# CHECK-NEXT:          ">",
+# CHECK-NEXT:          ":"
+# CHECK-NEXT:        ]
+# CHECK-NEXT:      },
+# CHECK-NEXT:      "declarationProvider": true,
+# CHECK-NEXT:      "definitionProvider": true,
+# CHECK-NEXT:      "documentFormattingProvider": true,
+# CHECK-NEXT:      "documentHighlightProvider": true,
+# CHECK-NEXT:      "documentOnTypeFormattingProvider": {
+# CHECK-NEXT:        "firstTriggerCharacter": "\n",
+# CHECK-NEXT:        "moreTriggerCharacter": []
+# CHECK-NEXT:      },
+# CHECK-NEXT:      "documentRangeFormattingProvider": true,
+# CHECK-NEXT:      "documentSymbolProvider": true,
+# CHECK-NEXT:      "executeCommandProvider": {
+# CHECK-NEXT:        "commands": [
+# CHECK-NEXT:          "clangd.applyFix",
+# CHECK-NEXT:          "clangd.applyTweak"
+# CHECK-NEXT:        ]
+# CHECK-NEXT:      },
+# CHECK-NEXT:      "hoverProvider": true,
+# CHECK-NEXT:      "referencesProvider": true,
+# CHECK-NEXT:      "renameProvider": true,
+# CHECK-NEXT:      "semanticHighlighting": {
+# CHECK-NEXT:        "scopes": [
+# CHECK-NEXT:          [
+# CHECK-NEXT:            "variable.cpp"
+# CHECK-NEXT:          ],
+# CHECK-NEXT:          [
+# CHECK-NEXT:            "entity.name.function.cpp"
+# CHECK-NEXT:          ]
+# CHECK-NEXT:        ]
+# CHECK-NEXT:      },
+# CHECK-NEXT:      "signatureHelpProvider": {
+# CHECK-NEXT:        "triggerCharacters": [
+# CHECK-NEXT:          "(",
+# CHECK-NEXT:          ","
+# CHECK-NEXT:        ]
+# CHECK-NEXT:      },
+# CHECK-NEXT:      "textDocumentSync": 2,
+# CHECK-NEXT:      "typeHierarchyProvider": true
+# CHECK-NEXT:      "workspaceSymbolProvider": true
+# CHECK-NEXT:    }
+# CHECK-NEXT:  }
+---
+{"jsonrpc":"2.0","method":"textDocument/didOpen","params":{"textDocument":{"uri":"test:///foo.cpp","languageId":"cpp","version":1,"text":"int x = 2;"}}}
+#      CHECK:  "method": "textDocument/semanticHighlighting",
+# CHECK-NEXT:  "params": {
+# CHECK-NEXT:    "lines": [
+# CHECK-NEXT:      {
+# CHECK-NEXT:        "line": 0,
+# CHECK-NEXT:        "tokens": "AAAABAABAAA="
+# CHECK-NEXT:      }
+# CHECK-NEXT:    ],
+# CHECK-NEXT:    "textDocument": {
+# CHECK-NEXT:      "uri": "file:///clangd-test/foo.cpp"
+# CHECK-NEXT:    }
+# CHECK-NEXT:  }
+# CHECK-NEXT:}
+---
+{"jsonrpc":"2.0","id":3,"method":"shutdown"}
+---
+{"jsonrpc":"2.0","method":"exit"}

clang-tools-extra/trunk/clangd/unittests/SemanticHighlightingTests.cpp

 //==- SemanticHighlightingTests.cpp - SemanticHighlighting tests-*- C++ -* -==//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "Annotations.h"
 #include "ClangdServer.h"
+#include "Protocol.h"
 #include "SemanticHighlighting.h"
 #include "TestFS.h"
 #include "TestTU.h"
 #include "gmock/gmock.h"
 
 namespace clang {
 namespace clangd {
 namespace {
)cpp",
       R"cpp(
     void $Function[[foo]](int);
   )cpp"};
   for (const auto &TestCase : TestCases) {
     checkHighlightings(TestCase);
   }
 }
 
-TEST(ClangdSemanticHighlightingTest, GeneratesHighlightsWhenFileChange) {
+TEST(SemanticHighlighting, GeneratesHighlightsWhenFileChange) {
   class HighlightingsCounterDiagConsumer : public DiagnosticsConsumer {
   public:
     std::atomic<int> Count = {0};
 
     void onDiagnosticsReady(PathRef, std::vector<Diag>) override {}
     void onHighlightingsReady(
         PathRef File, std::vector<HighlightingToken> Highlightings) override {
       ++Count;
     }
   };
 
   auto FooCpp = testPath("foo.cpp");
   MockFSProvider FS;
   FS.Files[FooCpp] = "";
 
   MockCompilationDatabase MCD;
   HighlightingsCounterDiagConsumer DiagConsumer;
   ClangdServer Server(MCD, FS, DiagConsumer, ClangdServer::optsForTest());
   Server.addDocument(FooCpp, "int a;");
   ASSERT_TRUE(Server.blockUntilIdleForTest()) << "Waiting for server";
   ASSERT_EQ(DiagConsumer.Count, 1);
 }
 
+TEST(SemanticHighlighting, toSemanticHighlightingInformation) {
+  auto CreatePosition = [](int Line, int Character) -> Position {
+    Position Pos;
+    Pos.line = Line;
+    Pos.character = Character;
+    return Pos;
+  };
+
+  std::vector<HighlightingToken> Tokens{
+      {HighlightingKind::Variable,
+                        Range{CreatePosition(3, 8), CreatePosition(3, 12)}},
+      {HighlightingKind::Function,
+                        Range{CreatePosition(3, 4), CreatePosition(3, 7)}},
+      {HighlightingKind::Variable,
+                        Range{CreatePosition(1, 1), CreatePosition(1, 5)}}};
+  std::vector<SemanticHighlightingInformation> ActualResults =
+      toSemanticHighlightingInformation(Tokens);
+  std::vector<SemanticHighlightingInformation> ExpectedResults = {
+      {1, "AAAAAQAEAAA="},
+      {3, "AAAACAAEAAAAAAAEAAMAAQ=="}};
+  EXPECT_EQ(ActualResults, ExpectedResults);
+}
+
 } // namespace
 } // namespace clangd
 } // namespace clang