[clangd] Added option to enable semantic highlighting via an experimental capability

[jvikstrom]

Summary

Adds option to enable semantic highlighting as an experimental capability. The reason for this is that it is not possible to add textDocument capabilities in the vscode extension as it is a strongly typed argument. The experimental capability is a normal JS like object and we can therefore send the capability there. However we can't remove the textDocument capability as theia relies on the capability to be in textDocument. (The same reasoning is why the server sends the TM scopes in the experimental capability if it got the capability as experimental).

[jvikstrom]

Made Protocol comment be doxygen.

[sammccall]

clang-tools-extra/clangd/ClangdLSPServer.cpp

428

these comments (and also the one in Protocol.h) lack context: it's easy to understand what they're saying when looking at this patch, but hard when just looking at the code.

Maybe something like "There are two slightly different versions of the semantic highlighting capabilities: one in the 'main' namespace (supported by Theia), and one in the 'experimental' namespace (supported by VSCode)"

[nridge]

I believe this is not necessary. You can add text document capabilities in the vscode client extension like this:

class SemanticHighlightingFeature implements vscodelc.StaticFeature {
  fillClientCapabilities(capabilities: vscodelc.ClientCapabilities): void {
    const textDocumentCapabilities:
        vscodelc.TextDocumentClientCapabilities & { semanticHighlightingCapabilities?: { semanticHighlighting: boolean } }
        = capabilities.textDocument;
    textDocumentCapabilities.semanticHighlightingCapabilities = { semanticHighlighting: true };
  }
}

...

clangdClient.registerFeature(new SemanticHighlightingFeature());

What this is doing is creating a new type on the fly which extends vscodelc.TextDocumentCapabilities with a new optional field semanticHighlightingCapabilities, and then casts capabilities.textDocument to that type.

[jvikstrom]

In D64922#1603759, @nridge wrote:

I believe this is not necessary. You can add text document capabilities in the vscode client extension like this:

class SemanticHighlightingFeature implements vscodelc.StaticFeature {
  fillClientCapabilities(capabilities: vscodelc.ClientCapabilities): void {
    const textDocumentCapabilities:
        vscodelc.TextDocumentClientCapabilities & { semanticHighlightingCapabilities?: { semanticHighlighting: boolean } }
        = capabilities.textDocument;
    textDocumentCapabilities.semanticHighlightingCapabilities = { semanticHighlighting: true };
  }
}

...

clangdClient.registerFeature(new SemanticHighlightingFeature());

What this is doing is creating a new type on the fly which extends vscodelc.TextDocumentCapabilities with a new optional field semanticHighlightingCapabilities, and then casts capabilities.textDocument to that type.

That definitely works, I had no idea that was a feature you could use in typescript. Thanks

Abandoning in favor of using type intersection in typescript instead.

clang-tools-extra/clangd/ClangdLSPServer.cpp

for (OffsetEncoding Supported : *Params.capabilities.offsetEncoding)
       }
   }
   llvm::Optional<WithContextValue> WithOffsetEncoding;
   if (NegotiatedOffsetEncoding)
     WithOffsetEncoding.emplace(kCurrentOffsetEncoding,
                                *NegotiatedOffsetEncoding);
 
   ClangdServerOpts.SemanticHighlighting =
-      Params.capabilities.SemanticHighlighting;
+      Params.capabilities.SemanticHighlighting |
+      Params.capabilities.ExperimentalSemanticHighlighting;
   if (Params.rootUri && *Params.rootUri)
     ClangdServerOpts.WorkspaceRoot = Params.rootUri->file();
   else if (Params.rootPath && !Params.rootPath->empty())
     ClangdServerOpts.WorkspaceRoot = *Params.rootPath;
   if (Server)
     return Reply(llvm::make_error<LSPError>("server already initialized",
                                             ErrorCode::InvalidRequest));
   if (const auto &Dir = Params.initializationOptions.compilationDatabasePath)
llvm::json::Object Result{
                  {"commands",
                   {ExecuteCommandParams::CLANGD_APPLY_FIX_COMMAND,
                    ExecuteCommandParams::CLANGD_APPLY_TWEAK}},
              }},
             {"typeHierarchyProvider", true},
         }}}};
   if (NegotiatedOffsetEncoding)
     Result["offsetEncoding"] = *NegotiatedOffsetEncoding;
+  llvm::json::Object TMScopes{{"scopes", buildHighlightScopeLookupTable()}};
+  // Theia needs the text mate scopes in the normal capabilities object.
   if (Params.capabilities.SemanticHighlighting)
     Result.getObject("capabilities")
-        ->insert(
-            {"semanticHighlighting",
-             llvm::json::Object{{"scopes", buildHighlightScopeLookupTable()}}});
+        ->insert({"semanticHighlighting", (llvm::json::Object)TMScopes});
+  // The vscode language client can currently only get the text mate scopes via
+  // the experimental server capability.
+  if (Params.capabilities.ExperimentalSemanticHighlighting)
+    Result.getObject("capabilities")
+        ->insert({"experimental",
+                  llvm::json::Object{
+                      {"semanticHighlighting", (llvm::json::Object)TMScopes}}});
   Reply(std::move(Result));
 }
 
 void ClangdLSPServer::onShutdown(const ShutdownParams &Params,
                                  Callback<std::nullptr_t> Reply) {
   // Do essentially nothing, just say we're ready to exit.
   ShutdownRequestReceived = true;
   Reply(nullptr);

clang-tools-extra/clangd/Protocol.h

struct ClientCapabilities {
   /// Client supports CodeAction return value for textDocument/codeAction.
   /// textDocument.codeAction.codeActionLiteralSupport.
   bool CodeActionStructure = false;
 
   /// Client supports semantic highlighting.
   /// textDocument.semanticHighlightingCapabilities.semanticHighlighting
   bool SemanticHighlighting = false;
 
+  /// Only used because there is no other way to pass the text mate scopes to
+  /// the vscode extension other than as experimental server capabilities.
+  bool ExperimentalSemanticHighlighting = false;
+
   /// Supported encodings for LSP character offsets. (clangd extension).
   llvm::Optional<std::vector<OffsetEncoding>> offsetEncoding;
 
   /// The content format that should be used for Hover requests.
   /// textDocument.hover.contentEncoding
   MarkupKind HoverContentFormat = MarkupKind::PlainText;
 };
 bool fromJSON(const llvm::json::Value &, ClientCapabilities &);

clang-tools-extra/clangd/Protocol.cpp

SymbolKind indexSymbolKindToSymbolKind(index::SymbolKind Kind) {
   }
   llvm_unreachable("invalid symbol kind");
 }
 
 bool fromJSON(const llvm::json::Value &Params, ClientCapabilities &R) {
   const llvm::json::Object *O = Params.getAsObject();
   if (!O)
     return false;
+  if (auto *Experimental = O->getObject("experimental"))
+    if (auto *SemanticHighlighting =
+            Experimental->getObject("semanticHighlightingCapabilities"))
+      if (auto SemanticHighlightingSupport =
+              SemanticHighlighting->getBoolean("semanticHighlighting"))
+        R.ExperimentalSemanticHighlighting = *SemanticHighlightingSupport;
   if (auto *TextDocument = O->getObject("textDocument")) {
     if (auto *SemanticHighlighting =
             TextDocument->getObject("semanticHighlightingCapabilities")) {
       if (auto SemanticHighlightingSupport =
               SemanticHighlighting->getBoolean("semanticHighlighting"))
         R.SemanticHighlighting = *SemanticHighlightingSupport;
     }
     if (auto *Diagnostics = TextDocument->getObject("publishDiagnostics")) {

clang-tools-extra/clangd/test/experimental-semantic-highlighting.test

+# RUN: clangd -lit-test < %s | FileCheck -strict-whitespace %s
+{"jsonrpc":"2.0","id":0,"method":"initialize","params":{"processId":123,"rootPath":"clangd","capabilities":{"experimental":{"semanticHighlightingCapabilities":{"semanticHighlighting":true}}},"trace":"off"}}
+---
+#      CHECK:  "id": 0,
+#      CHECK:      "experimental": {
+# CHECK-NEXT:        "semanticHighlighting": {
+# CHECK-NEXT:          "scopes": [
+# CHECK-NEXT:            [
+# CHECK-NEXT:              "variable.other.cpp"
+# CHECK-NEXT:            ],
+# CHECK-NEXT:            [
+# CHECK-NEXT:              "entity.name.function.cpp"
+# CHECK-NEXT:            ],
+# CHECK-NEXT:            [
+# CHECK-NEXT:              "entity.name.function.method.cpp"
+# CHECK-NEXT:            ],
+# CHECK-NEXT:            [
+# CHECK-NEXT:              "variable.other.field.cpp"
+# CHECK-NEXT:            ],
+# CHECK-NEXT:            [
+# CHECK-NEXT:              "entity.name.type.class.cpp"
+# CHECK-NEXT:            ],
+# CHECK-NEXT:            [
+# CHECK-NEXT:              "entity.name.type.enum.cpp"
+# CHECK-NEXT:            ],
+# CHECK-NEXT:            [
+# CHECK-NEXT:              "variable.other.enummember.cpp"
+# CHECK-NEXT:            ],
+# CHECK-NEXT:            [
+# CHECK-NEXT:              "entity.name.namespace.cpp"
+# CHECK-NEXT:            ],
+# CHECK-NEXT:            [
+# CHECK-NEXT:              "entity.name.type.template.cpp"
+# CHECK-NEXT:            ]
+# CHECK-NEXT:          ]
+# CHECK-NEXT:        }
+# CHECK-NEXT:      },
+---
+{"jsonrpc":"2.0","id":3,"method":"shutdown"}
+---
+{"jsonrpc":"2.0","method":"exit"}

clang-tools-extra/clangd/test/semantic-highlighting.test

 # RUN: clangd -lit-test < %s | FileCheck -strict-whitespace %s
 {"jsonrpc":"2.0","id":0,"method":"initialize","params":{"processId":123,"rootPath":"clangd","capabilities":{"textDocument":{"semanticHighlightingCapabilities":{"semanticHighlighting":true}}},"trace":"off"}}
 ---
 #      CHECK:  "id": 0,
-# CHECK:      "semanticHighlighting": {
+#      CHECK:      "semanticHighlighting": {
 # CHECK-NEXT:        "scopes": [
 # CHECK-NEXT:          [
 # CHECK-NEXT:            "variable.other.cpp"
 # CHECK-NEXT:          ],
 # CHECK-NEXT:          [
 # CHECK-NEXT:            "entity.name.function.cpp"
 # CHECK-NEXT:          ],
 # CHECK-NEXT:          [