[clangd] Introduce client state invalidation

[kadircet]

Summary

Clangd can invalidate client state of features like semantic higlighting
without client explicitly triggering, for example after a preamble build
caused by an onSave notification on a different file.

This patch introduces a mechanism to let client know of such actions,
and also calls the workspace/semanticTokens/refresh request to
demonstrate the situation after each preamble build.

Fixes https://github.com/clangd/clangd/issues/699

[kadircet]

clang-tools-extra/clangd/test/semantic-tokens-refresh.test

13

lsp spec says params: none for this request. i am not sure if this is acceptable or if we should really change our transport layer to accommodate optional parameters. can't really test it on vscode atm unfortunately :/

[kadircet]

ping

[sammccall]

clang-tools-extra/clangd/ClangdServer.h

78

I think "state change", "client information" and "requesting it" might be a bit too vague.
Some ideas to take or leave...

"onSemanticsMaybeChanged"

Called when the meaning of a source code may have changed without an edit.
Usually clients assume that responses to requests are valid until they next
edit the file. If they're invalidated at other times, we should tell the client.
In particular, when an asynchronous preamble build finishes, we can provide
more accurate semantic tokens, so we should tell the client to refresh.

[kadircet]

• Rename method and update comment

clang-tools-extra/clangd/ClangdLSPServer.h

public:
   void profile(MemoryTree &MT) const;
 
 private:
   // Implement ClangdServer::Callbacks.
   void onDiagnosticsReady(PathRef File, llvm::StringRef Version,
                           std::vector<Diag> Diagnostics) override;
   void onFileUpdated(PathRef File, const TUStatus &Status) override;
   void onBackgroundIndexProgress(const BackgroundQueue::Stats &Stats) override;
+  void onClientInformationInvalidated(PathRef File) override;
 
   // LSP methods. Notifications have signature void(const Params&).
   // Calls have signature void(const Params&, Callback<Response>).
   void onInitialize(const InitializeParams &, Callback<llvm::json::Value>);
   void onInitialized(const InitializedParams &);
   void onShutdown(const NoParams &, Callback<std::nullptr_t>);
   void onSync(const NoParams &, Callback<std::nullptr_t>);
   void onDocumentDidOpen(const DidOpenTextDocumentParams &);
private:
   LSPBinder::OutgoingMethod<WorkDoneProgressCreateParams, std::nullptr_t>
       CreateWorkDoneProgress;
   LSPBinder::OutgoingNotification<ProgressParams<WorkDoneProgressBegin>>
       BeginWorkDoneProgress;
   LSPBinder::OutgoingNotification<ProgressParams<WorkDoneProgressReport>>
       ReportWorkDoneProgress;
   LSPBinder::OutgoingNotification<ProgressParams<WorkDoneProgressEnd>>
       EndWorkDoneProgress;
+  LSPBinder::OutgoingMethod<NoParams, std::nullptr_t> SemanticTokensRefresh;
 
   void applyEdit(WorkspaceEdit WE, llvm::json::Value Success,
                  Callback<llvm::json::Value> Reply);
 
-  void bindMethods(LSPBinder &);
+  void bindMethods(LSPBinder &, const ClientCapabilities &Caps);
   std::vector<Fix> getFixes(StringRef File, const clangd::Diagnostic &D);
 
   /// Checks if completion request should be ignored. We need this due to the
   /// limitation of the LSP. Per LSP, a client sends requests for all "trigger
   /// character" we specify, but for '>' and ':' we need to check they actually
   /// produce '->' and '::', respectively.
   bool shouldRunCompletion(const CompletionParams &Params) const;
 

clang-tools-extra/clangd/ClangdLSPServer.cpp

llvm::json::Object ServerCaps{
       {"memoryUsageProvider", true}, // clangd extension
       {"compilationDatabase",        // clangd extension
        llvm::json::Object{{"automaticReload", true}}},
       {"callHierarchyProvider", true},
   };
 
   {
     LSPBinder Binder(Handlers, *this);
-    bindMethods(Binder);
+    bindMethods(Binder, Params.capabilities);
     if (Opts.Modules)
       for (auto &Mod : *Opts.Modules)
         Mod.initializeLSP(Binder, Params.rawCapabilities, ServerCaps);
   }
 
   // Per LSP, renameProvider can be either boolean or RenameOptions.
   // RenameOptions will be specified if the client states it supports prepare.
   ServerCaps["renameProvider"] =
assert(!Opts.ContextProvider &&
            "Only one of ConfigProvider and ContextProvider allowed!");
     this->Opts.ContextProvider = ClangdServer::createConfiguredContextProvider(
         Opts.ConfigProvider, this);
   }
   LSPBinder Bind(this->Handlers, *this);
   Bind.method("initialize", this, &ClangdLSPServer::onInitialize);
 }
 
-void ClangdLSPServer::bindMethods(LSPBinder &Bind) {
+void ClangdLSPServer::bindMethods(LSPBinder &Bind,
+                                  const ClientCapabilities &Caps) {
   // clang-format off
   Bind.notification("initialized", this, &ClangdLSPServer::onInitialized);
   Bind.method("shutdown", this, &ClangdLSPServer::onShutdown);
   Bind.method("sync", this, &ClangdLSPServer::onSync);
   Bind.method("textDocument/rangeFormatting", this, &ClangdLSPServer::onDocumentRangeFormatting);
   Bind.method("textDocument/onTypeFormatting", this, &ClangdLSPServer::onDocumentOnTypeFormatting);
   Bind.method("textDocument/formatting", this, &ClangdLSPServer::onDocumentFormatting);
   Bind.method("textDocument/codeAction", this, &ClangdLSPServer::onCodeAction);
void ClangdLSPServer::bindMethods(LSPBinder &Bind,
   ApplyWorkspaceEdit = Bind.outgoingMethod("workspace/applyEdit");
   PublishDiagnostics = Bind.outgoingNotification("textDocument/publishDiagnostics");
   ShowMessage = Bind.outgoingNotification("window/showMessage");
   NotifyFileStatus = Bind.outgoingNotification("textDocument/clangd.fileStatus");
   CreateWorkDoneProgress = Bind.outgoingMethod("window/workDoneProgress/create");
   BeginWorkDoneProgress = Bind.outgoingNotification("$/progress");
   ReportWorkDoneProgress = Bind.outgoingNotification("$/progress");
   EndWorkDoneProgress = Bind.outgoingNotification("$/progress");
+  if(Caps.SemanticTokenRefreshSupport)
+    SemanticTokensRefresh = Bind.outgoingMethod("workspace/semanticTokens/refresh");
   // clang-format on
 }
 
 ClangdLSPServer::~ClangdLSPServer() {
   IsBeingDestroyed = true;
   // Explicitly destroy ClangdServer first, blocking on threads it owns.
   // This ensures they don't access any other members.
   Server.reset();
void ClangdLSPServer::reparseOpenFilesIfNeeded(
   for (const Path &FilePath : DraftMgr.getActiveFiles())
     if (Filter(FilePath))
       if (auto Draft = DraftMgr.getDraft(FilePath)) // else disappeared in race?
         Server->addDocument(FilePath, std::move(Draft->Contents),
                             encodeVersion(Draft->Version),
                             WantDiagnostics::Auto);
 }
 
+void ClangdLSPServer::onClientInformationInvalidated(PathRef File) {
+  if (SemanticTokensRefresh) {
+    SemanticTokensRefresh(NoParams{}, [](llvm::Expected<std::nullptr_t> E) {
+      if (E)
+        return;
+      elog("Failed to refresh semantic tokens: {0}", E.takeError());
+    });
+  }
+}
+
 } // namespace clangd
 } // namespace clang

clang-tools-extra/clangd/ClangdServer.h

 #include "index/Background.h"
 #include "index/FileIndex.h"
 #include "index/Index.h"
 #include "refactor/Rename.h"
 #include "refactor/Tweak.h"
 #include "support/Cancellation.h"
 #include "support/Function.h"
 #include "support/MemoryTree.h"
+#include "support/Path.h"
 #include "support/ThreadsafeFS.h"
 #include "clang/Tooling/CompilationDatabase.h"
 #include "clang/Tooling/Core/Replacement.h"
 #include "llvm/ADT/FunctionExtras.h"
 #include "llvm/ADT/IntrusiveRefCntPtr.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/StringRef.h"
 #include <functional>
public:
     /// Called whenever the file status is updated.
     /// May be called concurrently for separate files, not for a single file.
     virtual void onFileUpdated(PathRef File, const TUStatus &Status) {}
 
     /// Called when background indexing tasks are enqueued/started/completed.
     /// Not called concurrently.
     virtual void
     onBackgroundIndexProgress(const BackgroundQueue::Stats &Stats) {}
+
+    /// Called when server triggers a state change without client requesting it.
+    /// For example a preamble build, which might happen on its own,
+    /// invalidating information like semantic tokens on client side. This is
+    /// used to notify clients that they might have stale information.
+    virtual void onClientInformationInvalidated(PathRef File) {}
   };
   /// Creates a context provider that loads and installs config.
   /// Errors in loading config are reported as diagnostics via Callbacks.
   /// (This is typically used as ClangdServer::Options::ContextProvider).
   static std::function<Context(PathRef)>
   createConfiguredContextProvider(const config::Provider *Provider,
                                   ClangdServer::Callbacks *);
 

clang-tools-extra/clangd/ClangdServer.cpp

UpdateIndexCallbacks(FileIndex *FIndex,
                        ClangdServer::Callbacks *ServerCallbacks)
       : FIndex(FIndex), ServerCallbacks(ServerCallbacks) {}
 
   void onPreambleAST(PathRef Path, llvm::StringRef Version, ASTContext &Ctx,
                      std::shared_ptr<clang::Preprocessor> PP,
                      const CanonicalIncludes &CanonIncludes) override {
     if (FIndex)
       FIndex->updatePreamble(Path, Version, Ctx, std::move(PP), CanonIncludes);
+    if (ServerCallbacks)
+      ServerCallbacks->onClientInformationInvalidated(Path);
   }
 
   void onMainAST(PathRef Path, ParsedAST &AST, PublishFn Publish) override {
     if (FIndex)
       FIndex->updateMain(Path, AST);
 
     std::vector<Diag> Diagnostics = AST.getDiagnostics();
     if (ServerCallbacks)

clang-tools-extra/clangd/Protocol.h

 enum class TraceLevel {
   Off = 0,
   Messages = 1,
   Verbose = 2,
 };
 bool fromJSON(const llvm::json::Value &E, TraceLevel &Out, llvm::json::Path);
 
 struct NoParams {};
+inline llvm::json::Value toJSON(const NoParams &) { return nullptr; }
 inline bool fromJSON(const llvm::json::Value &, NoParams &, llvm::json::Path) {
   return true;
 }
 using InitializedParams = NoParams;
 
 /// Defines how the host (editor) should sync document changes to the language
 /// server.
 enum class TextDocumentSyncKind {
struct ClientCapabilities {
   /// window.workDoneProgress
   bool WorkDoneProgress = false;
 
   /// The client supports implicit $/progress work-done progress streams,
   /// without a preceding window/workDoneProgress/create.
   /// This is a clangd extension.
   /// window.implicitWorkDoneProgressCreate
   bool ImplicitProgressCreation = false;
+
+  /// Whether the client implementation supports a refresh request sent from the
+  /// server to the client.
+  bool SemanticTokenRefreshSupport = false;
 };
 bool fromJSON(const llvm::json::Value &, ClientCapabilities &,
               llvm::json::Path);
 
 /// Clangd extension that's used in the 'compilationDatabaseChanges' in
 /// workspace/didChangeConfiguration to record updates to the in-memory
 /// compilation database.
 struct ClangdCompileCommand {

clang-tools-extra/clangd/Protocol.cpp

if (auto *Symbol = Workspace->getObject("symbol")) {
                         P.field("workspace")
                             .field("symbol")
                             .field("symbolKind")
                             .field("valueSet")))
             return false;
         }
       }
     }
+    if (auto *SemanticTokens = Workspace->getObject("semanticTokens")) {
+      if (auto RefreshSupport = SemanticTokens->getBoolean("refreshSupport"))
+        R.SemanticTokenRefreshSupport = *RefreshSupport;
+    }
   }
   if (auto *Window = O->getObject("window")) {
     if (auto WorkDoneProgress = Window->getBoolean("workDoneProgress"))
       R.WorkDoneProgress = *WorkDoneProgress;
     if (auto Implicit = Window->getBoolean("implicitWorkDoneProgressCreate"))
       R.ImplicitProgressCreation = *Implicit;
   }
   if (auto *OffsetEncoding = O->get("offsetEncoding")) {

clang-tools-extra/clangd/test/semantic-tokens-refresh.test

+# RUN: clangd -lit-test < %s | FileCheck -strict-whitespace %s
+{"jsonrpc":"2.0","id":0,"method":"initialize","params":{"capabilities":{
+  "workspace":{"semanticTokens":{"refreshSupport":true}}
+}}}
+---
+{"jsonrpc":"2.0","method":"textDocument/didOpen","params":{"textDocument":{
+  "uri": "test:///foo.cpp",
+  "languageId": "cpp",
+  "text": "int x = 2;"
+}}}
+# Expect a request after initial preamble build.
+# CHECK:        "method": "workspace/semanticTokens/refresh",
+# CHECK-NEXT:   "params": null
+# CHECK-NEXT:  }
+---
+# Reply with success.
+{"jsonrpc":"2.0","id":0}
+---
+# Preamble stays the same, no refresh requests.
+{"jsonrpc":"2.0","method":"textDocument/didChange","params":{
+  "textDocument": {"uri":"test:///foo.cpp","version":2},
+  "contentChanges":[{"text":"int x = 2;\nint y = 3;"}]
+}}
+# CHECK-NOT:  "method": "workspace/semanticTokens/refresh"
+---
+# Preamble changes
+{"jsonrpc":"2.0","method":"textDocument/didChange","params":{
+  "textDocument": {"uri":"test:///foo.cpp","version":2},
+  "contentChanges":[{"text":"#define FOO"}]
+}}
+# Expect a request after initial preamble build.
+# CHECK:        "method": "workspace/semanticTokens/refresh",
+# CHECK-NEXT:   "params": null
+# CHECK-NEXT:  }
+---
+# Reply with error, to make sure there are no crashes.
+{"jsonrpc":"2.0","id":1,"error":{"code": 0, "message": "msg"}}
+---
+{"jsonrpc":"2.0","id":3,"method":"shutdown"}
+---
+{"jsonrpc":"2.0","method":"exit"}
+