[clangd] Handle workspace/didChangeWatchedFiles

[malaperle]

Summary

The client can send notifications when it detects watched files have
changed. This patch adds the protocol handling for this type of notification.
For now, the notification will be passed down to the ClangdServer, but it will
not be acted upon. However, this will become useful for the indexer to react
to file changes.
The events could also potentially be used to invalidate other caches
(compilation database, etc).

This change also updates the VSCode extension so that it sends the events.

Signed-off-by: Marc-Andre Laperle <[e-mail]>

[ilya-biryukov]

clangd/ClangdServer.cpp

429

Code style: use FIXME instead of TODO.

clangd/clients/clangd-vscode/src/extension.ts

37

Won't this option make VSCode slow for very large or network-mounted projects?

I'm asking because we may want to have an option to disable it in that case.

38

Maybe extract a helper variable with array of supported extensions and use it both here and in documentSelector?

So that we always keep those lists in sync.

[malaperle]

Add VSCode config, address comments.

[malaperle]

clangd/clients/clangd-vscode/src/extension.ts

37

I haven't noticed any slowdowns in large projects (LLVM) but I haven't tested network-mounted. In fact, it seems VSCode stops sending the events when your workspace gets to a certain size, possibly when it reaches the limit of watched files (fs.inotify.max_user_watches on Linux). In any case, I added a config for disabling the file events.

[ilya-biryukov]

clangd/clients/clangd-vscode/src/extension.ts

37

Thanks.

I wonder if all LSP clients support watching for changes and whether it makes sense for clangd to implement file watching internally at some point.

[malaperle]

clangd/clients/clangd-vscode/src/extension.ts

37

I was thinking that we could eventually add an option (InitializeParams?) to tell Clangd to watch the files instead of the client. However, I think a lot of clients have to watch the files anyway so it makes sense to reuse that if it's available. I don't think it would be good for both to be watching files as it can consumes a lot of system resources.
You can find some interesting discussions about this subject here:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=516216
https://github.com/Microsoft/language-server-protocol/issues/237

I think adding event-based file watching is going to be a non-trivial as this is done differently on Linux, Windows and Mac. I think it would make sense to add that to llvm::sys ideally.

[ilya-biryukov]

clangd/clients/clangd-vscode/src/extension.ts

37

Sure, if clients have it, we should definitely reuse it.

I don't think current protocol methods cover clangd's needs, though.
One important thing missing feature is watching changes outside workspace root (#include may easily point to any file on the filesystem).

clangd/ClangdLSPServer.h

private:
   void onCodeAction(CodeActionParams Params, StringRef ID,
                     JSONOutput &Out) override;
   void onCompletion(TextDocumentPositionParams Params, StringRef ID,
                     JSONOutput &Out) override;
   void onGoToDefinition(TextDocumentPositionParams Params, StringRef ID,
                         JSONOutput &Out) override;
   void onSwitchSourceHeader(TextDocumentIdentifier Params, StringRef ID,
                             JSONOutput &Out) override;
+  void onFileEvent(const DidChangeWatchedFilesParams &Params) override;
 
   std::vector<clang::tooling::Replacement>
   getFixIts(StringRef File, const clangd::Diagnostic &D);
 
   /// Function that will be called on a separate thread when diagnostics are
   /// ready. Sends the Dianostics to LSP client via Out.writeMessage and caches
   /// corresponding fixits in the FixItsMap.
   void consumeDiagnostics(PathRef File,

clangd/ClangdLSPServer.cpp

 
 void ClangdLSPServer::onDocumentDidChange(DidChangeTextDocumentParams Params,
                                           JSONOutput &Out) {
   // We only support full syncing right now.
   Server.addDocument(Params.textDocument.uri.file,
                      Params.contentChanges[0].text);
 }
 
+void ClangdLSPServer::onFileEvent(const DidChangeWatchedFilesParams &Params) {
+  Server.onFileEvent(Params);
+}
+
 void ClangdLSPServer::onDocumentDidClose(DidCloseTextDocumentParams Params,
                                          JSONOutput &Out) {
   Server.removeDocument(Params.textDocument.uri.file);
 }
 
 void ClangdLSPServer::onDocumentOnTypeFormatting(
     DocumentOnTypeFormattingParams Params, StringRef ID, JSONOutput &Out) {
   auto File = Params.textDocument.uri.file;

clangd/ClangdServer.h

public:
   /// FIXME(ibiryukov): This function is here to allow offset-to-Position
   /// conversions in outside code, maybe there's a way to get rid of it.
   std::string getDocument(PathRef File);
 
   /// Only for testing purposes.
   /// Waits until all requests to worker thread are finished and dumps AST for
   /// \p File. \p File must be in the list of added documents.
   std::string dumpAST(PathRef File);
+  /// Called when an event occurs for a watched file in the workspace.
+  void onFileEvent(const DidChangeWatchedFilesParams &Params);
 
 private:
   std::future<void>
   scheduleReparseAndDiags(PathRef File, VersionedDraft Contents,
                           std::shared_ptr<CppFile> Resources,
                           Tagged<IntrusiveRefCntPtr<vfs::FileSystem>> TaggedFS);
 
   std::future<void> scheduleCancelRebuild(std::shared_ptr<CppFile> Resources);

clangd/ClangdServer.cpp

auto CancelReparses = [Resources](std::promise<void> DonePromise,
                                     std::future<void> DeferredCancel) {
     FulfillPromiseGuard Guard(DonePromise);
     DeferredCancel.get();
   };
   WorkScheduler.addToFront(std::move(CancelReparses), std::move(DonePromise),
                            std::move(DeferredCancel));
   return DoneFuture;
 }
+
+void ClangdServer::onFileEvent(const DidChangeWatchedFilesParams &Params) {
+  // FIXME: Do nothing for now. This will be used for indexing and potentially
+  // invalidating other caches.
+}

clangd/Protocol.h

struct DidChangeTextDocumentParams {
 
   /// The actual content changes.
   std::vector<TextDocumentContentChangeEvent> contentChanges;
 
   static llvm::Optional<DidChangeTextDocumentParams>
   parse(llvm::yaml::MappingNode *Params, clangd::Logger &Logger);
 };
 
+enum class FileChangeType {
+  /// The file got created.
+  Created = 1,
+  /// The file got changed.
+  Changed = 2,
+  /// The file got deleted.
+  Deleted = 3
+};
+
+struct FileEvent {
+  /// The file's URI.
+  URI uri;
+  /// The change type.
+  FileChangeType type;
+
+  static llvm::Optional<FileEvent> parse(llvm::yaml::MappingNode *Params,
+                                         clangd::Logger &Logger);
+};
+
+struct DidChangeWatchedFilesParams {
+  /// The actual file events.
+  std::vector<FileEvent> changes;
+
+  static llvm::Optional<DidChangeWatchedFilesParams>
+  parse(llvm::yaml::MappingNode *Params, clangd::Logger &Logger);
+};
+
 struct FormattingOptions {
   /// Size of a tab in spaces.
   int tabSize;
 
   /// Prefer spaces over tabs.
   bool insertSpaces;
 
   static llvm::Optional<FormattingOptions>

clangd/Protocol.cpp

if (KeyValue == "textDocument") {
       }
     } else {
       logIgnoredField(KeyValue, Logger);
     }
   }
   return Result;
 }
 
+llvm::Optional<FileEvent> FileEvent::parse(llvm::yaml::MappingNode *Params,
+                                           clangd::Logger &Logger) {
+  llvm::Optional<FileEvent> Result = FileEvent();
+  for (auto &NextKeyValue : *Params) {
+    // We have to consume the whole MappingNode because it doesn't support
+    // skipping and we want to be able to parse further valid events.
+    if (!Result)
+      continue;
+
+    auto *KeyString = dyn_cast<llvm::yaml::ScalarNode>(NextKeyValue.getKey());
+    if (!KeyString) {
+      Result.reset();
+      continue;
+    }
+
+    llvm::SmallString<10> KeyStorage;
+    StringRef KeyValue = KeyString->getValue(KeyStorage);
+    auto *Value =
+        dyn_cast_or_null<llvm::yaml::ScalarNode>(NextKeyValue.getValue());
+    if (!Value) {
+      Result.reset();
+      continue;
+    }
+    llvm::SmallString<10> Storage;
+    if (KeyValue == "uri") {
+      Result->uri = URI::parse(Value);
+    } else if (KeyValue == "type") {
+      long long Val;
+      if (llvm::getAsSignedInteger(Value->getValue(Storage), 0, Val)) {
+        Result.reset();
+        continue;
+      }
+      Result->type = static_cast<FileChangeType>(Val);
+      if (Result->type < FileChangeType::Created ||
+          Result->type > FileChangeType::Deleted)
+        Result.reset();
+    } else {
+      logIgnoredField(KeyValue, Logger);
+    }
+  }
+  return Result;
+}
+
+llvm::Optional<DidChangeWatchedFilesParams>
+DidChangeWatchedFilesParams::parse(llvm::yaml::MappingNode *Params,
+                                   clangd::Logger &Logger) {
+  DidChangeWatchedFilesParams Result;
+  for (auto &NextKeyValue : *Params) {
+    auto *KeyString = dyn_cast<llvm::yaml::ScalarNode>(NextKeyValue.getKey());
+    if (!KeyString)
+      return llvm::None;
+
+    llvm::SmallString<10> KeyStorage;
+    StringRef KeyValue = KeyString->getValue(KeyStorage);
+    auto *Value = NextKeyValue.getValue();
+
+    llvm::SmallString<10> Storage;
+    if (KeyValue == "changes") {
+      auto *Seq = dyn_cast<llvm::yaml::SequenceNode>(Value);
+      if (!Seq)
+        return llvm::None;
+      for (auto &Item : *Seq) {
+        auto *I = dyn_cast<llvm::yaml::MappingNode>(&Item);
+        if (!I)
+          return llvm::None;
+        auto Parsed = FileEvent::parse(I, Logger);
+        if (Parsed)
+          Result.changes.push_back(std::move(*Parsed));
+        else
+          Logger.log("Failed to decode a FileEvent.\n");
+      }
+    } else {
+      logIgnoredField(KeyValue, Logger);
+    }
+  }
+  return Result;
+}
+
 llvm::Optional<TextDocumentContentChangeEvent>
 TextDocumentContentChangeEvent::parse(llvm::yaml::MappingNode *Params,
                                       clangd::Logger &Logger) {
   TextDocumentContentChangeEvent Result;
   for (auto &NextKeyValue : *Params) {
     auto *KeyString = dyn_cast<llvm::yaml::ScalarNode>(NextKeyValue.getKey());
     if (!KeyString)
       return llvm::None;

clangd/ProtocolHandlers.h

virtual void onDocumentRangeFormatting(DocumentRangeFormattingParams Params,
                                          StringRef ID, JSONOutput &Out) = 0;
   virtual void onCodeAction(CodeActionParams Params, StringRef ID,
                             JSONOutput &Out) = 0;
   virtual void onCompletion(TextDocumentPositionParams Params, StringRef ID,
                             JSONOutput &Out) = 0;
   virtual void onGoToDefinition(TextDocumentPositionParams Params, StringRef ID,
                                 JSONOutput &Out) = 0;
   virtual void onSwitchSourceHeader(TextDocumentIdentifier Params, StringRef ID,
-                                    JSONOutput &Out) = 0;                              
+                                    JSONOutput &Out) = 0;
+  virtual void onFileEvent(const DidChangeWatchedFilesParams &Params) = 0;
 };
 
 void registerCallbackHandlers(JSONRPCDispatcher &Dispatcher, JSONOutput &Out,
                              ProtocolCallbacks &Callbacks);
 
 } // namespace clangd
 } // namespace clang
 
 #endif

clangd/ProtocolHandlers.cpp

void handleMethod(llvm::yaml::MappingNode *Params, StringRef ID) override {
 
     Callbacks.onSwitchSourceHeader(*TDPP, ID, Output);
   }
 
 private:
   ProtocolCallbacks &Callbacks;
 };
 
+struct WorkspaceDidChangeWatchedFilesHandler : Handler {
+  WorkspaceDidChangeWatchedFilesHandler(JSONOutput &Output,
+                                        ProtocolCallbacks &Callbacks)
+      : Handler(Output), Callbacks(Callbacks) {}
+
+  void handleNotification(llvm::yaml::MappingNode *Params) {
+    auto DCWFP = DidChangeWatchedFilesParams::parse(Params, Output);
+    if (!DCWFP) {
+      Output.log("Failed to decode DidChangeWatchedFilesParams.\n");
+      return;
+    }
+
+    Callbacks.onFileEvent(*DCWFP);
+  }
+
+private:
+  ProtocolCallbacks &Callbacks;
+};
+
 } // namespace
 
 void clangd::registerCallbackHandlers(JSONRPCDispatcher &Dispatcher,
                                      JSONOutput &Out,
                                      ProtocolCallbacks &Callbacks) {
   Dispatcher.registerHandler(
       "initialize", llvm::make_unique<InitializeHandler>(Out, Callbacks));
   Dispatcher.registerHandler(
void clangd::registerCallbackHandlers(JSONRPCDispatcher &Dispatcher,
   Dispatcher.registerHandler(
       "textDocument/completion",
       llvm::make_unique<CompletionHandler>(Out, Callbacks));
   Dispatcher.registerHandler(
       "textDocument/definition",
       llvm::make_unique<GotoDefinitionHandler>(Out, Callbacks));
   Dispatcher.registerHandler(
       "textDocument/switchSourceHeader",
-      llvm::make_unique<SwitchSourceHeaderHandler>(Out, Callbacks));    
+      llvm::make_unique<SwitchSourceHeaderHandler>(Out, Callbacks));
+  Dispatcher.registerHandler(
+      "workspace/didChangeWatchedFiles",
+      llvm::make_unique<WorkspaceDidChangeWatchedFilesHandler>(Out, Callbacks));
 }

clangd/clients/clangd-vscode/package.json

"contributes": {
                 },
                 "clangd.arguments": {
                     "type": "array",
                     "default": [],
                     "items": {
                         "type": "string"
                     },
                     "description": "Arguments for clangd server"
+                },
+                "clangd.syncFileEvents": {
+                    "type": "boolean",
+                    "default": true,
+                    "description": "Whether or not to send file events to clangd (File created, changed or deleted). This can be disabled for performance consideration."
                 }
             }
         }
     }
 }

clangd/clients/clangd-vscode/src/extension.ts

 
 /**
  *  this method is called when your extension is activate
  *  your extension is activated the very first time the command is executed
  */
 export function activate(context: vscode.ExtensionContext) {
     const clangdPath = getConfig<string>('path');
     const clangdArgs = getConfig<string[]>('arguments');
+    const syncFileEvents = getConfig<boolean>('syncFileEvents', true);
 
     const serverOptions: vscodelc.ServerOptions = { command: clangdPath, args: clangdArgs };
 
+    const cppFileExtensions: string[] = ['cpp', 'c', 'cc', 'cxx', 'c++', 'm', 'mm', 'h', 'hh', 'hpp', 'hxx', 'inc'];
+    const cppFileExtensionsPattern = cppFileExtensions.join();
     const clientOptions: vscodelc.LanguageClientOptions = {
         // Register the server for C/C++ files
-        documentSelector: ['c', 'cc', 'cpp', 'h', 'hh', 'hpp'],
+        documentSelector: cppFileExtensions,
         uriConverters: {
             // FIXME: by default the URI sent over the protocol will be percent encoded (see rfc3986#section-2.1)
             //        the "workaround" below disables temporarily the encoding until decoding
             //        is implemented properly in clangd
             code2Protocol: (uri: vscode.Uri) : string => uri.toString(true),
             protocol2Code: (uri: string) : vscode.Uri => vscode.Uri.parse(uri)
+        },
+        synchronize: !syncFileEvents ? undefined : {
+            fileEvents: vscode.workspace.createFileSystemWatcher('**/*.{' + cppFileExtensionsPattern + '}')
         }
     };
 
     const clangdClient = new vscodelc.LanguageClient('Clang Language Server', serverOptions, clientOptions);
 
     function applyTextEdits(uri: string, edits: vscodelc.TextEdit[]) {
         let textEditor = vscode.window.activeTextEditor;
 

test/clangd/did-change-watch-files.test

+# RUN: clangd -run-synchronously < %s 2>&1 | FileCheck -check-prefix=STDERR %s
+# It is absolutely vital that this file has CRLF line endings.
+#
+# Test initialize request parameters with rootUri
+Content-Length: 143
+
+{"jsonrpc":"2.0","id":0,"method":"initialize","params":{"processId":123,"rootUri":"file:///path/to/workspace","capabilities":{},"trace":"off"}}
+# CHECK: Content-Length: 466
+# CHECK: {"jsonrpc":"2.0","id":0,"result":{"capabilities":{
+# CHECK:   "textDocumentSync": 1,
+# CHECK:   "documentFormattingProvider": true,
+# CHECK:   "documentRangeFormattingProvider": true,
+# CHECK:   "documentOnTypeFormattingProvider": {"firstTriggerCharacter":"}","moreTriggerCharacter":[]},
+# CHECK:   "codeActionProvider": true,
+# CHECK:   "completionProvider": {"resolveProvider": false, "triggerCharacters": [".",">",":"]},
+# CHECK:   "definitionProvider": true
+# CHECK: }}}
+#
+#Normal case
+Content-Length: 217
+
+{"jsonrpc":"2.0","method":"workspace/didChangeWatchedFiles","params":{"changes":[{"uri":"file:///path/to/file.cpp","type":1},{"uri":"file:///path/to/file2.cpp","type":2},{"uri":"file:///path/to/file3.cpp","type":3}]}}
+
+# Wrong event type, integer
+Content-Length: 173
+
+{"jsonrpc":"2.0","method":"workspace/didChangeWatchedFiles","params":{"changes":[{"uri":"file:///path/to/file2.cpp","type":0},{"uri":"file:///path/to/file3.cpp","type":4}]}}
+# STDERR: Failed to decode a FileEvent.
+# Wrong event type, string
+Content-Length: 132
+
+{"jsonrpc":"2.0","method":"workspace/didChangeWatchedFiles","params":{"changes":[{"uri":"file:///path/to/file2.cpp","type":"foo"}]}}
+# STDERR: Failed to decode a FileEvent.
+#Custom event field
+Content-Length: 143
+
+{"jsonrpc":"2.0","method":"workspace/didChangeWatchedFiles","params":{"changes":[{"uri":"file:///path/to/file2.cpp","type":1,"custom":"foo"}]}}
+# STDERR: Failed to decode a FileEvent.
+#Event field with object
+Content-Length: 140
+
+{"jsonrpc":"2.0","method":"workspace/didChangeWatchedFiles","params":{"changes":[{"uri":"file:///path/to/file2.cpp","type":{"foo":"bar"}}]}}
+# STDERR: Failed to decode a FileEvent.
+# Changes field with sequence but no object
+Content-Length: 86
+
+{"jsonrpc":"2.0","method":"workspace/didChangeWatchedFiles","params":{"changes":[""]}}
+# STDERR: Failed to decode DidChangeWatchedFilesParams.
+# Changes field with no sequence
+Content-Length: 84
+
+{"jsonrpc":"2.0","method":"workspace/didChangeWatchedFiles","params":{"changes":""}}
+# STDERR: Failed to decode DidChangeWatchedFilesParams.
+# Custom field
+Content-Length: 86
+
+{"jsonrpc":"2.0","method":"workspace/didChangeWatchedFiles","params":{"custom":"foo"}}
+# STDERR: Ignored unknown field "custom"
+Content-Length: 44
+
+{"jsonrpc":"2.0","id":3,"method":"shutdown"}