[clangd] Expose FileStatus in LSP.

[hokein]

Summary

Add an LSP extension "textDocument/fileStatus" to emit file-status information.

[hokein]

Update.

[ilya-biryukov]

Sorry if I missed any important design discussions here, but wanted to clear up what information we are trying to convey to the user with the status messages?
E.g. why seeing "building preamble", "building file" or "queued" in the status bar can be useful to the user? Those messages mention internals of clangd, I'm not sure how someone unfamiliar with internals of clangd should interpret this information.

[simark]

In D55363#1329652, @ilya-biryukov wrote:

Sorry if I missed any important design discussions here, but wanted to clear up what information we are trying to convey to the user with the status messages?
E.g. why seeing "building preamble", "building file" or "queued" in the status bar can be useful to the user? Those messages mention internals of clangd, I'm not sure how someone unfamiliar with internals of clangd should interpret this information.

I agree about not necessarily needing to expose clang internals, but I think the end goal of providing some feedback to the user (still thinking/done) is worthy.

If this is a clangd-specific event, shouldn't the name of the event reflect it? As-is, it looks like an official LSP feature and can be confusing.

[ilya-biryukov]

Good point, we should definitely state it's a clangd-specific extension. We could come up with a naming scheme for our extensions. I would propose something like clangd:$methodName, i.e. the current extensions would be clangd:textDocument/fileStatus.
WDYT?

[hokein]

Update:

• avoid using terms that C++ programmers are unfamiliar with.
• textDocument/fileStatus => textDocument/clangd.fileStatus

[hokein]

Sorry if I missed any important design discussions here, but wanted to clear up what information we are trying to convey to the user with the status messages?
E.g. why seeing "building preamble", "building file" or "queued" in the status bar can be useful to the user? Those messages mention internals of clangd, I'm not sure how someone unfamiliar with internals of clangd should interpret this information.

Leaking some details to users is fine -- it can help users build the model of how clangd work, even though they don't understand all details, users will workout "building file" will take a while before "idle", and when the file is idle, clangd is responsive for upcoming requests.
I also agree that we should try to avoid using terms that C++ programmers are unfamiliar with.

Good point, we should definitely state it's a clangd-specific extension. We could come up with a naming scheme for our extensions. I would propose something like clangd:$methodName, i.e. the current extensions would be clangd:textDocument/fileStatus.

Adding clang-specific words to clangd-extension methods sounds good to me (to avoid confusion, the extension textDocument/switchFromHeader confused me before). I'd vote for textDocument/clangd.FileStatus.

[ilya-biryukov]

Could we add a capability to the init request to check for this extension? We don't want to send those notifications to clients that don't have the code to handle them.

Another observation: when I played around with the previous version of the patch, RunningAction and BuildingAST where constantly "blinking", i.e. it changes faster than I can read it.
This lead to them being irritating and not providing any actual value.

Maybe consider sending an update after some period of time, e.g. 0.5s? (I expect this particular strategy to be a bit complicated and out of scope of this patch, so another alternative is to simply not send them to the clients in the first version). WDYT?

clangd/ClangdLSPServer.cpp

806

It's somewhat easy to miss the clangd in the middle, would personally prefer the clangd.textDocument/fileStatus. But up to you.

clangd/Protocol.h

1002

Why not vector<string>? What's the use of the MessageType?

clangd/TUScheduler.cpp

741

Maybe start with a lower case?
Arguably clangd: queued looks more natural than clangd: Queued

Also consider maybe change this to file is queued to make it clear it's not clangd that's queuing up somewhere.

744

Maybe make it running ${action name}, e.g. running find references?

[hokein]

address review comments.

[hokein]

In D55363#1336315, @ilya-biryukov wrote:

Could we add a capability to the init request to check for this extension? We don't want to send those notifications to clients that don't have the code to handle them.

yes, I think we should add it to initializationOptions, since LSP clients don't provide interfaces to customize TextDocumentClientCapabilities.

Another observation: when I played around with the previous version of the patch, RunningAction and BuildingAST where constantly "blinking", i.e. it changes faster than I can read it.
This lead to them being irritating and not providing any actual value.

This depends on the file, for some huge files, BuildingAST RunningAction may take a while, but I agree that for most small files, these two are fast :).

Maybe consider sending an update after some period of time, e.g. 0.5s? (I expect this particular strategy to be a bit complicated and out of scope of this patch, so another alternative is to simply not send them to the clients in the first version). WDYT?

Sounds fair, keeping status bar blinking seems annony to users. Added a FIXME.

clangd/Protocol.h

1002

MessageType indicates the importance of the message. Editors might choose different UI for showing different message (e.g. in vscode, error message are prompted with a red icon; while Info are prompted with a yellow icon).

[ilya-biryukov]

clangd/Protocol.cpp

769

Ah, there's a non-zero chance of name clash here in case the protocol implements something similar.
Maybe name the JSON field "clangdFileStatus" to be completely sure this won't clash later.

clangd/Protocol.h

1002

As discussed offline, maybe leave out the 'details' field in the first version?
It does not seem to provide much value as of now. Could be totally useful if we showed a fallback compile command used or similar.

clangd/index/Merge.cpp

137

Accidental change?

unittests/clangd/IndexTests.cpp

222

Accidental change?

[hokein]

Remove details.

[hokein]

Rebase

[hokein]

clangd/index/Merge.cpp

137

oops, I rebased this branch on top of my another branch. Fixed.

[ilya-biryukov]

LGTM

[jkorous]

In D55363#1337376, @hokein wrote:

In D55363#1336315, @ilya-biryukov wrote:

Maybe consider sending an update after some period of time, e.g. 0.5s? (I expect this particular strategy to be a bit complicated and out of scope of this patch, so another alternative is to simply not send them to the clients in the first version). WDYT?

Sounds fair, keeping status bar blinking seems annony to users. Added a FIXME.

I am a bit late to the discussion but isn't this something to be dealt with on the client's side?

[ilya-biryukov]

In D55363#1343611, @jkorous wrote:

I am a bit late to the discussion but isn't this something to be dealt with on the client's side?

That would mean more complex code in every client and more communication between the server and the client, which is not particularly useful.
Doing it on the server means keeping the complexity in one place, this looks strictly better, unless there are actual use-cases for high-frequency updates.

clangd/ClangdLSPServer.h

public:
   /// constructor. This method must not be executed more than once.
   ///
   /// \return Whether we shut down cleanly with a 'shutdown' -> 'exit' sequence.
   bool run();
 
 private:
   // Implement DiagnosticsConsumer.
   void onDiagnosticsReady(PathRef File, std::vector<Diag> Diagnostics) override;
+  void onFileUpdated(PathRef File, const TUStatus &Status) override;
 
   // LSP methods. Notifications have signature void(const Params&).
   // Calls have signature void(const Params&, Callback<Response>).
   void onInitialize(const InitializeParams &, Callback<llvm::json::Value>);
   void onShutdown(const ShutdownParams &, Callback<std::nullptr_t>);
   void onSync(const NoParams &, Callback<std::nullptr_t>);
   void onDocumentDidOpen(const DidOpenTextDocumentParams &);
   void onDocumentDidChange(const DidChangeTextDocumentParams &);
private:
   /// Options used for code completion
   clangd::CodeCompleteOptions CCOpts;
   /// Options used for diagnostics.
   ClangdDiagnosticOptions DiagOpts;
   /// The supported kinds of the client.
   SymbolKindBitset SupportedSymbolKinds;
   /// The supported completion item kinds of the client.
   CompletionItemKindBitset SupportedCompletionItemKinds;
-  // Whether the client supports CodeAction response objects.
+  /// Whether the client supports CodeAction response objects.
   bool SupportsCodeAction = false;
   /// From capabilities of textDocument/documentSymbol.
   bool SupportsHierarchicalDocumentSymbol = false;
-
+  /// Whether the client supports showing file status.
+  bool SupportFileStatus = false;
   // Store of the current versions of the open documents.
   DraftStore DraftMgr;
 
   // The CDB is created by the "initialize" LSP method.
   bool UseDirBasedCDB;                     // FIXME: make this a capability.
   llvm::Optional<Path> CompileCommandsDir; // FIXME: merge with capability?
   std::unique_ptr<GlobalCompilationDatabase> BaseCDB;
   // CDB is BaseCDB plus any comands overridden via LSP extensions.

clangd/ClangdLSPServer.cpp

void ClangdLSPServer::onInitialize(const InitializeParams &Params,
   DiagOpts.SendDiagnosticCategory = Params.capabilities.DiagnosticCategory;
   if (Params.capabilities.WorkspaceSymbolKinds)
     SupportedSymbolKinds |= *Params.capabilities.WorkspaceSymbolKinds;
   if (Params.capabilities.CompletionItemKinds)
     SupportedCompletionItemKinds |= *Params.capabilities.CompletionItemKinds;
   SupportsCodeAction = Params.capabilities.CodeActionStructure;
   SupportsHierarchicalDocumentSymbol =
       Params.capabilities.HierarchicalDocumentSymbol;
-
+  SupportFileStatus = Params.initializationOptions.FileStatus;
   Reply(json::Object{
       {{"capabilities",
         json::Object{
             {"textDocumentSync", (int)TextDocumentSyncKind::Incremental},
             {"documentFormattingProvider", true},
             {"documentRangeFormattingProvider", true},
             {"documentOnTypeFormattingProvider",
              json::Object{
void ClangdLSPServer::onDiagnosticsReady(PathRef File,
   // Publish diagnostics.
   notify("textDocument/publishDiagnostics",
          json::Object{
              {"uri", URI},
              {"diagnostics", std::move(LSPDiagnostics)},
          });
 }
 
+void ClangdLSPServer::onFileUpdated(PathRef File, const TUStatus &Status) {
+  if (!SupportFileStatus)
+    return;
+  // FIXME: we don't emit "BuildingFile" and `RunningAction`, as these
+  // two statuses are running faster in practice, which leads the UI constantly
+  // changing, and doesn't provide much value. We may want to emit status at a
+  // reasonable time interval (e.g. 0.5s).
+  if (Status.Action.S == TUAction::BuildingFile ||
+      Status.Action.S == TUAction::RunningAction)
+    return;
+  notify("textDocument/clangd.fileStatus", Status.render(File));
+}
+
 void ClangdLSPServer::reparseOpenedFiles() {
   for (const Path &FilePath : DraftMgr.getActiveFiles())
     Server->addDocument(FilePath, *DraftMgr.getDraft(FilePath),
                         WantDiagnostics::Auto);
 }
 
 } // namespace clangd
 } // namespace clang

clangd/Protocol.h

struct InitializationOptions {
   // also set through the initialize request (initializationOptions field).
   ConfigurationSettings ConfigSettings;
 
   llvm::Optional<std::string> compilationDatabasePath;
   // Additional flags to be included in the "fallback command" used when
   // the compilation database doesn't describe an opened file.
   // The command used will be approximately `clang $FILE $fallbackFlags`.
   std::vector<std::string> fallbackFlags;
+
+  /// Clients supports show file status for textDocument/clangd.fileStatus.
+  bool FileStatus = false;
 };
 bool fromJSON(const llvm::json::Value &, InitializationOptions &);
 
 struct InitializeParams {
   /// The process Id of the parent process that started
   /// the server. Is null if the process has not been started by another
   /// process. If the parent process is not alive then the server should exit
   /// (see exit notification) its process.

 llvm::json::Value toJSON(const DocumentHighlight &DH);
 llvm::raw_ostream &operator<<(llvm::raw_ostream &, const DocumentHighlight &);
 
 struct ReferenceParams : public TextDocumentPositionParams {
   // For now, no options like context.includeDeclaration are supported.
 };
 bool fromJSON(const llvm::json::Value &, ReferenceParams &);
 
+enum class MessageType {
+  Error = 1,
+  Warning = 2,
+  Info = 3,
+  Log = 4,
+};
+struct ShowMessageParams {
+  /// The message type.
+  MessageType type;
+  /// The actual message.
+  std::string message;
+};
+llvm::json::Value toJSON(const ShowMessageParams &);
+
+/// Clangd extension: indicates the current state of the file in clangd,
+/// sent from server via the `textDocument/fileStatus` notification.
+struct FileStatus {
+  /// The text document's URI.
+  URIForFile uri;
+  /// The human-readable string presents the current state of the file, can be
+  /// shown in the UI (e.g. status bar).
+  std::string state;
+  /// Details of the state that are worth sufacing to users.
+  std::vector<ShowMessageParams> details;
+};
+llvm::json::Value toJSON(const FileStatus &FStatus);
+
 } // namespace clangd
 } // namespace clang
 
 namespace llvm {
 template <> struct format_provider<clang::clangd::Position> {
   static void format(const clang::clangd::Position &Pos, raw_ostream &OS,
                      StringRef Style) {
     assert(Style.empty() && "style modifiers for this type are not supported");
     OS << Pos;
   }
 };
 } // namespace llvm
 
 #endif

clangd/Protocol.cpp

 
 json::Value toJSON(const DocumentHighlight &DH) {
   return json::Object{
       {"range", toJSON(DH.range)},
       {"kind", static_cast<int>(DH.kind)},
   };
 }
 
+llvm::json::Value toJSON(const ShowMessageParams &S) {
+  return json::Object{
+      {"type", static_cast<int>(S.type)}, {"message", S.message},
+  };
+}
+
+llvm::json::Value toJSON(const FileStatus &FStatus) {
+  return json::Object{
+      {"uri", FStatus.uri},
+      {"state", FStatus.state},
+      {"details", FStatus.details},
+  };
+}
+
 raw_ostream &operator<<(raw_ostream &O, const DocumentHighlight &V) {
   O << V.range;
   if (V.kind == DocumentHighlightKind::Read)
     O << "(r)";
   if (V.kind == DocumentHighlightKind::Write)
     O << "(w)";
   return O;
 }

 bool fromJSON(const json::Value &Params, InitializationOptions &Opts) {
   json::ObjectMapper O(Params);
   if (!O)
     return true; // 'any' type in LSP.
 
   fromJSON(Params, Opts.ConfigSettings);
   O.map("compilationDatabasePath", Opts.compilationDatabasePath);
   O.map("fallbackFlags", Opts.fallbackFlags);
+  O.map("fileStatus", Opts.FileStatus);
   return true;
 }
 
 bool fromJSON(const json::Value &Params, ReferenceParams &R) {
   TextDocumentPositionParams &Base = R;
   return fromJSON(Params, Base);
 }
 
 } // namespace clangd
 } // namespace clang

clangd/TUScheduler.h

 // Internal status of the TU in TUScheduler.
 struct TUStatus {
   struct BuildDetails {
     /// Indicates whether clang failed to build the TU.
     bool BuildFailed = false;
     /// Indicates whether we reused the prebuilt AST.
     bool ReuseAST = false;
   };
+  /// Serialize this to an LSP file status item.
+  FileStatus render(PathRef File) const;
 
   TUAction Action;
   BuildDetails Details;
 };
 
 class ParsingCallbacks {
 public:
   virtual ~ParsingCallbacks() = default;

clangd/TUScheduler.cpp

bool ASTWorker::shouldSkipHeadLocked() const {
   llvm_unreachable("Unknown WantDiagnostics");
 }
 
 bool ASTWorker::blockUntilIdle(Deadline Timeout) const {
   std::unique_lock<std::mutex> Lock(Mutex);
   return wait(Lock, RequestsCV, Timeout, [&] { return Requests.empty(); });
 }
 
+// Render a TUAction to a user-facing string representation.
+// TUAction represents clangd-internal states, we don't intend to expose them
+// to users (say C++ programmers) directly to avoid confusion, we use terms that
+// are familiar by C++ programmers.
+std::string renderTUAction(const TUAction& Action) {
+  std::string Result;
+  raw_string_ostream OS(Result);
+  switch (Action.S) {
+  case TUAction::Queued:
+    OS << "file is queued";
+    break;
+  case TUAction::RunningAction:
+    OS << "running " << Action.Name;
+    break;
+  case TUAction::BuildingPreamble:
+    OS << "parsing includes";
+    break;
+  case TUAction::BuildingFile:
+    OS << "parsing main file";
+    break;
+  case TUAction::Idle:
+    OS << "idle";
+    break;
+  }
+  return OS.str();
+}
+
 } // namespace
 
 unsigned getDefaultAsyncThreadsCount() {
   unsigned HardwareConcurrency = std::thread::hardware_concurrency();
   // C++ standard says that hardware_concurrency()
   // may return 0, fallback to 1 worker thread in
   // that case.
   if (HardwareConcurrency == 0)
     return 1;
   return HardwareConcurrency;
 }
 
+FileStatus TUStatus::render(PathRef File) const {
+  FileStatus FStatus;
+  FStatus.uri = URIForFile::canonicalize(File, /*TUPath=*/File);
+  FStatus.state = renderTUAction(Action);
+  if (Details.BuildFailed)
+    FStatus.details.push_back({MessageType::Error, "failed to build the file"});
+  if (Details.ReuseAST)
+    FStatus.details.push_back({MessageType::Info, "reused the AST"});
+  return FStatus;
+}
+
 struct TUScheduler::FileData {
   /// Latest inputs, passed to TUScheduler::update().
   std::string Contents;
   tooling::CompileCommand Command;
   ASTWorkerHandle Worker;
 };
 
 TUScheduler::TUScheduler(unsigned AsyncThreadsCount,

clangd/index/Merge.cpp

Symbol mergeSymbol(const Symbol &L, const Symbol &R) {
   if (S.Signature == "")
     S.Signature = O.Signature;
   if (S.CompletionSnippetSuffix == "")
     S.CompletionSnippetSuffix = O.CompletionSnippetSuffix;
   if (S.Documentation == "")
     S.Documentation = O.Documentation;
   if (S.ReturnType == "")
     S.ReturnType = O.ReturnType;
+  if (S.Type == "")
+    S.Type = O.Type;
   for (const auto &OI : O.IncludeHeaders) {
     bool Found = false;
     for (auto &SI : S.IncludeHeaders) {
       if (SI.IncludeHeader == OI.IncludeHeader) {
         Found = true;
         SI.References += OI.References;
         break;
       }

test/clangd/filestatus.test

+# RUN: clangd -lit-test < %s | FileCheck -strict-whitespace %s
+{"jsonrpc":"2.0","id":0,"method":"initialize","params":{"processId":123,"rootPath":"clangd","capabilities":{},"initializationOptions":{"fileStatus": true},"trace":"off"}}
+---
+{"jsonrpc":"2.0","method":"textDocument/didOpen","params":{"textDocument":{"uri":"test:///main.cpp","languageId":"cpp","version":1,"text":"int x; int y = x;"}}}
+#      CHECK:  "method": "textDocument/clangd.fileStatus",
+# CHECK-NEXT:  "params": {
+# CHECK-NEXT:    "details": [],
+# CHECK-NEXT:    "state": "parsing includes",
+# CHECK-NEXT:    "uri": "{{.*}}/main.cpp"
+# CHECK-NEXT:  }
+---
+{"jsonrpc":"2.0","id":3,"method":"shutdown"}
+---
+{"jsonrpc":"2.0","method":"exit"}

unittests/clangd/IndexTests.cpp

TEST(MergeTest, Merge) {
   R.CanonicalDeclaration.FileURI = "file:///right.h";
   L.References = 1;
   R.References = 2;
   L.Signature = "()";                   // present in left only
   R.CompletionSnippetSuffix = "{$1:0}"; // present in right only
   R.Documentation = "--doc--";
   L.Origin = SymbolOrigin::Dynamic;
   R.Origin = SymbolOrigin::Static;
+  R.Type = "expectedType";
 
   Symbol M = mergeSymbol(L, R);
   EXPECT_EQ(M.Name, "Foo");
   EXPECT_EQ(StringRef(M.CanonicalDeclaration.FileURI), "file:///left.h");
   EXPECT_EQ(M.References, 3u);
   EXPECT_EQ(M.Signature, "()");
   EXPECT_EQ(M.CompletionSnippetSuffix, "{$1:0}");
   EXPECT_EQ(M.Documentation, "--doc--");
+  EXPECT_EQ(M.Type, "expectedType");
   EXPECT_EQ(M.Origin,
             SymbolOrigin::Dynamic | SymbolOrigin::Static | SymbolOrigin::Merge);
 }
 
 TEST(MergeTest, PreferSymbolWithDefn) {
   Symbol L, R;
 
   L.ID = R.ID = SymbolID("hello");