[clangd] Add $/dumpMemoryTree LSP extension

[kadircet]

Summary

Performs a detailed profiling of clangd lsp server and conveys the
result to the client via showMessage.

[nridge]

This is neat! Are there plans to add vscode client support to invoke this?

[kadircet]

In D89277#2326376, @nridge wrote:

This is neat! Are there plans to add vscode client support to invoke this?

Yes, since we control the plugin in vscode we should have a way to invoke it there. (we already have support for switchHeaderSource).

We should first agree on the interaction though. This currently uses showMessage to display the dump, which is the most generic way LSP provides us with, but usually isn't good for multi-line output.
Other options we have are:

• Write it into logs - Hard to find, but at least will look reasonable on every editor.
• Insert as a commented block into the code. (like dumpast) - Clutters the code, but again looks reasonable on every editor and easier to find than logs.
• Provide a json object in clangd and use fancy client side features like tree views. - Probably the best output, but will only work on a handful of editors and will require a lot of effort to implement.

I am leaning towards using showMessage and also logging the tree. Do others have opinions on other possibilities or pros/cons of the options i've listed above?

[sammccall]

(sorry out today and haven't looked at code yet)

If it's a custom method, I think it should return the data as a json structure - the client already has to have custom support to invoke it, displaying the result isn't much extra work.

And I would really love to add a tree view to vscode, I think it wouldn't be hard (vs call hierarchy: no laziness and no direction-flipping) and could be reused for an AST viewer.

[kadircet]

In D89277#2329947, @sammccall wrote:

(sorry out today and haven't looked at code yet)

no worries it is a prototype, I wouldn't spend time looking at the implementation until we agree on the interaction :D
OTHO, checking out the lit test for output would probably be useful.

If it's a custom method, I think it should return the data as a json structure - the client already has to have custom support to invoke it, displaying the result isn't much extra work.

SGTM. WDYT about a json object in the form of:

interface MemoryTreeNode {
  name: string;
  totalSize: int;
  children?: Node[];
};

And I would really love to add a tree view to vscode, I think it wouldn't be hard (vs call hierarchy: no laziness and no direction-flipping) and could be reused for an AST viewer.

right, vscode already has APIs for it, https://code.visualstudio.com/api/extension-guides/tree-view.

[kadircet]

• As discussed offline moving with compact version of the output, while preserving names starting with an _ to be leaves.

This is also ready for an implementation-wise review now.

[sammccall]

This looks really nice!

clang-tools-extra/clangd/ClangdLSPServer.cpp

615

can you add {"memoryTreeProvider", true} // clangd extension?

I'd like to add client support in VSCode and it's useful if the command is only visible with versions of clangd that support it.

1390

Putting the serialization code inline is a tempting option here because it's (mostly) just a map.
It's not something we do often though, so it does bury the protocol details in an unexpected place.

A couple of alternatives to consider:

• map to a struct UsageTree { unsigned Self; unsigned Total; Map<string, UsageTree> Children; } defined in protocol.h, and define marshalling in the usual way
• skip the struct and use MemoryTree as the type, providing a toJSON(const MemoryTree&) function in protocol.h

I lean towards the last alternative because it provides a bit more regularity without adding much more redundancy, but this is up to you.

1392

nit: Alloc -> DetailAlloc

1440

I think "dump" is a bit casual and also redundant - LSP methods to obtain information tend to have noun names. (executeCommand is the exception that proves the rule - it's used for side effects).

I went through the same question with the AST-dumping and landed on textDocument/ast as my preferred option, for what it's worth.

clang-tools-extra/clangd/ClangdLSPServer.h

145

(if you decide to move toJSON to protocol.h, then the second sentence onward probably belongs there)

[kadircet]

• Add memoryTreeProvider to initialization params
• Move serialization logic to Protocol.h and return MemoryTree directly.
• Drop dump from method name.

[sammccall]

clang-tools-extra/clangd/ClangdLSPServer.cpp

623

one last naming nit: I think memoryUsage would put appropriately more weight on the semantics rather than the data structure

[kadircet]

s/memoryTree/memoryUsage/g

clang-tools-extra/clangd/ClangdLSPServer.h

 #include "support/Context.h"
 #include "support/MemoryTree.h"
 #include "support/Path.h"
 #include "clang/Tooling/Core/Replacement.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/StringSet.h"
 #include "llvm/Support/JSON.h"
 #include <chrono>
+#include <cstddef>
 #include <memory>
 
 namespace clang {
 namespace clangd {
 
 class SymbolIndex;
 
 /// This class exposes ClangdServer's capabilities via Language Server Protocol.
void onSymbolInfo(const TextDocumentPositionParams &,
                     Callback<std::vector<SymbolDetails>>);
   void onSelectionRange(const SelectionRangeParams &,
                         Callback<std::vector<SelectionRange>>);
   void onDocumentLink(const DocumentLinkParams &,
                       Callback<std::vector<DocumentLink>>);
   void onSemanticTokens(const SemanticTokensParams &, Callback<SemanticTokens>);
   void onSemanticTokensDelta(const SemanticTokensDeltaParams &,
                              Callback<SemanticTokensOrDelta>);
+  /// This is a clangd extension. Provides a json tree representing memory usage
+  /// hierarchy.
+  void onMemoryUsage(const NoParams &, Callback<MemoryTree>);
 
   std::vector<Fix> getFixes(StringRef File, const clangd::Diagnostic &D);
 
   /// Checks if completion request should be ignored. We need this due to the
   /// limitation of the LSP. Per LSP, a client sends requests for all "trigger
   /// character" we specify, but for '>' and ':' we need to check they actually
   /// produce '->' and '::', respectively.
   bool shouldRunCompletion(const CompletionParams &Params) const;

clang-tools-extra/clangd/ClangdLSPServer.cpp

 #include "llvm/Support/Allocator.h"
 #include "llvm/Support/Errc.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/FormatVariadic.h"
 #include "llvm/Support/JSON.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/SHA1.h"
 #include "llvm/Support/ScopedPrinter.h"
+#include "llvm/Support/raw_ostream.h"
 #include <chrono>
 #include <cstddef>
+#include <cstdint>
+#include <functional>
 #include <memory>
 #include <mutex>
 #include <string>
 #include <vector>
 
 namespace clang {
 namespace clangd {
 namespace {
llvm::json::Object Result{
              llvm::json::Object{
                  {"resolveProvider", false},
              }},
             {"hoverProvider", true},
             {"renameProvider", std::move(RenameProvider)},
             {"selectionRangeProvider", true},
             {"documentSymbolProvider", true},
             {"workspaceSymbolProvider", true},
             {"referencesProvider", true},
             {"executeCommandProvider",
              llvm::json::Object{
                  {"commands",
                   {ExecuteCommandParams::CLANGD_APPLY_FIX_COMMAND,
                    ExecuteCommandParams::CLANGD_APPLY_TWEAK}},
              }},
             {"typeHierarchyProvider", true},
+            {"memoryUsageProvider", true}, // clangd extension.
         }}}};
   if (Opts.Encoding)
     Result["offsetEncoding"] = *Opts.Encoding;
   if (Opts.TheiaSemanticHighlighting)
     Result.getObject("capabilities")
         ->insert(
             {"semanticHighlighting",
              llvm::json::Object{{"scopes", buildHighlightScopeLookupTable()}}});
Server->semanticHighlights(
           increment(Last.resultId);
           Result.resultId = Last.resultId;
         }
 
         CB(std::move(Result));
       });
 }
 
+void ClangdLSPServer::onMemoryUsage(const NoParams &,
+                                    Callback<MemoryTree> Reply) {
+  llvm::BumpPtrAllocator DetailAlloc;
+  MemoryTree MT(&DetailAlloc);
+  profile(MT);
+  Reply(std::move(MT));
+}
+
 ClangdLSPServer::ClangdLSPServer(class Transport &Transp,
                                  const ThreadsafeFS &TFS,
                                  const ClangdLSPServer::Options &Opts)
     : BackgroundContext(Context::current().clone()), Transp(Transp),
       MsgHandler(new MessageHandler(*this)), TFS(TFS),
       SupportedSymbolKinds(defaultSymbolKinds()),
       SupportedCompletionItemKinds(defaultCompletionItemKinds()), Opts(Opts) {
   // clang-format off
ClangdLSPServer::ClangdLSPServer(class Transport &Transp,
   MsgHandler->bind("workspace/didChangeConfiguration", &ClangdLSPServer::onChangeConfiguration);
   MsgHandler->bind("textDocument/symbolInfo", &ClangdLSPServer::onSymbolInfo);
   MsgHandler->bind("textDocument/typeHierarchy", &ClangdLSPServer::onTypeHierarchy);
   MsgHandler->bind("typeHierarchy/resolve", &ClangdLSPServer::onResolveTypeHierarchy);
   MsgHandler->bind("textDocument/selectionRange", &ClangdLSPServer::onSelectionRange);
   MsgHandler->bind("textDocument/documentLink", &ClangdLSPServer::onDocumentLink);
   MsgHandler->bind("textDocument/semanticTokens/full", &ClangdLSPServer::onSemanticTokens);
   MsgHandler->bind("textDocument/semanticTokens/full/delta", &ClangdLSPServer::onSemanticTokensDelta);
+  MsgHandler->bind("$/memoryUsage", &ClangdLSPServer::onMemoryUsage);
   if (Opts.FoldingRanges)
     MsgHandler->bind("textDocument/foldingRange", &ClangdLSPServer::onFoldingRange);
   // clang-format on
 
   // Delay first profile until we've finished warming up.
   NextProfileTime = std::chrono::steady_clock::now() + std::chrono::minutes(1);
 }
 

clang-tools-extra/clangd/Protocol.h

 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_PROTOCOL_H
 #define LLVM_CLANG_TOOLS_EXTRA_CLANGD_PROTOCOL_H
 
 #include "URI.h"
 #include "index/SymbolID.h"
+#include "support/MemoryTree.h"
 #include "clang/Index/IndexSymbol.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/Support/JSON.h"
 #include "llvm/Support/raw_ostream.h"
 #include <bitset>
 #include <memory>
 #include <string>
 #include <vector>
struct FoldingRange {
   unsigned startLine = 0;
   unsigned startCharacter;
   unsigned endLine = 0;
   unsigned endCharacter;
   llvm::Optional<std::string> kind;
 };
 llvm::json::Value toJSON(const FoldingRange &Range);
 
+/// Keys starting with an underscore(_) represent leaves, e.g. _total or _self
+/// for memory usage of whole subtree or only that specific node in bytes. All
+/// other keys represents children. An example:
+///   {
+///     "_self": 0,
+///     "_total": 8,
+///     "child1": {
+///       "_self": 4,
+///       "_total": 4,
+///     }
+///     "child2": {
+///       "_self": 2,
+///       "_total": 4,
+///       "child_deep": {
+///         "_self": 2,
+///         "_total": 2,
+///       }
+///     }
+///   }
+llvm::json::Value toJSON(const MemoryTree &MT);
+
 } // namespace clangd
 } // namespace clang
 
 namespace llvm {
 template <> struct format_provider<clang::clangd::Position> {
   static void format(const clang::clangd::Position &Pos, raw_ostream &OS,
                      StringRef Style) {
     assert(Style.empty() && "style modifiers for this type are not supported");
     OS << Pos;
   }
 };
 } // namespace llvm
 
 #endif

clang-tools-extra/clangd/Protocol.cpp

if (Range.startCharacter)
     Result["startCharacter"] = Range.startCharacter;
   if (Range.endCharacter)
     Result["endCharacter"] = Range.endCharacter;
   if (Range.kind)
     Result["kind"] = *Range.kind;
   return Result;
 }
 
+llvm::json::Value toJSON(const MemoryTree &MT) {
+  llvm::json::Object Out;
+  int64_t Total = MT.self();
+  Out["_self"] = Total;
+  for (const auto &Entry : MT.children()) {
+    auto Child = toJSON(Entry.getSecond());
+    Total += *Child.getAsObject()->getInteger("_total");
+    Out[Entry.first] = std::move(Child);
+  }
+  Out["_total"] = Total;
+  return Out;
+}
 } // namespace clangd
 } // namespace clang

clang-tools-extra/clangd/test/initialize-params.test

 # CHECK-NEXT:      "documentSymbolProvider": true,
 # CHECK-NEXT:      "executeCommandProvider": {
 # CHECK-NEXT:        "commands": [
 # CHECK-NEXT:          "clangd.applyFix",
 # CHECK-NEXT:          "clangd.applyTweak"
 # CHECK-NEXT:        ]
 # CHECK-NEXT:      },
 # CHECK-NEXT:      "hoverProvider": true,
+# CHECK-NEXT:      "memoryUsageProvider": true
 # CHECK-NEXT:      "referencesProvider": true,
 # CHECK-NEXT:      "renameProvider": true,
 # CHECK-NEXT:      "selectionRangeProvider": true,
 # CHECK-NEXT:      "semanticTokensProvider": {
 # CHECK-NEXT:        "full": {
 # CHECK-NEXT:          "delta": true
 # CHECK-NEXT:        },
 # CHECK-NEXT:        "legend": {

clang-tools-extra/clangd/test/memory_tree.test

+# RUN: clangd -lit-test < %s | FileCheck -strict-whitespace %s
+{"jsonrpc":"2.0","id":0,"method":"initialize","params":{"processId":123,"rootPath":"clangd","capabilities":{},"trace":"off"}}
+---
+{"jsonrpc":"2.0","method":"textDocument/didOpen","params":{"textDocument":{"uri":"test:///main.cpp","languageId":"cpp","version":1,"text":"void func() {}"}}}
+---
+{"jsonrpc":"2.0","id":1,"method":"$/memoryUsage","params":{}}
+# CHECK:        "id": 1,
+# CHECK-NEXT:   "jsonrpc": "2.0",
+# CHECK-NEXT:   "result": {
+# CHECK-NEXT:     "_self": {{[0-9]+}},
+# CHECK-NEXT:     "_total": {{[0-9]+}},
+# CHECK-NEXT:     "clangd_server": {
+# CHECK-NEXT:       "_self": {{[0-9]+}},
+# CHECK-NEXT:       "_total": {{[0-9]+}},
+# CHECK-NEXT:       "dynamic_index": {
+# CHECK-NEXT:         "_self": {{[0-9]+}},
+# CHECK-NEXT:         "_total": {{[0-9]+}},
+# CHECK-NEXT:         "main_file": {
+# CHECK-NEXT:           "_self": {{[0-9]+}},
+# CHECK-NEXT:           "_total": {{[0-9]+}},
+# CHECK-NEXT:           "index": {
+# CHECK-NEXT:             "_self": {{[0-9]+}},
+# CHECK-NEXT:             "_total": {{[0-9]+}}
+# CHECK-NEXT:           },
+# CHECK-NEXT:           "symbols": {
+# CHECK-NEXT:             "{{.*}}main.cpp": {
+# CHECK-NEXT:               "_self": {{[0-9]+}},
+# CHECK-NEXT:               "_total": {{[0-9]+}},
+# CHECK-NEXT:               "references": {
+# CHECK-NEXT:                 "_self": {{[0-9]+}},
+# CHECK-NEXT:                 "_total": {{[0-9]+}}
+# CHECK-NEXT:               },
+# CHECK-NEXT:               "relations": {
+# CHECK-NEXT:                 "_self": {{[0-9]+}},
+# CHECK-NEXT:                 "_total": {{[0-9]+}}
+# CHECK-NEXT:               },
+# CHECK-NEXT:               "symbols": {
+# CHECK-NEXT:                 "_self": {{[0-9]+}},
+# CHECK-NEXT:                 "_total": {{[0-9]+}}
+# CHECK-NEXT:               }
+# CHECK-NEXT:             },
+# CHECK-NEXT:             "_self": {{[0-9]+}},
+# CHECK-NEXT:             "_total": {{[0-9]+}}
+# CHECK-NEXT:           }
+# CHECK-NEXT:         },
+# CHECK-NEXT:         "preamble": {
+# CHECK-NEXT:           "_self": {{[0-9]+}},
+# CHECK-NEXT:           "_total": {{[0-9]+}},
+# CHECK-NEXT:           "index": {
+# CHECK-NEXT:             "_self": {{[0-9]+}},
+# CHECK-NEXT:             "_total": {{[0-9]+}}
+# CHECK-NEXT:           },
+# CHECK-NEXT:           "symbols": {
+# CHECK-NEXT:             "_self": {{[0-9]+}},
+# CHECK-NEXT:             "_total": {{[0-9]+}}
+# CHECK-NEXT:           }
+# CHECK-NEXT:         }
+# CHECK-NEXT:       },
+# CHECK-NEXT:       "tuscheduler": {
+# CHECK-NEXT:         "{{.*}}main.cpp": {
+# CHECK-NEXT:           "_self": {{[0-9]+}},
+# CHECK-NEXT:           "_total": {{[0-9]+}},
+# CHECK-NEXT:           "ast": {
+# CHECK-NEXT:             "_self": {{[0-9]+}},
+# CHECK-NEXT:             "_total": {{[0-9]+}}
+# CHECK-NEXT:           },
+# CHECK-NEXT:           "preamble": {
+# CHECK-NEXT:             "_self": {{[0-9]+}},
+# CHECK-NEXT:             "_total": {{[0-9]+}}
+# CHECK-NEXT:           }
+# CHECK-NEXT:         },
+# CHECK-NEXT:         "_self": {{[0-9]+}},
+# CHECK-NEXT:         "_total": {{[0-9]+}}
+# CHECK-NEXT:       }
+# CHECK-NEXT:     }
+# CHECK-NEXT:   }
+---
+{"jsonrpc":"2.0","id":3,"method":"shutdown"}
+---
+{"jsonrpc":"2.0","method":"exit"}
+