[clangd] codeComplete returns more structured completion items, LSP. NFC.

[sammccall]

Summary

LSP has some presentational fields with limited semantics (e.g. 'detail') and
doesn't provide a good place to return information like namespace.

Some places where more detailed information is useful:

• tools like quality analysis
• alternate frontends that aren't strictly LSP
• code completion unit tests

In this patch, ClangdServer::codeComplete still return LSP CompletionList, but
I plan to switch that soon (should be a no-op for ClangdLSPServer).

Deferring this makes it clear that we don't change behavior (tests stay the
same) and also keeps the API break to a small patch which can be reverted.

[sammccall]

@jkorous FYI: this may be interesting from an XPC perspective - it'd be easy to serialize the LSP-style struct, or the more detailed one if that's useful.
LSP will be more stable of course :-)

[ioeric]

Looks great! Mostly nits.

clangd/CodeComplete.cpp

259

nit: shall we keep assert(bool(SemaResult) == bool(SemaCCS));?

279

remove?

280

Inserted can be an absolute path which might have long uninteresting prefix. It might make sense to use the result from calculateIncludePath?

309

nit: It's unclear why add is a public interface given that the constructor also adds candidate. Might worth a comment. Also, it seems that the interface design has been driven by by the bundling logic, so it might also make sense to explain a bit on the class level.

322

nit: the documentation for the bundling says Documentation may contain a combined docstring from all comments. The implementation seems to take the documentation from the last comment (which seems fine to me), but they should be consistent.

346

nit: this might worth a comment. it's unclear what this does by just looking at the name.

1152

nit: this seems to assume that NameMatch is a factor of the final score, which seems to be changeable given the abstraction of evaluateSymbolAndRelevance. Not sure if there is an alternative to the current approach, but this probably deserves a comment.

1169–1179

nit: toCompletionItem might be ambiguous now that this returns CodeCompletion.

clangd/CodeComplete.h

88

Give that this field has caused confusion before, would it make sense to make the name a bit more concrete like RequiredQualifier or InsertedQualifier?

106

Would this be an absolute path or an include path?

140

I found Result.More a bit not straightforward. Maybe HasMore?

[sammccall]

Address review comments.

clangd/CodeComplete.cpp

259

Done (in add(), and hoisted the call to add() to the top here)

322

This was meant to be ambiguous ("may") to allow a better implementation later. But rewrote it to mention the current behavior as a possibility :-)

1152

Yeah, client-side filtering is the great abstraction-breaker :-(
Added the comment, not sure there's much more to be done.

[ioeric]

lgtm

clangd/ClangdServer.cpp

auto Task = [PCHs, Pos, FS,
                                  llvm::Expected<InputsAndPreamble> IP) {
     if (!IP)
       return CB(IP.takeError());
 
     auto PreambleData = IP->Preamble;
 
     // FIXME(ibiryukov): even if Preamble is non-null, we may want to check
     // both the old and the new version in case only one of them matches.
-    CompletionList Result = clangd::codeComplete(
+    CodeCompleteResult Result = clangd::codeComplete(
         File, IP->Command, PreambleData ? &PreambleData->Preamble : nullptr,
         PreambleData ? PreambleData->Inclusions : std::vector<Inclusion>(),
         IP->Contents, Pos, FS, PCHs, CodeCompleteOpts);
-    CB(std::move(Result));
+    CompletionList LSPResult;
+    LSPResult.isIncomplete = Result.HasMore;
+    for (const auto &Completion : Result.Completions)
+      LSPResult.items.push_back(Completion.render(CodeCompleteOpts));
+    CB(std::move(LSPResult));
   };
 
   WorkScheduler.runWithPreamble("CodeComplete", File,
                                 Bind(Task, File.str(), std::move(CB)));
 }
 
 void ClangdServer::signatureHelp(PathRef File, Position Pos,
                                  Callback<SignatureHelp> CB) {

clangd/CodeComplete.h

struct CodeCompleteOptions {
   // Populated internally by clangd, do not set.
   /// If `Index` is set, it is used to augment the code completion
   /// results.
   /// FIXME(ioeric): we might want a better way to pass the index around inside
   /// clangd.
   const SymbolIndex *Index = nullptr;
 };
 
+// Semi-structured representation of a code-complete suggestion for our C++ API.
+// We don't use the LSP structures here (unlike most features) as we want
+// to expose more data to allow for more precise testing and evaluation.
+struct CodeCompletion {
+  // The unqualified name of the symbol or other completion item.
+  std::string Name;
+  // The scope qualifier for the symbol name. e.g. "ns1::ns2::"
+  // Empty for non-symbol completions. Not inserted, but may be displayed.
+  std::string Scope;
+  // Text that must be inserted before the name, and displayed (e.g. base::).
+  std::string RequiredQualifier;
+  // Details to be displayed following the name. Not inserted.
+  std::string Signature;
+  // Text to be inserted following the name, in snippet format.
+  std::string SnippetSuffix;
+  // Type to be displayed for this completion.
+  std::string ReturnType;
+  std::string Documentation;
+  CompletionItemKind Kind = CompletionItemKind::Missing;
+  // This completion item may represent several symbols that can be inserted in
+  // the same way, such as function overloads. In this case BundleSize > 1, and
+  // the following fields are summaries:
+  //  - Signature is e.g. "(...)" for functions.
+  //  - SnippetSuffix is similarly e.g. "(${0})".
+  //  - ReturnType may be empty
+  //  - Documentation may be from one symbol, or a combination of several
+  // Other fields should apply equally to all bundled completions.
+  unsigned BundleSize;
+  // The header through which this symbol could be included.
+  // Quoted string as expected by an #include directive, e.g. "<memory>".
+  // Empty for non-symbol completions, or when not known.
+  std::string Header;
+  // Present if Header is set and should be inserted to use this item.
+  llvm::Optional<TextEdit> HeaderInsertion;
+
+  // Scores are used to rank completion items.
+  struct Scores {
+    // The score that items are ranked by.
+    float Total = 0.f;
+
+    // The finalScore with the fuzzy name match score excluded.
+    // When filtering client-side, editors should calculate the new fuzzy score,
+    // whose scale is 0-1 (with 1 = prefix match, special case 2 = exact match),
+    // and recompute finalScore = fuzzyScore * symbolScore.
+    float ExcludingName = 0.f;
+
+    // Component scores that contributed to the final score:
+
+    // Quality describes how important we think this candidate is,
+    // independent of the query.
+    // e.g. symbols with lots of incoming references have higher quality.
+    float Quality = 0.f;
+    // Relevance describes how well this candidate matched the query.
+    // e.g. symbols from nearby files have higher relevance.
+    float Relevance = 0.f;
+  };
+  Scores Score;
+
+  // Serialize this to an LSP completion item. This is a lossy operation.
+  CompletionItem render(const CodeCompleteOptions &) const;
+};
+struct CodeCompleteResult {
+  std::vector<CodeCompletion> Completions;
+  bool HasMore = false;
+};
+
 /// Get code completions at a specified \p Pos in \p FileName.
-CompletionList codeComplete(PathRef FileName,
-                            const tooling::CompileCommand &Command,
-                            PrecompiledPreamble const *Preamble,
-                            const std::vector<Inclusion> &PreambleInclusions,
-                            StringRef Contents, Position Pos,
-                            IntrusiveRefCntPtr<vfs::FileSystem> VFS,
-                            std::shared_ptr<PCHContainerOperations> PCHs,
-                            CodeCompleteOptions Opts);
+CodeCompleteResult codeComplete(
+    PathRef FileName, const tooling::CompileCommand &Command,
+    const std::vector<Inclusion> &PreambleInclusions, StringRef Contents,
+    Position Pos, IntrusiveRefCntPtr<vfs::FileSystem> VFS,
+    std::shared_ptr<PCHContainerOperations> PCHs, CodeCompleteOptions Opts);
 
 /// Get signature help at a specified \p Pos in \p FileName.
 SignatureHelp signatureHelp(PathRef FileName,
                             const tooling::CompileCommand &Command,
                             PrecompiledPreamble const *Preamble,
                             StringRef Contents, Position Pos,
                             IntrusiveRefCntPtr<vfs::FileSystem> VFS,
                             std::shared_ptr<PCHContainerOperations> PCHs);

clangd/CodeComplete.cpp

 #include "clang/Index/USRGeneration.h"
 #include "clang/Sema/CodeCompleteConsumer.h"
 #include "clang/Sema/Sema.h"
 #include "clang/Tooling/Core/Replacement.h"
 #include "llvm/Support/Format.h"
 #include <queue>
 
 // We log detailed candidate here if you run with -debug-only=codecomplete.
-#define DEBUG_TYPE "codecomplete"
+#define DEBUG_TYPE "CodeComplete"
 
 namespace clang {
 namespace clangd {
 namespace {
 
 CompletionItemKind toCompletionItemKind(index::SymbolKind Kind) {
   using SK = index::SymbolKind;
   switch (Kind) {
if (SemaResult && SemaResult->Declaration) {
       auto &SM = SemaResult->Declaration->getASTContext().getSourceManager();
       for (const Decl *RD : SemaResult->Declaration->redecls())
         if (SM.isInMainFile(SM.getExpansionLoc(RD->getLocStart())))
           return llvm::None;
     }
     return IndexResult->Detail->IncludeHeader;
   }
 
-  // Builds an LSP completion item.
-  CompletionItem build(StringRef FileName, const CompletionItemScores &Scores,
-                       const CodeCompleteOptions &Opts,
-                       CodeCompletionString *SemaCCS,
-                       const IncludeInserter &Includes,
-                       llvm::StringRef SemaDocComment) const {
-    assert(bool(SemaResult) == bool(SemaCCS));
-    assert(SemaResult || IndexResult);
-
-    CompletionItem I;
-    bool InsertingInclude = false; // Whether a new #include will be added.
-    if (SemaResult) {
-      llvm::StringRef Name(SemaCCS->getTypedText());
-      std::string Signature, SnippetSuffix, Qualifiers;
-      getSignature(*SemaCCS, &Signature, &SnippetSuffix, &Qualifiers);
-      I.label = (Qualifiers + Name + Signature).str();
-      I.filterText = Name;
-      I.insertText = (Qualifiers + Name).str();
-      if (Opts.EnableSnippets)
-        I.insertText += SnippetSuffix;
-      I.documentation = formatDocumentation(*SemaCCS, SemaDocComment);
-      I.detail = getReturnType(*SemaCCS);
-      if (SemaResult->Kind == CodeCompletionResult::RK_Declaration)
-        if (const auto *D = SemaResult->getDeclaration())
-          if (const auto *ND = llvm::dyn_cast<NamedDecl>(D))
-            I.SymbolScope = splitQualifiedName(printQualifiedName(*ND)).first;
-      I.kind = toCompletionItemKind(SemaResult->Kind, SemaResult->Declaration);
-    }
-    if (IndexResult) {
-      if (I.SymbolScope.empty())
-        I.SymbolScope = IndexResult->Scope;
-      if (I.kind == CompletionItemKind::Missing)
-        I.kind = toCompletionItemKind(IndexResult->SymInfo.Kind);
-      // FIXME: reintroduce a way to show the index source for debugging.
-      if (I.label.empty())
-        I.label = (IndexResult->Name + IndexResult->Signature).str();
-      if (I.filterText.empty())
-        I.filterText = IndexResult->Name;
-
-      // FIXME(ioeric): support inserting/replacing scope qualifiers.
-      if (I.insertText.empty()) {
-        I.insertText = IndexResult->Name;
-        if (Opts.EnableSnippets)
-          I.insertText += IndexResult->CompletionSnippetSuffix;
-      }
+  using Bundle = llvm::SmallVector<CompletionCandidate, 4>;
+};
+using ScoredBundle =
+    std::pair<CompletionCandidate::Bundle, CodeCompletion::Scores>;
+struct ScoredBundleGreater {
+  bool operator()(const ScoredBundle &L, const ScoredBundle &R) {
+    if (L.second.Total != R.second.Total)
+      return L.second.Total > R.second.Total;
+    return L.first.front().Name <
+           R.first.front().Name; // Earlier name is better.
+};
 
-      if (auto *D = IndexResult->Detail) {
-        if (I.documentation.empty())
-          I.documentation = D->Documentation;
-        if (I.detail.empty())
-          I.detail = D->ReturnType;
-        if (auto Inserted = headerToInsertIfNotPresent()) {
-          auto IncludePath = [&]() -> Expected<std::string> {
-            auto ResolvedDeclaring = toHeaderFile(
-                IndexResult->CanonicalDeclaration.FileURI, FileName);
-            if (!ResolvedDeclaring)
-              return ResolvedDeclaring.takeError();
-            auto ResolvedInserted = toHeaderFile(*Inserted, FileName);
-            if (!ResolvedInserted)
-              return ResolvedInserted.takeError();
-            if (!Includes.shouldInsertInclude(*ResolvedDeclaring,
-                                              *ResolvedInserted))
-              return "";
-            return Includes.calculateIncludePath(*ResolvedDeclaring,
-                                                 *ResolvedInserted);
-          }();
-          if (!IncludePath) {
-            std::string ErrMsg =
-                ("Failed to generate include insertion edits for adding header "
-                 "(FileURI=\"" +
-                 IndexResult->CanonicalDeclaration.FileURI +
-                 "\", IncludeHeader=\"" + D->IncludeHeader + "\") into " +
-                 FileName)
-                    .str();
-            log(ErrMsg + ":" + llvm::toString(IncludePath.takeError()));
-          } else if (!IncludePath->empty()) {
-            // FIXME: consider what to show when there is no #include insertion,
-            // and for sema results, for consistency.
-            if (auto Edit = Includes.insert(*IncludePath)) {
-              I.detail += ("\n" + StringRef(*IncludePath).trim('"')).str();
-              I.additionalTextEdits = {std::move(*Edit)};
-              InsertingInclude = true;
-            }
-          }
-        }
-      }
-    }
-    I.label = (InsertingInclude ? Opts.IncludeIndicator.Insert
-                                : Opts.IncludeIndicator.NoInsert) +
-              I.label;
-    I.scoreInfo = Scores;
-    I.sortText = sortText(Scores.finalScore, Name);
-    I.insertTextFormat = Opts.EnableSnippets ? InsertTextFormat::Snippet
-                                             : InsertTextFormat::PlainText;
-    return I;
+// Assembles a code completion out of a bundle of >=1 completion candidates.
+// Many of the expensive strings are only computed at this point, once we know
+// the candidate bundle is going to be returned.
+//
+// Many fields are the same for all candidates in a bundle (e.g. name), and are
+// computed from the first candidate, in the constructor.
+// Others vary per candidate, so add() must be called for remaining candidates.
+struct CodeCompletionBuilder {
+  CodeCompletionBuilder(ASTContext &ASTCtx, const CompletionCandidate &C,
+                        CodeCompletionString *SemaCCS,
+                        const IncludeInserter &Includes, StringRef FileName,
+                        const CodeCompleteOptions &Opts)
+      : ASTCtx(ASTCtx), ExtractDocumentation(Opts.IncludeComments) {
+    add(C, SemaCCS);
+    if (C.SemaResult) {
+      Completion.Name = llvm::StringRef(SemaCCS->getTypedText());
+      if (Completion.Scope.empty())
+        if (C.SemaResult->Kind == CodeCompletionResult::RK_Declaration)
+          if (const auto *D = C.SemaResult->getDeclaration())
+            if (const auto *ND = llvm::dyn_cast<NamedDecl>(D))
+              Completion.Scope =
+                  splitQualifiedName(printQualifiedName(*ND)).first;
+      Completion.Kind =
+          toCompletionItemKind(C.SemaResult->Kind, C.SemaResult->Declaration);
+    }
+    if (C.IndexResult) {
+      if (Completion.Scope.empty())
+        Completion.Scope = C.IndexResult->Scope;
+      if (Completion.Kind == CompletionItemKind::Missing)
+        Completion.Kind = toCompletionItemKind(C.IndexResult->SymInfo.Kind);
+      if (Completion.Name.empty())
+        Completion.Name = C.IndexResult->Name;
+    }
+    if (auto Inserted = C.headerToInsertIfNotPresent()) {
+      // Turn absolute path into a literal string that can be #included.
+      auto Include = [&]() -> Expected<std::pair<std::string, bool>> {
+        auto ResolvedDeclaring =
+            toHeaderFile(C.IndexResult->CanonicalDeclaration.FileURI, FileName);
+        return std::make_pair(Includes.calculateIncludePath(*ResolvedDeclaring,
+                                                            *ResolvedInserted),
+                              Includes.shouldInsertInclude(*ResolvedDeclaring,
+                                                           *ResolvedInserted));
+      if (Include) {
+        Completion.Header = Include->first;
+        if (Include->second)
+          Completion.HeaderInsertion = Includes.insert(Include->first);
+      } else
+        log(llvm::formatv(
+            "Failed to generate include insertion edits for adding header "
+            "(FileURI='{0}', IncludeHeader='{1}') into {2}",
+            C.IndexResult->CanonicalDeclaration.FileURI,
+            C.IndexResult->Detail->IncludeHeader, FileName));
+
+  void add(const CompletionCandidate &C, CodeCompletionString *SemaCCS) {
+    assert(bool(C.SemaResult) == bool(SemaCCS));
+    Bundled.emplace_back();
+    BundledEntry &S = Bundled.back();
+    if (C.SemaResult) {
+      getSignature(*SemaCCS, &S.Signature, &S.SnippetSuffix,
+                   &Completion.RequiredQualifier);
+      S.ReturnType = getReturnType(*SemaCCS);
+    } else if (C.IndexResult) {
+      S.Signature = C.IndexResult->Signature;
+      S.SnippetSuffix = C.IndexResult->CompletionSnippetSuffix;
+      if (auto *D = C.IndexResult->Detail)
+        S.ReturnType = D->ReturnType;
+    }
+    if (ExtractDocumentation && Completion.Documentation.empty()) {
+      if (C.IndexResult && C.IndexResult->Detail)
+        Completion.Documentation = C.IndexResult->Detail->Documentation;
+      else if (C.SemaResult)
+        Completion.Documentation = getDocComment(ASTCtx, *C.SemaResult,
+                                                 /*CommentsFromHeader=*/false);
+
+  CodeCompletion build() {
+    Completion.ReturnType = summarizeReturnType();
+    Completion.Signature = summarizeSignature();
+    Completion.SnippetSuffix = summarizeSnippet();
+    Completion.BundleSize = Bundled.size();
+    return std::move(Completion);
   }
 
-  using Bundle = llvm::SmallVector<CompletionCandidate, 4>;
+private:
+  struct BundledEntry {
+    std::string SnippetSuffix;
+    std::string Signature;
+    std::string ReturnType;
+  };
 
-  static CompletionItem build(const Bundle &Bundle, CompletionItem First,
-                              const clangd::CodeCompleteOptions &Opts) {
-    if (Bundle.size() == 1)
-      return First;
-    // Patch up the completion item to make it look like a bundle.
-    // This is a bit of a hack but most things are the same.
-
-    // Need to erase the signature. All bundles are function calls.
-    llvm::StringRef Name = Bundle.front().Name;
-    First.insertText =
-        Opts.EnableSnippets ? (Name + "(${0})").str() : Name.str();
-    // Keep the visual indicator of the original label.
-    bool InsertingInclude =
-        StringRef(First.label).startswith(Opts.IncludeIndicator.Insert);
-    First.label = (Twine(InsertingInclude ? Opts.IncludeIndicator.Insert
-                                          : Opts.IncludeIndicator.NoInsert) +
-                   Name + "(…)")
-                      .str();
-    First.detail = llvm::formatv("[{0} overloads]", Bundle.size());
-    return First;
+  // If all BundledEntrys have the same value for a property, return it.
+  template <std::string BundledEntry::*Member>
+  const std::string *onlyValue() const {
+    auto B = Bundled.begin(), E = Bundled.end();
+    for (auto I = B + 1; I != E; ++I)
+      if (I->*Member != B->*Member)
+        return nullptr;
+    return &(B->*Member);
   }
-};
-using ScoredBundle =
-    std::pair<CompletionCandidate::Bundle, CompletionItemScores>;
-struct ScoredBundleGreater {
-  bool operator()(const ScoredBundle &L, const ScoredBundle &R) {
-    if (L.second.finalScore != R.second.finalScore)
-      return L.second.finalScore > R.second.finalScore;
-    return L.first.front().Name <
-           R.first.front().Name; // Earlier name is better.
+
+  std::string summarizeReturnType() const {
+    if (auto *RT = onlyValue<&BundledEntry::ReturnType>())
+      return *RT;
+    return "";
   }
+
+  std::string summarizeSnippet() const {
+    if (auto *Snippet = onlyValue<&BundledEntry::SnippetSuffix>())
+      return *Snippet;
+    // All bundles are function calls.
+    return "(${0})";
+  }
+
+  std::string summarizeSignature() const {
+    if (auto *Signature = onlyValue<&BundledEntry::Signature>())
+      return *Signature;
+    // All bundles are function calls.
+    return "(…)";
+  }
+
+  ASTContext &ASTCtx;
+  CodeCompletion Completion;
+  SmallVector<BundledEntry, 1> Bundled;
+  bool ExtractDocumentation;
 };
 
 // Determine the symbol ID for a Sema code completion result, if possible.
 llvm::Optional<SymbolID> getSymbolID(const CodeCompletionResult &R) {
   switch (R.Kind) {
   case CodeCompletionResult::RK_Declaration:
   case CodeCompletionResult::RK_Pattern: {
     llvm::SmallString<128> USR;
public:
   CodeCompleteFlow(PathRef FileName, const CodeCompleteOptions &Opts)
       : FileName(FileName), Opts(Opts),
         // FIXME: also use path of the main header corresponding to FileName to
         // calculate the file proximity, which would capture include/ and src/
         // project setup where headers and implementations are not in the same
         // directory.
         FileProximityMatch(ArrayRef<StringRef>({FileName})) {}
 
-  CompletionList run(const SemaCompleteInput &SemaCCInput) && {
+  CodeCompleteResult run(const SemaCompleteInput &SemaCCInput) && {
     trace::Span Tracer("CodeCompleteFlow");
 
     // We run Sema code completion first. It builds an AST and calculates:
     //   - completion results based on the AST.
     //   - partial identifier and context. We need these for the index query.
-    CompletionList Output;
+    CodeCompleteResult Output;
     auto RecorderOwner = llvm::make_unique<CompletionRecorder>(Opts, [&]() {
       assert(Recorder && "Recorder is not set");
       assert(Includes && "Includes is not set");
       // If preprocessor was run, inclusions from preprocessor callback should
       // already be added to Inclusions.
       Output = runWithSema();
       Includes.reset(); // Make sure this doesn't out-live Clang.
       SPAN_ATTACH(Tracer, "sema_completion_kind",
                   getCompletionKindString(Recorder->CCContext.getKind()));
     });
 
     Recorder = RecorderOwner.get();
     semaCodeComplete(std::move(RecorderOwner), Opts.getClangCompleteOpts(),
                      SemaCCInput, &Includes);
 
     SPAN_ATTACH(Tracer, "sema_results", NSema);
     SPAN_ATTACH(Tracer, "index_results", NIndex);
     SPAN_ATTACH(Tracer, "merged_results", NBoth);
-    SPAN_ATTACH(Tracer, "returned_results", Output.items.size());
-    SPAN_ATTACH(Tracer, "incomplete", Output.isIncomplete);
+    SPAN_ATTACH(Tracer, "returned_results", Output.Completions.size());
+    SPAN_ATTACH(Tracer, "incomplete", Output.HasMore);
     log(llvm::formatv("Code complete: {0} results from Sema, {1} from Index, "
                       "{2} matched, {3} returned{4}.",
-                      NSema, NIndex, NBoth, Output.items.size(),
-                      Output.isIncomplete ? " (incomplete)" : ""));
-    assert(!Opts.Limit || Output.items.size() <= Opts.Limit);
+                      NSema, NIndex, NBoth, Output.Completions.size(),
+                      Output.HasMore ? " (incomplete)" : ""));
+    assert(!Opts.Limit || Output.Completions.size() <= Opts.Limit);
     // We don't assert that isIncomplete means we hit a limit.
     // Indexes may choose to impose their own limits even if we don't have one.
     return Output;
   }
 
 private:
   // This is called by run() once Sema code completion is done, but before the
   // Sema data structures are torn down. It does all the real work.
-  CompletionList runWithSema() {
+  CodeCompleteResult runWithSema() {
     Filter = FuzzyMatcher(
         Recorder->CCSema->getPreprocessor().getCodeCompletionFilter());
     // Sema provides the needed context to query the index.
     // FIXME: in addition to querying for extra/overlapping symbols, we should
     //        explicitly request symbols corresponding to Sema results.
     //        We can use their signals even if the index can't suggest them.
     // We must copy index results to preserve them, but there are at most Limit.
     auto IndexResults = (Opts.Index && allowIndex(Recorder->CCContext))
                             ? queryIndex()
                             : SymbolSlab();
     // Merge Sema and Index results, score them, and pick the winners.
     auto Top = mergeResults(Recorder->Results, IndexResults);
-    // Convert the results to the desired LSP structs.
-    CompletionList Output;
-    for (auto &C : Top)
-      Output.items.push_back(toCompletionItem(C.first, C.second));
-    Output.isIncomplete = Incomplete;
+    // Convert the results to final form, assembling the expensive strings.
+    CodeCompleteResult Output;
+    for (auto &C : Top) {
+      Output.Completions.push_back(toCodeCompletion(C.first));
+      Output.Completions.back().Score = C.second;
+    }
+    Output.HasMore = Incomplete;
     return Output;
   }
 
   SymbolSlab queryIndex() {
     trace::Span Tracer("Query index");
     SPAN_ATTACH(Tracer, "limit", Opts.Limit);
 
     SymbolSlab::Builder ResultsBuilder;
for (const auto &Candidate : Bundle) {
       }
       if (Candidate.SemaResult) {
         Quality.merge(*Candidate.SemaResult);
         Relevance.merge(*Candidate.SemaResult);
         ++SemaResult;
       }
     }
 
-    float QualScore = Quality.evaluate(), RelScore = Relevance.evaluate();
-    CompletionItemScores Scores;
-    Scores.finalScore = evaluateSymbolAndRelevance(QualScore, RelScore);
-    // The purpose of exporting component scores is to allow NameMatch to be
-    // replaced on the client-side. So we export (NameMatch, final/NameMatch)
-    // rather than (RelScore, QualScore).
-    Scores.filterScore = Relevance.NameMatch;
-    Scores.symbolScore =
-        Scores.filterScore ? Scores.finalScore / Scores.filterScore : QualScore;
+    CodeCompletion::Scores Scores;
+    Scores.Quality = Quality.evaluate();
+    Scores.Relevance = Relevance.evaluate();
+    Scores.Total = evaluateSymbolAndRelevance(Scores.Quality, Scores.Relevance);
+    // NameMatch is in fact a multiplier on total score, so rescoring is sound.
+    Scores.ExcludingName = Relevance.NameMatch
+                               ? Scores.Total / Relevance.NameMatch
+                               : Scores.Quality;
 
     LLVM_DEBUG(llvm::dbgs() << "CodeComplete: " << First.Name << "("
                             << IndexResult << " index) "
                             << "(" << SemaResult << " sema)"
-                            << " = " << Scores.finalScore << "\n"
+                            << " = " << Scores.Total << "\n"
                             << Quality << Relevance << "\n");
 
     NSema += bool(SemaResult);
     NIndex += bool(IndexResult);
     NBoth += SemaResult && IndexResult;
     if (Candidates.push({std::move(Bundle), Scores}))
       Incomplete = true;
   }
 
-  CompletionItem toCompletionItem(const CompletionCandidate::Bundle &Bundle,
-                                  const CompletionItemScores &Scores) {
-    CodeCompletionString *SemaCCS = nullptr;
-    std::string FrontDocComment;
-    if (auto *SR = Bundle.front().SemaResult) {
-      SemaCCS = Recorder->codeCompletionString(*SR);
-      if (Opts.IncludeComments) {
-        assert(Recorder->CCSema);
-        FrontDocComment = getDocComment(Recorder->CCSema->getASTContext(), *SR,
-                                        /*CommentsFromHeader=*/false);
-      }
+  CodeCompletion toCodeCompletion(const CompletionCandidate::Bundle &Bundle) {
+    llvm::Optional<CodeCompletionBuilder> Builder;
+    for (const auto &Item : Bundle) {
+      CodeCompletionString *SemaCCS =
+          Item.SemaResult ? Recorder->codeCompletionString(*Item.SemaResult)
+                          : nullptr;
+      if (!Builder)
+        Builder.emplace(Recorder->CCSema->getASTContext(), Item, SemaCCS,
+                        *Includes, FileName, Opts);
+      else
+        Builder->add(Item, SemaCCS);
     }
-    return CompletionCandidate::build(
-        Bundle,
-        Bundle.front().build(FileName, Scores, Opts, SemaCCS, *Includes,
-                             FrontDocComment),
-        Opts);
+    return Builder->build();
   }
 };
 
-CompletionList codeComplete(PathRef FileName,
-                            const tooling::CompileCommand &Command,
-                            PrecompiledPreamble const *Preamble,
-                            const std::vector<Inclusion> &PreambleInclusions,
-                            StringRef Contents, Position Pos,
-                            IntrusiveRefCntPtr<vfs::FileSystem> VFS,
-                            std::shared_ptr<PCHContainerOperations> PCHs,
-                            CodeCompleteOptions Opts) {
+CodeCompleteResult codeComplete(
+    PathRef FileName, const tooling::CompileCommand &Command,
+    const std::vector<Inclusion> &PreambleInclusions, StringRef Contents,
+    Position Pos, IntrusiveRefCntPtr<vfs::FileSystem> VFS,
+    std::shared_ptr<PCHContainerOperations> PCHs, CodeCompleteOptions Opts) {
   return CodeCompleteFlow(FileName, Opts)
       .run({FileName, Command, Preamble, PreambleInclusions, Contents, Pos, VFS,
             PCHs});
 }
 
 SignatureHelp signatureHelp(PathRef FileName,
                             const tooling::CompileCommand &Command,
                             PrecompiledPreamble const *Preamble,
auto InTopLevelScope = hasDeclContext(
       anyOf(namespaceDecl(), translationUnitDecl(), linkageSpecDecl()));
   return !match(decl(anyOf(InTopLevelScope,
                            hasDeclContext(
                                enumDecl(InTopLevelScope, unless(isScoped()))))),
                 ND, ASTCtx)
               .empty();
 }
 
+CompletionItem CodeCompletion::render(const CodeCompleteOptions &Opts) const {
+  CompletionItem LSP;
+  LSP.label = (HeaderInsertion ? Opts.IncludeIndicator.Insert
+                               : Opts.IncludeIndicator.NoInsert) +
+              RequiredQualifier + Name + Signature;
+  LSP.kind = Kind;
+  LSP.detail = BundleSize > 1 ? llvm::formatv("[{0} overloads]", BundleSize)
+                              : ReturnType;
+  if (!Header.empty())
+    LSP.detail += "\n" + Header;
+  LSP.documentation = Documentation;
+  LSP.sortText = sortText(Score.Total, Name);
+  LSP.filterText = Name;
+  LSP.insertText = RequiredQualifier + Name;
+  if (Opts.EnableSnippets)
+    LSP.insertText += SnippetSuffix;
+  LSP.insertTextFormat = Opts.EnableSnippets ? InsertTextFormat::Snippet
+                                             : InsertTextFormat::PlainText;
+  if (HeaderInsertion)
+    LSP.additionalTextEdits = {*HeaderInsertion};
+  LSP.SymbolScope = Scope;
+  return LSP;
+}
+
 } // namespace clangd
 } // namespace clang

clangd/Protocol.h

enum class InsertTextFormat {
   /// of the snippet. Placeholders with equal identifiers are linked, that is
   /// typing in one will update others too.
   ///
   /// See also:
   /// https//github.com/Microsoft/vscode/blob/master/src/vs/editor/contrib/snippet/common/snippet.md
   Snippet = 2,
 };
 
-/// Provides details for how a completion item was scored.
-/// This can be used for client-side filtering of completion items as the
-/// user keeps typing.
-/// This is a clangd extension.
-struct CompletionItemScores {
-  /// The score that items are ranked by.
-  /// This is filterScore * symbolScore.
-  float finalScore = 0.f;
-  /// How the partial identifier matched filterText. [0-1]
-  float filterScore = 0.f;
-  /// How the symbol fits, ignoring the partial identifier.
-  float symbolScore = 0.f;
-};
-
 struct CompletionItem {
   /// The label of this completion item. By default also the text that is
   /// inserted when selecting this completion.
   std::string label;
 
   /// The kind of this completion item. Based of the kind an icon is chosen by
   /// the editor.
   CompletionItemKind kind = CompletionItemKind::Missing;
 
   /// A human-readable string with additional information about this item, like
   /// type or symbol information.
   std::string detail;
 
   /// A human-readable string that represents a doc-comment.
   std::string documentation;
 
   /// A string that should be used when comparing this item with other items.
   /// When `falsy` the label is used.
   std::string sortText;
 
-  /// Details about the quality of this completion item. (clangd extension)
-  llvm::Optional<CompletionItemScores> scoreInfo;
-
   /// A string that should be used when filtering a set of completion items.
   /// When `falsy` the label is used.
   std::string filterText;
 
   /// A string that should be inserted to a document when selecting this
   /// completion. When `falsy` the label is used.
   std::string insertText;