[clangd] Add instrumentation mode in clangd for metrics collection.

[usaxena95]

Summary

This patch adds an instrumentation mode for clangd (enabled by
corresponding option in cc_opts).
If this mode is enabled then user can specify callbacks to run on the
final code completion result.

Moreover the CodeCompletion::Score will contain the detailed Quality and
Relevance signals used to compute the score when this mode is enabled.
These are required because we do not any place in which the final
candidates (scored and sorted) are available along with the above
signals. The signals are temporary structures in addCandidate.

The callback is needed as it gives access to many data structures that
are internal to CodeCompleteFlow and are available once Sema has run. Eg:
ScopeDistnace and FileDistance.

If this mode is disabled (as in default) then Score would just contain 2
shared pointers (null). Thus cost(memory/time) increase for the default
mode would be fairly cheap and insignificant.

[usaxena95]

Addressed linter issues.

[sammccall]

clang-tools-extra/clangd/CodeComplete.h

140

I'm wondering if we can simplify this interface a bit. I'm not sure why we need another callback rather than just returning the CodeCompleteResult in the usual way.

Another option:

• if we invoke a callback for each completion instead of for the result set as a whole, we don't need to work out where to stash anything.
• the work done after addCandidate is pretty trivial, so invoking a callback there provides basically all the information about the result set. The Top-N truncation is probably something you'd rather *not* have for analysis.
• code completion always ends with the callback being invoked, so cross-result analysis can be done at that point.

So I think this could just be a single
std::function<void(const CodeCompletion&, const SymbolQualitySignals &, const SymbolRelevanceSignals &)>.
If it's non-null, addCandidate would call toCodeCompletion on the bundle and pass it to the callback at the end.

220

why shared rather than unique?

[kadircet]

clang-tools-extra/clangd/CodeComplete.cpp

1463

can't we make use of the trace::Span instead ?

[usaxena95]

Changed to invoke callback on all code completion items.

[usaxena95]

clang-tools-extra/clangd/CodeComplete.cpp

1463

CMIIW. I believe with trace::Span we can send only JSON messages from clangd to the tool running it. This doesn't allow us to get access to internal DS of CodeCompleteFlow that are used along with Quality and Relevance signals (like Distances of file and scopes).
You can argue that all this information can be serialized as a JSON (including features derived from these internal DS) but this then must be done on clangd side (not on tools side).

IMO this gives more freedom to the tool to use and derive more features which makes experimentation easier.

clang-tools-extra/clangd/CodeComplete.h

140

why we need another callback rather than just returning the CodeCompleteResult in the usual way.

There are some data structures from CodeCompleteFlow referred to in Reference signals like ScopeDistance which were needed to compute distances. But think can be addressed in your suggested "per completion" callback approach.

Another option: ...

I had given this approach some thought previously and had concerns about mapping the items to the final ranked items in the TopN result. But on a second thought we can completely ignore the final result (assuming no significant changes are done after addCandidate) and score and rank the results ourselves in the FlumeTool.

220

A not-so-proud hack to keep Score copyable (this is removed now).

[usaxena95]

Remove ununsed import.

[sammccall]

clang-tools-extra/clangd/CodeComplete.h

136

First say what the behavior/API is (called once for each result...), Then justify it :)

139

I'd suggest including the final score in the signature rather than recompute it, just so the contract is really clear and simple (results yielded in arbitrary order, will be ranked by -score). Please spell this out.

140

This doesn't need to be a pointer, std::function is copy/movable and supports nullptr as a sentinel value.

clang-tools-extra/clangd/unittests/CodeCompleteTests.cpp

1049

Nit: typically auto here (the anonymous lambda type) and let it convert to function implicitly when needed.

No need for -> type in trivial cases

[sammccall]

Forgot to mention: I also think the trace approach certainly has things going for it, or even parsing out the messages from the existing logs.
But in this particular case the callback happens to be extremely convenient and also not invasive (since the data structures are already exposed, code complete has an opts struct etc). And since this is for analysis we have a lot of flexibility to rework if it stops being easy to maintain.

[kadircet]

In D75603#1906418, @sammccall wrote:

Forgot to mention: I also think the trace approach certainly has things going for it, or even parsing out the messages from the existing logs.
But in this particular case the callback happens to be extremely convenient and also not invasive (since the data structures are already exposed, code complete has an opts struct etc). And since this is for analysis we have a lot of flexibility to rework if it stops being easy to maintain.

as discussed offline, I was rather afraid of the initial version of the patch, but the final version seems ok as it only adds a single field to codecompleteopts.

[usaxena95]

Addressed comments.

• Populated score in CodeCompletion before invoking the callback.
• Tested that CodeCompletion is scored
• Updated comment for callback.

[usaxena95]

clang-tools-extra/clangd/CodeComplete.h

139

Couldn't add it to the signature since inner classes cannot be forward declared.
Since CodeCompletion contains the Score, I have populated this field in the CodeCompletion (and also CompletionTokenRange) as done in toCodeCompleteResult to be consistent.

Also tested that the CodeCompletion is scored.

[usaxena95]

Passed score as a float as an explicit argument of the callback.

clang-tools-extra/clangd/CodeComplete.h

 
 #ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_CODECOMPLETE_H
 #define LLVM_CLANG_TOOLS_EXTRA_CLANGD_CODECOMPLETE_H
 
 #include "Headers.h"
 #include "Logger.h"
 #include "Path.h"
 #include "Protocol.h"
+#include "Quality.h"
 #include "index/Index.h"
 #include "index/Symbol.h"
 #include "index/SymbolOrigin.h"
 #include "clang/Sema/CodeCompleteConsumer.h"
 #include "clang/Sema/CodeCompleteOptions.h"
 #include "clang/Tooling/CompilationDatabase.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/Error.h"
+#include <functional>
 #include <future>
 
 namespace clang {
 class NamedDecl;
 namespace clangd {
 struct PreambleData;
+struct CodeCompletion;
 
 struct CodeCompleteOptions {
   /// Returns options that can be passed to clang's completion engine.
   clang::CodeCompleteOptions getClangCompleteOpts() const;
 
   /// When true, completion items will contain expandable code snippets in
   /// completion (e.g.  `return ${1:expression}` or `foo(${1:int a}, ${2:int
   /// b})).
enum CodeCompletionParse {
     /// Return an error if this fails.
     AlwaysParse,
     /// Run the parser if inputs (preamble) are ready.
     /// Otherwise, use text-based completion.
     ParseIfReady,
     /// Always use text-based completion.
     NeverParse,
   } RunParser = ParseIfReady;
+
+  /// Callback invoked on all CompletionCandidate after they are scored and
+  /// before they are ranked (by -Score). Thus the results are yielded in
+  /// arbitrary order. CodeCompletion also contains the computed Score.
+  ///
+  /// This callbacks allows capturing various internal structures used by clangd
+  /// during code completion. Eg: Symbol quality and relevance signals.
+  std::function<void(const CodeCompletion &, const SymbolQualitySignals &,
+                     const SymbolRelevanceSignals &)>
+      RecordCCResult;
 };
 
 // Semi-structured representation of a code-complete suggestion for our C++ API.
 // We don't use the LSP structures here (unlike most features) as we want
 // to expose more data to allow for more precise testing and evaluation.
 struct CodeCompletion {
   // The unqualified name of the symbol or other completion item.
   std::string Name;
struct Scores {
     // independent of the query.
     // e.g. symbols with lots of incoming references have higher quality.
     float Quality = 0.f;
     // Relevance describes how well this candidate matched the query.
     // e.g. symbols from nearby files have higher relevance.
     float Relevance = 0.f;
   };
   Scores Score;
 
   /// Indicates if this item is deprecated.
   bool Deprecated = false;
 
   // Serialize this to an LSP completion item. This is a lossy operation.
   CompletionItem render(const CodeCompleteOptions &) const;
 };
 raw_ostream &operator<<(raw_ostream &, const CodeCompletion &);
 struct CodeCompleteResult {

clang-tools-extra/clangd/CodeComplete.cpp

auto IndexResults = (Opts.Index && allowIndex(Recorder->CCContext))
                             ? queryIndex()
                             : SymbolSlab();
     trace::Span Tracer("Populate CodeCompleteResult");
     // Merge Sema and Index results, score them, and pick the winners.
     auto Top =
         mergeResults(Recorder->Results, IndexResults, /*Identifiers*/ {});
     return toCodeCompleteResult(Top);
   }
 
   CodeCompleteResult
   toCodeCompleteResult(const std::vector<ScoredBundle> &Scored) {
     CodeCompleteResult Output;
 
     // Convert the results to final form, assembling the expensive strings.
     for (auto &C : Scored) {
       Output.Completions.push_back(toCodeCompletion(C.first));
       Output.Completions.back().Score = C.second;
void addCandidate(TopN<ScoredBundle, ScoredBundleGreater> &Candidates,
     Scores.Quality = Quality.evaluate();
     Scores.Relevance = Relevance.evaluate();
     Scores.Total = evaluateSymbolAndRelevance(Scores.Quality, Scores.Relevance);
     // NameMatch is in fact a multiplier on total score, so rescoring is sound.
     Scores.ExcludingName = Relevance.NameMatch
                                ? Scores.Total / Relevance.NameMatch
                                : Scores.Quality;
 
+    if (Opts.RecordCCResult) {
+      CodeCompletion Completion = toCodeCompletion(Bundle);
+      Completion.Score = Scores;
+      Completion.CompletionTokenRange = ReplacedRange;
+      Opts.RecordCCResult(Completion, Quality, Relevance);
+    }
+
     dlog("CodeComplete: {0} ({1}) = {2}\n{3}{4}\n", First.Name,
          llvm::to_string(Origin), Scores.Total, llvm::to_string(Quality),
          llvm::to_string(Relevance));
 
     NSema += bool(Origin & SymbolOrigin::AST);
     NIndex += FromIndex;
     NSemaAndIndex += bool(Origin & SymbolOrigin::AST) && FromIndex;
     NIdent += bool(Origin & SymbolOrigin::Identifier);

clang-tools-extra/clangd/unittests/CodeCompleteTests.cpp

 #include "clang/Sema/CodeCompleteConsumer.h"
 #include "clang/Tooling/CompilationDatabase.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Testing/Support/Error.h"
 #include "gmock/gmock.h"
 #include "gtest/gtest.h"
 #include <condition_variable>
+#include <functional>
 #include <mutex>
+#include <vector>
 
 namespace clang {
 namespace clangd {
 
 namespace {
 using ::llvm::Failed;
 using ::testing::AllOf;
 using ::testing::Contains;

 }
 MATCHER_P(InsertInclude, IncludeHeader, "") {
   return !arg.Includes.empty() && arg.Includes[0].Header == IncludeHeader &&
          bool(arg.Includes[0].Insertion);
 }
 MATCHER(InsertInclude, "") {
   return !arg.Includes.empty() && bool(arg.Includes[0].Insertion);
 }
+MATCHER(Scored, "") { return arg.Score.Total > 0; }
 MATCHER_P(SnippetSuffix, Text, "") { return arg.SnippetSuffix == Text; }
 MATCHER_P(Origin, OriginSet, "") { return arg.Origin == OriginSet; }
 MATCHER_P(Signature, S, "") { return arg.Signature == S; }
 
 // Shorthand for Contains(Named(Name)).
 Matcher<const std::vector<CodeCompletion> &> Has(std::string Name) {
   return Contains(Named(std::move(Name)));
 }

 template <template <class> class TT> int foo() {
   int a = ^
 }
 )cpp")
                          .Completions;
   EXPECT_THAT(Completions, Contains(Named("TT")));
 }
 
+TEST(CompletionTest, EnableInstrumentationMode) {
+  std::vector<CodeCompletion> RecordedCompletions;
+  CodeCompleteOptions Opts;
+  Opts.RecordCCResult = [&RecordedCompletions](const CodeCompletion &CC,
+                                               const SymbolQualitySignals &,
+                                               const SymbolRelevanceSignals &) {
+    RecordedCompletions.push_back(CC);
+  };
+
+  completions("int xy1, xy2; int a = xy^", /*IndexSymbols=*/{}, Opts);
+  EXPECT_THAT(RecordedCompletions,
+              UnorderedElementsAre(AllOf(Named("xy1"), Scored()),
+                                   AllOf(Named("xy2"), Scored())));
+}
+
 SignatureHelp signatures(llvm::StringRef Text, Position Point,
                          std::vector<Symbol> IndexSymbols = {}) {
   std::unique_ptr<SymbolIndex> Index;
   if (!IndexSymbols.empty())
     Index = memIndex(IndexSymbols);
 
   MockFSProvider FS;
   MockCompilationDatabase CDB;