[clangd] Use ML Code completion ranking as default.

[usaxena95]

Summary

This makes code completion use a Decision Forest based ranking algorithm to rank
completion candidates. [Estimated 6% accuracy boost]. This was
previously hidden behind the flag --ranking-model=decision_forest. This
patch makes it the default ranking algorithm.

Note: this is a generic model, not specialized for any particular
project. clangd does not collect or upload data to train code completion.

Also treat Keywords separately as they are not recorded by the training set generator.

[hokein]

clang-tools-extra/clangd/unittests/CodeCompleteTests.cpp

650

sorry, I didn't infer the motivation of this change func->ns from this patch, could you explain?

650–656

looks like the Results here is not verified, as you remove the line below, is it intentional?

[usaxena95]

Addressed comments.

clang-tools-extra/clangd/unittests/CodeCompleteTests.cpp

650

Sorry about not highlighting this.
The ML model doesn't rank the function before the namespace. I removed this and made both of these as functions to independently validate the effect of references.

650–656

Oops. Thanks!.

[hokein]

maybe add some data (improvement DecisionForest vs heuristic) in the patch description.

clang-tools-extra/clangd/unittests/CodeCompleteTests.cpp

650

ah, I see, that makes sense, thanks!

clang-tools-extra/clangd/CodeComplete.h

struct CodeCompleteOptions {
   std::function<void(const CodeCompletion &, const SymbolQualitySignals &,
                      const SymbolRelevanceSignals &, float Score)>
       RecordCCResult;
 
   /// Model to use for ranking code completion candidates.
   enum CodeCompletionRankingModel {
     Heuristics,
     DecisionForest,
-  } RankingModel = Heuristics;
+  } RankingModel = DecisionForest;
 
   /// Callback used to score a CompletionCandidate if DecisionForest ranking
   /// model is enabled.
   /// This allows us to inject experimental models and compare them with
   /// baseline model using A/B testing.
   std::function<DecisionForestScores(
       const SymbolQualitySignals &, const SymbolRelevanceSignals &, float Base)>
       DecisionForestScorer = &evaluateDecisionForest;

clang-tools-extra/clangd/Quality.cpp

evaluateDecisionForest(const SymbolQualitySignals &Quality,
   E.setHadSymbolType(Relevance.HadSymbolType);
   E.setTypeMatchesPreferred(Relevance.TypeMatchesPreferred);
 
   DecisionForestScores Scores;
   // Exponentiating DecisionForest prediction makes the score of each tree a
   // multiplciative boost (like NameMatch). This allows us to weigh the
   // prediciton score and NameMatch appropriately.
   Scores.ExcludingName = pow(Base, Evaluate(E));
-  // NeedsFixIts is not part of the DecisionForest as generating training
-  // data that needs fixits is not-feasible.
+  // Following cases are not part of the generated training dataset:
+  //  - Symbols with `NeedsFixIts`.
+  //  - Forbidden symbols.
+  //  - Keywords: Dataset contains only macros and decls.
   if (Relevance.NeedsFixIts)
     Scores.ExcludingName *= 0.5;
   if (Relevance.Forbidden)
     Scores.ExcludingName *= 0;
+  if (Quality.Category == SymbolQualitySignals::Keyword)
+    Scores.ExcludingName *= 4;
 
   // NameMatch should be a multiplier on total score to support rescoring.
   Scores.Total = Relevance.NameMatch * Scores.ExcludingName;
   return Scores;
 }
 
 // Produces an integer that sorts in the same order as F.
 // That is: a < b <==> encodeFloat(a) < encodeFloat(b).

clang-tools-extra/clangd/unittests/CodeCompleteTests.cpp

auto Results = completions(
       R"cpp(
           void f() { ns::x^ }
       )cpp",
       {cls("ns::XYZ"), func("ns::foo")});
   EXPECT_THAT(Results.Completions, UnorderedElementsAre(Named("XYZ")));
 }
 
 TEST(CompletionTest, ReferencesAffectRanking) {
-  auto Results = completions("int main() { abs^ }", {ns("absl"), func("absb")});
-  EXPECT_THAT(Results.Completions,
-              HasSubsequence(Named("absb"), Named("absl")));
-  Results = completions("int main() { abs^ }",
-                        {withReferences(10000, ns("absl")), func("absb")});
-  EXPECT_THAT(Results.Completions,
-              HasSubsequence(Named("absl"), Named("absb")));
+  EXPECT_THAT(completions("int main() { abs^ }", {func("absA"), func("absB")})
+                  .Completions,
+              HasSubsequence(Named("absA"), Named("absB")));
+  EXPECT_THAT(completions("int main() { abs^ }",
+                          {func("absA"), withReferences(1000, func("absB"))})
+                  .Completions,
+              HasSubsequence(Named("absB"), Named("absA")));
 }
 
 TEST(CompletionTest, ContextWords) {
   auto Results = completions(R"cpp(
   enum class Color { RED, YELLOW, BLUE };
 
   // (blank lines so the definition above isn't "context")