Closed with commit 158af0d3d165c0382a6a291e81ffecf0b18ffe77

Addressed comments.

Fixed output_dir cmake variable. Clean build succeeds now.
Ready to land.

Removed generated (for review) files.

Addressed comments.

Addressed comments.

Fixed namespace ending.

Fixed namespace.

Added README.md for the code completion model.

Do you guys intend to open-source also the training part of the model pipeline ?

Open sourcing the training part (both dataset generation and using an open sourced DecisionForest based framework for training) has been on our radar. Although gathering capacity for this task has been difficult lately.

Addressed comments.

Better formatting in generated files.

Addressed offline comments.

The features refers to the code completion signals in https://github.com/llvm/llvm-project/blob/master/clang-tools-extra/clangd/Quality.h
These signals are currently used to map the code completion candidates to a relevance score using hand-coded heuristics.
We intend to replace the heuristics with a Decision forest model. This patch introduces a dummy model and corresponding runtime that will be used to inference this model.

• Added DerivedSignals struct containing all the derived signals.
• Added NameMatchesContext and proximtiy signals to this struct.
• We need to call computeDerivedSignals() before calling evaluate() if we set non-concrete utilites (e.g. ContextWords and Name).
• This is logically equivalent to the previous version (both when the utilites are explicitly set and when the default signals are used).
• The utilites are not marked as null in computeDerivedSignals. This is due to 2 reasons:
  • Current scoring function checks whether ScopeProximityMatch is set or not to decide whether to multiply with scopeProxitiyScore. Possible solutions:
    • Have scopeProxitiyScore as a derived signal itself.
    • Or have a different derived signal HasScopeProximityMatch.
  • Having these utilities available for debug purposes is great. We can try to compute other derived signals (e.g. ContextMatchesName) and test out it's value without even adding them concretely to clangd. Once their value is justified, we can add it to Quality/Relevance signals.

Thanks. LG.

Mostly looks good. Few nits. Thanks.

Passed score as a float as an explicit argument of the callback.

Addressed comments.

• Populated score in CodeCompletion before invoking the callback.
• Tested that CodeCompletion is scored
• Updated comment for callback.

Remove ununsed import.

Changed to invoke callback on all code completion items.

Addressed linter issues.

Thanks. LG.

Removed repeated check from the test.

Addressed comments.

Added tests.

Removed extra tests.

Addressed comments.

Added another version of handleMacroOccurence to handle MacroReferences from main file.

Updated benchmarks:
Before this patch

Loaded Dex from static-index.idx with estimated memory usage 354410880 bytes
  - number of symbols: 462463
  - number of refs: 6361763
  - numnber of relations: 20322

After this patch
Num refs increases by 5.7%
Disk increase: 2.5%

Loaded Dex from with-macro-xref.idx with estimated memory usage 363390426 bytes
  - number of symbols: 462482
  - number of refs: 6727051
  - numnber of relations: 20325

Addressed comments.

Removed unintended space.

Addressed comments.

Addressed comments.

Explicitly turn off CollectMacro in test.

Before this patch
CollectMacro is disabled by default

Loaded Dex from static-index.idx with estimated memory usage 354410880 bytes
  - number of symbols: 462463
  - number of refs: 6361763
  - numnber of relations: 20322

Before this patch
CollectMacros=true enabled in IndexerMain.cpp
Num symbols increases by 3%
Disk increase: 1.2%

Loaded Dex from no-macro-xref.idx with estimated memory usage 358596403 bytes
  - number of symbols: 476056
  - number of refs: 6361764
  - numnber of relations: 20322

Addressed comments.
Will update the increase in size of index shortly.

Added correct documentation.

Addressed comments.

Addressed comments.

Minor change: Resued variable.

Modified tests for better error messages.

Uploading latest patch