[clangd] Fetch documentation from the Index during signature help

[ilya-biryukov]

Summary

Sema can only be used for documentation in the current file, other doc
comments should be fetched from the index.

[ilya-biryukov]

• run clang-format

[kadircet]

LGTM

[hokein]

looks good, a few nits.

clangd/CodeComplete.cpp

742

nit: do we want to log anything here? It may be useful for debug.

clangd/index/Index.h

65

We already have this similar function in clangd/AST.h.

[ilya-biryukov]

• Log on index request, remove FIXME that was addressed
• Remove SymbolID::forDecl, use existing helper instead

[ioeric]

clangd/CodeComplete.cpp

756

drive by: I think this should be vlog or dlog.

[ilya-biryukov]

clangd/CodeComplete.cpp

742

Definitely useful. Thanks!

clangd/index/Index.h

65

Thanks for pointing this out.
It's somewhat hard to find. WDYT about moving it to Index.h? The concern would probably be a somewhat broken layering, right? E.g. Index.h should not directly depend on anything AST-specific

[hokein]

clangd/index/Index.h

65

Yes, I think we will not add any AST-specific stuff to Index.h, that's why we have AST.h.

[ilya-biryukov]

clangd/CodeComplete.cpp

756

Code completion also logs the number of results from sema, index, etc. using the log() call.
The added log message looks similar, so trying to be consistent with the rest of the code in this file.

Maybe we should turn all of them into vlog or dlog, but I'd rather leave it to a separate patch.

[ioeric]

clangd/CodeComplete.cpp

756

I'm not sure which level code completion log messages should be in, but I think we should follow the guidelines in the logger documentation instead of the existing uses.

Quote from Logger.h

// log() is used for information important to understanding a clangd session.
// e.g. the names of LSP messages sent are logged at this level.
// This level could be enabled in production builds to allow later inspection.

// vlog() is used for details often needed for debugging clangd sessions.
// This level would typically be enabled for clangd developers.

[hokein]

clangd/CodeComplete.cpp

756

My recent experience of debugging some weird GotoDef issues tells me that log of index should be on production (since it is a non-trivial part in a clangd session), it would be really helpful to understand what is going on.

[ioeric]

clangd/CodeComplete.cpp

756

I agree that they are useful for debugging, but they might not be interesting to end-users. And I think that's why there is vlog. Clangd developers could use a different log level with --log flag.

[ilya-biryukov]

clangd/CodeComplete.cpp

756

I agree that vlog is probably a better fit here, but still think we should change it across CodeComplete.cpp in a single commit, rather than partially apply the guidelines to different parts of the file.

However, if everyone agrees we should use vlog here, happy to use vlog too and also send a patch to update all the rest of the usages.
The situation I'm trying to avoid is:

1. We commit vlog here.
2. Someone argues that using log is actually better and we decide to not change other usages to log.
3. We end up with inconsistent choices across this file: vlog for signature help and log for code completion.

But if there's an agreement that we should use vlog everywhere, more than happy to change it :-)

[ioeric]

clangd/CodeComplete.cpp

756

That sounds good to me. Thanks!

clangd/ClangdServer.cpp

WorkScheduler.runWithPreamble("CodeComplete", File,
                                 Bind(Task, File.str(), std::move(CB)));
 }
 
 void ClangdServer::signatureHelp(PathRef File, Position Pos,
                                  Callback<SignatureHelp> CB) {
 
   auto PCHs = this->PCHs;
   auto FS = FSProvider.getFileSystem();
-  auto Action = [Pos, FS, PCHs](Path File, Callback<SignatureHelp> CB,
+  auto *Index = this->Index;
+  auto Action = [Pos, FS, PCHs, Index](Path File, Callback<SignatureHelp> CB,
                                        llvm::Expected<InputsAndPreamble> IP) {
     if (!IP)
       return CB(IP.takeError());
 
     auto PreambleData = IP->Preamble;
     CB(clangd::signatureHelp(File, IP->Command,
                              PreambleData ? &PreambleData->Preamble : nullptr,
-                             IP->Contents, Pos, FS, PCHs));
+                             IP->Contents, Pos, FS, PCHs, Index));
   };
 
   WorkScheduler.runWithPreamble("SignatureHelp", File,
                                 Bind(Action, File.str(), std::move(CB)));
 }
 
 llvm::Expected<tooling::Replacements>
 ClangdServer::formatRange(StringRef Code, PathRef File, Range Rng) {

clangd/CodeComplete.h

CodeCompleteResult codeComplete(PathRef FileName,
                                 PrecompiledPreamble const *Preamble,
                                 const IncludeStructure &PreambleInclusions,
                                 StringRef Contents, Position Pos,
                                 IntrusiveRefCntPtr<vfs::FileSystem> VFS,
                                 std::shared_ptr<PCHContainerOperations> PCHs,
                                 CodeCompleteOptions Opts);
 
 /// Get signature help at a specified \p Pos in \p FileName.
-SignatureHelp signatureHelp(PathRef FileName,
-                            const tooling::CompileCommand &Command,
-                            PrecompiledPreamble const *Preamble,
-                            StringRef Contents, Position Pos,
-                            IntrusiveRefCntPtr<vfs::FileSystem> VFS,
-                            std::shared_ptr<PCHContainerOperations> PCHs);
+SignatureHelp
+signatureHelp(PathRef FileName, const tooling::CompileCommand &Command,
+              PrecompiledPreamble const *Preamble, StringRef Contents,
+              Position Pos, IntrusiveRefCntPtr<vfs::FileSystem> VFS,
+              std::shared_ptr<PCHContainerOperations> PCHs, SymbolIndex *Index);
 
 // For index-based completion, we only consider:
 //   * symbols in namespaces or translation unit scopes (e.g. no class
 //     members, no locals)
 //   * enum constants in unscoped enum decl (e.g. "red" in "enum {red};")
 //   * primary templates (no specializations)
 // For the other cases, we let Clang do the completion because it does not
 // need any non-local information and it will be much better at following
 // lookup rules. Other symbols still appear in the index for other purposes,
 // like workspace/symbols or textDocument/definition, but are not used for code
 // completion.
 bool isIndexedForCodeCompletion(const NamedDecl &ND, ASTContext &ASTCtx);
 } // namespace clangd
 } // namespace clang
 
 #endif

clangd/CodeComplete.cpp

 
 private:
   CodeCompleteOptions Opts;
   std::shared_ptr<GlobalCodeCompletionAllocator> CCAllocator;
   CodeCompletionTUInfo CCTUInfo;
   llvm::unique_function<void()> ResultsCallback;
 };
 
-using ScoredSignature =
-    std::pair<SignatureQualitySignals, SignatureInformation>;
+struct ScoredSignature {
+  // When set, requires documentation to be requested from the index with this
+  // ID.
+  llvm::Optional<SymbolID> IDForDoc;
+  SignatureInformation Signature;
+  SignatureQualitySignals Quality;
+};
 
 class SignatureHelpCollector final : public CodeCompleteConsumer {
-
 public:
   SignatureHelpCollector(const clang::CodeCompleteOptions &CodeCompleteOpts,
-                         SignatureHelp &SigHelp)
-      : CodeCompleteConsumer(CodeCompleteOpts, /*OutputIsBinary=*/false),
+                         SymbolIndex *Index, SignatureHelp &SigHelp)
+      : CodeCompleteConsumer(CodeCompleteOpts,
+                             /*OutputIsBinary=*/false),
         SigHelp(SigHelp),
         Allocator(std::make_shared<clang::GlobalCodeCompletionAllocator>()),
-        CCTUInfo(Allocator) {}
+        CCTUInfo(Allocator), Index(Index) {}
 
   void ProcessOverloadCandidates(Sema &S, unsigned CurrentArg,
                                  OverloadCandidate *Candidates,
                                  unsigned NumCandidates) override {
     std::vector<ScoredSignature> ScoredSignatures;
     SigHelp.signatures.reserve(NumCandidates);
     ScoredSignatures.reserve(NumCandidates);
     // FIXME(rwols): How can we determine the "active overload candidate"?
for (unsigned I = 0; I < NumCandidates; ++I) {
           CurrentArg, S, *Allocator, CCTUInfo, true);
       assert(CCS && "Expected the CodeCompletionString to be non-null");
       // FIXME: for headers, we need to get a comment from the index.
       ScoredSignatures.push_back(processOverloadCandidate(
           Candidate, *CCS,
           getParameterDocComment(S.getASTContext(), Candidate, CurrentArg,
                                  /*CommentsFromHeaders=*/false)));
     }
-    std::sort(ScoredSignatures.begin(), ScoredSignatures.end(),
-              [](const ScoredSignature &L, const ScoredSignature &R) {
-                // Ordering follows:
-                // - Less number of parameters is better.
-                // - Function is better than FunctionType which is better than
-                // Function Template.
-                // - High score is better.
-                // - Shorter signature is better.
-                // - Alphebatically smaller is better.
-                if (L.first.NumberOfParameters != R.first.NumberOfParameters)
-                  return L.first.NumberOfParameters <
-                         R.first.NumberOfParameters;
-                if (L.first.NumberOfOptionalParameters !=
-                    R.first.NumberOfOptionalParameters)
-                  return L.first.NumberOfOptionalParameters <
-                         R.first.NumberOfOptionalParameters;
-                if (L.first.Kind != R.first.Kind) {
-                  using OC = CodeCompleteConsumer::OverloadCandidate;
-                  switch (L.first.Kind) {
-                  case OC::CK_Function:
-                    return true;
-                  case OC::CK_FunctionType:
-                    return R.first.Kind != OC::CK_Function;
-                  case OC::CK_FunctionTemplate:
-                    return false;
-                  }
-                  llvm_unreachable("Unknown overload candidate type.");
-                }
-                if (L.second.label.size() != R.second.label.size())
-                  return L.second.label.size() < R.second.label.size();
-                return L.second.label < R.second.label;
-              });
-    for (const auto &SS : ScoredSignatures)
-      SigHelp.signatures.push_back(SS.second);
+
+    llvm::DenseMap<SymbolID, std::string> FetchedDocs;
+    if (Index) {
+      LookupRequest IndexRequest;
+      for (const auto &S : ScoredSignatures) {
+        if (!S.IDForDoc)
+          continue;
+        IndexRequest.IDs.insert(*S.IDForDoc);
+      }
+      Index->lookup(IndexRequest, [&](const Symbol &S) {
+        if (!S.Detail)
+          return;
+        FetchedDocs[S.ID] = S.Detail->Documentation;
+      });
+    }
+
+    std::sort(
+        ScoredSignatures.begin(), ScoredSignatures.end(),
+          if (L.Quality.NumberOfParameters != R.Quality.NumberOfParameters)
+            return L.Quality.NumberOfParameters < R.Quality.NumberOfParameters;
+          if (L.Quality.NumberOfOptionalParameters !=
+              R.Quality.NumberOfOptionalParameters)
+            return L.Quality.NumberOfOptionalParameters <
+                   R.Quality.NumberOfOptionalParameters;
+          if (L.Quality.Kind != R.Quality.Kind) {
+            switch (L.Quality.Kind) {
+              return R.Quality.Kind != OC::CK_Function;
+          if (L.Signature.label.size() != R.Signature.label.size())
+            return L.Signature.label.size() < R.Signature.label.size();
+          return L.Signature.label < R.Signature.label;
+
+    for (auto &SS : ScoredSignatures) {
+      auto IndexDocIt =
+          SS.IDForDoc ? FetchedDocs.find(*SS.IDForDoc) : FetchedDocs.end();
+      if (IndexDocIt != FetchedDocs.end())
+        SS.Signature.documentation = IndexDocIt->second;
+
+      SigHelp.signatures.push_back(std::move(SS.Signature));
+    }
   }
 
   GlobalCodeCompletionAllocator &getAllocator() override { return *Allocator; }
 
   CodeCompletionTUInfo &getCodeCompletionTUInfo() override { return CCTUInfo; }
 
 private:
   // FIXME(ioeric): consider moving CodeCompletionString logic here to
   // CompletionString.h.
   ScoredSignature processOverloadCandidate(const OverloadCandidate &Candidate,
                                            const CodeCompletionString &CCS,
                                            llvm::StringRef DocComment) const {
-    SignatureInformation Result;
+    SignatureInformation Signature;
     SignatureQualitySignals Signal;
     const char *ReturnType = nullptr;
 
-    Result.documentation = formatDocumentation(CCS, DocComment);
+    Signature.documentation = formatDocumentation(CCS, DocComment);
     Signal.Kind = Candidate.getKind();
 
     for (const auto &Chunk : CCS) {
       switch (Chunk.Kind) {
       case CodeCompletionString::CK_ResultType:
         // A piece of text that describes the type of an entity or,
         // for functions and methods, the return type.
         assert(!ReturnType && "Unexpected CK_ResultType");
         ReturnType = Chunk.Text;
         break;
       case CodeCompletionString::CK_Placeholder:
         // A string that acts as a placeholder for, e.g., a function call
         // argument.
         // Intentional fallthrough here.
       case CodeCompletionString::CK_CurrentParameter: {
         // A piece of text that describes the parameter that corresponds to
         // the code-completion location within a function call, message send,
         // macro invocation, etc.
-        Result.label += Chunk.Text;
+        Signature.label += Chunk.Text;
         ParameterInformation Info;
         Info.label = Chunk.Text;
-        Result.parameters.push_back(std::move(Info));
+        Signature.parameters.push_back(std::move(Info));
         Signal.NumberOfParameters++;
         Signal.ContainsActiveParameter = true;
         break;
       }
       case CodeCompletionString::CK_Optional: {
         // The rest of the parameters are defaulted/optional.
         assert(Chunk.Optional &&
                "Expected the optional code completion string to be non-null.");
-        Result.label +=
-            getOptionalParameters(*Chunk.Optional, Result.parameters, Signal);
+        Signature.label += getOptionalParameters(*Chunk.Optional,
+                                                 Signature.parameters, Signal);
         break;
       }
       case CodeCompletionString::CK_VerticalSpace:
         break;
       default:
-        Result.label += Chunk.Text;
+        Signature.label += Chunk.Text;
         break;
       }
     }
     if (ReturnType) {
-      Result.label += " -> ";
-      Result.label += ReturnType;
+      Signature.label += " -> ";
+      Signature.label += ReturnType;
     }
-    dlog("Signal for {0}: {1}", Result, Signal);
-    return {Signal, Result};
+    dlog("Signal for {0}: {1}", Signature, Signal);
+    ScoredSignature Result;
+    Result.Signature = std::move(Signature);
+    Result.Quality = Signal;
+    Result.IDForDoc =
+        Result.Signature.documentation.empty() && Candidate.getFunction()
+            ? SymbolID::forDecl(*Candidate.getFunction())
+            : llvm::None;
+    return Result;
   }
 
   SignatureHelp &SigHelp;
   std::shared_ptr<clang::GlobalCodeCompletionAllocator> Allocator;
   CodeCompletionTUInfo CCTUInfo;
-
+  const SymbolIndex *Index;
 }; // SignatureHelpCollector
 
 struct SemaCompleteInput {
   PathRef FileName;
   const tooling::CompileCommand &Command;
   PrecompiledPreamble const *Preamble;
   StringRef Contents;
   Position Pos;
return CodeCompleteFlow(FileName, PreambleInclusions, Opts)
       .run({FileName, Command, Preamble, Contents, Pos, VFS, PCHs});
 }
 
 SignatureHelp signatureHelp(PathRef FileName,
                             const tooling::CompileCommand &Command,
                             PrecompiledPreamble const *Preamble,
                             StringRef Contents, Position Pos,
                             IntrusiveRefCntPtr<vfs::FileSystem> VFS,
-                            std::shared_ptr<PCHContainerOperations> PCHs) {
+                            std::shared_ptr<PCHContainerOperations> PCHs,
+                            SymbolIndex *Index) {
   SignatureHelp Result;
   clang::CodeCompleteOptions Options;
   Options.IncludeGlobals = false;
   Options.IncludeMacros = false;
   Options.IncludeCodePatterns = false;
   Options.IncludeBriefComments = false;
   IncludeStructure PreambleInclusions; // Unused for signatureHelp
-  semaCodeComplete(llvm::make_unique<SignatureHelpCollector>(Options, Result),
-                   Options,
-                   {FileName, Command, Preamble, Contents, Pos, std::move(VFS),
-                    std::move(PCHs)});
+  semaCodeComplete(
+      llvm::make_unique<SignatureHelpCollector>(Options, Index, Result),
   return Result;
 }
 
 bool isIndexedForCodeCompletion(const NamedDecl &ND, ASTContext &ASTCtx) {
   using namespace clang::ast_matchers;
   auto InTopLevelScope = hasDeclContext(
       anyOf(namespaceDecl(), translationUnitDecl(), linkageSpecDecl()));
   return !match(decl(anyOf(InTopLevelScope,

clangd/index/Index.h

 #include "llvm/ADT/DenseSet.h"
 #include "llvm/ADT/Hashing.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/StringExtras.h"
 #include <array>
 #include <string>
 
 namespace clang {
+class Decl;
+
 namespace clangd {
 
 struct SymbolLocation {
   // Specify a position (Line, Column) of symbol. Using Line/Column allows us to
   // build LSP responses without reading the file content.
   struct Position {
     uint32_t Line = 0; // 0-based
     // Using UTF-16 code units.

 // As USRs (Unified Symbol Resolution) could be large, especially for functions
 // with long type arguments, SymbolID is using 160-bits SHA1(USR) values to
 // guarantee the uniqueness of symbols while using a relatively small amount of
 // memory (vs storing USRs directly).
 //
 // SymbolID can be used as key in the symbol indexes to lookup the symbol.
 class SymbolID {
 public:
+  static llvm::Optional<SymbolID> forDecl(const Decl &D);
+
   SymbolID() = default;
   explicit SymbolID(llvm::StringRef USR);
 
   bool operator==(const SymbolID &Sym) const {
     return HashValue == Sym.HashValue;
   }
   bool operator<(const SymbolID &Sym) const {
     return HashValue < Sym.HashValue;

clangd/index/Index.cpp

 //===--- Index.cpp -----------------------------------------------*- C++-*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "Index.h"
+#include "clang/Index/USRGeneration.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/Support/SHA1.h"
 #include "llvm/Support/raw_ostream.h"
 
 namespace clang {
 namespace clangd {
 using namespace llvm;
 
 raw_ostream &operator<<(raw_ostream &OS, const SymbolLocation &L) {
   if (!L)
     return OS << "(none)";
   return OS << L.FileURI << "[" << L.Start.Line << ":" << L.Start.Column << "-"
             << L.End.Line << ":" << L.End.Column << ")";
 }
 
+llvm::Optional<SymbolID> SymbolID::forDecl(const Decl &D) {
+  llvm::SmallString<128> USR;
+  if (index::generateUSRForDecl(&D, USR))
+    return llvm::None;
+  return SymbolID(USR);
+}
+
 SymbolID::SymbolID(StringRef USR)
     : HashValue(SHA1::hash(arrayRefFromStringRef(USR))) {}
 
 raw_ostream &operator<<(raw_ostream &OS, const SymbolID &ID) {
   OS << toHex(toStringRef(ID.HashValue));
   return OS;
 }
 

unittests/clangd/CodeCompleteTests.cpp

#if 0
   par^
 #endif
     }
 )cpp");
 
   EXPECT_TRUE(Results.Completions.empty());
 }
 
-SignatureHelp signatures(StringRef Text) {
+SignatureHelp signatures(StringRef Text,
+                         std::vector<Symbol> IndexSymbols = {}) {
+  std::unique_ptr<SymbolIndex> Index;
+  if (!IndexSymbols.empty())
+    Index = memIndex(IndexSymbols);
+
   MockFSProvider FS;
   MockCompilationDatabase CDB;
   IgnoreDiagnostics DiagConsumer;
-  ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
+  ClangdServer::Options Opts = ClangdServer::optsForTest();
+  Opts.StaticIndex = Index.get();
+
+  ClangdServer Server(CDB, FS, DiagConsumer, Opts);
   auto File = testPath("foo.cpp");
   Annotations Test(Text);
   runAddDocument(Server, File, Test.code());
   return cantFail(runSignatureHelp(Server, File, Test.point()));
 }
 
 MATCHER_P(ParamsAre, P, "") {
   if (P.size() != arg.parameters.size())
     return false;
   for (unsigned I = 0; I < P.size(); ++I)
     if (P[I] != arg.parameters[I].label)
       return false;
   return true;
 }
+MATCHER_P(SigDoc, Doc, "") { return arg.documentation == Doc; }
 
 Matcher<SignatureInformation> Sig(std::string Label,
                                   std::vector<std::string> Params) {
   return AllOf(SigHelpLabeled(Label), ParamsAre(Params));
 }
 
 TEST(SignatureHelpTest, Overloads) {
   auto Results = signatures(R"cpp(
EXPECT_THAT(signatures(R"cpp(
     int main() {
       X<int>().foo<double>(^)
     }
   )cpp")
                   .signatures,
               ElementsAre(Sig("foo(T, U) -> void", {"T", "U"})));
 }
 
+TEST(SignatureHelpTest, IndexDocumentation) {
+  Symbol::Details DocDetails;
+  DocDetails.Documentation = "Doc from the index";
+
+  Symbol Foo0 = sym("foo", index::SymbolKind::Function, "@F@\\0#");
+  Foo0.Detail = &DocDetails;
+  Symbol Foo1 = sym("foo", index::SymbolKind::Function, "@F@\\0#I#");
+  Foo1.Detail = &DocDetails;
+  Symbol Foo2 = sym("foo", index::SymbolKind::Function, "@F@\\0#I#I#");
+
+  EXPECT_THAT(
+      signatures(R"cpp(
+    int foo();
+    int foo(double);
+
+    void test() {
+      foo(^);
+    }
+  )cpp",
+                 {Foo0})
+          .signatures,
+      ElementsAre(AllOf(Sig("foo() -> int", {}), SigDoc("Doc from the index")),
+                  AllOf(Sig("foo(double) -> int", {"double"}), SigDoc(""))));
+
+  EXPECT_THAT(
+      signatures(R"cpp(
+    int foo();
+    // Overriden doc from sema
+    int foo(int);
+    // Doc from sema
+    int foo(int, int);
+
+    void test() {
+      foo(^);
+    }
+  )cpp",
+                 {Foo0, Foo1, Foo2})
+          .signatures,
+      ElementsAre(AllOf(Sig("foo() -> int", {}), SigDoc("Doc from the index")),
+                  AllOf(Sig("foo(int) -> int", {"int"}),
+                        SigDoc("Overriden doc from sema")),
+                  AllOf(Sig("foo(int, int) -> int", {"int", "int"}),
+                        SigDoc("Doc from sema"))));
+}
+
 } // namespace
 } // namespace clangd
 } // namespace clang