Wiring of standard library indexing into clangd.

[kuhnel]

Summary

Plumbing code to enable standard library indexing to enable end-to-end testing.
This does not change the stdlib indexing behavior.

I added an experimental flag to enable this feature. This flag would probably be removed (or negated) for the final product.

This is part of https://github.com/clangd/clangd/issues/618 and depends on D105177

[sammccall]

(First just the bug that I suspect is blocking you from testing this, design feedback to come)

clang-tools-extra/clangd/ClangdServer.cpp

184

this pointer will dangle after the if block finishes (the local auto owns the index)

[kuhnel]

fixed lifecycle of StdLibIndex variable

[kuhnel]

clang-tools-extra/clangd/ClangdServer.cpp

180

I'm not sure if returning a Expected<T> makes sense here. I suppose logging could be done from StandardLibraryIndex just as well.

184

Ah yes, you're right. The other indexes are kept in member variables...

[sammccall]

clang-tools-extra/clangd/ClangdServer.cpp

180

A couple of issues with indexing here and placing it in the stack:

1. it's synchronous and blocks all other operations, we shouldn't do that. This is a latency sensitive path: the user just opened the first file in their editor. Our usual trick here is using a SwapIndex as a placeholder and loading asynchronously, but...

2. there's no way to disable it or use a different configuration for e.g. C files, or according to config.

I think ultimately the triggering logic will want to live in TUScheduler so it can be sensitive to CompilerInvocation (and thus LangOpts) as well as config. Essentially it will wrap ParseInputs.Index after calling buildCompilerInvocation.

That's more complicated to get right, and if you want to delay it and still be able to use this locally it makes sense. Some FIXME comments should describe how this needs to be moved before the feature is usable by end-users though.

[kuhnel]

addressed some review comments

question of integration with main functionality still open.

[sammccall]

LG with the understanding that we'll have to move this to be more sophisticated later later, but it's useful to have the simple version now.
Similarly I think it's OK to add the tests later too as this version is experimental & safe.

clang-tools-extra/clangd/ClangdServer.cpp

180

add a FIXME that this happens synchronously, and can't respond to langopts or config and I think we're good to go as an experimental feature

clang-tools-extra/clangd/ClangdServer.h

412

You can push this into MergedIdx for storage. The reason for DynamicIdx and BackgroundIdx having dedicated variables is that they have special APIs we need to access.

clang-tools-extra/clangd/ClangdServer.h

 #include "Hover.h"
 #include "Protocol.h"
 #include "SemanticHighlighting.h"
 #include "TUScheduler.h"
 #include "XRefs.h"
 #include "index/Background.h"
 #include "index/FileIndex.h"
 #include "index/Index.h"
+#include "index/StdLib.h"
 #include "refactor/Rename.h"
 #include "refactor/Tweak.h"
 #include "support/Cancellation.h"
 #include "support/Function.h"
 #include "support/MemoryTree.h"
 #include "support/Path.h"
 #include "support/ThreadsafeFS.h"
 #include "clang/Tooling/CompilationDatabase.h"
struct Options {
     std::vector<std::string> QueryDriverGlobs;
 
     /// Enable preview of FoldingRanges feature.
     bool FoldingRanges = false;
 
     FeatureModuleSet *FeatureModules = nullptr;
 
     explicit operator TUScheduler::Options() const;
+
+    // Automatically index the standard library - experimental feature
+    bool IndexStandardLibrary = false;
   };
+
   // Sensible default options for use in tests.
   // Features like indexing must be enabled if desired.
   static Options optsForTest();
 
   /// Creates a new ClangdServer instance.
   ///
   /// ClangdServer uses \p CDB to obtain compilation arguments for parsing. Note
   /// that ClangdServer only obtains compilation arguments once for each newly
private:
   //   - a merged view of a static and dynamic index (MergedIndex)
   const SymbolIndex *Index = nullptr;
   // If present, an index of symbols in open files. Read via *Index.
   std::unique_ptr<FileIndex> DynamicIdx;
   // If present, the new "auto-index" maintained in background threads.
   std::unique_ptr<BackgroundIndex> BackgroundIdx;
   // Storage for merged views of the various indexes.
   std::vector<std::unique_ptr<SymbolIndex>> MergedIdx;
+  // If present, the index of the standard library.
+  std::unique_ptr<SymbolIndex> StandardLibraryIdx;
 
   // When set, provides clang-tidy options for a specific file.
   TidyProviderRef ClangTidyProvider;
 
   // GUARDED_BY(CachedCompletionFuzzyFindRequestMutex)
   llvm::StringMap<llvm::Optional<FuzzyFindRequest>>
       CachedCompletionFuzzyFindRequestByFile;
   mutable std::mutex CachedCompletionFuzzyFindRequestMutex;

clang-tools-extra/clangd/ClangdServer.cpp

 #include "SemanticHighlighting.h"
 #include "SemanticSelection.h"
 #include "SourceCode.h"
 #include "TUScheduler.h"
 #include "XRefs.h"
 #include "index/CanonicalIncludes.h"
 #include "index/FileIndex.h"
 #include "index/Merge.h"
+#include "index/StdLib.h"
 #include "refactor/Rename.h"
 #include "refactor/Tweak.h"
 #include "support/Logger.h"
 #include "support/Markup.h"
 #include "support/MemoryTree.h"
 #include "support/ThreadsafeFS.h"
 #include "support/Trace.h"
 #include "clang/Format/Format.h"
ClangdServer::ClangdServer(const GlobalCompilationDatabase &CDB,
   auto AddIndex = [&](SymbolIndex *Idx) {
     if (this->Index != nullptr) {
       MergedIdx.push_back(std::make_unique<MergedIndex>(Idx, this->Index));
       this->Index = MergedIdx.back().get();
     } else {
       this->Index = Idx;
     }
   };
+  if (Opts.IndexStandardLibrary) {
+    StandardLibraryIdx = indexStandardLibrary(TFS);
+    if (!StandardLibraryIdx) {
+      elog("Unable to build standard library index. Not using it.");
+    } else {
+      AddIndex(StandardLibraryIdx.get());
+    }
+  }
   if (Opts.StaticIndex)
     AddIndex(Opts.StaticIndex);
   if (Opts.BackgroundIndex) {
     BackgroundIndex::Options BGOpts;
     BGOpts.ThreadPoolSize = std::max(Opts.AsyncThreadsCount, 1u);
     BGOpts.OnProgress = [Callbacks](BackgroundQueue::Stats S) {
       if (Callbacks)
         Callbacks->onBackgroundIndexProgress(S);

clang-tools-extra/clangd/tool/ClangdMain.cpp

 #if defined(__GLIBC__) && CLANGD_MALLOC_TRIM
 opt<bool> EnableMallocTrim{
     "malloc-trim",
     cat(Misc),
     desc("Release memory periodically via malloc_trim(3)."),
     init(true),
 };
 
+opt<bool> IndexStandardLibrary{
+    "index-standard-library",
+    cat(Features),
+    desc("Background index should always include the STL headers even if \n"
+         "not used at the moment. This will enable code completion and \n"
+         "include fixer."
+         "WARNING: This option is experimental only, and will be removed "
+         "eventually. Don't rely on it"),
+    init(false),
+    Hidden,
+};
+
 std::function<void()> getMemoryCleanupFunction() {
   if (!EnableMallocTrim)
     return nullptr;
   // Leave a few MB at the top of the heap: it is insignificant
   // and will most likely be needed by the main thread
   constexpr size_t MallocTrimPad = 20'000'000;
   return []() {
     if (malloc_trim(MallocTrimPad))
if (T.hidden() && !HiddenFeatures)
       return false;
     if (TweakList.getNumOccurrences())
       return llvm::is_contained(TweakList, T.id());
     return true;
   };
   if (ForceOffsetEncoding != OffsetEncoding::UnsupportedEncoding)
     Opts.Encoding = ForceOffsetEncoding;
 
+  Opts.IndexStandardLibrary = IndexStandardLibrary;
   if (CheckFile.getNumOccurrences()) {
     llvm::SmallString<256> Path;
     if (auto Error =
             llvm::sys::fs::real_path(CheckFile, Path, /*expand_tilde=*/true)) {
       elog("Failed to resolve path {0}: {1}", CheckFile, Error.message());
       return 1;
     }
     log("Entering check mode (no LSP server)");