[clangd] Indexing of standard library

[sammccall]

Summary

This provides a nice "warm start" with all headers indexed, not just
those included so far.

The standard library is indexed after a preamble is parsed, using that
file's configuration. The result is pushed into the dynamic index.
If we later see a higher language version, we reindex it.

It's configurable as Index.StandardLibrary, off by default for now.

Based on D105177 by @kuhnel

Fixes https://github.com/clangd/clangd/issues/618

[sammccall]

remove parts split into other patches

[sammccall]

Tests, polish, comments

[kadircet]

thanks LG, mostly nits and a couple of questions

clang-tools-extra/clangd/ClangdServer.cpp

93

I suppose this should be rare hence won't bite us in practice, but might worth having a comment mentioning this creates tasks with no barriers.

clang-tools-extra/clangd/index/StdLib.cpp

45

llvm_uncreachable instead?

50

nit: this feels a little bit hard to read, what about:

if(2b) return 2b;
if(20) return 20;
...
return 98;

57

same here

93

maybe move second half of this comment into buildUmbrella ?

122

s/our our/to our/

132

drop static, and move into previous anonymous namespace ?

143

s/Name/Header/ ?

161

StandardHeaders are always in verbatim format, but are we sure if that holds for the IncludeHeader ?

I suppose that should be the case since we map known path suffixes back to a verbatim umbrella header, I just wonder how exhaustive the list is. (it might be nice to manually check no implementation detail headers falls out of cracks)

178

seems like debugging artifact, either drop or put in #ifndef NDEBUG

205

why log if we think we can't hit this case?

217

why are we doing this exactly? once we override the same file multiple times, I believe the last one takes precedence. it's definitely safer to clear the remapped files here rather than relying on that fact, but I am afraid of losing other mappings, possibly set outside of clangd's knowledge (e.g. through flags like -remap-file?)

234

what does the location refer to here? I think we should also stress the fact that even when indexing the same file, we have a good chance of seeing different symbols due to PP directives (and different std versions)

238

s/containers/include graph

257

i'd make this part of the next log

300

maybe move this check to the top

314

why do we resolve the symlinks ?

324

any reason for going with Noncached version? (clangd doesn't set one up today, but not relying on that would be nice if we don't have a particular concern here)

330

s/bust/must

clang-tools-extra/clangd/index/StdLib.h

67

maybe drop the optinal and bail out in indexing when Paths are empty ?

69

s/a built/built an/

[sammccall]

address review comments

[sammccall]

(sorry about the long delay, would still love to merge this)

clang-tools-extra/clangd/index/StdLib.cpp

93

I think that removes the context of why we're #including them.

161

Right, I looked at these manually and the headers (and symbols) we were dropping seemed reasonable.

Note that we're not filtering symbols in this loop, just building a list of blessed files to add to the StdLibLocation.

So we only drop symbols that:

• aren't recognized by the indexer as being in the standard library
• aren't in the standard library directory
• don't share a file with anything recognized as being in the standard library

178

This is useful for debugging, and fits well with the other dlog()s in this file, I'd like to check it in.

I was going to call you paranoid, being sure this would get compiled out.
but indeed not: https://godbolt.org/z/hKWhro3Mv (gcc manages it)

Added #ifndef NDEBUG

205

Because I'm not certain, and would much rather get a user bug report with this log line in it than without!

(Assert because I'd most rather find out before release of course)

217

We map dirty buffers ourselves. Conceivably, it may be part of the standard library itself that was remapped to some dirty buffer content!

We're reusing the CompilerInvocation from building a preamble, where we remap the main-file buffer to the dirty contents. (We do this in prepareCompilerInstance, but the PPOptions are shared).
This isn't part of the "compilerinvocation-as-proxy-for-build-flags" that we're trying to index.

I don't think it's a realistic possibility that anyone would rely on -Xclang -remap-file to find the standard library (note that it's a cc1 flag, not a public one...).

234

what does the location refer to here?

It refers to the StdLibLocation Loc, made that explicit.

I think we should also stress the fact that even when indexing the same file, we have a good chance of seeing different symbols due to PP directives (and different std versions)

Different than what? Do you mean "why might different calls to indexStandardLibrary see different symbols" from the same file?

257

Can you say why? I generally like one thought per line. Scanning vertically through familiar lines, it's easy to miss something unfamiliar tacked onto the end. This message should be rare, and log lines aren't precious.

(I reordered them, which seems a bit more natural)

314

Oops, because I read the documentation of getCanonicalPath() instead of the implementation, and they diverged in https://github.com/llvm/llvm-project/commit/dd67793c0c549638cd93cad1142d324b979ba682 :-D

Ultimately we're forming URIs to lexically compare with the ones emitted by the indexer, which uses getCanonicalPath(). Today getCanonicalPath() wants a FileEntry and we don't have one, but I think there's no fundamental reason for that, I can fix it.

(I'll do it as a separate patch, for now it's just calling getCanonicalPath with the wrong arguments)

324

Because the cached version is more complicated with no benefits:

• any entanglement between the IO of the preamble indexing and that of the translation unit that happened to trigger it seems like a complicated idea, that's worth understanding before doing
• as you say, we don't actually install a statcache, so there's no concrete benefit
• in fact there exists no caching implementation of FileSystemStatCache, so the idea that we might be able to implement that interface and gain benefits is *extremely* speculative

clang-tools-extra/clangd/index/StdLib.h

67

Why? This would definitely be using an empty vector as a sentinel value:

• 2 paths -> index
• 1 path -> index
• 0 paths -> don't index

And it's not as if "probe for a standard library" is the main point of this function so the interpretation of the return value is obvious - that's only one of three criteria.

None seems to be a clearer way to communicate this than {}, and performance doesn't seem to be an issue here.

[sammccall]

Revert StdLibLocation to realpath, document why

[sammccall]

revert to previous version of realpath code

clang-tools-extra/clangd/index/StdLib.cpp

314

Actually, nevermind, the code is correct and I'd just forgotten why :-) Added a comment to StdLibLocation.

getCanonicalPath() does actually resolve symlinks and so on: it asks the FileManager for the directory entry and grabs the its "canonical name" which is just FS->getRealPath behind a cache.
So the strings are going to match the indexer after all.

It'd be possible to modify getCanonicalPath() so we can call it here, but I don't think it helps. Calling it with (path, filemanager) is inconvenient for the (many) existing callsites, so it'd have to be a new overload just for this case. And the FileManager caching we'd gain doesn't matter here.
I can still do it if you like, though.

(Also, relevant to your interests, realpath is probably taking care of case mismatches too!)

[sammccall]

This had a "LG" comment above... want to take another pass?
(Not urgent, just checking)

[kadircet]

sorry for the long turn around here, LGTM. let's ship it!

clang-tools-extra/clangd/index/StdLib.cpp

234

Different than what? Do you mean "why might different calls to indexStandardLibrary see different symbols" from the same file?

yes, i meant compared to a previous runs. but i don't think it's as relevant here. i believe i was thinking about caching indexing status across runs and using that cache to implement filefilter, so that we don't index the same file twice (as we normally do in bgindex).

257

i was rather implying to add it as a (in)complete field into the current log line you have. usually when clangd is printing lots of logs across threads it might be hard to correlate these. hence having them printed as a single log would help.

314

So the strings are going to match the indexer after all.

thanks, this makes sense.

It'd be possible to modify getCanonicalPath() so we can call it here, but I don't think it helps. Calling it with (path, filemanager) is inconvenient for the (many) existing callsites, so it'd have to be a new overload just for this case. And the FileManager caching we'd gain doesn't matter here.
I can still do it if you like, though.

No need. We can take a look at that if the logic is likely to change (or get more complicated) in the future.

clang-tools-extra/clangd/index/StdLib.h

67

okay, makes sense.

[sammccall]

Address comments
Add end-to-end test
Move ownership of AsyncTaskRunner to allow blockUntilIdle() in test
Fix bugs caught by end-to-end-test

[sammccall]

fix HasSubstr matcher type issue

[sammccall]

Hmm, the test keeps crashing on the GN bot: http://45.33.8.238/win/58316/step_9.txt
Unfortunately the stacktrace is not symbolized, and I'm not seeing this elsewhere (e.g. premerge bot).

@thakis, any idea why unittests no longer manage to symbolize stack traces on crash on the windows bot? I believe this used to work...

[thakis]

In D115232#3520461, @sammccall wrote:

Hmm, the test keeps crashing on the GN bot: http://45.33.8.238/win/58316/step_9.txt
Unfortunately the stacktrace is not symbolized, and I'm not seeing this elsewhere (e.g. premerge bot).

@thakis, any idea why unittests no longer manage to symbolize stack traces on crash on the windows bot? I believe this used to work...

I do not know. Maybe related to the "run many unit tests in a single process" lit change from a month ago?

Anyways, looks like this relanded and broke tests yet again. Maybe find a win box before relanding the next time?

[sammccall]

In D115232#3522514, @thakis wrote:

In D115232#3520461, @sammccall wrote:

Hmm, the test keeps crashing on the GN bot: http://45.33.8.238/win/58316/step_9.txt
Unfortunately the stacktrace is not symbolized, and I'm not seeing this elsewhere (e.g. premerge bot).

@thakis, any idea why unittests no longer manage to symbolize stack traces on crash on the windows bot? I believe this used to work...

I do not know. Maybe related to the "run many unit tests in a single process" lit change from a month ago?

I suspected that, and verified locally that:

• llvm-symbolizer on PATH still works
• LLVM_SYMBOLIZER_PATH env variable didn't work, but I fixed it in 1236b66a98197109ed40141329d6056dfbe25967 along with this reland, still no dice.

Anyways, looks like this relanded and broke tests yet again. Maybe find a win box before relanding the next time?

I found one, but the test doesn't crash (nor on the premerge bots).

I'll revert again, but I have no idea how to proceed. Only this bot and llvm-avr-linux show the failure, and neither of them have a working symbolizer.

[thakis]

In D115232#3522571, @sammccall wrote:

In D115232#3522514, @thakis wrote:

In D115232#3520461, @sammccall wrote:

Hmm, the test keeps crashing on the GN bot: http://45.33.8.238/win/58316/step_9.txt
Unfortunately the stacktrace is not symbolized, and I'm not seeing this elsewhere (e.g. premerge bot).

@thakis, any idea why unittests no longer manage to symbolize stack traces on crash on the windows bot? I believe this used to work...

I do not know. Maybe related to the "run many unit tests in a single process" lit change from a month ago?

I suspected that, and verified locally that:

• llvm-symbolizer on PATH still works
• LLVM_SYMBOLIZER_PATH env variable didn't work, but I fixed it in 1236b66a98197109ed40141329d6056dfbe25967 along with this reland, still no dice.

(My bot neither has llvm-symbolizer on path, nor sets LLVM_SYMBOLIZER_PATH fwiw.)

Anyways, looks like this relanded and broke tests yet again. Maybe find a win box before relanding the next time?

I found one, but the test doesn't crash (nor on the premerge bots).

I'll revert again, but I have no idea how to proceed. Only this bot and llvm-avr-linux show the failure, and neither of them have a working symbolizer.

I wouldn't be super surprised if this is related to windows and delayed template parsing. I haven't found any bots on llvm's official waterfall that run ClangdTests.exe – maybe that's why there aren't more bots finding this?

I'd recommend building and running the test on a win box and see if it repros locally.

[sammccall]

I managed to get a stack trace from a bot (by leaving the broken commit up for longer this time).

In D115232#3522598, @thakis wrote:

In D115232#3522571, @sammccall wrote:

In D115232#3522514, @thakis wrote:

In D115232#3520461, @sammccall wrote:

Hmm, the test keeps crashing on the GN bot: http://45.33.8.238/win/58316/step_9.txt
Unfortunately the stacktrace is not symbolized, and I'm not seeing this elsewhere (e.g. premerge bot).

@thakis, any idea why unittests no longer manage to symbolize stack traces on crash on the windows bot? I believe this used to work...

I do not know. Maybe related to the "run many unit tests in a single process" lit change from a month ago?

I suspected that, and verified locally that:

• llvm-symbolizer on PATH still works
• LLVM_SYMBOLIZER_PATH env variable didn't work, but I fixed it in 1236b66a98197109ed40141329d6056dfbe25967 along with this reland, still no dice.

(My bot neither has llvm-symbolizer on path, nor sets LLVM_SYMBOLIZER_PATH fwiw.)

Is this something you could add? I'd much rather revert quickly when after seeing a problem than wait for the slower official bots to catch it, but the stacktrace is pretty key.

Anyways, looks like this relanded and broke tests yet again. Maybe find a win box before relanding the next time?

I found one, but the test doesn't crash (nor on the premerge bots).

I'll revert again, but I have no idea how to proceed. Only this bot and llvm-avr-linux show the failure, and neither of them have a working symbolizer.

I wouldn't be super surprised if this is related to windows and delayed template parsing.

It seems possible. The other other failure I'd seen was the avr-linux bot (not windows).
However it definitely passes in some configurations: both the premerge tests and my local build.

I haven't found any bots on llvm's official waterfall that run ClangdTests.exe – maybe that's why there aren't more bots finding this?

clang-x64-windows-msvc does, but it's slow.

I'd recommend building and running the test on a win box and see if it repros locally.

Again, I have done this, and cannot reproduce (on 19.29.30038.1).

Now I have some idea where the crash is I can try some blind fixes, though.

[sammccall]

Landed finally as 03ea140b3a285c9a4400ee007b1790b110cbf984

[bnbarham]

clang-tools-extra/clangd/ClangdServer.cpp

982–984

@sammccall shouldn't we also be waiting for this to finish when ClangdServer is destroyed? IIUC right now the both FileIndex itself (stored in ClangdServer) and the actual UpdateIndexCallbacks (stored in TUScheduler) can be freed while indexStdlib is running asynchronously, resulting in a use-after-free on eg. FIndex->updatePreamble(std::move(IF)). I was confused as to why this wasn't happening in the tests, but these lines would explain it 😅

Adding a IndexTasks->wait() to ~ClangdServer fixes the crash I'm seeing in the sourcekit-lsp tests (see https://github.com/apple/llvm-project/pull/5837), though ideally we (sourcekit-lsp) wouldn't be running any indexing at all. As far as I can tell there's no way to turn off dynamic indexing now though, except for StandardLibrary indexing through the config file (but not from clangd args)?

[sammccall]

clang-tools-extra/clangd/ClangdServer.cpp

982–984

Thanks for flagging this!

We *almost* have the sequencing we need in ~ClangdServer:

• when we fall off the end of ~ClangdServer it destroys all its members
• ClangdServer::IndexTasks is declared after FIndex, so is destroyed first
• ~AsyncTaskRunner calls wait()

But the task we schedule on IndexTasks captures a ref to UpdateIndexCallbacks, which is owned by the TUScheduler, which we explicitly destroy at the beginning of ~ClangdServer.

However I think your patch is *also* not quite correct: we can wait for the tasks to be empty, but then the TUScheduler could fill it up again before we destroy TUScheduler.

Options include adding an explicit stop() to TUScheduler, changing TUScheduler to not (exclusively) own UpdateIndexCallbacks, or have the task not capture the callbacks by reference.
I'll try the latter first, which seems least invasive.

---

ideally we (sourcekit-lsp) wouldn't be running any indexing at all. As far as I can tell there's no way to turn off dynamic indexing now though, except for StandardLibrary indexing through the config file (but not from clangd args)?

Clangd won't provide any top-level/namespace-level completions at all without dynamic index (index of preambles), and various other features won't work (docs on hover, include-fixer, type/call-hierarchy). We dropped support for disabling this at some point, as it didn't really seem usable and made features more complex if we tried to accommodate it. At a technical level it would be possible to disable I think, but I'd be really surprised if completion worked well, or if a language server without completion was useful.

StandardLibrary indexing through the config file (but not from clangd args)

We've tried to move away from flags for options that are interesting to users, as config files are more flexible, more forgiving on errors, and allow different settings per-project in a consistent way. (We don't own the editors, so cross-editor consistency is important to being able to support users at all...)

I can see how requiring config to be materialized on disk could be inconvenient for IDEs though. I think we could add a general-purpose --config-inline=<YAML/JSON goes here> flag, and/or the ability to set config over LSP (this can be dynamic, accordingly bigger design space that might be hard to get right).

[bnbarham]

clang-tools-extra/clangd/ClangdServer.cpp

982–984

Ah, I didn't actually check AsyncTaskRunner. Makes sense it would wait though :). Thanks for looking into this in detail!

or have the task not capture the callbacks by reference. I'll try the latter first, which seems least invasive.

This + moving FIndex after IndexTasks seems reasonable to me.

Clangd won't provide any top-level/namespace-level completions at all without dynamic index (index of preambles), and various other features won't work (docs on hover, include-fixer, type/call-hierarchy).

That's good to know - I assume this extends to indexing the stdlib as well, ie. the stdlib would be missing from top/namespace level completion if not indexed? Does the dynamic index grow with every opened file, or is it just the currently opened file? If disabling breaks everything it's not something we'd want to do, we just don't need it for find refs/etc.

[sammccall]

clang-tools-extra/clangd/ClangdServer.cpp

982–984

This + moving FIndex after IndexTasks seems reasonable to me.

I sent D140486. FIndex should be before IndexTasks in order to outlive it, unless I'm missing something.

I assume this extends to indexing the stdlib as well, ie. the stdlib would be missing from top/namespace level completion if not indexed?

Any parts of the standard library that you haven't (transitively) included from an open file would be missing. In particular, if you start up clangd, open a blank file, and type std:: you'll get nothing.

Does the dynamic index grow with every opened file, or is it just the currently opened file?

It contains symbols for all the transitively included headers of every file you've had open. So it does grow for each opened file, but in practice usually only by a little bit: if you've opened three files usually >90% of the headers visible from the fourth file are already in the index.

[bnbarham]

clang-tools-extra/clangd/ClangdServer.cpp

982–984

I sent D140486. FIndex should be before IndexTasks in order to outlive it, unless I'm missing something.

Thanks! No, that's right. And it is already so 👍.

And thanks for the extra information too.

clang-tools-extra/clangd/AST.h

 //===--- AST.h - Utility AST functions  -------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Various code that examines C++ source code using AST.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_AST_H_
 #define LLVM_CLANG_TOOLS_EXTRA_CLANGD_AST_H_
 
+#include "SourceCode.h"
 #include "index/SymbolID.h"
 #include "clang/AST/Decl.h"
 #include "clang/AST/DeclObjC.h"
 #include "clang/AST/NestedNameSpecifier.h"
 #include "clang/Basic/SourceLocation.h"
 #include "clang/Lex/MacroInfo.h"
 #include "llvm/ADT/DenseSet.h"
 #include "llvm/ADT/StringRef.h"
 #include <string>
 #include <vector>
 
 namespace clang {
 class SourceManager;
 class Decl;
 class DynTypedNode;
 
 namespace clangd {
 
 /// Returns true if the declaration is considered implementation detail based on
 /// heuristics. For example, a declaration whose name is not explicitly spelled
 /// in code is considered implementation detail.
 bool isImplementationDetail(const Decl *D);
 
+/// True if the decl has a reserved name like __Vector_base.
+inline bool hasReservedName(const NamedDecl &ND) {
+  if (ND.getDeclName().isIdentifier() && isReservedIdentifier(ND.getName()))
+    llvm::errs() << ND.getName() << " dropped\n";
+  return ND.getDeclName().isIdentifier() && isReservedIdentifier(ND.getName());
+}
+/// True if the DeclContext would be specified with a reserved name.
+/// e.g. std::__Vector_base::iterator
+bool hasReservedScopeName(const DeclContext &DC);
+
 /// Find the source location of the identifier for \p D.
 /// Transforms macro locations to locations spelled inside files. All code
 /// that needs locations of declaration names (e.g. the index) should go through
 /// this function.
 SourceLocation nameLocation(const clang::Decl &D, const SourceManager &SM);
 
 /// Returns the qualified name of ND. The scope doesn't contain unwritten scopes
 /// like inline namespaces.

clang-tools-extra/clangd/AST.cpp

bool isExplicitTemplateSpecialization(const NamedDecl *D) {
   return isTemplateSpecializationKind(D, TSK_ExplicitSpecialization);
 }
 
 bool isImplementationDetail(const Decl *D) {
   return !isSpelledInSource(D->getLocation(),
                             D->getASTContext().getSourceManager());
 }
 
+bool hasReservedScopeName(const DeclContext &DC) {
+  bool IsLocallyReserved = [&] {
+    if (const auto *NS = llvm::dyn_cast<NamespaceDecl>(&DC))
+      if (const NamedDecl *ND = llvm::dyn_cast<NamedDecl>(&DC)) {
+        if (DC.isTransparentContext())
+          return false;
+        if (NS->isInlineNamespace())
+          return false;
+        return hasReservedName(*ND);
+      }
+    return false;
+  }();
+  if (IsLocallyReserved)
+    return true;
+  if (DC.getParent() == nullptr)
+    return false;
+  return hasReservedScopeName(*DC.getParent());
+}
+
 SourceLocation nameLocation(const clang::Decl &D, const SourceManager &SM) {
   auto L = D.getLocation();
   if (isSpelledInSource(L, SM))
     return SM.getSpellingLoc(L);
   return SM.getExpansionLoc(L);
 }
 
 std::string printQualifiedName(const NamedDecl &ND) {

clang-tools-extra/clangd/CMakeLists.txt

add_clang_library(clangDaemon
   index/Index.cpp
   index/IndexAction.cpp
   index/MemIndex.cpp
   index/Merge.cpp
   index/ProjectAware.cpp
   index/Ref.cpp
   index/Relation.cpp
   index/Serialization.cpp
+  index/StdLib.cpp
   index/Symbol.cpp
   index/SymbolCollector.cpp
   index/SymbolID.cpp
   index/SymbolLocation.cpp
   index/SymbolOrigin.cpp
   index/YAMLSerialization.cpp
 
   index/dex/Dex.cpp

clang-tools-extra/clangd/ClangdServer.cpp

 //===--- ClangdServer.cpp - Main clangd server code --------------*- C++-*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===-------------------------------------------------------------------===//
 
 #include "ClangdServer.h"
 #include "CodeComplete.h"
 #include "Config.h"
 #include "Diagnostics.h"
 #include "DumpAST.h"
 #include "FindSymbols.h"
 #include "Format.h"
 #include "HeaderSourceSwitch.h"
-#include "Headers.h"
 #include "InlayHints.h"
 #include "ParsedAST.h"
 #include "Preamble.h"
 #include "Protocol.h"
 #include "SemanticHighlighting.h"
 #include "SemanticSelection.h"
 #include "SourceCode.h"
 #include "TUScheduler.h"
 #include "XRefs.h"
 #include "index/CanonicalIncludes.h"
 #include "index/FileIndex.h"
 #include "index/Merge.h"
+#include "index/StdLib.h"
 #include "refactor/Rename.h"
 #include "refactor/Tweak.h"
 #include "support/Logger.h"
-#include "support/Markup.h"
 #include "support/MemoryTree.h"
 #include "support/ThreadsafeFS.h"
 #include "support/Trace.h"
 #include "clang/Format/Format.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/CompilerInvocation.h"
 #include "clang/Lex/Preprocessor.h"
 #include "clang/Tooling/CompilationDatabase.h"
 #include "clang/Tooling/Core/Replacement.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/STLExtras.h"
-#include "llvm/ADT/ScopeExit.h"
-#include "llvm/ADT/StringExtras.h"
 #include "llvm/ADT/StringRef.h"
-#include "llvm/Support/Errc.h"
 #include "llvm/Support/Error.h"
-#include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Path.h"
-#include "llvm/Support/ScopedPrinter.h"
 #include "llvm/Support/raw_ostream.h"
 #include <algorithm>
 #include <chrono>
 #include <future>
 #include <memory>
 #include <mutex>
 #include <string>
 #include <type_traits>
+#include <unordered_set>
 
 namespace clang {
 namespace clangd {
 namespace {
 
 // Update the FileIndex with new ASTs and plumb the diagnostics responses.
 struct UpdateIndexCallbacks : public ParsingCallbacks {
   UpdateIndexCallbacks(FileIndex *FIndex,
-                       ClangdServer::Callbacks *ServerCallbacks)
-      : FIndex(FIndex), ServerCallbacks(ServerCallbacks) {}
+                       ClangdServer::Callbacks *ServerCallbacks,
+                       const ThreadsafeFS &TFS, bool Sync)
+      : FIndex(FIndex), ServerCallbacks(ServerCallbacks), TFS(TFS) {
+    if (!Sync)
+      Tasks.emplace();
+  }
 
-  void onPreambleAST(PathRef Path, llvm::StringRef Version, ASTContext &Ctx,
+  void onPreambleAST(PathRef Path, llvm::StringRef Version,
+                     const CompilerInvocation &CI, ASTContext &Ctx,
                      std::shared_ptr<clang::Preprocessor> PP,
                      const CanonicalIncludes &CanonIncludes) override {
+    // If this preamble uses a standard library we haven't seen yet, index it.
+    if (auto Loc = Stdlib.add(*CI.getLangOpts(), PP->getHeaderSearchInfo()))
+      indexStdlib(CI, std::move(*Loc));
+
     if (FIndex)
       FIndex->updatePreamble(Path, Version, Ctx, std::move(PP), CanonIncludes);
   }
 
+  void indexStdlib(const CompilerInvocation &CI, StdLibLocation Loc) {
+    auto Task = [this, LO(*CI.getLangOpts()), Loc(std::move(Loc)),
+                 CI(std::make_unique<CompilerInvocation>(CI))]() mutable {
+      IndexFileIn IF;
+      IF.Symbols = indexStandardLibrary(std::move(CI), Loc, TFS);
+      if (Stdlib.isBest(LO))
+        FIndex->updatePreamble(std::move(IF));
+    };
+    if (Tasks)
+      Tasks->runAsync("IndexStdlib", std::move(Task));
+    else
+      Task();
+  }
+
   void onMainAST(PathRef Path, ParsedAST &AST, PublishFn Publish) override {
     if (FIndex)
       FIndex->updateMain(Path, AST);
 
     assert(AST.getDiagnostics().hasValue() &&
            "We issue callback only with fresh preambles");
     std::vector<Diag> Diagnostics = *AST.getDiagnostics();
     if (ServerCallbacks)
struct UpdateIndexCallbacks : public ParsingCallbacks {
   void onPreamblePublished(PathRef File) override {
     if (ServerCallbacks)
       ServerCallbacks->onSemanticsMaybeChanged(File);
   }
 
 private:
   FileIndex *FIndex;
   ClangdServer::Callbacks *ServerCallbacks;
+  const ThreadsafeFS &TFS;
+  StdLibSet Stdlib;
+  llvm::Optional<AsyncTaskRunner> Tasks;
 };
 
 class DraftStoreFS : public ThreadsafeFS {
 public:
   DraftStoreFS(const ThreadsafeFS &Base, const DraftStore &Drafts)
       : Base(Base), DirtyFiles(Drafts) {}
 
 private:
: FeatureModules(Opts.FeatureModules), CDB(CDB), TFS(TFS),
       ClangTidyProvider(Opts.ClangTidyProvider),
       WorkspaceRoot(Opts.WorkspaceRoot),
       Transient(Opts.ImplicitCancellation ? TUScheduler::InvalidateOnUpdate
                                           : TUScheduler::NoInvalidation),
       DirtyFS(std::make_unique<DraftStoreFS>(TFS, DraftMgr)) {
   // Pass a callback into `WorkScheduler` to extract symbols from a newly
   // parsed file and rebuild the file index synchronously each time an AST
   // is parsed.
-  WorkScheduler.emplace(
-      CDB, TUScheduler::Options(Opts),
-      std::make_unique<UpdateIndexCallbacks>(DynamicIdx.get(), Callbacks));
+  WorkScheduler.emplace(CDB, TUScheduler::Options(Opts),
+                        std::make_unique<UpdateIndexCallbacks>(
+                            DynamicIdx.get(), Callbacks, TFS,
+                            /*Sync=*/Opts.AsyncThreadsCount == 0));
   // Adds an index to the stack, at higher priority than existing indexes.
   auto AddIndex = [&](SymbolIndex *Idx) {
     if (this->Index != nullptr) {
       MergedIdx.push_back(std::make_unique<MergedIndex>(Idx, this->Index));
       this->Index = MergedIdx.back().get();
     } else {
       this->Index = Idx;
     }
void ClangdServer::getAST(PathRef File, llvm::Optional<Range> R,
   auto Action =
       [R, CB(std::move(CB))](llvm::Expected<InputsAndAST> Inputs) mutable {
         if (!Inputs)
           return CB(Inputs.takeError());
         if (!R) {
           // It's safe to pass in the TU, as dumpAST() does not
           // deserialize the preamble.
           auto Node = DynTypedNode::create(
-                *Inputs->AST.getASTContext().getTranslationUnitDecl());
           return CB(dumpAST(Node, Inputs->AST.getTokens(),
                             Inputs->AST.getASTContext()));
         }
         unsigned Start, End;
         if (auto Offset = positionToOffset(Inputs->Inputs.Contents, R->start))
           Start = *Offset;
         else
           return CB(Offset.takeError());

 ClangdServer::blockUntilIdleForTest(llvm::Optional<double> TimeoutSeconds) {
   // Order is important here: we don't want to block on A and then B,
   // if B might schedule work on A.
 
   // Nothing else can schedule work on TUScheduler, because it's not threadsafe
   // and we're blocking the main thread.
   if (!WorkScheduler->blockUntilIdle(timeoutSeconds(TimeoutSeconds)))
     return false;
 
   // Unfortunately we don't have strict topological order between the rest of
   // the components. E.g. CDB broadcast triggers backrgound indexing.
   // This queries the CDB which may discover new work if disk has changed.
   //
   // So try each one a few times in a loop.
   // If there are no tricky interactions then all after the first are no-ops.
   // Then on the last iteration, verify they're idle without waiting.
   //
   // There's a small chance they're juggling work and we didn't catch them :-(
   for (llvm::Optional<double> Timeout :

clang-tools-extra/clangd/Config.h

struct ExternalIndexSpec {
     /// This is one of:
     /// - Address of a clangd-index-server, in the form of "ip:port".
     /// - Absolute path to an index produced by clangd-indexer.
     std::string Location;
     /// Absolute path to source root this index is associated with, uses
     /// forward-slashes.
     std::string MountPoint;
   };
-  /// Controls background-index behavior.
+  /// Controls index behavior.
   struct {
-    /// Whether this TU should be indexed.
+    /// Whether this TU should be background-indexed.
     BackgroundPolicy Background = BackgroundPolicy::Build;
     ExternalIndexSpec External;
+    bool StandardLibrary = false;
   } Index;
 
   enum UnusedIncludesPolicy { Strict, None };
   /// Controls warnings and errors when parsing code.
   struct {
     bool SuppressAll = false;
     llvm::StringSet<> Suppress;
 

clang-tools-extra/clangd/ConfigCompile.cpp

if (F.Background) {
                          .map("Build", Config::BackgroundPolicy::Build)
                          .map("Skip", Config::BackgroundPolicy::Skip)
                          .value())
         Out.Apply.push_back(
             [Val](const Params &, Config &C) { C.Index.Background = *Val; });
     }
     if (F.External)
       compile(std::move(**F.External), F.External->Range);
+    if (F.StandardLibrary)
+      Out.Apply.push_back(
+          [Val(**F.StandardLibrary)](const Params &, Config &C) {
+            C.Index.StandardLibrary = Val;
+          });
   }
 
   void compile(Fragment::IndexBlock::ExternalBlock &&External,
                llvm::SMRange BlockRange) {
     if (External.Server && !Trusted) {
       diag(Error,
            "Remote index may not be specified by untrusted configuration. "
            "Copy this into user config to use it.",

clang-tools-extra/clangd/ConfigFragment.h

struct ExternalBlock {
       /// `123.1.1.1:13337`.
       llvm::Optional<Located<std::string>> Server;
       /// Source root governed by this index. Default is the directory
       /// associated with the config fragment. Absolute in case of user config
       /// and relative otherwise. Should always use forward-slashes.
       llvm::Optional<Located<std::string>> MountPoint;
     };
     llvm::Optional<Located<ExternalBlock>> External;
+    // Whether the standard library visible from this file should be indexed.
+    // This makes all standard library symbols available, included or not.
+    llvm::Optional<Located<bool>> StandardLibrary;
   };
   IndexBlock Index;
 
   /// Controls behavior of diagnostics (errors and warnings).
   struct DiagnosticsBlock {
     /// Diagnostic codes that should be suppressed.
     ///
     /// Valid values are:

clang-tools-extra/clangd/ConfigYAML.cpp

Dict.handle("External", [&](Node &N) {
         parse(External, scalarValue(N, "External").getValue());
       } else {
         error("External must be either a scalar or a mapping.", N);
         return;
       }
       F.External.emplace(std::move(External));
       F.External->Range = N.getSourceRange();
     });
+    Dict.handle("StandardLibrary", [&](Node &N) {
+      if (auto StandardLibrary = boolValue(N, "StandardLibrary"))
+        F.StandardLibrary = *StandardLibrary;
+    });
     Dict.parse(N);
   }
 
   void parse(Fragment::IndexBlock::ExternalBlock &F,
              Located<std::string> ExternalVal) {
     if (!llvm::StringRef(*ExternalVal).equals_insensitive("none")) {
       error("Only scalar value supported for External is 'None'",
             ExternalVal.Range);
void parse(Fragment::IndexBlock::ExternalBlock &F, Node &N) {
     Dict.handle("MountPoint",
                 [&](Node &N) { F.MountPoint = scalarValue(N, "MountPoint"); });
     Dict.parse(N);
   }
 
   void parse(Fragment::CompletionBlock &F, Node &N) {
     DictParser Dict("Completion", this);
     Dict.handle("AllScopes", [&](Node &N) {
-      if (auto Value = scalarValue(N, "AllScopes")) {
-        if (auto AllScopes = llvm::yaml::parseBool(**Value))
-          F.AllScopes = *AllScopes;
-        else
-          warning("AllScopes should be a boolean", N);
-      }
+      if (auto AllScopes = boolValue(N, "AllScopes"))
     });
     Dict.parse(N);
   }
 
   // Helper for parsing mapping nodes (dictionaries).
   // We don't use YamlIO as we want to control over unknown keys.
   class DictParser {
     llvm::StringRef Description;
if (auto *S = llvm::dyn_cast<ScalarNode>(&N)) {
       }
     } else {
       warning("Expected scalar or list of scalars", N);
       return llvm::None;
     }
     return Result;
   }
 
+  llvm::Optional<Located<bool>> boolValue(Node &N, llvm::StringRef Desc) {
+    if (auto Scalar = scalarValue(N, Desc)) {
+      if (auto StandardLibrary = llvm::yaml::parseBool(**Scalar))
+        return Located<bool>(*StandardLibrary, Scalar->Range);
+      else
+        warning(Desc + " should be a boolean", N);
+    }
+    return llvm::None;
+  }
+
   // Report a "hard" error, reflecting a config file that can never be valid.
   void error(const llvm::Twine &Msg, llvm::SMRange Range) {
     HadError = true;
     SM.PrintMessage(Range.Start, llvm::SourceMgr::DK_Error, Msg, Range);
   }
   void error(const llvm::Twine &Msg, const Node &N) {
     return error(Msg, N.getSourceRange());
   }

clang-tools-extra/clangd/FS.h

 
 /// Returns a version of \p File that doesn't contain dots and dot dots.
 /// e.g /a/b/../c -> /a/c
 ///     /a/b/./c -> /a/b/c
 /// FIXME: We should avoid encountering such paths in clangd internals by
 /// filtering everything we get over LSP, CDB, etc.
 Path removeDots(PathRef File);
 
+/// Get a virtual root directory for the filesystem depending on the OS
+/// This is useful when a plausible absolute path name is needed.
+const llvm::StringLiteral virtualRoot();
+
 } // namespace clangd
 } // namespace clang
 
 #endif // LLVM_CLANG_TOOLS_EXTRA_CLANGD_FS_H

clang-tools-extra/clangd/FS.cpp

 }
 
 Path removeDots(PathRef File) {
   llvm::SmallString<128> CanonPath(File);
   llvm::sys::path::remove_dots(CanonPath, /*remove_dot_dot=*/true);
   return CanonPath.str().str();
 }
 
+const llvm::StringLiteral virtualRoot() {
+#ifdef _WIN32
+  return "C:\\virtual\\";
+#else
+  // This path must exist on the current file system. The only place we can
+  // safely assume to exist is "/".
+  return "/";
+#endif
+}
+
 } // namespace clangd
 } // namespace clang

clang-tools-extra/clangd/SourceCode.h

 //
 //===----------------------------------------------------------------------===//
 #ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_SOURCECODE_H
 #define LLVM_CLANG_TOOLS_EXTRA_CLANGD_SOURCECODE_H
 
 #include "Protocol.h"
 #include "support/Context.h"
 #include "support/ThreadsafeFS.h"
+#include "clang/Basic/CharInfo.h"
 #include "clang/Basic/Diagnostic.h"
 #include "clang/Basic/LangOptions.h"
 #include "clang/Basic/SourceLocation.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Format/Format.h"
 #include "clang/Lex/HeaderSearch.h"
 #include "clang/Tooling/Core/Replacement.h"
 #include "clang/Tooling/Syntax/Tokens.h"

 /// Returns true if the given location is in a generated protobuf file.
 bool isProtoFile(SourceLocation Loc, const SourceManager &SourceMgr);
 
 /// This scans source code, and should not be called when using a preamble.
 /// Prefer to access the cache in IncludeStructure::isSelfContained if you can.
 bool isSelfContainedHeader(const FileEntry *FE, FileID ID,
                            const SourceManager &SM, HeaderSearch &HeaderInfo);
 
+/// Whether a name is reserved for the implementation, e.g. __Vector_base
+/// Not exhaustive, may return false for some reserved names!
+inline bool isReservedIdentifier(llvm::StringRef N) {
+  // There are other classes of reserved identifiers, but this is common.
+  return N.size() >= 2 && N[0] == '_' && (isUppercase(N[1]) || N[1] == '_');
+}
+
 } // namespace clangd
 } // namespace clang
 #endif

clang-tools-extra/clangd/TUScheduler.h

 class ParsingCallbacks {
 public:
   virtual ~ParsingCallbacks() = default;
 
   /// Called on the AST that was built for emitting the preamble. The built AST
   /// contains only AST nodes from the #include directives at the start of the
   /// file. AST node in the current file should be observed on onMainAST call.
   virtual void onPreambleAST(PathRef Path, llvm::StringRef Version,
-                             ASTContext &Ctx,
+                             const CompilerInvocation &CI, ASTContext &Ctx,
                              std::shared_ptr<clang::Preprocessor> PP,
                              const CanonicalIncludes &) {}
 
   /// The argument function is run under the critical section guarding against
   /// races when closing the files.
   using PublishFn = llvm::function_ref<void(llvm::function_ref<void()>)>;
   /// Called on the AST built for the file itself. Note that preamble AST nodes
   /// are not deserialized and should be processed in the onPreambleAST call

clang-tools-extra/clangd/TUScheduler.cpp

void PreambleThread::build(Request Req) {
 
   ThreadCrashReporter ScopedReporter([&Inputs]() {
     llvm::errs() << "Signalled while building preamble\n";
     crashDumpParseInputs(llvm::errs(), Inputs);
   });
 
   LatestBuild = clang::clangd::buildPreamble(
       FileName, *Req.CI, Inputs, StoreInMemory,
-      [this, Version(Inputs.Version)](ASTContext &Ctx,
-                                      std::shared_ptr<clang::Preprocessor> PP,
-                                      const CanonicalIncludes &CanonIncludes) {
-        Callbacks.onPreambleAST(FileName, Version, Ctx, std::move(PP),
-                                CanonIncludes);
+      [&](ASTContext &Ctx, std::shared_ptr<clang::Preprocessor> PP,
+        Callbacks.onPreambleAST(FileName, Inputs.Version, *Req.CI, Ctx,
+                                std::move(PP), CanonIncludes);
       });
   if (LatestBuild && isReliable(LatestBuild->CompileCommand))
     HeaderIncluders.update(FileName, LatestBuild->Includes.allHeaders());
 }
 
 void ASTWorker::updatePreamble(std::unique_ptr<CompilerInvocation> CI,
                                ParseInputs PI,
                                std::shared_ptr<const PreambleData> Preamble,

clang-tools-extra/clangd/index/Background.cpp

llvm::Error BackgroundIndex::index(tooling::CompileCommand Cmd) {
 
   auto Clang =
       prepareCompilerInstance(std::move(CI), /*Preamble=*/nullptr,
                               std::move(*Buf), std::move(FS), IgnoreDiags);
   if (!Clang)
     return error("Couldn't build compiler instance");
 
   SymbolCollector::Options IndexOpts;
+  IndexOpts.Origin = SymbolOrigin::Background;
   // Creates a filter to not collect index results from files with unchanged
   // digests.
   IndexOpts.FileFilter = [&ShardVersionsSnapshot](const SourceManager &SM,
                                                   FileID FID) {
     const auto *F = SM.getFileEntryForID(FID);
     if (!F)
       return false; // Skip invalid files.
     auto AbsPath = getCanonicalPath(F, SM);

clang-tools-extra/clangd/index/FileIndex.h

public:
   FileIndex();
 
   /// Update preamble symbols of file \p Path with all declarations in \p AST
   /// and macros in \p PP.
   void updatePreamble(PathRef Path, llvm::StringRef Version, ASTContext &AST,
                       std::shared_ptr<Preprocessor> PP,
                       const CanonicalIncludes &Includes);
 
+  void updatePreamble(IndexFileIn);
+
   /// Update symbols and references from main file \p Path with
   /// `indexMainDecls`.
   void updateMain(PathRef Path, ParsedAST &AST);
 
   void profile(MemoryTree &MT) const;
 
 private:
   // Contains information from each file's preamble only. Symbols and relations

clang-tools-extra/clangd/index/FileIndex.cpp

 namespace clang {
 namespace clangd {
 namespace {
 
 SlabTuple indexSymbols(ASTContext &AST, std::shared_ptr<Preprocessor> PP,
                        llvm::ArrayRef<Decl *> DeclsToIndex,
                        const MainFileMacros *MacroRefsToIndex,
                        const CanonicalIncludes &Includes, bool IsIndexMainAST,
-                       llvm::StringRef Version, bool CollectMainFileRefs) {
+                       llvm::StringRef Version) {
   SymbolCollector::Options CollectorOpts;
   CollectorOpts.CollectIncludePath = true;
   CollectorOpts.Includes = &Includes;
   CollectorOpts.CountReferences = false;
-  CollectorOpts.Origin = SymbolOrigin::Dynamic;
-  CollectorOpts.CollectMainFileRefs = CollectMainFileRefs;
+  CollectorOpts.Origin =
+      IsIndexMainAST ? SymbolOrigin::Open : SymbolOrigin::Preamble;
+  CollectorOpts.CollectMainFileRefs = IsIndexMainAST;
+  CollectorOpts.CollectReservedSymbols = IsIndexMainAST;
 
   index::IndexingOptions IndexOpts;
   // We only need declarations, because we don't count references.
   IndexOpts.SystemSymbolFilter =
       index::IndexingOptions::SystemSymbolFilterKind::DeclarationsOnly;
   // We index function-local classes and its member functions only.
   IndexOpts.IndexFunctionLocals = true;
   if (IsIndexMainAST) {
for (const auto *Rel : It->getValue().Relations) {
     RelB.insert(*Rel);
   }
   IF.Relations = std::move(RelB).build();
   // Explicit move here is needed by some compilers.
   return std::move(IF);
 }
 
 SlabTuple indexMainDecls(ParsedAST &AST) {
-  return indexSymbols(
-      AST.getASTContext(), AST.getPreprocessorPtr(),
-      AST.getLocalTopLevelDecls(), &AST.getMacros(), AST.getCanonicalIncludes(),
-      /*IsIndexMainAST=*/true, AST.version(), /*CollectMainFileRefs=*/true);
+  return indexSymbols(AST.getASTContext(), AST.getPreprocessorPtr(),
+                      AST.getLocalTopLevelDecls(), &AST.getMacros(),
+                      AST.getCanonicalIncludes(),
+                      /*IsIndexMainAST=*/true, AST.version());
 }
 
 SlabTuple indexHeaderSymbols(llvm::StringRef Version, ASTContext &AST,
                              std::shared_ptr<Preprocessor> PP,
                              const CanonicalIncludes &Includes) {
   std::vector<Decl *> DeclsToIndex(
       AST.getTranslationUnitDecl()->decls().begin(),
       AST.getTranslationUnitDecl()->decls().end());
   return indexSymbols(AST, std::move(PP), DeclsToIndex,
                       /*MainFileMacros=*/nullptr, Includes,
-                      /*IsIndexMainAST=*/false, Version,
-                      /*CollectMainFileRefs=*/false);
+                      /*IsIndexMainAST=*/false, Version);
 }
 
 FileSymbols::FileSymbols(IndexContents IdxContents)
     : IdxContents(IdxContents) {}
 
 void FileSymbols::update(llvm::StringRef Key,
                          std::unique_ptr<SymbolSlab> Symbols,
                          std::unique_ptr<RefSlab> Refs,

 
 FileIndex::FileIndex()
     : MergedIndex(&MainFileIndex, &PreambleIndex),
       PreambleSymbols(IndexContents::Symbols | IndexContents::Relations),
       PreambleIndex(std::make_unique<MemIndex>()),
       MainFileSymbols(IndexContents::All),
       MainFileIndex(std::make_unique<MemIndex>()) {}
 
-void FileIndex::updatePreamble(PathRef Path, llvm::StringRef Version,
-                               ASTContext &AST,
-                               std::shared_ptr<Preprocessor> PP,
-                               const CanonicalIncludes &Includes) {
-  IndexFileIn IF;
-  std::tie(IF.Symbols, std::ignore, IF.Relations) =
-      indexHeaderSymbols(Version, AST, std::move(PP), Includes);
+void FileIndex::updatePreamble(IndexFileIn IF) {
   FileShardedIndex ShardedIndex(std::move(IF));
   for (auto Uri : ShardedIndex.getAllSources()) {
     auto IF = ShardedIndex.getShard(Uri);
     // We are using the key received from ShardedIndex, so it should always
     // exist.
     assert(IF);
     PreambleSymbols.update(
         Uri, std::make_unique<SymbolSlab>(std::move(*IF->Symbols)),
auto NewIndex = PreambleSymbols.buildIndex(
     PreambleIndex.reset(std::move(NewIndex));
     vlog(
         "Build dynamic index for header symbols with estimated memory usage of "
         "{0} bytes",
         PreambleIndex.estimateMemoryUsage());
   }
 }
 
+void FileIndex::updatePreamble(PathRef Path, llvm::StringRef Version,
+                               ASTContext &AST,
+                               std::shared_ptr<Preprocessor> PP,
+                               const CanonicalIncludes &Includes) {
+  IndexFileIn IF;
+  std::tie(IF.Symbols, std::ignore, IF.Relations) =
+      indexHeaderSymbols(Version, AST, std::move(PP), Includes);
+  updatePreamble(std::move(IF));
+}
+
 void FileIndex::updateMain(PathRef Path, ParsedAST &AST) {
   auto Contents = indexMainDecls(AST);
   MainFileSymbols.update(
       URI::create(Path).toString(),
       std::make_unique<SymbolSlab>(std::move(std::get<0>(Contents))),
       std::make_unique<RefSlab>(std::move(std::get<1>(Contents))),
       std::make_unique<RelationSlab>(std::move(std::get<2>(Contents))),
       /*CountReferences=*/true);

clang-tools-extra/clangd/index/StdLib.h

+//===--- StdLib.h ------------------------------------------------*- C++-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+// Clangd indexer for the C++ standard library.
+//
+// The index only contains symbols that are part of the translation unit. So
+// if your translation unit does not yet #include <string>, you do not get
+// auto completion for std::string. However we expect that many users would
+// like to use the the standard library anyway, so we could index that by
+// default an offer e.g. code completion without requiring #includes.
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_INDEX_STDLIB_H
+#define LLVM_CLANG_TOOLS_EXTRA_CLANGD_INDEX_STDLIB_H
+
+#include "index/Symbol.h"
+#include "support/ThreadsafeFS.h"
+#include "clang/Basic/LangStandard.h"
+#include "llvm/ADT/StringRef.h"
+#include <string>
+
+namespace clang {
+class CompilerInvocation;
+class LangOptions;
+class HeaderSearch;
+namespace clangd {
+
+struct StdLibLocation {
+  llvm::SmallVector<std::string> Paths;
+};
+
+class StdLibSet {
+  std::atomic<int> Best[2] = {{-1}, {-1}};
+
+public:
+  // Determines if we should index the standard library in a configuration.
+  //
+  // This is true if:
+  //  - standard library indexing is enabled for the file
+  //  - the language version is higher than any previous add() for the language
+  //  - the standard library headers exist on the search path
+  //
+  // This function is threadsafe.
+  llvm::Optional<StdLibLocation> add(const LangOptions &, const HeaderSearch &);
+
+  // Indicates whether we a built index should be used.
+  // It should not be used if a newer version has subsequently been added.
+  //
+  // Intended pattern is:
+  //   if (add()) {
+  //     symbols = indexStandardLibrary();
+  //     if (isBest())
+  //       index.update(symbols);
+  //   }
+  //
+  // This is still technically racy: we could return true here, then another
+  // thread could add->index->update a better library before we can update.
+  // We'd then overwrite it with the older version.
+  // However, it's very unlikely: indexing takes a long time.
+  bool isBest(const LangOptions &) const;
+};
+
+/// Generate a index of the standard library index for a given variant of
+/// the standard library. This index can be included if the translation unit
+/// does not (yet) contain any standard library headers.
+SymbolSlab indexStandardLibrary(std::unique_ptr<CompilerInvocation> Invocation,
+                                const StdLibLocation &Loc,
+                                const ThreadsafeFS &TFS);
+
+/// Index the given umbrella header file using the standard library from the
+/// given file system.
+SymbolSlab indexUmbrellaHeaders(llvm::StringRef HeaderSources,
+                                std::unique_ptr<CompilerInvocation> CI,
+                                const StdLibLocation &Loc,
+                                const ThreadsafeFS &TFS);
+
+/// generate header containing #includes for all standard library headers
+llvm::StringRef generateUmbrellaHeaders(LangStandard::Kind LibraryVariant);
+
+} // namespace clangd
+} // namespace clang
+
+#endif // LLVM_CLANG_TOOLS_EXTRA_CLANGD_INDEX_STDLIB_H

clang-tools-extra/clangd/index/StdLib.cpp

+//===-- StdLib.cpp ----------------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+#include "StdLib.h"
+#include <fstream>
+#include <memory>
+#include <string>
+#include <vector>
+
+#include "Compiler.h"
+#include "Config.h"
+#include "SymbolCollector.h"
+#include "index/IndexAction.h"
+#include "support/Logger.h"
+#include "support/ThreadsafeFS.h"
+#include "support/Trace.h"
+#include "clang/Basic/LangOptions.h"
+#include "clang/Frontend/CompilerInvocation.h"
+#include "clang/Lex/PreprocessorOptions.h"
+#include "llvm/ADT/IntrusiveRefCntPtr.h"
+#include "llvm/ADT/None.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/Path.h"
+
+namespace clang {
+namespace clangd {
+namespace {
+
+enum Lang { C, CXX };
+
+llvm::StringLiteral mandatoryHeader(Lang L) {
+  switch (L) {
+  case C:
+    return "stdio.h";
+  case CXX:
+    return "vector";
+  }
+  return L == CXX ? llvm::StringLiteral("vector") : "stdio.h";
+}
+
+std::string buildUmbrella(llvm::StringLiteral Mandatory,
+                          std::vector<llvm::StringLiteral> Headers) {
+  std::string Result;
+  llvm::raw_string_ostream OS(Result);
+
+  // We __has_include guard all our #includes to avoid errors when using older
+  // stdlib version that don't have headers for the newest language standards.
+  // But make sure we get *some* error if things are totally broken.
+  OS << llvm::formatv(
+      "#if !__has_include(<{0}>)\n"
+      "#error Mandatory header <{0}> not found in standard library!\n"
+      "#endif\n",
+      Mandatory);
+
+  llvm::sort(Headers.begin(), Headers.end());
+  auto Last = std::unique(Headers.begin(), Headers.end());
+  for (auto Header = Headers.begin(); Header != Last; ++Header) {
+    OS << llvm::formatv("#if __has_include({0})\n"
+                        "#include {0}\n"
+                        "#endif\n",
+                        *Header);
+  }
+  OS.flush();
+  return Result;
+}
+
+} // namespace
+
+// Build umbrella header for the the standard library.
+//
+// The umbrella header is the same for all versions of each language.
+// Headers that are unsupported in old lang versions are usually guarded by #if.
+// Some headers may be not present in old stdlib versions, the umbrella header
+// guards with __has_include for this purpose.
+llvm::StringRef generateUmbrellaHeaders(Lang L) {
+  switch (L) {
+  case CXX:
+    static std::string *UmbrellaCXX =
+        new std::string(buildUmbrella(mandatoryHeader(L), {
+#define SYMBOL(Name, NameSpace, Header) #Header,
+#include "StdSymbolMap.inc"
+#undef SYMBOL
+                                                          }));
+    return *UmbrellaCXX;
+  case C:
+    static std::string *UmbrellaC =
+        new std::string(buildUmbrella(mandatoryHeader(L), {
+#define SYMBOL(Name, NameSpace, Header) #Header,
+#include "CSymbolMap.inc"
+#undef SYMBOL
+                                                          }));
+    return *UmbrellaC;
+  }
+}
+
+namespace {
+
+// Including the standard library leaks a lot of symbols/names that are not
+// part of its interface, and we want to drop these. There are two main groups:
+//
+// Implementation details. These tend to have reserved names like __foo.
+// We drop symbols with reserved names unless they're on our symbol list.
+//
+// Transitive includes that are not part of the stdlib (low-level platform deps)
+// These are surprisingly tricky to identify:
+//  - we don't want to limit to symbols our our list, as our list has only
+//    top-level symbols (and there may be legitimate stdlib extensions).
+//  - we can't limit to only symbols defined by known stdlib headers, as stdlib
+//    internal structure is murky
+//  - we can't strictly require symbols to come from a particular path, e.g.
+//      libstdc++ is mostly under /usr/include/c++/10/...
+//      but std::ctype_base is under /usr/include/<platform>/c++/10/...
+// Instead we require the symbol to come from a header that is *either* from
+// the standard library path (as identified by the location of <vector>), or
+// another header that defines a symbol from our stdlib list.
+static SymbolSlab filter(SymbolSlab Slab, const StdLibLocation &Loc,
+                         llvm::StringRef TUPath) {
+  SymbolSlab::Builder Result;
+
+  static auto &StandardHeaders = *[] {
+    auto Set = new llvm::DenseSet<llvm::StringRef>();
+    for (llvm::StringRef Name : {
+#define SYMBOL(Name, NameSpace, Header) #Header,
+#include "CSymbolMap.inc"
+#include "StdSymbolMap.inc"
+#undef SYMBOL
+         })
+      Set->insert(Name);
+    return Set;
+  }();
+
+  // Form prefixes like file:///usr/include/c++/10/
+  // These can be trivially prefix-compared with URIs in the indexed symbols.
+  llvm::SmallVector<std::string> StdLibURIPrefixes;
+  for (const auto &Path : Loc.Paths) {
+    StdLibURIPrefixes.push_back(URI::create(Path).toString());
+    if (StdLibURIPrefixes.back().back() != '/')
+      StdLibURIPrefixes.back().push_back('/');
+  }
+  // For each header URI, is it *either* prefixed by StdLibURIPrefixes *or*
+  // owner of a symbol whose insertable header is in StandardHeaders?
+  // Pointer key because strings in a SymbolSlab are interned.
+  llvm::DenseMap<const char *, bool> GoodHeader;
+  for (const Symbol &S : Slab) {
+    if (!S.IncludeHeaders.empty() &&
+        StandardHeaders.contains(S.IncludeHeaders.front().IncludeHeader)) {
+      GoodHeader[S.CanonicalDeclaration.FileURI] = true;
+      GoodHeader[S.Definition.FileURI] = true;
+      continue;
+    }
+    for (const char *URI :
+         {S.CanonicalDeclaration.FileURI, S.Definition.FileURI}) {
+      auto R = GoodHeader.try_emplace(URI, false);
+      if (R.second) {
+        R.first->second = llvm::any_of(
+            StdLibURIPrefixes,
+            [&, URIStr(llvm::StringRef(URI))](const std::string &Prefix) {
+              return URIStr.startswith(Prefix);
+            });
+      }
+    }
+  }
+  for (const auto &Good : GoodHeader)
+    if (Good.second)
+      dlog("Stdlib header: {0}", Good.first);
+  // Empty URIs aren't considered good. (Definition can be blank).
+  auto IsGoodHeader = [&](const char *C) { return *C && GoodHeader.lookup(C); };
+
+  for (const Symbol &S : Slab) {
+    if (!(IsGoodHeader(S.CanonicalDeclaration.FileURI) ||
+          IsGoodHeader(S.Definition.FileURI))) {
+      dlog("Ignoring wrong-header symbol {0}{1} in {2}", S.Scope, S.Name,
+           S.CanonicalDeclaration.FileURI);
+      continue;
+    }
+    Result.insert(S);
+  }
+
+  return std::move(Result).build();
+}
+
+LangStandard::Kind standardFromOpts(const LangOptions &LO) {
+  if (LO.CPlusPlus) {
+    return !LO.CPlusPlus11   ? LangStandard::lang_cxx98
+           : !LO.CPlusPlus14 ? LangStandard::lang_cxx11
+           : !LO.CPlusPlus17 ? LangStandard::lang_cxx14
+           : !LO.CPlusPlus20 ? LangStandard::lang_cxx17
+           : !LO.CPlusPlus2b ? LangStandard::lang_cxx20
+                             : LangStandard::lang_cxx2b;
+  }
+  return !LO.C11 ? LangStandard::lang_c99
+         // C17 has no new features, so treat C14 as C17.
+         : !LO.C2x ? LangStandard::lang_c17
+                   : LangStandard::lang_c2x;
+}
+
+Lang langFromOpts(const LangOptions &LO) { return LO.CPlusPlus ? CXX : C; }
+
+} // namespace
+
+SymbolSlab indexUmbrellaHeaders(llvm::StringRef HeaderSources,
+                                std::unique_ptr<CompilerInvocation> CI,
+                                const StdLibLocation &Loc,
+                                const ThreadsafeFS &TFS) {
+  if (CI->getFrontendOpts().Inputs.size() != 1 ||
+      !CI->getPreprocessorOpts().ImplicitPCHInclude.empty()) {
+    elog("Indexing standard library failed: bad CompilerInvocation");
+    assert(false && "indexing stdlib with a dubious CompilerInvocation!");
+    return SymbolSlab();
+  }
+  const FrontendInputFile &Input = CI->getFrontendOpts().Inputs.front();
+  trace::Span Tracer("StandardLibraryIndex");
+  LangStandard::Kind LangStd = standardFromOpts(*CI->getLangOpts());
+  log("Indexing {0} standard library in the context of {1}",
+      LangStandard::getLangStandardForKind(LangStd).getName(), Input.getFile());
+
+  SymbolSlab Symbols;
+  IgnoreDiagnostics IgnoreDiags;
+  CI->getPreprocessorOpts().clearRemappedFiles();
+  auto Clang = prepareCompilerInstance(
+      std::move(CI), /*Preamble=*/nullptr,
+      llvm::MemoryBuffer::getMemBuffer(HeaderSources, Input.getFile()),
+      TFS.view(/*CWD=*/llvm::None), IgnoreDiags);
+  if (!Clang) {
+    elog("Standard Library Index: Couldn't build compiler instance");
+    return Symbols;
+  }
+
+  SymbolCollector::Options IndexOpts;
+  IndexOpts.Origin = SymbolOrigin::StdLib;
+  IndexOpts.CollectMainFileSymbols = false;
+  IndexOpts.CollectMainFileRefs = false;
+  IndexOpts.CollectMacro = true;
+  IndexOpts.StoreAllDocumentation = true;
+  // Sadly we can't use IndexOpts.FileFilter to restrict indexing scope.
+  // Files from outside the location may define true std symbols anyway.
+  // We end up "blessing" such headers, and can only do that by indexing
+  // everything first.
+
+  // we only care about the symbols, so not storing the other attributes
+  auto Action = createStaticIndexingAction(
+      IndexOpts, [&](SymbolSlab S) { Symbols = std::move(S); }, nullptr,
+      nullptr, nullptr);
+
+  if (!Action->BeginSourceFile(*Clang, Input)) {
+    elog("Standard Library Index: BeginSourceFile() failed");
+    return Symbols;
+  }
+
+  if (llvm::Error Err = Action->Execute()) {
+    elog("Standard Library Index: Execute failed: {0}", std::move(Err));
+    return Symbols;
+  }
+
+  Action->EndSourceFile();
+  bool HadErrors = Clang->hasDiagnostics() &&
+                   Clang->getDiagnostics().hasUncompilableErrorOccurred();
+  if (HadErrors) {
+    log("Errors when generating the standard library index, index may be "
+        "incomplete");
+  }
+
+  unsigned SymbolsBeforeFilter = Symbols.size();
+  Symbols = filter(std::move(Symbols), Loc, Input.getFile());
+  log("Indexed {0} standard library: ({1} symbols, {2} filtered)",
+      LangStandard::getLangStandardForKind(LangStd).getName(), Symbols.size(),
+      SymbolsBeforeFilter - Symbols.size());
+  SPAN_ATTACH(Tracer, "symbols", int(Symbols.size()));
+  return Symbols;
+}
+
+SymbolSlab indexStandardLibrary(std::unique_ptr<CompilerInvocation> Invocation,
+                                const StdLibLocation &Loc,
+                                const ThreadsafeFS &TFS) {
+  return indexUmbrellaHeaders(
+      generateUmbrellaHeaders(langFromOpts(*Invocation->getLangOpts())),
+      std::move(Invocation), Loc, TFS);
+}
+
+bool StdLibSet::isBest(const LangOptions &LO) const {
+  return standardFromOpts(LO) >=
+         Best[langFromOpts(LO)].load(std::memory_order_acquire);
+}
+
+llvm::Optional<StdLibLocation> StdLibSet::add(const LangOptions &LO,
+                                              const HeaderSearch &HS) {
+  Lang L = langFromOpts(LO);
+  int OldVersion = Best[L].load(std::memory_order_acquire);
+  int NewVersion = standardFromOpts(LO);
+  dlog("Index stdlib? {0}",
+       LangStandard::getLangStandardForKind(standardFromOpts(LO)).getName());
+
+  if (NewVersion <= OldVersion) {
+    dlog("No: have {0}, {1}>={2}",
+         LangStandard::getLangStandardForKind(
+             static_cast<LangStandard::Kind>(NewVersion))
+             .getName(),
+         OldVersion, NewVersion);
+    return llvm::None;
+  }
+
+  if (!Config::current().Index.StandardLibrary) {
+    dlog("No: disabled in config");
+    return llvm::None;
+  }
+
+  // We'd like to index a standard library here if there is one.
+  // Check for the existence of <vector> on the search path.
+  // We could cache this, but we only get here repeatedly when there's no
+  // stdlib, and even then only once per preamble build.
+  llvm::StringLiteral ProbeHeader = mandatoryHeader(L);
+  llvm::SmallString<256> Path; // Scratch space.
+  llvm::SmallVector<std::string> SearchPaths;
+  auto RecordHeaderPath = [&](llvm::StringRef HeaderPath) {
+    llvm::StringRef DirPath = llvm::sys::path::parent_path(HeaderPath);
+    if (!HS.getFileMgr().getVirtualFileSystem().getRealPath(DirPath, Path))
+      SearchPaths.emplace_back(Path);
+  };
+  for (const auto &DL :
+       llvm::make_range(HS.search_dir_begin(), HS.search_dir_end())) {
+    switch (DL.getLookupType()) {
+    case DirectoryLookup::LT_NormalDir: {
+      Path = DL.getDir()->getName();
+      llvm::sys::path::append(Path, ProbeHeader);
+      llvm::vfs::Status Stat;
+      if (!HS.getFileMgr().getNoncachedStatValue(Path, Stat) &&
+          Stat.isRegularFile())
+        RecordHeaderPath(Path);
+      break;
+    }
+    case DirectoryLookup::LT_Framework:
+      // stdlib can't be a framework (framework includes bust have a slash)
+      continue;
+    case DirectoryLookup::LT_HeaderMap:
+      llvm::StringRef Target =
+          DL.getHeaderMap()->lookupFilename(ProbeHeader, Path);
+      if (!Target.empty())
+        RecordHeaderPath(Target);
+      break;
+    }
+  }
+  if (SearchPaths.empty()) {
+    dlog("No: didn't find <{0}>)", ProbeHeader);
+    return llvm::None;
+  }
+  dlog("Found standard library in {0}", llvm::join(SearchPaths, ", "));
+
+  while (!Best[L].compare_exchange_weak(OldVersion, NewVersion,
+                                        std::memory_order_acq_rel))
+    if (OldVersion >= NewVersion) {
+      dlog("No: lost the race");
+      return llvm::None; // Another thread won the race while we were checking.
+    }
+
+  dlog("Yes, index stdlib!");
+  return StdLibLocation{std::move(SearchPaths)};
+}
+
+} // namespace clangd
+} // namespace clang

clang-tools-extra/clangd/index/SymbolCollector.h

struct Options {
     /// functions.
     bool CollectMainFileSymbols = true;
     /// Collect references to main-file symbols.
     bool CollectMainFileRefs = false;
     /// If set to true, SymbolCollector will collect doc for all symbols.
     /// Note that documents of symbols being indexed for completion will always
     /// be collected regardless of this option.
     bool StoreAllDocumentation = false;
+    /// Collect symbols with reserved names like __Vector_base.
+    /// These are often private parts of library implementations.
+    /// This *doesn't* currently affect macros, which are more often interfaces!
+    bool CollectReservedSymbols = false;
     /// If this is set, only collect symbols/references from a file if
     /// `FileFilter(SM, FID)` is true. If not set, all files are indexed.
     std::function<bool(const SourceManager &, FileID)> FileFilter = nullptr;
   };
 
   SymbolCollector(Options Opts);
   ~SymbolCollector();
 

clang-tools-extra/clangd/index/SymbolCollector.cpp

void SymbolCollector::initialize(ASTContext &Ctx) {
   ASTCtx = &Ctx;
   HeaderFileURIs = std::make_unique<HeaderFileURICache>(
       PP, ASTCtx->getSourceManager(), Opts);
   CompletionAllocator = std::make_shared<GlobalCodeCompletionAllocator>();
   CompletionTUInfo =
       std::make_unique<CodeCompletionTUInfo>(CompletionAllocator);
 }
 
+static bool shouldCollectDC(const DeclContext &DC,
+                            const SymbolCollector::Options &Opts) {
+  // We want most things but not "local" symbols such as symbols inside
+  // FunctionDecl, BlockDecl, ObjCMethodDecl and OMPDeclareReductionDecl.
+  // FIXME: Need a matcher for ExportDecl in order to include symbols declared
+  // within an export.
+  switch (DC.getDeclKind()) {
+  case Decl::TranslationUnit:
+  case Decl::Namespace:
+  case Decl::LinkageSpec:
+  case Decl::Enum:
+  case Decl::ObjCProtocol:
+  case Decl::ObjCInterface:
+  case Decl::ObjCCategory:
+  case Decl::ObjCCategoryImpl:
+  case Decl::ObjCImplementation:
+    break;
+  default:
+    // Record has a few derivations (e.g. CXXRecord, Class specialization), it's
+    // easier to cast.
+    if (!isa<RecordDecl>(DC))
+      return false;
+  }
+
+  if (!Opts.CollectReservedSymbols && hasReservedScopeName(DC))
+    return false;
+
+  return true;
+}
+
 bool SymbolCollector::shouldCollectSymbol(const NamedDecl &ND,
                                           const ASTContext &ASTCtx,
                                           const Options &Opts,
                                           bool IsMainFileOnly) {
   // Skip anonymous declarations, e.g (anonymous enum/class/struct).
   if (ND.getDeclName().isEmpty())
     return false;
 
   // Skip main-file symbols if we are not collecting them.
   if (IsMainFileOnly && !Opts.CollectMainFileSymbols)
     return false;
 
   // Skip symbols in anonymous namespaces in header files.
   if (!IsMainFileOnly && ND.isInAnonymousNamespace())
     return false;
 
   // For function local symbols, index only classes and its member functions.
   if (index::isFunctionLocalSymbol(&ND))
     return isa<RecordDecl>(ND) ||
            (ND.isCXXInstanceMember() && ND.isFunctionOrFunctionTemplate());
 
-  // We want most things but not "local" symbols such as symbols inside
-  // FunctionDecl, BlockDecl, ObjCMethodDecl and OMPDeclareReductionDecl.
-  // FIXME: Need a matcher for ExportDecl in order to include symbols declared
-  // within an export.
-  const auto *DeclCtx = ND.getDeclContext();
-  switch (DeclCtx->getDeclKind()) {
-  case Decl::TranslationUnit:
-  case Decl::Namespace:
-  case Decl::LinkageSpec:
-  case Decl::Enum:
-  case Decl::ObjCProtocol:
-  case Decl::ObjCInterface:
-  case Decl::ObjCCategory:
-  case Decl::ObjCCategoryImpl:
-  case Decl::ObjCImplementation:
-    break;
-  default:
-    // Record has a few derivations (e.g. CXXRecord, Class specialization), it's
-    // easier to cast.
-    if (!isa<RecordDecl>(DeclCtx))
-      return false;
-  }
-
   // Avoid indexing internal symbols in protobuf generated headers.
   if (isPrivateProtoDecl(ND))
     return false;
+  if (!Opts.CollectReservedSymbols && hasReservedName(ND))
+    return false;
+  if (!shouldCollectDC(*ND.getDeclContext(), Opts))
+    return false;
+
   return true;
 }
 
 // Always return true to continue indexing.
 bool SymbolCollector::handleDeclOccurrence(
     const Decl *D, index::SymbolRoleSet Roles,
     llvm::ArrayRef<index::SymbolRelation> Relations, SourceLocation Loc,
     index::IndexDataConsumer::ASTNodeInfo ASTNode) {

clang-tools-extra/clangd/index/SymbolOrigin.h

 #include <cstdint>
 
 namespace clang {
 namespace clangd {
 
 // Describes the source of information about a symbol.
 // Mainly useful for debugging, e.g. understanding code completion results.
 // This is a bitfield as information can be combined from several sources.
-enum class SymbolOrigin : uint8_t {
+enum class SymbolOrigin : uint16_t {
   Unknown = 0,
   AST = 1 << 0,        // Directly from the AST (indexes should not set this).
-  Dynamic = 1 << 1,    // From the dynamic index of opened files.
+  Preamble = 1 << 1,   // From the dynamic index of opened file preambles.
   Static = 1 << 2,     // From the static, externally-built index.
   Merge = 1 << 3,      // A non-trivial index merge was performed.
   Identifier = 1 << 4, // Raw identifiers in file.
   Remote = 1 << 5,     // Remote index.
-  // Remaining bits reserved for index implementations.
+  Open = 1 << 6,       // From the dynamic index of opened files.
+                       // 1 << 7 reserved
+  StdLib = 1 << 8,     // Standard library indexer
+  Background = 1 << 9, // Background indexer for project
 };
 
 inline SymbolOrigin operator|(SymbolOrigin A, SymbolOrigin B) {
-  return static_cast<SymbolOrigin>(static_cast<uint8_t>(A) |
-                                   static_cast<uint8_t>(B));
+  return static_cast<SymbolOrigin>(static_cast<uint16_t>(A) |
+                                   static_cast<uint16_t>(B));
 }
 inline SymbolOrigin &operator|=(SymbolOrigin &A, SymbolOrigin B) {
   return A = A | B;
 }
 inline SymbolOrigin operator&(SymbolOrigin A, SymbolOrigin B) {
-  return static_cast<SymbolOrigin>(static_cast<uint8_t>(A) &
-                                   static_cast<uint8_t>(B));
+  return static_cast<SymbolOrigin>(static_cast<uint16_t>(A) &
+                                   static_cast<uint16_t>(B));
 }
 
 llvm::raw_ostream &operator<<(llvm::raw_ostream &, SymbolOrigin);
 
 } // namespace clangd
 } // namespace clang
 
 #endif // LLVM_CLANG_TOOLS_EXTRA_CLANGD_INDEX_SYMBOL_ORIGIN_H

clang-tools-extra/clangd/index/SymbolOrigin.cpp

 //===--- SymbolOrigin.cpp ----------------------------------------*- C++-*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "SymbolOrigin.h"
 
 namespace clang {
 namespace clangd {
 
 llvm::raw_ostream &operator<<(llvm::raw_ostream &OS, SymbolOrigin O) {
   if (O == SymbolOrigin::Unknown)
     return OS << "unknown";
-  constexpr static char Sigils[] = "ADSMIR67";
+  constexpr static char Sigils[] = "APSMIRO7LB012345";
   for (unsigned I = 0; I < sizeof(Sigils); ++I)
-    if (static_cast<uint8_t>(O) & 1u << I)
+    if (static_cast<uint16_t>(O) & 1u << I)
       OS << Sigils[I];
   return OS;
 }
 
 } // namespace clangd
 } // namespace clang

clang-tools-extra/clangd/unittests/CMakeLists.txt

add_unittest(ClangdUnitTests ClangdTests
   QualityTests.cpp
   RenameTests.cpp
   RIFFTests.cpp
   SelectionTests.cpp
   SemanticHighlightingTests.cpp
   SemanticSelectionTests.cpp
   SerializationTests.cpp
   SourceCodeTests.cpp
+  StdLibIndexTests.cpp
   SymbolCollectorTests.cpp
   SymbolInfoTests.cpp
   SyncAPI.cpp
   TUSchedulerTests.cpp
   TestFS.cpp
   TestIndex.cpp
   TestTU.cpp
   TestWorkspace.cpp

clang-tools-extra/clangd/unittests/StdLibIndexTests.cpp

+//===-- StdLibIndexTests.cpp ------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "Compiler.h"
+#include "Config.h"
+#include "TestFS.h"
+#include "index/StdLib.h"
+#include "clang/Basic/LangStandard.h"
+#include "gmock/gmock.h"
+#include "gtest/gtest.h"
+#include <memory>
+
+using namespace testing;
+
+namespace clang {
+namespace clangd {
+namespace {
+
+/// check that the generated header sources contains some usual standard library
+/// headers
+TEST(StdLibIndexTests, generateUmbrellaHeader) {
+  auto CXX = generateUmbrellaHeaders(LangStandard::lang_cxx14).str();
+  EXPECT_THAT(CXX, HasSubstr("#include <string>"));
+  EXPECT_THAT(CXX, HasSubstr("#include <cstdio>"));
+  EXPECT_THAT(CXX, Not(HasSubstr("#include <stdio.h>")));
+
+  auto C = generateUmbrellaHeaders(LangStandard::lang_c11).str();
+  EXPECT_THAT(C, Not(HasSubstr("#include <string>")));
+  EXPECT_THAT(C, Not(HasSubstr("#include <cstdio>")));
+  EXPECT_THAT(C, HasSubstr("#include <stdio.h>"));
+}
+
+MATCHER_P(Named, Name, "") { return arg.Name == Name; }
+
+/// build the index and check if it contains the right symbols
+TEST(StdLibIndexTests, buildIndex) {
+  MockFS FS;
+  // TODO: maybe find a way to use a local libcxx for testing if that is
+  //       available on the machine
+  std::string HeaderMock = R"CPP(
+  #if __cplusplus >= 201703L
+    int func17();
+  #elif __cplusplus >= 201402L
+    int func14();
+  #else
+    bool func98();
+  #endif
+  int __implementation_details();
+  )CPP";
+  StdlibIndexSpec Spec;
+  Spec.LangStd = LangStandard::lang_cxx14;
+  auto Symbols = indexUmbrellaHeaders(HeaderMock, FS, Spec);
+
+  EXPECT_THAT(Symbols, ElementsAre(Named("func14")));
+}
+
+CompilerInvocation parse(std::vector<std::string> Flags) {
+  ParseInputs Inputs;
+  Inputs.CompileCommand.Filename = testPath(Flags.back());
+  Inputs.CompileCommand.Directory = testRoot();
+  Inputs.CompileCommand.CommandLine = std::move(Flags);
+  MockFS FS;
+  Inputs.TFS = &FS;
+  IgnoreDiagnostics IgnoreDiags;
+  auto Result = buildCompilerInvocation(Inputs, IgnoreDiags);
+  EXPECT_TRUE(Result);
+  return *Result;
+}
+
+TEST(StdLibIndexTests, Detect) {
+  {
+    Config Cfg;
+    Cfg.Index.StandardLibrary = false;
+    WithContextValue WithCfg(Config::Key, std::move(Cfg));
+    EXPECT_FALSE(StdlibIndexSpec::detect(parse({"clang++", "test.cc"})));
+  }
+  Config Cfg;
+  Cfg.Index.StandardLibrary = true;
+  WithContextValue WithCfg(Config::Key, std::move(Cfg));
+  EXPECT_TRUE(StdlibIndexSpec::detect(parse({"clang++", "test.cc"})));
+
+  auto CXX11 =
+      StdlibIndexSpec::detect(parse({"clang++", "-std=gnu++11", "test.cc"}));
+  EXPECT_EQ(CXX11->LangStd, LangStandard::lang_cxx11);
+
+  auto C11 = StdlibIndexSpec::detect(parse({"clang", "-std=c11", "test.c"}));
+  EXPECT_EQ(C11->LangStd, LangStandard::lang_c17); // No new features in 17.
+
+  auto WithSysroot = StdlibIndexSpec::detect(
+      parse({"clang++", "-isysroot", testPath("blah"), "test.cc"}));
+  EXPECT_EQ(testPath("blah"), WithSysroot->Sysroot);
+}
+
+} // namespace
+} // namespace clangd
+} // namespace clang

clang-tools-extra/clangd/unittests/TUSchedulerTests.cpp

 
 TEST_F(TUSchedulerTests, AsyncPreambleThread) {
   // Blocks preamble thread while building preamble with \p BlockVersion until
   // \p N is notified.
   class BlockPreambleThread : public ParsingCallbacks {
   public:
     BlockPreambleThread(llvm::StringRef BlockVersion, Notification &N)
         : BlockVersion(BlockVersion), N(N) {}
-    void onPreambleAST(PathRef Path, llvm::StringRef Version, ASTContext &Ctx,
+    void onPreambleAST(PathRef Path, llvm::StringRef Version,
+                       const CompilerInvocation &, ASTContext &Ctx,
                        std::shared_ptr<clang::Preprocessor> PP,
                        const CanonicalIncludes &) override {
       if (Version == BlockVersion)
         N.wait();
     }
 
   private:
     llvm::StringRef BlockVersion;