[clangd] Remove the direct use of StdSymbolMapping.inc usage.

[hokein]

Summary

Replace them with the library APIs.

[kadircet]

clang-tools-extra/clangd/index/CanonicalIncludes.cpp

753–754

what about getting rid of StdSymbolMapping completely and changing CanonicalIncludes::mapSymbol to use tooling::stdlib ?
only call site is through SymbolCollector, which has access to LangOptions.

[hokein]

address review comment: remove StdSymbolMapping in the CanonicalIncluedes.

clang-tools-extra/clangd/index/CanonicalIncludes.cpp

753–754

Yeah, good idea. This simplifies the code further, thanks!

[kadircet]

clang-tools-extra/clangd/index/CanonicalIncludes.cpp

711

i think the comment is misleading. as if we had some alternatives in the tooling::stdlib, it would already pick one for us. the issue is we don't have it in the mapping at all.

717

this is a slight behaviour change. we'd actually return "" whenever the language isn't CPlusPlus or C11. could you retain that behaviour?

clang-tools-extra/clangd/index/SymbolCollector.cpp

837–838

nit: rather than creating a new copy of the symbol name everytime, i'd also update the mapSymbol interface to take in StringRef Scope, StringRef Name instead.

[hokein]

address review comments

[hokein]

update

[kadircet]

thanks a lot. since this is the last (and only) upstream user of the raw mappings. can you also move them into clang/lib/Tooling/Inclusions/Stdlib/ as part of this patch?

clang-tools-extra/clangd/index/CanonicalIncludes.h

43

these parameter comments are a little bit unconventional, maybe just mention that Scope should have trailing colons (e.g. std::) and name shouldn't have any (e.g. vector) ? also mention that it should be empty ("") for global namespace?

[hokein]

In D143274#4106449, @kadircet wrote:

thanks a lot. since this is the last (and only) upstream user of the raw mappings. can you also move them into clang/lib/Tooling/Inclusions/Stdlib/ as part of this patch?

As discussed, I will address this in a followup patch.

clang-tools-extra/clangd/index/CanonicalIncludes.cpp

711

yeah, this comment was moved from the old version. I removed it as it is covered by the above FIXME.

clang-tools-extra/clangd/index/CanonicalIncludes.h

43

Removed the parameter comments, and refined the doc comment of the function.

[hokein]

refined the comment.

[kadircet]

clang-tools-extra/clangd/index/CanonicalIncludes.h

43

s/full/fully

45

s/for global namespace symbol/for global namespace/

clang-tools-extra/clangd/index/CanonicalIncludes.h

 /// Maps a definition location onto an #include file, based on a set of filename
 /// rules.
 /// Only const methods (i.e. mapHeader) in this class are thread safe.
 class CanonicalIncludes {
 public:
   /// Adds a file-to-string mapping from \p ID to \p CanonicalPath.
   void addMapping(FileEntryRef Header, llvm::StringRef CanonicalPath);
 
-  /// Returns the overridden include for symbol with \p QualifiedName, or "".
-  llvm::StringRef mapSymbol(llvm::StringRef QualifiedName) const;
+  /// Returns the overridden include for a qualified symbol with, or "".
+  /// \p Scope and \p Name concatenation forms the fully qualified name.
+  /// \p Scope is the qualifier with the trailing "::" (e.g. "std::") or empty
+  /// (for global namespace).
+  llvm::StringRef mapSymbol(llvm::StringRef Scope, llvm::StringRef Name,
+                            const LangOptions &L) const;
 
   /// Returns the overridden include for files in \p Header, or "".
   llvm::StringRef mapHeader(FileEntryRef Header) const;
 
   /// Adds mapping for system headers and some special symbols (e.g. STL symbols
   /// in <iosfwd> need to be mapped individually). Approximately, the following
   /// system headers are handled:
   ///   - C++ standard library e.g. bits/basic_string.h$ -> <string>
   ///   - Posix library e.g. bits/pthreadtypes.h$ -> <pthread.h>
   ///   - Compiler extensions, e.g. include/avx512bwintrin.h$ -> <immintrin.h>
   /// The mapping is hardcoded and hand-maintained, so it might not cover all
   /// headers.
   void addSystemHeadersMapping(const LangOptions &Language);
 
 private:
   /// A map from the private header to a canonical include path.
   llvm::DenseMap<llvm::sys::fs::UniqueID, std::string> FullPathMapping;
   /// A map from a suffix (one or components of a path) to a canonical path.
   /// Used only for mapping standard headers.
   const llvm::StringMap<llvm::StringRef> *StdSuffixHeaderMapping = nullptr;
-  /// A map from fully qualified symbol names to header names.
-  /// Used only for mapping standard symbols.
-  const llvm::StringMap<llvm::StringRef> *StdSymbolMapping = nullptr;
 };
 
 /// Returns a CommentHandler that parses pragma comment on include files to
 /// determine when we should include a different header from the header that
 /// directly defines a symbol. Mappinps are registered with \p Includes.
 ///
 /// Currently it only supports IWYU private pragma:
 /// https://github.com/include-what-you-use/include-what-you-use/blob/master/docs/IWYUPragmas.md#iwyu-pragma-private

clang-tools-extra/clangd/index/CanonicalIncludes.cpp

 //===-- CanonicalIncludes.h - remap #include headers-------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "CanonicalIncludes.h"
 #include "Headers.h"
 #include "clang/Basic/FileEntry.h"
 #include "clang/Tooling/Inclusions/HeaderAnalysis.h"
+#include "clang/Tooling/Inclusions/StandardLibrary.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/FileSystem/UniqueID.h"
 #include "llvm/Support/Path.h"
 #include <algorithm>
 
 namespace clang {
 namespace clangd {
 namespace {
for (auto It = llvm::sys::path::rbegin(Filename),
     auto SubPath = Filename.substr(It->data() - Filename.begin());
     auto MappingIt = StdSuffixHeaderMapping->find(SubPath);
     if (MappingIt != StdSuffixHeaderMapping->end())
       return MappingIt->second;
   }
   return "";
 }
 
-llvm::StringRef CanonicalIncludes::mapSymbol(llvm::StringRef QName) const {
-  return StdSymbolMapping ? StdSymbolMapping->lookup(QName) : "";
+llvm::StringRef CanonicalIncludes::mapSymbol(llvm::StringRef Scope,
+                                             llvm::StringRef Name,
+                                             const LangOptions &L) const {
+  tooling::stdlib::Lang Lang;
+  if (L.CPlusPlus)
+    Lang = tooling::stdlib::Lang::CXX;
+  else if (L.C11)
+    Lang = tooling::stdlib::Lang::C;
+  else
+    return "";
+  // FIXME: remove the following special cases when the tooling stdlib supports
+  // them.
+  // There are two std::move()s, this is by far the most common.
+  if (Scope == "std::" && Name == "move")
+    return "<utility>";
+  if (Scope == "std::" && Name == "size_t")
+    return "<cstddef>";
+  if (auto StdSym = tooling::stdlib::Symbol::named(Scope, Name, Lang))
+    return StdSym->header().name();
+  return "";
 }
 
 std::unique_ptr<CommentHandler>
 collectIWYUHeaderMaps(CanonicalIncludes *Includes) {
   class PragmaCommentHandler : public clang::CommentHandler {
   public:
     PragmaCommentHandler(CanonicalIncludes *Includes) : Includes(Includes) {}
 
public:
     }
 
   private:
     CanonicalIncludes *const Includes;
   };
   return std::make_unique<PragmaCommentHandler>(Includes);
 }
 
 void CanonicalIncludes::addSystemHeadersMapping(const LangOptions &Language) {
-  if (Language.CPlusPlus) {
-    static const auto *Symbols = new llvm::StringMap<llvm::StringRef>({
-#define SYMBOL(Name, NameSpace, Header) {#NameSpace #Name, #Header},
-#include "clang/Tooling/Inclusions/StdSymbolMap.inc"
-        // There are two std::move()s, this is by far the most common.
-        SYMBOL(move, std::, <utility>)
-        // There are multiple headers for size_t, pick one.
-        SYMBOL(size_t, std::, <cstddef>)
-#undef SYMBOL
-    });
-    StdSymbolMapping = Symbols;
-  } else if (Language.C11) {
-    static const auto *CSymbols = new llvm::StringMap<llvm::StringRef>({
-#define SYMBOL(Name, NameSpace, Header) {#Name, #Header},
-#include "clang/Tooling/Inclusions/CSymbolMap.inc"
-        // There are multiple headers for size_t, pick one.
-        SYMBOL(size_t, None, <stddef.h>)
-#undef SYMBOL
-    });
-    StdSymbolMapping = CSymbols;
-  }
-
   // FIXME: remove the std header mapping once we support ambiguous symbols, now
   // it serves as a fallback to disambiguate:
   //   - symbols with multiple headers (e.g. std::move)
   static const auto *SystemHeaderMap = [] {
     const size_t Size = sizeof(IncludeMappings) / sizeof(IncludeMappings[0]);
     auto *HeaderMap = new llvm::StringMap<llvm::StringRef>(Size);
     for (size_t I = 0; I < Size; I++) {
       auto Result = HeaderMap->insert(IncludeMappings[I]);
       (void)Result;

clang-tools-extra/clangd/index/StdLib.cpp

 #include "SymbolCollector.h"
 #include "index/IndexAction.h"
 #include "support/Logger.h"
 #include "support/ThreadsafeFS.h"
 #include "support/Trace.h"
 #include "clang/Basic/LangOptions.h"
 #include "clang/Frontend/CompilerInvocation.h"
 #include "clang/Lex/PreprocessorOptions.h"
+#include "clang/Tooling/Inclusions/StandardLibrary.h"
 #include "llvm/ADT/IntrusiveRefCntPtr.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Path.h"
 
 namespace clang {
 namespace clangd {
 namespace {
if (LO.C2x)
     return LangStandard::lang_c2x;
   // C17 has no new features, so treat {C11,C17} as C17.
   if (LO.C11)
     return LangStandard::lang_c17;
   return LangStandard::lang_c99;
 }
 
 std::string buildUmbrella(llvm::StringLiteral Mandatory,
-                          std::vector<llvm::StringLiteral> Headers) {
+                          llvm::ArrayRef<tooling::stdlib::Header> Headers) {
   std::string Result;
   llvm::raw_string_ostream OS(Result);
 
   // We __has_include guard all our #includes to avoid errors when using older
   // stdlib version that don't have headers for the newest language standards.
   // But make sure we get *some* error if things are totally broken.
   OS << llvm::formatv(
       "#if !__has_include(<{0}>)\n"
       "#error Mandatory header <{0}> not found in standard library!\n"
       "#endif\n",
       Mandatory);
 
-  llvm::sort(Headers);
-  auto Last = std::unique(Headers.begin(), Headers.end());
-  for (auto Header = Headers.begin(); Header != Last; ++Header) {
+  for (auto Header : Headers) {
     OS << llvm::formatv("#if __has_include({0})\n"
                         "#include {0}\n"
                         "#endif\n",
-                        *Header);
+                        Header);
   }
   OS.flush();
   return Result;
 }
 
 } // namespace
 
 llvm::StringRef getStdlibUmbrellaHeader(const LangOptions &LO) {
   // The umbrella header is the same for all versions of each language.
   // Headers that are unsupported in old lang versions are usually guarded by
   // #if. Some headers may be not present in old stdlib versions, the umbrella
   // header guards with __has_include for this purpose.
   Lang L = langFromOpts(LO);
   switch (L) {
   case CXX:
-    static std::string *UmbrellaCXX =
-        new std::string(buildUmbrella(mandatoryHeader(L), {
-#define SYMBOL(Name, NameSpace, Header) #Header,
-#include "clang/Tooling/Inclusions/StdSymbolMap.inc"
-#undef SYMBOL
-                                                          }));
+    static std::string *UmbrellaCXX = new std::string(buildUmbrella(
+        mandatoryHeader(L),
+        tooling::stdlib::Header::all(tooling::stdlib::Lang::CXX)));
     return *UmbrellaCXX;
   case C:
-    static std::string *UmbrellaC =
-        new std::string(buildUmbrella(mandatoryHeader(L), {
-#define SYMBOL(Name, NameSpace, Header) #Header,
-#include "clang/Tooling/Inclusions/CSymbolMap.inc"
-#undef SYMBOL
-                                                          }));
+    static std::string *UmbrellaC = new std::string(
+        buildUmbrella(mandatoryHeader(L),
+                      tooling::stdlib::Header::all(tooling::stdlib::Lang::C)));
     return *UmbrellaC;
   }
   llvm_unreachable("invalid Lang in langFromOpts");
 }
 
 namespace {
 
 // Including the standard library leaks unwanted transitively included symbols.

 // We require the symbol to come from a header that is *either* from
 // the standard library path (as identified by the location of <vector>), or
 // another header that defines a symbol from our stdlib list.
 SymbolSlab filter(SymbolSlab Slab, const StdLibLocation &Loc) {
   SymbolSlab::Builder Result;
 
   static auto &StandardHeaders = *[] {
     auto *Set = new llvm::DenseSet<llvm::StringRef>();
-    for (llvm::StringRef Header : {
-#define SYMBOL(Name, NameSpace, Header) #Header,
-#include "clang/Tooling/Inclusions/CSymbolMap.inc"
-#include "clang/Tooling/Inclusions/StdSymbolMap.inc"
-#undef SYMBOL
-         })
-      Set->insert(Header);
+    for (auto Header : tooling::stdlib::Header::all(tooling::stdlib::Lang::CXX))
+      Set->insert(Header.name());
+    for (auto Header : tooling::stdlib::Header::all(tooling::stdlib::Lang::C))
+      Set->insert(Header.name());
     return Set;
   }();
 
   // Form prefixes like file:///usr/include/c++/10/
   // These can be trivially prefix-compared with URIs in the indexed symbols.
   llvm::SmallVector<std::string> StdLibURIPrefixes;
   for (const auto &Path : Loc.Paths) {
     StdLibURIPrefixes.push_back(URI::create(Path).toString());

clang-tools-extra/clangd/index/SymbolCollector.cpp

for (const IdentifierInfo *II : IndexedMacros) {
                 getSymbolIDCached(II->getName(), MI, PP->getSourceManager()))
           if (MI->isUsedForHeaderGuard())
             Symbols.erase(ID);
     }
   }
   llvm::DenseMap<FileID, bool> FileToContainsImportsOrObjC;
   // Fill in IncludeHeaders.
   // We delay this until end of TU so header guards are all resolved.
-  llvm::SmallString<128> QName;
   for (const auto &[SID, FID] : IncludeFiles) {
     if (const Symbol *S = Symbols.find(SID)) {
       llvm::StringRef IncludeHeader;
       // Look for an overridden include header for this symbol specifically.
       if (Opts.Includes) {
-        QName = S->Scope;
-        QName.append(S->Name);
-        IncludeHeader = Opts.Includes->mapSymbol(QName);
+        IncludeHeader =
+            Opts.Includes->mapSymbol(S->Scope, S->Name, ASTCtx->getLangOpts());
         if (!IncludeHeader.empty()) {
           if (IncludeHeader.front() != '"' && IncludeHeader.front() != '<')
             IncludeHeader = HeaderFileURIs->toURI(IncludeHeader);
-          else if (IncludeHeader == "<utility>" && QName == "std::move" &&
-                   S->Signature.contains(','))
+          else if (IncludeHeader == "<utility>" && S->Scope == "std::" &&
+                   S->Name == "move" && S->Signature.contains(','))
             IncludeHeader = "<algorithm>";
         }
       }
       // Otherwise find the approprate include header for the defining file.
       if (IncludeHeader.empty())
         IncludeHeader = HeaderFileURIs->getIncludeHeader(FID);
 
       // Symbols in slabs aren't mutable, insert() has to walk all the strings

clang-tools-extra/clangd/unittests/CanonicalIncludesTests.cpp

 }
 
 TEST(CanonicalIncludesTest, CStandardLibrary) {
   CanonicalIncludes CI;
   auto Language = LangOptions();
   Language.C11 = true;
   CI.addSystemHeadersMapping(Language);
   // Usual standard library symbols are mapped correctly.
-  EXPECT_EQ("<stdio.h>", CI.mapSymbol("printf"));
-  EXPECT_EQ("", CI.mapSymbol("unknown_symbol"));
+  EXPECT_EQ("<stdio.h>", CI.mapSymbol("", "printf", Language));
+  EXPECT_EQ("", CI.mapSymbol("", "unknown_symbol", Language));
 }
 
 TEST(CanonicalIncludesTest, CXXStandardLibrary) {
   CanonicalIncludes CI;
   auto Language = LangOptions();
   Language.CPlusPlus = true;
   CI.addSystemHeadersMapping(Language);
 
   // Usual standard library symbols are mapped correctly.
-  EXPECT_EQ("<vector>", CI.mapSymbol("std::vector"));
-  EXPECT_EQ("<cstdio>", CI.mapSymbol("std::printf"));
+  EXPECT_EQ("<vector>", CI.mapSymbol("std::", "vector", Language));
+  EXPECT_EQ("<cstdio>", CI.mapSymbol("std::", "printf", Language));
   // std::move is ambiguous, currently always mapped to <utility>
-  EXPECT_EQ("<utility>", CI.mapSymbol("std::move"));
+  EXPECT_EQ("<utility>", CI.mapSymbol("std::", "move", Language));
   // Unknown std symbols aren't mapped.
-  EXPECT_EQ("", CI.mapSymbol("std::notathing"));
+  EXPECT_EQ("", CI.mapSymbol("std::", "notathing", Language));
   // iosfwd declares some symbols it doesn't own.
-  EXPECT_EQ("<ostream>", CI.mapSymbol("std::ostream"));
+  EXPECT_EQ("<ostream>", CI.mapSymbol("std::", "ostream", Language));
   // And (for now) we assume it owns the others.
   auto InMemFS = llvm::makeIntrusiveRefCnt<llvm::vfs::InMemoryFileSystem>();
   FileManager Files(FileSystemOptions(), InMemFS);
   auto File = addFile(*InMemFS, Files, testPath("iosfwd"));
   EXPECT_EQ("<iosfwd>", CI.mapHeader(File));
 }
 
 TEST(CanonicalIncludesTest, PathMapping) {