[include-mapping] Implement language separation in stdlib recognizer library

[VitaNuo]

Remove comments.

[VitaNuo]

Minor improvements.

[hokein]

Can you add some tests in StandardLibraryTest.cpp?

clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp

31

using a map here seems like an overkill, we have just 2 elements, I'd just use two separate variables (CMapping, and CXXMapping)

100

nit: just for (Lang Language: {Lang::C, Lang::CXX}) or two statements initilize(Lang::C); and initialize(Lang::CXX);.

164

Do we need the ensureInitialized() here? looks like no needed, we have called it in the Recognizer constructor,

188

nit: just use LangOpts.CPlusPlus to check the language.

[VitaNuo]

Address review comments.

[VitaNuo]

Thanks for the comments!

clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp

31

what about the design idea that we might potentially want to extend this to multiple standards etc.? The idea is that it's extensible to ObjC, OpenCL... and so on and so forth, as has been discussed offline.

100

yes, same argument as above. I remember extensive discussions about the idea that we might want to extend this to multiple language versions etc. in the future.

164

you're right, not needed.

188

There's LangStandard::isCPlusPlus method that I've just discovered. That's probably even better.

[VitaNuo]

Add a couple more test cases to StandardLibraryTest.cpp.

[VitaNuo]

Added a couple of test cases.

[hokein]

clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp

31

I think having a generic Lang enum structure is sufficient for the future extension, and I don't think we're going to add other languages in the foreseeable future (that's why I value the simple implementation at the beginning).

But you're right, getting the implementation right is probably a good idea. I'd like to remove the DenseMap, just use a raw array, something like below should work

enum Lang {
   C = 0,
   CXX,

   LastValue = CXX,
};

// access by e.g. LanguageMappings[static_cast<unsigned>(Lang::C)].
static SymbolHeaderMapping* LanguageMappings[static_cast<unsigned>(Lang::LastValue) + 1];

188

sorry if I was not clearer -- there is a bit CPlusPlus in the LangOptions which already does the equivalent thing (see https://github.com/llvm/llvm-project/blob/main/clang/lib/Basic/LangOptions.cpp#L107).

So we can just use if (D->getLangOpts().CPlusPlus).

clang/unittests/Tooling/StandardLibraryTest.cpp

115 ↗

(On Diff #493918)

nit: just EXPECT_FALSE(stdlib::Symbol::named("", "int16_t"))), following the existing pattern L46.

122 ↗

(On Diff #493918)

The existing TEST(StdlibTest, All) test seems like a good umbrella for the this test and the above test, how about moving them to the All test?

[VitaNuo]

Address review comments.

[VitaNuo]

Remove extra include.

[VitaNuo]

Thanks for the comments!

clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp

31

Ok, this will need some casting, but I don't have a strong opinion.

188

Ah ok, thanks.

clang/unittests/Tooling/StandardLibraryTest.cpp

115 ↗

(On Diff #493918)

Right, thanks.

122 ↗

(On Diff #493918)

Ok, I've moved everything to existing test cases.

[hokein]

Thanks, this looks good to me!

This requires some work to rebase for https://github.com/llvm/llvm-project/commit/e1aaa314a46cd303019da117bfd330611d5b7a84, I will rebase it and land it for you.

clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp

31

yeah, it is the sad bit, but I think it is OK.

[hokein]

rebase to main

clang/include/clang/Tooling/Inclusions/StandardLibrary.h

 namespace clang {
 class Decl;
 class NamespaceDecl;
 class DeclContext;
 namespace tooling {
 namespace stdlib {
 
 class Symbol;
+enum class Lang { C, CXX };
+
+constexpr std::initializer_list<Lang> AllLangs = {Lang::C, Lang::CXX};
 
 // A standard library header, such as <iostream>
 // Lightweight class, in fact just an index into a table.
 // C++ and C Library compatibility headers are considered different: e.g.
 // "<cstdio>" and "<stdio.h>" (and their symbols) are treated differently.
 class Header {
 public:
   // Name should contain the angle brackets, e.g. "<vector>".
-  static std::optional<Header> named(llvm::StringRef Name);
+  static std::optional<Header> named(llvm::StringRef Name,
+                                     Lang Language = Lang::CXX);
 
   friend llvm::raw_ostream &operator<<(llvm::raw_ostream &OS, const Header &H) {
     return OS << H.name();
   }
   llvm::StringRef name() const;
 
 private:
-  Header(unsigned ID) : ID(ID) {}
+  Header(unsigned ID, Lang Language = Lang::CXX) : ID(ID), Language(Language) {}
   unsigned ID;
+  Lang Language;
+
   friend Symbol;
   friend llvm::DenseMapInfo<Header>;
   friend bool operator==(const Header &L, const Header &R) {
     return L.ID == R.ID;
   }
 };
 
 // A top-level standard library symbol, such as std::vector
 // Lightweight class, in fact just an index into a table.
 // C++ and C Standard Library symbols are considered distinct: e.g. std::printf
 // and ::printf are not treated as the same symbol.
 // The symbols do not contain macros right now, we don't have a reliable index
 // for them.
 class Symbol {
 public:
   /// \p Scope should have the trailing "::", for example:
   /// named("std::chrono::", "system_clock")
-  static std::optional<Symbol> named(llvm::StringRef Scope,
-                                      llvm::StringRef Name);
+  static std::optional<Symbol>
+  named(llvm::StringRef Scope, llvm::StringRef Name, Lang Language = Lang::CXX);
 
   friend llvm::raw_ostream &operator<<(llvm::raw_ostream &OS, const Symbol &S) {
     return OS << S.scope() << S.name();
   }
   llvm::StringRef scope() const;
   llvm::StringRef name() const;
   // The preferred header for this symbol (e.g. the suggested insertion).
   Header header() const;
   // Some symbols may be provided by multiple headers.
   llvm::SmallVector<Header> headers() const;
 
 private:
-  Symbol(unsigned ID) : ID(ID) {}
+  Symbol(unsigned ID, Lang Language = Lang::CXX) : ID(ID), Language(Language) {}
   unsigned ID;
+  clang::tooling::stdlib::Lang Language;
   friend class Recognizer;
   friend llvm::DenseMapInfo<Symbol>;
   friend bool operator==(const Symbol &L, const Symbol &R) {
     return L.ID == R.ID;
   }
 };
 
 // A functor to find the stdlib::Symbol associated with a decl.
 //
 // For non-top-level decls (std::vector<int>::iterator), returns the top-level
 // symbol (std::vector).
 class Recognizer {
 public:
   Recognizer();
   std::optional<Symbol> operator()(const Decl *D);
 
 private:
   using NSSymbolMap = llvm::DenseMap<llvm::StringRef, unsigned>;
-  NSSymbolMap *namespaceSymbols(const NamespaceDecl *D);
+  NSSymbolMap *namespaceSymbols(const NamespaceDecl *D,
+                                Lang Language = Lang::CXX);
   llvm::DenseMap<const DeclContext *, NSSymbolMap *> NamespaceCache;
 };
 
 } // namespace stdlib
 } // namespace tooling
 } // namespace clang
 
 namespace llvm {
template <> struct DenseMapInfo<clang::tooling::stdlib::Symbol> {
   static unsigned getHashValue(const clang::tooling::stdlib::Symbol &S) {
     return hash_value(S.ID);
   }
   static bool isEqual(const clang::tooling::stdlib::Symbol &LHS,
                       const clang::tooling::stdlib::Symbol &RHS) {
     return LHS == RHS;
   }
 };
+
+template <> struct DenseMapInfo<clang::tooling::stdlib::Lang> {
+  static inline clang::tooling::stdlib::Lang getEmptyKey() {
+    return clang::tooling::stdlib::Lang(-1);
+  }
+  static inline clang::tooling::stdlib::Lang getTombstoneKey() {
+    return clang::tooling::stdlib::Lang(-2);
+  }
+  static unsigned getHashValue(const clang::tooling::stdlib::Lang &S) {
+    return hash_value(S);
+  }
+  static bool isEqual(const clang::tooling::stdlib::Lang &LHS,
+                      const clang::tooling::stdlib::Lang &RHS) {
+    return LHS == RHS;
+  }
+};
 } // namespace llvm
 
 #endif // LLVM_CLANG_TOOLING_INCLUSIONS_STANDARDLIBRARY_H

clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp

 //===--- StandardLibrary.cpp ------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "clang/Tooling/Inclusions/StandardLibrary.h"
 #include "clang/AST/Decl.h"
+#include "clang/Basic/LangOptions.h"
+#include "clang/Basic/LangStandard.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/Casting.h"
 
 namespace clang {
 namespace tooling {
 namespace stdlib {
 
-static llvm::StringRef *HeaderNames;
-static std::pair<llvm::StringRef, llvm::StringRef> *SymbolNames;
-static unsigned *SymbolHeaderIDs;
-static llvm::DenseMap<llvm::StringRef, unsigned> *HeaderIDs;
 // Maps symbol name -> Symbol::ID, within a namespace.
 using NSSymbolMap = llvm::DenseMap<llvm::StringRef, unsigned>;
-static llvm::DenseMap<llvm::StringRef, NSSymbolMap *> *NamespaceSymbols;
 
-static int initialize() {
+struct SymbolHeaderMapping {
+  llvm::StringRef *HeaderNames;
+  std::pair<llvm::StringRef, llvm::StringRef> *SymbolNames;
+  unsigned *SymbolHeaderIDs;
+  llvm::DenseMap<llvm::StringRef, unsigned> HeaderIDs;
+  llvm::DenseMap<llvm::StringRef, NSSymbolMap *> NamespaceSymbols;
+};
+
+static llvm::DenseMap<Lang, SymbolHeaderMapping *> LanguageMappings;
+
+static int countSymbols(Lang Language) {
   unsigned SymCount = 0;
 #define SYMBOL(Name, NS, Header) ++SymCount;
+  switch (Language) {
+  case Lang::C:
 #include "clang/Tooling/Inclusions/CSymbolMap.inc"
+    break;
+  case Lang::CXX:
 #include "clang/Tooling/Inclusions/StdSymbolMap.inc"
+    break;
+  }
 #undef SYMBOL
-  SymbolNames = new std::remove_reference_t<decltype(*SymbolNames)>[SymCount];
-  SymbolHeaderIDs =
-      new std::remove_reference_t<decltype(*SymbolHeaderIDs)>[SymCount];
-  NamespaceSymbols = new std::remove_reference_t<decltype(*NamespaceSymbols)>;
-  HeaderIDs = new std::remove_reference_t<decltype(*HeaderIDs)>;
+  return SymCount;
+}
+
+static void initialize(Lang Language) {
+  SymbolHeaderMapping *Mapping = new SymbolHeaderMapping();
+  LanguageMappings.try_emplace(Language, Mapping);
+
+  unsigned SymCount = countSymbols(Language);
+  Mapping->SymbolNames =
+      new std::remove_reference_t<decltype(*Mapping->SymbolNames)>[SymCount];
+  Mapping->SymbolHeaderIDs = new std::remove_reference_t<
+      decltype(*Mapping->SymbolHeaderIDs)>[SymCount];
 
   auto AddNS = [&](llvm::StringRef NS) -> NSSymbolMap & {
-    auto R = NamespaceSymbols->try_emplace(NS, nullptr);
+    auto R = Mapping->NamespaceSymbols.try_emplace(NS, nullptr);
     if (R.second)
       R.first->second = new NSSymbolMap();
     return *R.first->second;
   };
 
   auto AddHeader = [&](llvm::StringRef Header) -> unsigned {
-    return HeaderIDs->try_emplace(Header, HeaderIDs->size()).first->second;
+    return Mapping->HeaderIDs.try_emplace(Header, Mapping->HeaderIDs.size())
+        .first->second;
   };
 
   auto Add = [&, SymIndex(0)](llvm::StringRef Name, llvm::StringRef NS,
                               llvm::StringRef HeaderName) mutable {
     if (NS == "None")
       NS = "";
 
-    SymbolNames[SymIndex] = {NS, Name};
-    SymbolHeaderIDs[SymIndex] = AddHeader(HeaderName);
+    Mapping->SymbolNames[SymIndex] = {NS, Name};
+    Mapping->SymbolHeaderIDs[SymIndex] = AddHeader(HeaderName);
 
     NSSymbolMap &NSSymbols = AddNS(NS);
     NSSymbols.try_emplace(Name, SymIndex);
 
     ++SymIndex;
   };
 #define SYMBOL(Name, NS, Header) Add(#Name, #NS, #Header);
+  switch (Language) {
+  case Lang::C:
 #include "clang/Tooling/Inclusions/CSymbolMap.inc"
+    break;
+  case Lang::CXX:
 #include "clang/Tooling/Inclusions/StdSymbolMap.inc"
+    break;
+  }
 #undef SYMBOL
 
-  HeaderNames = new llvm::StringRef[HeaderIDs->size()];
-  for (const auto &E : *HeaderIDs)
-    HeaderNames[E.second] = E.first;
+  Mapping->HeaderNames = new llvm::StringRef[Mapping->HeaderIDs.size()];
+  for (const auto &E : Mapping->HeaderIDs)
+    Mapping->HeaderNames[E.second] = E.first;
+}
 
+static int initialize() {
+  for (Lang Language : AllLangs)
+    initialize(Language);
   return 0;
 }
 
 static void ensureInitialized() {
   static int Dummy = initialize();
   (void)Dummy;
 }
 
-std::optional<Header> Header::named(llvm::StringRef Name) {
+static SymbolHeaderMapping *GetMapping(Lang Language) {
+  auto MapIt = LanguageMappings.find(Language);
+  SymbolHeaderMapping *Mapping = MapIt->getSecond();
+  return Mapping;
+}
+
+std::optional<Header> Header::named(llvm::StringRef Name, Lang Language) {
   ensureInitialized();
-  auto It = HeaderIDs->find(Name);
-  if (It == HeaderIDs->end())
+  SymbolHeaderMapping *Mapping = GetMapping(Language);
+  auto It = Mapping->HeaderIDs.find(Name);
+  if (It == Mapping->HeaderIDs.end())
     return std::nullopt;
-  return Header(It->second);
+  return Header(It->second, Language);
+}
+
+llvm::StringRef Header::name() const {
+  SymbolHeaderMapping *Mapping = GetMapping(Language);
+  return Mapping->HeaderNames[ID];
 }
-llvm::StringRef Header::name() const { return HeaderNames[ID]; }
-llvm::StringRef Symbol::scope() const { return SymbolNames[ID].first; }
-llvm::StringRef Symbol::name() const { return SymbolNames[ID].second; }
-std::optional<Symbol> Symbol::named(llvm::StringRef Scope,
-                                     llvm::StringRef Name) {
+llvm::StringRef Symbol::scope() const {
+  SymbolHeaderMapping *Mapping = GetMapping(Language);
+  return Mapping->SymbolNames[ID].first;
+}
+llvm::StringRef Symbol::name() const {
+  SymbolHeaderMapping *Mapping = GetMapping(Language);
+  return Mapping->SymbolNames[ID].second;
+}
+std::optional<Symbol> Symbol::named(llvm::StringRef Scope, llvm::StringRef Name,
+                                    Lang Language) {
   ensureInitialized();
-  if (NSSymbolMap *NSSymbols = NamespaceSymbols->lookup(Scope)) {
+  SymbolHeaderMapping *Mapping = GetMapping(Language);
+  if (NSSymbolMap *NSSymbols = Mapping->NamespaceSymbols.lookup(Scope)) {
     auto It = NSSymbols->find(Name);
     if (It != NSSymbols->end())
-      return Symbol(It->second);
+      return Symbol(It->second, Language);
   }
   return std::nullopt;
 }
-Header Symbol::header() const { return Header(SymbolHeaderIDs[ID]); }
+Header Symbol::header() const {
+  SymbolHeaderMapping *Mapping = GetMapping(Language);
+  return Header(Mapping->SymbolHeaderIDs[ID], Language);
+}
 llvm::SmallVector<Header> Symbol::headers() const {
   return {header()}; // FIXME: multiple in case of ambiguity
 }
 
 Recognizer::Recognizer() { ensureInitialized(); }
 
-NSSymbolMap *Recognizer::namespaceSymbols(const NamespaceDecl *D) {
+NSSymbolMap *Recognizer::namespaceSymbols(const NamespaceDecl *D,
+                                          Lang Language) {
   auto It = NamespaceCache.find(D);
   if (It != NamespaceCache.end())
     return It->second;
 
+  ensureInitialized();
+  SymbolHeaderMapping *Mapping = GetMapping(Language);
+
   NSSymbolMap *Result = [&]() -> NSSymbolMap * {
     if (D && D->isAnonymousNamespace())
       return nullptr;
     // Print the namespace and its parents ommitting inline scopes.
     std::string Scope;
     for (const auto *ND = D; ND;
          ND = llvm::dyn_cast_or_null<NamespaceDecl>(ND->getParent()))
       if (!ND->isInlineNamespace() && !ND->isAnonymousNamespace())
         Scope = ND->getName().str() + "::" + Scope;
-    return NamespaceSymbols->lookup(Scope);
+    return Mapping->NamespaceSymbols.lookup(Scope);
   }();
   NamespaceCache.try_emplace(D, Result);
   return Result;
 }
 
 std::optional<Symbol> Recognizer::operator()(const Decl *D) {
+  const LangOptions &LangOpts = D->getLangOpts();
+  const LangStandard &Std =
+      LangStandard::getLangStandardForKind(LangOpts.LangStd);
+  clang::Language Language = Std.getLanguage();
+  Lang RecognizerLang = Lang::CXX;
+  if (Language == clang::Language::C) {
+    RecognizerLang = Lang::C;
+  }
+
   // If D is std::vector::iterator, `vector` is the outer symbol to look up.
   // We keep all the candidate DCs as some may turn out to be anon enums.
   // Do this resolution lazily as we may turn out not to have a std namespace.
   llvm::SmallVector<const DeclContext *> IntermediateDecl;
   const DeclContext *DC = D->getDeclContext();
   while (DC && !DC->isNamespace()) {
     if (NamedDecl::classofKind(DC->getDeclKind()))
       IntermediateDecl.push_back(DC);
     DC = DC->getParent();
   }
-  NSSymbolMap *Symbols = namespaceSymbols(cast_or_null<NamespaceDecl>(DC));
+  NSSymbolMap *Symbols =
+      namespaceSymbols(cast_or_null<NamespaceDecl>(DC), RecognizerLang);
   if (!Symbols)
     return std::nullopt;
 
   llvm::StringRef Name = [&]() -> llvm::StringRef {
     for (const auto *SymDC : llvm::reverse(IntermediateDecl)) {
       DeclarationName N = cast<NamedDecl>(SymDC)->getDeclName();
       if (const auto *II = N.getAsIdentifierInfo())
         return II->getName();