[lld] Add support for other demanglers other than Itanium

[ljmf00]

Summary

LLVM core library supports demangling other mangled symbols other than itanium, such as D and Rust. LLD should use those demanglers in order to output pretty demangled symbols on error messages.

[MaskRay]

lld/Common/Strings.cpp

27

The tests are in case demangle(...) is slow on non-mangled symbols.

I have linked chrome with --dynamic-list specifying a extern "C" version node and don't see much difference, so this seems fine.

lld/ELF/Symbols.cpp

31

Prefer config whenever possible.

[ljmf00]

lld/Common/Strings.cpp

27

What do you mean by this? Is this validation or asking to be kept as is? There are checks against this already on llvm::demangle. See https://github.com/llvm/llvm-project/blob/main/llvm/lib/Demangle/Demangle.cpp#L51 . This includes checks not only for Itanium encoding but also for D and Rust.

[ljmf00]

lld/ELF/Symbols.cpp

31

Changed.

[MaskRay]

lld/Common/Strings.cpp

27

No action needed.

See SymbolTable::getDemangledSyms. When extern "C++" is used for ld.lld --dynamic-list, every symbol is demangled. A large executable may have millions of symbols (chrome), so the performance matters.

1559632 of 1571293 symbols in chrome's symbol table starts with _, so the check may be redundant.

lld/include/lld/Common/Strings.h

21

llvm-project/lld nearly doesn't use doxygen \param \returns. The original comment style is sufficiently clear. The new \param just adds information a user can easily infer.

[ljmf00]

I built and checked with x86 and experimental WASM target locally on a Linux environment.

lld/Common/Strings.cpp

27

Ok, thanks for clarifying. Yes, right now this check is restrictive and redundant.

lld/include/lld/Common/Strings.h

21

Reverted. Just removed the C++ restriction.

[MaskRay]

Looks great!

[ljmf00]

I reverted this to not congestionante other builds and described the reason on the revert commit. I will do a clean build with all targets and try to reproduce this. I only tested this on x86 and WebAssembly targets since I haven't seen target specific code here. Additionally, I've tested this on an x86_64 environment.

[MaskRay]

The failure before the revert was:

tools/lld/MachO/CMakeFiles/lldMachO.dir/Symbols.cpp.o: In function `lld::demangle(llvm::StringRef, bool)':

This affects common code and all ports may be affected. You can use ninja check-lld to test all ports. not just check-lld-wasm or check-lld-elf

[ljmf00]

In D116279#3214173, @MaskRay wrote:

The failure before the revert was:

tools/lld/MachO/CMakeFiles/lldMachO.dir/Symbols.cpp.o: In function `lld::demangle(llvm::StringRef, bool)':

This affects common code and all ports may be affected. You can use ninja check-lld to test all ports. not just check-lld-wasm or check-lld-elf

I've done it but only with "X86" on LLVM_TARGETS_TO_BUILD, since this can reduce build times. I've done a clean build with all targets and additionally the WebAssembly experimental target and the testsuite ran successfully with ninja check-lld. That said, I can't reproduce the issue in my environment. I can investigate further, but I don't have a PowerPC64 environment.

[ljmf00]

I finally reproduced this locally. The buildbot is configured with BUILD_SHARED_LIBS and my builds are built without it and use LLVM_LINK_LLVM_DYLIB and CLANG_LINK_CLANG_DYLIB respectively. Since I made lld::demangle symbol inline, Demangle LLVM component was missing on MachO. I added that component to the link list on CMakeLists.txt.

[MaskRay]

If you don't contribute to llvm-project, LLVM_TARGETS_TO_BUILD=host or just X86 is fine. For contribution, it's a good idea to enable all targets, otherwise some tests may be skipped.

(The missing Demangle dependency issue)
A -DBUILD_SHARED_LIBS=on build uses -Wl,-z,defs to link shared objects. It has some dependency checking effects: https://maskray.me/blog/2021-06-13-dependency-related-linker-options#z-defs

Personally I use -DBUILD_SHARED_LIBS=on for Debug build and -DBUILD_SHARED_LIBS=off for Release build, and ensure I built both modes if I touch CMake files.

[ljmf00]

In D116279#3214270, @MaskRay wrote:

If you don't contribute to llvm-project, LLVM_TARGETS_TO_BUILD=host or just X86 is fine. For contribution, it's a good idea to enable all targets, otherwise some tests may be skipped.

Yeah, I'm going to start doing that.

(The missing Demangle dependency issue)
A -DBUILD_SHARED_LIBS=on build uses -Wl,-z,defs to link shared objects. It has some dependency checking effects: https://maskray.me/blog/2021-06-13-dependency-related-linker-options#z-defs

Personally I use -DBUILD_SHARED_LIBS=on for Debug build and -DBUILD_SHARED_LIBS=off for Release build, and ensure I built both modes if I touch CMake files.

Thanks for all that information. I didn't know about that. I will start building with it :)

lld/COFF/Symbols.cpp

static std::string maybeDemangleSymbol(StringRef symName) {
   if (config->demangle) {
     std::string prefix;
     StringRef prefixless = symName;
     if (prefixless.consume_front("__imp_"))
       prefix = "__declspec(dllimport) ";
     StringRef demangleInput = prefixless;
     if (config->machine == I386)
       demangleInput.consume_front("_");
-    std::string demangled = demangle(std::string(demangleInput));
+    std::string demangled = demangle(demangleInput, true);
     if (demangled != demangleInput)
-      return prefix + demangle(std::string(demangleInput));
+      return prefix + demangle(demangleInput, true);
     return (prefix + prefixless).str();
   }
   return std::string(symName);
 }
 std::string toString(coff::Symbol &b) {
   return maybeDemangleSymbol(b.getName());
 }
 std::string toCOFFString(const Archive::Symbol &b) {

lld/Common/Strings.cpp

 //===- Strings.cpp -------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "lld/Common/Strings.h"
 #include "lld/Common/ErrorHandler.h"
 #include "lld/Common/LLVM.h"
-#include "llvm/Demangle/Demangle.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/GlobPattern.h"
 #include <algorithm>
 #include <mutex>
 #include <vector>
 
 using namespace llvm;
 using namespace lld;
 
-// Returns the demangled C++ symbol name for name.
-std::string lld::demangleItanium(StringRef name) {
-  // demangleItanium() can be called for all symbols. Only demangle C++ symbols,
-  // to avoid getting unexpected result for a C symbol that happens to match a
-  // mangled type name such as "Pi" (which would demangle to "int*").
-  if (!name.startswith("_Z") && !name.startswith("__Z") &&
-      !name.startswith("___Z") && !name.startswith("____Z"))
-    return std::string(name);
-
-  return demangle(std::string(name));
-}
-
 SingleStringMatcher::SingleStringMatcher(StringRef Pattern) {
   if (Pattern.size() > 2 && Pattern.startswith("\"") &&
       Pattern.endswith("\"")) {
     ExactMatch = true;
     ExactPattern = Pattern.substr(1, Pattern.size() - 2);
   } else {
     Expected<GlobPattern> Glob = GlobPattern::create(Pattern);
     if (!Glob) {

lld/ELF/SymbolTable.cpp

StringMap<std::vector<Symbol *>> &SymbolTable::getDemangledSyms() {
   if (!demangledSyms) {
     demangledSyms.emplace();
     std::string demangled;
     for (Symbol *sym : symVector)
       if (canBeVersioned(*sym)) {
         StringRef name = sym->getName();
         size_t pos = name.find('@');
         if (pos == std::string::npos)
-          demangled = demangleItanium(name);
+          demangled = demangle(name, config->demangle);
         else if (pos + 1 == name.size() || name[pos + 1] == '@')
-          demangled = demangleItanium(name.substr(0, pos));
+          demangled = demangle(name.substr(0, pos), config->demangle);
         else
-          demangled =
-              (demangleItanium(name.substr(0, pos)) + name.substr(pos)).str();
+          demangled = (demangle(name.substr(0, pos), config->demangle) +
+                       name.substr(pos))
+                          .str();
         (*demangledSyms)[demangled].push_back(sym);
       }
   }
   return *demangledSyms;
 }
 
 std::vector<Symbol *> SymbolTable::findByVersion(SymbolVersion ver) {
   if (ver.isExternCpp)

lld/ELF/Symbols.cpp

 #include <cstring>
 
 using namespace llvm;
 using namespace llvm::object;
 using namespace llvm::ELF;
 using namespace lld;
 using namespace lld::elf;
 
-// Returns a symbol for an error message.
-static std::string demangle(StringRef symName) {
-  if (elf::config->demangle)
-    return demangleItanium(symName);
-  return std::string(symName);
-}
-
 std::string lld::toString(const elf::Symbol &sym) {
   StringRef name = sym.getName();
-  std::string ret = demangle(name);
+  std::string ret = demangle(name, config->demangle);
 
   const char *suffix = sym.getVersionSuffix();
   if (*suffix == '@')
     ret += suffix;
   return ret;
 }
 
 std::string lld::toELFString(const Archive::Symbol &b) {
-  return demangle(b.getName());
+  return demangle(b.getName(), config->demangle);
 }
 
 Defined *ElfSym::bss;
 Defined *ElfSym::etext1;
 Defined *ElfSym::etext2;
 Defined *ElfSym::edata1;
 Defined *ElfSym::edata2;
 Defined *ElfSym::end1;

lld/MachO/Symbols.cpp

 //===- Symbols.cpp --------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "Symbols.h"
 #include "InputFiles.h"
 #include "SyntheticSections.h"
+#include "lld/Common/Strings.h"
 
 using namespace llvm;
 using namespace lld;
 using namespace lld::macho;
 
 static_assert(sizeof(void *) != 8 || sizeof(Symbol) == 48,
               "Try to minimize Symbol's size; we create many instances");
 
 // The Microsoft ABI doesn't support using parent class tail padding for child
 // members, hence the _MSC_VER check.
 #if !defined(_MSC_VER)
 static_assert(sizeof(void *) != 8 || sizeof(Defined) == 80,
               "Try to minimize Defined's size; we create many instances");
 #endif
 
 static_assert(sizeof(SymbolUnion) == sizeof(Defined),
               "Defined should be the largest Symbol kind");
 
-// Returns a symbol for an error message.
-static std::string demangle(StringRef symName) {
-  if (config->demangle)
-    return demangleItanium(symName);
-  return std::string(symName);
+std::string lld::toString(const Symbol &sym) {
+  return demangle(sym.getName(), config->demangle);
 }
 
-std::string lld::toString(const Symbol &sym) { return demangle(sym.getName()); }
-
 std::string lld::toMachOString(const object::Archive::Symbol &b) {
-  return demangle(b.getName());
+  return demangle(b.getName(), config->demangle);
 }
 
 uint64_t Symbol::getStubVA() const { return in.stubs->getVA(stubsIndex); }
 uint64_t Symbol::getGotVA() const { return in.got->getVA(gotIndex); }
 uint64_t Symbol::getTlvVA() const { return in.tlvPointers->getVA(gotIndex); }
 
 Defined::Defined(StringRefZ name, InputFile *file, InputSection *isec,
                  uint64_t value, uint64_t size, bool isWeakDef, bool isExternal,

lld/include/lld/Common/Strings.h

 //===- Strings.h ------------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLD_STRINGS_H
 #define LLD_STRINGS_H
 
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/StringRef.h"
+#include "llvm/Demangle/Demangle.h"
 #include "llvm/Support/GlobPattern.h"
 #include <string>
 #include <vector>
 
 namespace lld {
-// Returns a demangled C++ symbol name. If Name is not a mangled
-// name, it returns name.
-std::string demangleItanium(llvm::StringRef name);
+/// Demangle a non-Microsoft mangled symbol.
+///
+/// \param symName mangled symbol name
+/// \param shouldDemangle flag to determine if the symbol should be demangled
+///
+/// \returns Demagled symbol or symName if it is not a valid mangled name.
+inline std::string demangle(llvm::StringRef symName, bool shouldDemangle) {
+  if (shouldDemangle)
+    return llvm::demangle(symName.str().c_str());
+  return std::string(symName);
+}
 
 std::vector<uint8_t> parseHex(llvm::StringRef s);
 bool isValidCIdentifier(llvm::StringRef s);
 
 // Write the contents of the a buffer to a file
 void saveBuffer(llvm::StringRef buffer, const llvm::Twine &path);
 
 // A single pattern to match against. A pattern can either be double-quoted

lld/test/ELF/undef.s

 # CHECK-NEXT: >>> referenced by undef.s
 # CHECK-NEXT: >>>               {{.*}}:(.text+0x15)
 # CHECK-NEXT: >>> the vtable symbol may be undefined because the class is missing its key function (see https://lld.llvm.org/missingkeyfunction)
 
 # CHECK:      error: undefined symbol: foo(int)
 # CHECK-NEXT: >>> referenced by undef.s
 # CHECK-NEXT: >>>               {{.*}}:(.text+0x1A)
 
+# CHECK:      error: undefined symbol: Pi
+# CHECK-NEXT: >>> referenced by undef.s
+# CHECK-NEXT: >>>               {{.*}}:(.text+0x1F)
+
+# CHECK:      error: undefined symbol: D main
+# CHECK-NEXT: >>> referenced by undef.s
+# CHECK-NEXT: >>>               {{.*}}:(.text+0x24)
+
+# CHECK:      error: undefined symbol: a::main
+# CHECK-NEXT: >>> referenced by undef.s
+# CHECK-NEXT: >>>               {{.*}}:(.text+0x29)
+
 # CHECK:      error: undefined symbol: zed2
 # CHECK-NEXT: >>> referenced by {{.*}}.o:(.text+0x0) in archive {{.*}}2.a
 
 # CHECK:      error: undefined symbol: zed3
 # CHECK-NEXT: >>> referenced by undef-debug.s:3 (dir{{/|\\}}undef-debug.s:3)
 # CHECK-NEXT: >>>               {{.*}}.o:(.text+0x0)
 
 # CHECK:      error: undefined symbol: zed4
.file "undef.s"
   .globl _start
 _start:
   call foo
   call bar
   call zed1
   call _Z3fooi
   call _ZTV3Foo
   call __Z3fooi
+  call Pi
+  call _Dmain
+  call _RNvC1a4main

lld/wasm/Symbols.cpp

 #include "lld/Common/Memory.h"
 #include "lld/Common/Strings.h"
 
 #define DEBUG_TYPE "lld"
 
 using namespace llvm;
 using namespace llvm::object;
 using namespace llvm::wasm;
+using namespace lld::wasm;
 
 namespace lld {
 std::string toString(const wasm::Symbol &sym) {
   return maybeDemangleSymbol(sym.getName());
 }
 
 std::string maybeDemangleSymbol(StringRef name) {
   // WebAssembly requires caller and callee signatures to match, so we mangle
   // `main` in the case where we need to pass it arguments.
   if (name == "__main_argc_argv")
     return "main";
-  if (wasm::config->demangle)
-    return demangleItanium(name);
-  return std::string(name);
+
+  return demangle(name, config->demangle);
 }
 
 std::string toString(wasm::Symbol::Kind kind) {
   switch (kind) {
   case wasm::Symbol::DefinedFunctionKind:
     return "DefinedFunction";
   case wasm::Symbol::DefinedDataKind:
     return "DefinedData";