[ELF] Make --version-script/--dynamic-list work for lazy symbols fetched by LTO libcalls

[MaskRay]

Summary

Fixes https://bugs.llvm.org/show_bug.cgi?id=45391

The LTO code generator happens after version script scanning and may
create references which will fetch some lazy symbols.

Currently a version script does not assign VER_NDX_LOCAL to lazy symbols
and such symbols will be made global after they are fetched.

Change findByVersion and findAllByVersion to work on lazy symbols.
For unfetched lazy symbols, we should keep them non-local (D35263).
Check isDefined() in computeBinding() as a compensation.

This patch fixes a companion bug that --dynamic-list does not export
such symbols.

[MaskRay]

State that this fixes --dynamic-list as well

[grimar]

lld/ELF/SymbolTable.cpp

128–129

Seems this change is untested.

156

What do you think about adding a helper like the following?
(to avoid the duplication in 3 places)

// <comment>
static bool canBeVersioned(const Symbol& Sym) {
 return b->isDefined() || b->isCommon() || b->isLazy()
}

[MaskRay]

Add canBeVersioned()

[MaskRay]

lld/ELF/SymbolTable.cpp

128–129

This can't be tested because no libcall has a C++ style mangled name.

I did this just for consistency.

[grimar]

I have no more comments, I'd like someone more familar with LTO to take a look.

[MaskRay]

In D77280#1959076, @grimar wrote:

I have no more comments, I'd like someone more familar with LTO to take a look.

There is not much about LTO. Version script scanning happens before LTO because LTO needs to know what symbols are local and can thus internalize them. Some library call undefined references (e.g. __udivti3) are not recorded in the symbol table of bitcode files but they may be generated on the fly. The reference will fetch the lazy __udivti3. We thus need to track the version of lazy libcall symbol as well.

if (!config->relocatable)
  symtab->scanVersionScript();

// Do link-time optimization if given files are LLVM bitcode files.
// This compiles bitcode files into real object files.
//
// With this the symbol table should be complete. After this, no new names
// except a few linker-synthesized ones will be added to the symbol table.
compileBitcodeFiles<ELFT>();

[grimar]

In D77280#1960813, @MaskRay wrote:

Some library call undefined references (e.g. __udivti3) are not recorded in the symbol table of bitcode files but they may be generated on the fly.

This detail filled some gap in my head, so LGTM, thanks!

lld/ELF/SymbolTable.cpp

Symbol *SymbolTable::find(StringRef name) {
   if (it == symMap.end())
     return nullptr;
   Symbol *sym = symVector[it->second];
   if (sym->isPlaceholder())
     return nullptr;
   return sym;
 }
 
+// A version script/dynamic list is only meaningful for a Defined symbol.
+// A CommonSymbol will be converted to a Defined in replaceCommonSymbols().
+// A lazy symbol may be made Defined if an LTO libcall fetches it.
+static bool canBeVersioned(const Symbol &sym) {
+  return sym.isDefined() || sym.isCommon() || sym.isLazy();
+}
+
 // Initialize demangledSyms with a map from demangled symbols to symbol
 // objects. Used to handle "extern C++" directive in version scripts.
 //
 // The map will contain all demangled symbols. That can be very large,
 // and in LLD we generally want to avoid do anything for each symbol.
 // Then, why are we doing this? Here's why.
 //
 // Users can use "extern C++ {}" directive to match against demangled
 // C++ symbols. For example, you can write a pattern such as
 // "llvm::*::foo(int, ?)". Obviously, there's no way to handle this
 // other than trying to match a pattern against all demangled symbols.
 // So, if "extern C++" feature is used, we need to demangle all known
 // symbols.
 StringMap<std::vector<Symbol *>> &SymbolTable::getDemangledSyms() {
   if (!demangledSyms) {
     demangledSyms.emplace();
-    for (Symbol *sym : symVector) {
-      if (!sym->isDefined() && !sym->isCommon())
-        continue;
+    for (Symbol *sym : symVector)
+      if (canBeVersioned(*sym))
         (*demangledSyms)[demangleItanium(sym->getName())].push_back(sym);
-    }
-  }
   return *demangledSyms;
 }
 
 std::vector<Symbol *> SymbolTable::findByVersion(SymbolVersion ver) {
   if (ver.isExternCpp)
     return getDemangledSyms().lookup(ver.name);
-  if (Symbol *b = find(ver.name))
-    if (b->isDefined() || b->isCommon())
-      return {b};
+  if (Symbol *sym = find(ver.name))
+    if (canBeVersioned(*sym))
+      return {sym};
   return {};
 }
 
 std::vector<Symbol *> SymbolTable::findAllByVersion(SymbolVersion ver) {
   std::vector<Symbol *> res;
   SingleStringMatcher m(ver.name);
 
   if (ver.isExternCpp) {
     for (auto &p : getDemangledSyms())
       if (m.match(p.first()))
         res.insert(res.end(), p.second.begin(), p.second.end());
     return res;
   }
 
   for (Symbol *sym : symVector)
-    if ((sym->isDefined() || sym->isCommon()) && m.match(sym->getName()))
+    if (canBeVersioned(*sym) && m.match(sym->getName()))
       res.push_back(sym);
   return res;
 }
 
 // Handles -dynamic-list.
 void SymbolTable::handleDynamicList() {
   for (SymbolVersion &ver : config->dynamicList) {
     std::vector<Symbol *> syms;

lld/ELF/Symbols.cpp

return CHECK(c.getMemoryBufferRef(),
                "could not get the buffer for the member defining symbol " +
                    toELFString(sym));
 }
 
 uint8_t Symbol::computeBinding() const {
   if (config->relocatable)
     return binding;
   if ((visibility != STV_DEFAULT && visibility != STV_PROTECTED) ||
-      versionId == VER_NDX_LOCAL)
+      (versionId == VER_NDX_LOCAL && isDefined()))
     return STB_LOCAL;
   if (!config->gnuUnique && binding == STB_GNU_UNIQUE)
     return STB_GLOBAL;
   return binding;
 }
 
 bool Symbol::includeInDynsym() const {
   if (!config->hasDynSymTab)

lld/test/ELF/lto/version-libcall.ll

+; REQUIRES: x86
+;; The LTO code generator may create references which will fetch lazy symbols.
+;; Test that version script local: directives can change the binding of such
+;; symbols to STB_LOCAL. This is a bit complex because the LTO code generator
+;; happens after version script scanning and can change symbols from Lazy to Defined.
+
+; RUN: llvm-as %s -o %t.bc
+; RUN: echo '.globl __udivti3; __udivti3:' | llvm-mc -filetype=obj -triple=x86_64 - -o %t1.o
+
+;; An exact pattern can localize a libcall.
+; RUN: echo '{ global: foo; local: __udivti3; };' > %t.exact.ver
+; RUN: ld.lld -shared --version-script %t.exact.ver %t.bc --start-lib %t1.o --end-lib -o %t.exact.so
+; RUN: llvm-nm %t.exact.so | FileCheck %s
+
+;; A wildcard pattern can localize a libcall.
+; RUN: echo '{ global: foo; local: *; };' > %t.wild.ver
+; RUN: ld.lld -shared --version-script %t.wild.ver %t.bc --start-lib %t1.o --end-lib -o %t.wild.so
+; RUN: llvm-nm %t.wild.so | FileCheck %s
+
+; CHECK: t __udivti3
+; CHECK: T foo
+
+;; Test that --dynamic-list works on such libcall fetched symbols.
+; RUN: echo '{ foo; __udivti3; };' > %t.exact.list
+; RUN: ld.lld -pie --dynamic-list %t.exact.list %t.bc --start-lib %t1.o --end-lib -o %t.exact
+; RUN: llvm-nm %t.exact | FileCheck --check-prefix=LIST %s
+; RUN: echo '{ foo; __udiv*; };' > %t.wild.list
+; RUN: ld.lld -pie --dynamic-list %t.wild.list %t.bc --start-lib %t1.o --end-lib -o %t.wild
+; RUN: llvm-nm %t.wild | FileCheck --check-prefix=LIST %s
+
+; LIST: T __udivti3
+; LIST: T foo
+
+target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+declare i64 @llvm.udiv.fix.i64(i64, i64, i32)
+
+;; The symbol table does not record __udivti3, but the reference will be created
+;; on the fly.
+define i64 @foo(i64 %x, i64 %y) {
+  %ret = call i64 @llvm.udiv.fix.i64(i64 %x, i64 %y, i32 31)
+  ret i64 %ret
+}