[ELF] Make --version-script/--dynamic-list work for lazy symbols fetched by LTO libcalls

[MaskRay]

Summary

Fixes https://bugs.llvm.org/show_bug.cgi?id=45391

The LTO code generator happens after version script scanning and may
create references which will fetch some lazy symbols.

Currently a version script does not assign VER_NDX_LOCAL to lazy symbols
and such symbols will be made global after they are fetched.

Change findByVersion and findAllByVersion to work on lazy symbols.
For unfetched lazy symbols, we should keep them non-local (D35263).
Check isDefined() in computeBinding() as a compensation.

This patch fixes a companion bug that --dynamic-list does not export
such symbols.

[MaskRay]

State that this fixes --dynamic-list as well

[grimar]

lld/ELF/SymbolTable.cpp

123

Seems this change is untested.

153

What do you think about adding a helper like the following?
(to avoid the duplication in 3 places)

// <comment>
static bool canBeVersioned(const Symbol& Sym) {
 return b->isDefined() || b->isCommon() || b->isLazy()
}

[MaskRay]

Add canBeVersioned()

[MaskRay]

lld/ELF/SymbolTable.cpp

123

This can't be tested because no libcall has a C++ style mangled name.

I did this just for consistency.

[grimar]

I have no more comments, I'd like someone more familar with LTO to take a look.

[MaskRay]

In D77280#1959076, @grimar wrote:

I have no more comments, I'd like someone more familar with LTO to take a look.

There is not much about LTO. Version script scanning happens before LTO because LTO needs to know what symbols are local and can thus internalize them. Some library call undefined references (e.g. __udivti3) are not recorded in the symbol table of bitcode files but they may be generated on the fly. The reference will fetch the lazy __udivti3. We thus need to track the version of lazy libcall symbol as well.

if (!config->relocatable)
  symtab->scanVersionScript();

// Do link-time optimization if given files are LLVM bitcode files.
// This compiles bitcode files into real object files.
//
// With this the symbol table should be complete. After this, no new names
// except a few linker-synthesized ones will be added to the symbol table.
compileBitcodeFiles<ELFT>();

[grimar]

In D77280#1960813, @MaskRay wrote:

Some library call undefined references (e.g. __udivti3) are not recorded in the symbol table of bitcode files but they may be generated on the fly.

This detail filled some gap in my head, so LGTM, thanks!

lld/ELF/SymbolTable.cpp

 //===- SymbolTable.cpp ----------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //

 // "llvm::*::foo(int, ?)". Obviously, there's no way to handle this
 // other than trying to match a pattern against all demangled symbols.
 // So, if "extern C++" feature is used, we need to demangle all known
 // symbols.
 StringMap<std::vector<Symbol *>> &SymbolTable::getDemangledSyms() {
   if (!demangledSyms) {
     demangledSyms.emplace();
     for (Symbol *sym : symVector) {
-      if (!sym->isDefined() && !sym->isCommon())
+      if (!sym->isDefined() && !sym->isCommon() && !sym->isLazy())
         continue;
       (*demangledSyms)[demangleItanium(sym->getName())].push_back(sym);
     }
   }
   return *demangledSyms;
 }
 
 std::vector<Symbol *> SymbolTable::findByVersion(SymbolVersion ver) {
   if (ver.isExternCpp)
     return getDemangledSyms().lookup(ver.name);
+  // A version script/dynamic list is only meaningful for a Defined symbol.
+  // A lazy symbol can be made Defined if an LTO libcall references it.
   if (Symbol *b = find(ver.name))
-    if (b->isDefined() || b->isCommon())
+    if (b->isDefined() || b->isCommon() || b->isLazy())
       return {b};
   return {};
 }
 
 std::vector<Symbol *> SymbolTable::findAllByVersion(SymbolVersion ver) {
   std::vector<Symbol *> res;
   SingleStringMatcher m(ver.name);
 
   if (ver.isExternCpp) {
     for (auto &p : getDemangledSyms())
       if (m.match(p.first()))
         res.insert(res.end(), p.second.begin(), p.second.end());
     return res;
   }
 
+  // See the comment in findByVersion() for the matched symbols.
   for (Symbol *sym : symVector)
-    if ((sym->isDefined() || sym->isCommon()) && m.match(sym->getName()))
+    if ((sym->isDefined() || sym->isCommon() || sym->isLazy()) &&
+        m.match(sym->getName()))
       res.push_back(sym);
   return res;
 }
 
 // Handles -dynamic-list.
 void SymbolTable::handleDynamicList() {
   for (SymbolVersion &ver : config->dynamicList) {
     std::vector<Symbol *> syms;

lld/ELF/Symbols.cpp

 //===- Symbols.cpp --------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
return CHECK(c.getMemoryBufferRef(),
                "could not get the buffer for the member defining symbol " +
                    toELFString(sym));
 }
 
 uint8_t Symbol::computeBinding() const {
   if (config->relocatable)
     return binding;
   if ((visibility != STV_DEFAULT && visibility != STV_PROTECTED) ||
-      versionId == VER_NDX_LOCAL)
+      (versionId == VER_NDX_LOCAL && isDefined()))
     return STB_LOCAL;
   if (!config->gnuUnique && binding == STB_GNU_UNIQUE)
     return STB_GLOBAL;
   return binding;
 }
 
 bool Symbol::includeInDynsym() const {
   if (!config->hasDynSymTab)

lld/test/ELF/lto/version-libcall.ll

+; REQUIRES: x86
+;; The LTO code generator may create references which will fetch lazy symbols.
+;; Test that version script local: directives can change the binding of such
+;; symbols to STB_LOCAL. This is a bit complex because the LTO code generator
+;; happens after version script scanning and can change symbols from Lazy to Defined.
+
+; RUN: llvm-as %s -o %t.bc
+; RUN: echo '.globl __udivti3; __udivti3:' | llvm-mc -filetype=obj -triple=x86_64 - -o %t1.o
+
+;; An exact pattern can localize a libcall.
+; RUN: echo '{ global: foo; local: __udivti3; };' > %t.exact.ver
+; RUN: ld.lld -shared --version-script %t.exact.ver %t.bc --start-lib %t1.o --end-lib -o %t.exact.so
+; RUN: llvm-nm %t.exact.so | FileCheck %s
+
+;; A wildcard pattern can localize a libcall.
+; RUN: echo '{ global: foo; local: *; };' > %t.wild.ver
+; RUN: ld.lld -shared --version-script %t.wild.ver %t.bc --start-lib %t1.o --end-lib -o %t.wild.so
+; RUN: llvm-nm %t.wild.so | FileCheck %s
+
+; CHECK: t __udivti3
+; CHECK: T foo
+
+;; Test that --dynamic-list works on such libcall fetched symbols.
+; RUN: echo '{ foo; __udivti3; };' > %t.exact.list
+; RUN: ld.lld -pie --dynamic-list %t.exact.list %t.bc --start-lib %t1.o --end-lib -o %t.exact
+; RUN: llvm-nm %t.exact | FileCheck --check-prefix=LIST %s
+; RUN: echo '{ foo; __udiv*; };' > %t.wild.list
+; RUN: ld.lld -pie --dynamic-list %t.wild.list %t.bc --start-lib %t1.o --end-lib -o %t.wild
+; RUN: llvm-nm %t.wild | FileCheck --check-prefix=LIST %s
+
+; LIST: T __udivti3
+; LIST: T foo
+
+target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+declare i64 @llvm.udiv.fix.i64(i64, i64, i32)
+
+define i64 @foo(i64 %x, i64 %y) {
+  %ret = call i64 @llvm.udiv.fix.i64(i64 %x, i64 %y, i32 31)
+  ret i64 %ret
+}