[lld][WebAssembly] Process stub libraries before and after LTO

[sbc100]

Summary

There are cases where stub library processing can trigger new exports
which might require them to be included at LTO time.

Specifically processStubLibraries marks symbols as forceExports
which even effect the LTO process.

And since the LTO process can generate new undefined symbols
(specifically libcall function) we need to also process the stub
libraries after LTO.

[sbc100]

• comments

[dschuff]

Do we really need to support stub libcalls (i.e. the kind that can be generated at codegen time)? These should just be compiler-rt and maybe a few core libc functions, right? Do we expect those to be implemented in JS and/or the environment?

[dschuff]

i.e. it seems like it would be simpler to just support processing the stubs once. or maybe more specifically, to handle them the same as other archive libraries (i.e. participate in the iterate-to-convergence phase of symbol resolution but not get special treatment for LTO?)

[sbc100]

In D147190#4234179, @dschuff wrote:

Do we really need to support stub libcalls (i.e. the kind that can be generated at codegen time)? These should just be compiler-rt and maybe a few core libc functions, right? Do we expect those to be implemented in JS and/or the environment?

Yes, we have use cases in emscripten where symbols that are created during LTO depend on other native symbols. The one I can remember was emscripten exception handling I believe. The compiler creates references to JS-defined functions at LTO time, which in tern require certain native exports (__cxa_is_pointer_type I believe).

[sbc100]

In D147190#4234186, @dschuff wrote:

i.e. it seems like it would be simpler to just support processing the stubs once. or maybe more specifically, to handle them the same as other archive libraries (i.e. participate in the iterate-to-convergence phase of symbol resolution but not get special treatment for LTO?)

I think that would require injecting the dependency graph of the stub library into the relocation graph somehow. I did consider trying to do that but this solution ended up being a lot simpler. I don't think we need to worry about the cost of processing these stubs, do you?

[dschuff]

I guess the new test covers the stub->bitcode lib dependence case. Is there a test that covers the codgen->libcall->stub case?

lld/test/wasm/lto/stub-library.s

7

[sbc100]

• feedback

[sbc100]

In D147190#4234228, @dschuff wrote:

I guess the new test covers the stub->bitcode lib dependence case. Is there a test that covers the codgen->libcall->stub case?

That specific type of test (where LTO generates new undefined symbols) is quite hard to write since it depends on the specifics of codegen/LTO... but I'll give it a go. We have at least one test already for that I think (libcall-archive.ll)

[sbc100]

• a test

lld/test/wasm/lto/Inputs/foo.ll

+target datalayout = "e-m:e-p:32:32-p10:8:8-p20:8:8-i64:64-n32:64-S128"
+target triple = "wasm32-unknown-unknown"
+
+define void @foo() local_unnamed_addr {
+entry:
+  ret void
+}

lld/test/wasm/lto/Inputs/libcall.ll

+target datalayout = "e-m:e-p:32:32-p10:8:8-p20:8:8-i64:64-n32:64-S128"
+target triple = "wasm32-unknown-unknown"
+
+; This function, when compiled will generate a new undefined reference to
+; memcpy, that is not present in the bitcode object
+define void @func_with_libcall(ptr %a, ptr %b) {
+entry:
+  call void @llvm.memcpy.p0.p0.i64(ptr %a, ptr %b, i64 1024, i1 false)
+  ret void
+}
+
+declare void @llvm.memcpy.p0.p0.i64(ptr nocapture, ptr nocapture, i64, i1)

lld/test/wasm/lto/Inputs/stub.so

+#STUB
+bar: foo
+memcpy: foo

lld/test/wasm/lto/stub-library-libcall.s

+# RUN: split-file %s %t
+# RUN: llvm-mc -filetype=obj -triple=wasm32-unknown-unknown -o %t_main.o %t/main.s
+# RUN: llvm-mc -filetype=obj -triple=wasm32-unknown-unknown -o %t_foo.o %t/foo.s
+# RUN: llvm-as %S/Inputs/libcall.ll -o %t_libcall.o
+# RUN: wasm-ld %t_main.o %t_libcall.o %t_foo.o %p/Inputs/stub.so -o %t.wasm
+# RUN: obj2yaml %t.wasm | FileCheck %s
+
+# The function `func_with_libcall` will generate an undefined reference to
+# `memcpy` at LTO time.  `memcpy` itself also declared in stub.so and depends
+# on `foo`
+
+# If %t_foo.o is not included in the link we get an undefined symbol reported
+# to the dependency of memcpy on the foo export:
+
+# RUN: not wasm-ld %t_main.o %t_libcall.o %p/Inputs/stub.so -o %t.wasm 2>&1 | FileCheck --check-prefix=MISSING %s
+# MISSING: stub.so: undefined symbol: foo. Required by memcpy
+
+#--- main.s
+.functype func_with_libcall (i32, i32) -> ()
+.globl _start
+_start:
+    .functype _start () -> ()
+    i32.const 0
+    i32.const 0
+    call func_with_libcall
+    end_function
+
+#--- foo.s
+.globl foo
+foo:
+  .functype foo () -> ()
+  end_function
+
+# CHECK:         Imports:
+# CHECK-NEXT:      - Module:          env
+# CHECK-NEXT:        Field:           memcpy
+# CHECK-NEXT:        Kind:            FUNCTION
+# CHECK-NEXT:        SigIndex:        0
+
+# CHECK:         Exports:
+# CHECK-NEXT:       - Name:            memory
+# CHECK-NEXT:         Kind:            MEMORY
+# CHECK-NEXT:         Index:           0
+# CHECK-NEXT:       - Name:            _start
+# CHECK-NEXT:         Kind:            FUNCTION
+# CHECK-NEXT:         Index:           1
+# CHECK-NEXT:       - Name:            foo
+# CHECK-NEXT:         Kind:            FUNCTION
+# CHECK-NEXT:         Index:           2

lld/test/wasm/lto/stub-library.s

+# RUN: llvm-mc -filetype=obj -triple=wasm32-unknown-unknown -o %t.o %s
+# RUN: llvm-as %S/Inputs/foo.ll -o %t1.o
+# RUN: wasm-ld %t.o %t1.o %p/Inputs/stub.so -o %t.wasm
+# RUN: obj2yaml %t.wasm | FileCheck %s
+
+# The function `bar` is declared in stub.so and depends on `foo`, which happens
+# be in an LTO object.
+# This verifies that stub library dependencies (required exports) can be defined
+# in LTO objects.
+.functype bar () -> ()
+
+.globl _start
+_start:
+    .functype _start () -> ()
+    call bar
+    end_function
+
+# CHECK:         Imports:
+# CHECK-NEXT:      - Module:          env
+# CHECK-NEXT:        Field:           bar
+# CHECK-NEXT:        Kind:            FUNCTION
+# CHECK-NEXT:        SigIndex:        0
+
+# CHECK:         Exports:
+# CHECK-NEXT:       - Name:            memory
+# CHECK-NEXT:         Kind:            MEMORY
+# CHECK-NEXT:         Index:           0
+# CHECK-NEXT:       - Name:            _start
+# CHECK-NEXT:         Kind:            FUNCTION
+# CHECK-NEXT:         Index:           1
+# CHECK-NEXT:       - Name:            foo
+# CHECK-NEXT:         Kind:            FUNCTION
+# CHECK-NEXT:         Index:           2

lld/wasm/Driver.cpp

 
 static void processStubLibraries() {
   log("-- processStubLibraries");
   for (auto &stub_file : symtab->stubFiles) {
     LLVM_DEBUG(llvm::dbgs()
                << "processing stub file: " << stub_file->getName() << "\n");
     for (auto [name, deps]: stub_file->symbolDependencies) {
       auto* sym = symtab->find(name);
-      if (!sym || !sym->isUndefined() || !sym->isUsedInRegularObj ||
-          sym->forceImport) {
+      if (!sym || !sym->isUndefined() || sym->forceImport) {
         LLVM_DEBUG(llvm::dbgs() << "stub not in needed: " << name << "\n");
         continue;
       }
       // The first stub library to define a given symbol sets this and
       // definitions in later stub libraries are ignored.
       sym->forceImport = true;
       if (sym->traced)
         message(toString(stub_file) + ": importing " + name);
       else
         LLVM_DEBUG(llvm::dbgs()
                    << toString(stub_file) << ": importing " << name << "\n");
       for (const auto dep : deps) {
         auto* needed = symtab->find(dep);
         if (!needed) {
           error(toString(stub_file) + ": undefined symbol: " + dep +
                 ". Required by " + toString(*sym));
         } else if (needed->isUndefined()) {
           error(toString(stub_file) +
                 ": undefined symbol: " + toString(*needed) +
                 ". Required by " + toString(*sym));
         } else {
           LLVM_DEBUG(llvm::dbgs()
                      << "force export: " << toString(*needed) << "\n");
           needed->forceExport = true;
-          needed->isUsedInRegularObj = true;
           if (auto *lazy = dyn_cast<LazySymbol>(needed)) {
             lazy->fetch();
             if (!config->whyExtract.empty())
               config->whyExtractRecords.emplace_back(stub_file->getName(),
                                                      sym->getFile(), *sym);
           }
         }
       }
void LinkerDriver::linkerMain(ArrayRef<const char *> argsArr) {
   // definition is in bitcode. Any other required libcall symbols will be added
   // to the link after LTO when we add the LTO object file to the link.
   if (!symtab->bitcodeFiles.empty())
     for (auto *s : lto::LTO::getRuntimeLibcallSymbols())
       handleLibcall(s);
   if (errorCount())
     return;
 
-  writeWhyExtract();
+  // processStubLibraries must happen before LTO because it can trigger the
+  // export of arbirary symbols that might themselves be defined in LTO objects.
+  processStubLibraries();
 
   // Do link-time optimization if given files are LLVM bitcode files.
   // This compiles bitcode files into real object files.
   symtab->compileBitcodeFiles();
   if (errorCount())
     return;
 
+  // The LTO process can generate new undefined symbols, specifically libcall
+  // functions.  Because those symbols might be declared in a stub library we
+  // need the process the stub libraries once again after LTO to handle any
+  // newly undefined symbols.
   processStubLibraries();
 
+  writeWhyExtract();
+
   createOptionalSymbols();
 
   // Resolve any variant symbols that were created due to signature
   // mismatchs.
   symtab->handleSymbolVariants();
   if (errorCount())
     return;