[lld][WebAssembly] Add --unresolved-symbols=import-dynamic

[sbc100]

Summary

This is a new mode for handling unresolved symbols that allows all
symbols to be imported in the same way they would be in the case of
-fpie or -shared but generating an otherwise fixes/non-relocatable
binary.

Code linked in this way should still be compiled with -fPIC so that
data symbols can be resolved via imports.

This essentially allow the building of static binaries that have dynamic
imports. See:
https://github.com/emscripten-core/emscripten/issues/12682

As with other used of dynamic linking ABI this beviour will produce a
warning unless run with --experimental-pic.

[dschuff]

What's the use case for 'import-functions' mode after we roll this out?

lld/test/wasm/unresolved-symbols.s

75

shouldn't these runs/checks go in unresolved-symbols-dynamic.s?

[sbc100]

In D91577#2401182, @dschuff wrote:

What's the use case for 'import-functions' mode after we roll this out?

It allows folks to have the semantics of --allow-undefined which has existing users. We've seen a it bunch of times in bug reports, mostly among users of wasi-sdk.

The semantics of this option are still useful because it allows the binary to import all undefined function while zero-ing out undefined data symbols. The new import-dynamic mode is not a suitable replacement for all users because (a) it requires use of the the experimental PIC ABI and (b) it requires all objects files that reference undefined data symbols to be built with -fPIC.

Indeed this is the default behaviour expected by emscripten, and will continue to be for a while now.

Are are you are asking.. "why bother allowing the --unresolved-symbols= version of this option when we can just use --allow-undefined"?

[dschuff]

I was curious about emscripten. You mean that this is the mode that links the final wasm? Does emscripten allow users to inject JS functions that are directly imported by the wasm? (I assume we could use that mode for JS functions that are directly packaged by emscripten if we wanted).

[sbc100]

In D91577#2401285, @dschuff wrote:

I was curious about emscripten. You mean that this is the mode that links the final wasm? Does emscripten allow users to inject JS functions that are directly imported by the wasm? (I assume we could use that mode for JS functions that are directly packaged by emscripten if we wanted).

Yes, emscripten (almost) always links with --allow-undefined and then reported its undefined symbols later on in the JS library processor. And yes emscripten fills in any imported functions with JS implementations if it can, and errors if it can't.

lld/test/wasm/unresolved-symbols.s

75

I guess you are right.. I'll move these.

[sbc100]

• feedback

[sbc100]

rebase

[sbc100]

rebase

[sbc100]

rebase

[sbc100]

• rebase

[sbc100]

• comments

[dschuff]

lld/docs/WebAssembly.rst

96

this is now slightly confusing since --unresolved symbols is no longer completely orthogonal to --import-undefined? Probably at least the doc for --import-undefined should go just above --unresolved-symbols

[sbc100]

• update docs

[sbc100]

lld/docs/WebAssembly.rst

96

I went back and forth between adding --import-dynamic and --unresolved-symbols=import-dynamic. In the end I think this is more clear since unlike --import-undefined this new option effects all undefined symbols.

This option is currently behind the --experimental-pic flag so revisiting this decision in the future is still possible.

[sbc100]

I re-wrote the documentation to make this a little more clear

[sbc100]

-docs

lld/docs/WebAssembly.rst

.. option:: --unresolved-symbols=<method>
   ignore-all:
 
      Resolve all undefined symbols to zero.  For data and function addresses
      this is trivial.  For direct function calls, the linker will generate a
      trapping stub function in place of the undefined function.
 
   import-functions:
 
      Generate WebAssembly imports for any undefined functions.  Undefined data
      symbols are resolved to zero as in ``ignore-all``.  This corresponds to
      the legacy ``--allow-undefined`` flag.
 
+  import-dynamic:
+
+     Like import-functions but also import the addresses of unresolved data
+     symbols.  This puts limitations on the type of relocations that are allowed
+     for imported data symbols.  Any relocations that require absolute data
+     addresses (i.e. All R_WASM_MEMORY_ADDR_I32) will generate an error if they
+     cannot be resolved statically.  For clang/llvm this means inputs should be
+     compiled with `-fPIC` (i.e. `pic` or `dynamic-no-pic` relocation models).
+     This is useful for linking binaries are themselves static but are compiled
+     with either `pic` or `dynamic-no-pic` reocation models, allowing undefined
+     symbols to be resolved dynamically.
+
 .. option:: --import-memory
 
   Import memory from the environment.
 
 .. option:: --initial-memory=<value>
 
   Initial size of the linear memory. Default: static data size.
 

lld/test/wasm/undefined-data.ll

 ; RUN: llc -filetype=obj %s -o %t.o
 ; RUN: not wasm-ld -o %t.wasm %t.o 2>&1 | FileCheck %s -check-prefix=UNDEF
 ; RUN: wasm-ld --allow-undefined -o %t.wasm %t.o
 ; RUN: not wasm-ld --shared -o %t.wasm %t.o 2>&1 | FileCheck %s -check-prefix=SHARED
 
 target triple = "wasm32-unknown-unknown"
 
 @data_external = external global i32
 
 define i32 @_start() {
 entry:
     %0 = load i32, i32* @data_external, align 4
     ret i32 %0
 }
 
 ; UNDEF: error: {{.*}}undefined-data.ll.tmp.o: undefined symbol: data_external
-; SHARED: error: {{.*}}undefined-data.ll.tmp.o: relocation R_WASM_MEMORY_ADDR_LEB cannot be used against symbol data_external; recompile with -fPIC
+; SHARED: error: {{.*}}undefined-data.ll.tmp.o: relocation R_WASM_MEMORY_ADDR_LEB cannot be used against symbol `data_external`; recompile with -fPIC

lld/test/wasm/unresolved-symbols-dynamic.s

+## import-dynamic should succeed if that input file contains GOT
+# relocations for external sybmols.
+# RUN: llvm-mc -filetype=obj -triple=wasm32-unknown-unknown %p/Inputs/unresolved-symbols-dynamic.s -o %t-dynamic.o
+# RUN: wasm-ld %t-dynamic.o -o %t.wasm --unresolved-symbols=import-dynamic 2>&1 | FileCheck -check-prefix=WARN %s
+# WARN: wasm-ld: warning: dynamic imports are not yet stable (--unresolved-symbols=import-dynamic)
+# RUN: obj2yaml %t.wasm | FileCheck %s
+#      CHECK:  - Type:            IMPORT
+# CHECK-NEXT:    Imports:
+# CHECK-NEXT:      - Module:          env
+# CHECK-NEXT:        Field:           undef
+# CHECK-NEXT:        Kind:            FUNCTION
+# CHECK-NEXT:        SigIndex:        0
+# CHECK-NEXT:      - Module:          GOT.mem
+# CHECK-NEXT:        Field:           undef_data
+# CHECK-NEXT:        Kind:            GLOBAL
+# CHECK-NEXT:        GlobalType:      I32
+# CHECK-NEXT:        GlobalMutable:   true
+# CHECK-NEXT:      - Module:          GOT.func
+# CHECK-NEXT:        Field:           undef
+# CHECK-NEXT:        Kind:            GLOBAL
+# CHECK-NEXT:        GlobalType:      I32
+# CHECK-NEXT:        GlobalMutable:   true
+
+.globl _start
+_start:
+    .functype _start () -> ()
+    call undef
+    call get_data_addr
+    call get_func_addr
+    end_function
+
+.globl get_data_addr
+get_data_addr:
+    .functype get_data_addr () -> (i32)
+    global.get undef_data@GOT
+    return
+    end_function
+
+.globl get_func_addr
+get_func_addr:
+    .functype get_func_addr () -> (i32)
+    global.get undef@GOT
+    return
+    end_function
+
+.functype undef () -> ()

lld/test/wasm/unresolved-symbols.s

 # IMPORT-NEXT:        Kind:            FUNCTION
 # IMPORT-NEXT:        SigIndex:        0
 # IMPORT-NEXT:  - Type:            FUNCTION
 
 ## Do not report undefines if linking relocatable.
 # RUN: wasm-ld -r %t1.o -o %t4.wasm --unresolved-symbols=report-all
 # RUN: llvm-readobj %t4.wasm > /dev/null 2>&1
 
+## import-dynamic should fail due to incompatible relocations.
+# RUN: not wasm-ld %t1.o -o %t5.wasm --unresolved-symbols=import-dynamic 2>&1 | FileCheck -check-prefix=ERRNOPIC %s
+# ERRNOPIC: relocation R_WASM_MEMORY_ADDR_SLEB cannot be used against symbol `undef_data`; recompile with -fPIC
+# ERRNOPIC: relocation R_WASM_TABLE_INDEX_SLEB cannot be used against symbol `undef`; recompile with -fPIC
+
 .globl _start
 _start:
     .functype _start () -> ()
     call undef
     call get_data_addr
     call get_func_addr
     end_function
 

lld/wasm/Config.h

 
 namespace lld {
 namespace wasm {
 
 // For --unresolved-symbols.
 // The `ImportFuncs` mode is an additional mode that corresponds to the
 // --allow-undefined flag which turns undefined functions in imports
 // as opposed ed to Ignore or Warn which turn them into unreachables.
-enum class UnresolvedPolicy { ReportError, Warn, Ignore, ImportFuncs };
+enum class UnresolvedPolicy {
+  ReportError,
+  Warn,
+  Ignore,
+  ImportFuncs,
+  ImportDynamic
+};
 
 // This struct contains the global configuration for the linker.
 // Most fields are direct mapping from the command line options
 // and such fields have the same name as the corresponding options.
 // Most fields are initialized by the driver.
 struct Configuration {
   bool bsymbolic;
   bool checkFeatures;

lld/wasm/Driver.cpp

UnresolvedPolicy errorOrWarn = args.hasFlag(OPT_error_unresolved_symbols,
                                      : UnresolvedPolicy::Warn;
 
   if (auto *arg = args.getLastArg(OPT_unresolved_symbols)) {
     StringRef s = arg->getValue();
     if (s == "ignore-all")
       return UnresolvedPolicy::Ignore;
     if (s == "import-functions")
       return UnresolvedPolicy::ImportFuncs;
+    if (s == "import-dynamic")
+      return UnresolvedPolicy::ImportDynamic;
     if (s == "report-all")
       return errorOrWarn;
     error("unknown --unresolved-symbols value: " + s);
   }
 
   // Legacy --allow-undefined flag which is equivalent to
   // --unresolve-symbols=ignore-all
   if (args.hasArg(OPT_allow_undefined))

 // of these values.
 static void setConfigs() {
   config->isPic = config->pie || config->shared;
 
   if (config->isPic) {
     if (config->exportTable)
       error("-shared/-pie is incompatible with --export-table");
     config->importTable = true;
+    config->unresolvedSymbols = UnresolvedPolicy::ImportDynamic;
   }
 
-  if (config->shared) {
+  if (config->shared)
     config->importMemory = true;
-    config->unresolvedSymbols = UnresolvedPolicy::ImportFuncs;
-  }
 }
 
 // Some command line options or some combinations of them are not allowed.
 // This function checks for such errors.
 static void checkOptions(opt::InputArgList &args) {
   if (!config->stripDebug && !config->stripAll && config->compressRelocations)
     error("--compress-relocations is incompatible with output debug"
           " information. Please pass --strip-debug or --strip-all");
static void checkOptions(opt::InputArgList &args) {
 
   // To begin to prepare for Module Linking-style shared libraries, start
   // warning about uses of `-shared` and related flags outside of Experimental
   // mode, to give anyone using them a heads-up that they will be changing.
   //
   // Also, warn about flags which request explicit exports.
   if (!config->experimentalPic) {
     // -shared will change meaning when Module Linking is implemented.
-    if (config->shared) {
+    if (config->shared)
       warn("creating shared libraries, with -shared, is not yet stable");
-    }
 
     // -pie will change meaning when Module Linking is implemented.
-    if (config->pie) {
+    if (config->pie)
       warn("creating PIEs, with -pie, is not yet stable");
-    }
+
+    if (config->unresolvedSymbols == UnresolvedPolicy::ImportDynamic)
+      warn("dynamic imports are not yet stable "
+           "(--unresolved-symbols=import-dynamic)");
   }
 
   if (config->bsymbolic && !config->shared) {
     warn("-Bsymbolic is only meaningful when combined with -shared");
   }
 }
 
 // Force Sym to be entered in the output. Used for -u or equivalent.

lld/wasm/Relocations.cpp

 
 using namespace llvm;
 using namespace llvm::wasm;
 
 namespace lld {
 namespace wasm {
 
 static bool requiresGOTAccess(const Symbol *sym) {
-  if (!config->isPic)
+  if (!config->isPic &&
+      config->unresolvedSymbols != UnresolvedPolicy::ImportDynamic)
     return false;
   if (sym->isHidden() || sym->isLocal())
     return false;
   // With `-Bsymbolic` (or when building an executable) as don't need to use
   // the GOT for symbols that are defined within the current module.
   if (sym->isDefined() && (!config->shared || config->bsymbolic))
     return false;
   return true;
case UnresolvedPolicy::Ignore:
                      << "ignoring undefined symbol: " + toString(*sym) + "\n");
           f->stubFunction = symtab->createUndefinedStub(*f->getSignature());
           f->stubFunction->markLive();
           f->setTableIndex(0);
         }
       }
       break;
     case UnresolvedPolicy::ImportFuncs:
+    case UnresolvedPolicy::ImportDynamic:
       break;
     }
   }
 }
 
 static void addGOTEntry(Symbol *sym) {
   if (requiresGOTAccess(sym))
     out.importSec->addGOTEntry(sym);
case R_WASM_MEMORY_ADDR_TLS_SLEB:
                 relocTypeToString(reloc.Type) + " cannot be used against `" +
                 toString(*sym) +
                 "` in non-TLS section: " + D->segment->outputSeg->name);
         }
       }
       break;
     }
 
-    if (config->isPic) {
+    if (config->isPic ||
+        (sym->isUndefined() &&
+         config->unresolvedSymbols == UnresolvedPolicy::ImportDynamic)) {
       switch (reloc.Type) {
       case R_WASM_TABLE_INDEX_SLEB:
       case R_WASM_TABLE_INDEX_SLEB64:
       case R_WASM_MEMORY_ADDR_SLEB:
       case R_WASM_MEMORY_ADDR_LEB:
       case R_WASM_MEMORY_ADDR_SLEB64:
       case R_WASM_MEMORY_ADDR_LEB64:
         // Certain relocation types can't be used when building PIC output,
         // since they would require absolute symbol addresses at link time.
         error(toString(file) + ": relocation " + relocTypeToString(reloc.Type) +
-              " cannot be used against symbol " + toString(*sym) +
-              "; recompile with -fPIC");
+              " cannot be used against symbol `" + toString(*sym) +
+              "`; recompile with -fPIC");
         break;
       case R_WASM_TABLE_INDEX_I32:
       case R_WASM_TABLE_INDEX_I64:
       case R_WASM_MEMORY_ADDR_I32:
       case R_WASM_MEMORY_ADDR_I64:
         // These relocation types are only present in the data section and
         // will be converted into code by `generateRelocationCode`.  This code
         // requires the symbols to have GOT entires.

lld/wasm/Writer.cpp

 
 static bool shouldImport(Symbol *sym) {
   // We don't generate imports for data symbols. They however can be imported
   // as GOT entries.
   if (isa<DataSymbol>(sym))
     return false;
 
   if (config->relocatable ||
-      config->unresolvedSymbols == UnresolvedPolicy::ImportFuncs)
+      config->unresolvedSymbols == UnresolvedPolicy::ImportFuncs ||
+      config->unresolvedSymbols == UnresolvedPolicy::ImportDynamic)
     return true;
   if (config->allowUndefinedSymbols.count(sym->getName()) != 0)
     return true;
   if (auto *g = dyn_cast<UndefinedGlobal>(sym))
     return g->importName.hasValue();
   if (auto *f = dyn_cast<UndefinedFunction>(sym))
     return f->importName.hasValue();