[lld][WebAssembly] Initial support for stub libraries

[sbc100]

Summary

See the docs in lld/docs/WebAssembly.rst for more on this.

This feature unlocks a lot of simplification in the emscripten toolchain
since we can represent the JS libraries to the compiler using stub
objects.

See https://github.com/emscripten-core/emscripten/issues/18875

[sbc100]

• typos

[sbc100]

• dead code

[sbc100]

Fixes

[pmatos]

lld/docs/WebAssembly.rst

81

typo: s/care/are/

201

typo: s/This features/This feature/

lld/wasm/Driver.cpp

886

Here you mean "stub symbol" , right? Same issue in other messages in this hunk.

lld/wasm/SymbolTable.cpp

44

What happens with stub files that have duplicates? Say

foo: x, y
foo: z

Should the dependencies be x, y, z or should this be disallowed?

[sbc100]

• feedback

lld/wasm/SymbolTable.cpp

44

Good point. I modified the implementation so that the dependencies are dictated only by the first stub object that contains that symbol.

[dschuff]

lld/docs/WebAssembly.rst

78

Can this be split out from the stub support? or is it tightly coupled somehow?

217

Can you clarify this a little? which module must export malloc and free? The module that imports foo? Any module loaded before foo is loaded at runtime?

Why do we need to have this per-symbol dependence rather than just having a list of symbols that the dylib imports and exports? Is our linking model actually that granular at runtime?

lld/test/wasm/Inputs/libstub.so

5

lld/wasm/Driver.cpp

282

Any particular reason not to just make #STUB the file magic for a new kind of file? or is that not done for text files?

lld/wasm/InputFiles.cpp

684

is there a more generic isspace-type function that would be better here?

[sbc100]

Feedback

[sbc100]

lld/docs/WebAssembly.rst

78

I just noticed it was missing from the docs. Want me to split it out?

217

Yes it needs to be on a per-symbol basis. For example, imagine I just use one symbol from libemscripten.stub.so... and that symbol has no dependencies. But there are 1000 other symbols in there with various dependencies, including malloc and free.

If we just did this on a per-shared-object bases any program that uses any symbol from libemscripten.stub.so would be forced to define and export the superset of all possible dependencies. In that case of my tiny program I don't want to be forces to export anything, just to use emscripten_get_now, for example.

217

I tried to clarify a little. I use the term "output module" to refer to the module being linked. Perhaps you can think of a more clear way to express that?

[sbc100]

• rebase

[dschuff]

lld/docs/WebAssembly.rst

217

But isn't it the case that a stub library represents a real DSO that will be loaded at runtime? (I think that's the case with ELF stubs). In which case the real DSO has whatever symbols and dependencies it has.

I guess the answer is actually "no", we are depending on the behavior that libemscripten isn't really a DSO, but it acts more like an archive with one function per member, i.e. each function will only be linked if it is needed.

I think in that sense, this isn't really a "stub object" at all, and has kind of weird linking behavior.

Is there any way we can improve this, to make stub objects behave more like regular objects or DSOs? I don't think we want to create a stub object for every JS function, that would get unwieldy very fast. Maybe a "stub archive" that can define multiple objects (and we'd just have one such object for every JS function).
Am I making sense here? I would be happier if this functionality were of more general use that only emscripten's JS use case.

[sbc100]

lld/docs/WebAssembly.rst

217

Yes, I agree. What we really want here is something closer the behaviour of the library. So we should call this "Stub library" or "Stub archive" I guess. SGTM

I think this functionality will be useful to anyone wanting to express external runtime imports that come with required exports (malloc and free being the canonical such requirements).

[sbc100]

• stub libraries

[dschuff]

I think this feature could also be useful in non-emscripten environments, e.g. you could have a stub library for each WASI API provided by the embedder. Does wasi-sdk just use --allow-undefined for this now? Stub libraries seem nicer than that.
@sunfish WDYT, does this look useful?

[sbc100]

In D145308#4181854, @dschuff wrote:

I think this feature could also be useful in non-emscripten environments, e.g. you could have a stub library for each WASI API provided by the embedder. Does wasi-sdk just use --allow-undefined for this now? Stub libraries seem nicer than that.
@sunfish WDYT, does this look useful?

I think wasi sdk currently relies on __attribute__(import_name) for all the WASI syscalls, but users who want to add more/custom imports often do --allow-undefined or --allow-undefined-file=.

I don't think there is any way to express the reverse dependencies (such as malloc/free), that imports have, and this would allow that. Having said that WASI itself doesn't have such reverse dependencies (at least not that I know of). I believe the component model bakes in the code dependency on malloc/free, so this could be useful for making process towards supporting the the component module in wasm-ld.

[sbc100]

WDYT @sunfish ? My guess is that wasi-sdk doesn't currently need to express these reverse dependencies, so its not currently relevant.

[sbc100]

• error -> assert

[sbc100]

• remove unused method

[dschuff]

Otherwise this LGTM, but I'd still like to hear Dan's opinion if he's got one.

lld/wasm/Driver.cpp

877

can we rename the methods and variables/comments to match the renaming of the feature?

[sbc100]

• feedback