This is an archive of the discontinued LLVM Phabricator instance.

[lld][WebAssembly] Initial support for stub libraries
ClosedPublic

Authored by sbc100 on Mar 4 2023, 10:40 AM.

Details

Summary

See the docs in lld/docs/WebAssembly.rst for more on this.

This feature unlocks a lot of simplification in the emscripten toolchain
since we can represent the JS libraries to the compiler using stub
objects.

See https://github.com/emscripten-core/emscripten/issues/18875

Diff Detail

Event Timeline

sbc100 created this revision.Mar 4 2023, 10:40 AM
Herald added a project: Restricted Project. · View Herald Transcript
Herald added subscribers: pmatos, asb, wingo and 5 others. · View Herald Transcript
sbc100 requested review of this revision.Mar 4 2023, 10:40 AM
Herald added a project: Restricted Project. · View Herald TranscriptMar 4 2023, 10:40 AM
sbc100 updated this revision to Diff 502382.Mar 4 2023, 10:44 AM
  • dead code
pmatos added inline comments.Mar 6 2023, 7:17 AM
lld/docs/WebAssembly.rst
81

typo: s/care/are/

201

typo: s/This features/This feature/

lld/wasm/Driver.cpp
831

Here you mean "stub symbol" , right? Same issue in other messages in this hunk.

lld/wasm/SymbolTable.cpp
45

What happens with stub files that have duplicates? Say

foo: x, y
foo: z

Should the dependencies be x, y, z or should this be disallowed?

sbc100 updated this revision to Diff 502860.Mar 6 2023, 4:44 PM
sbc100 marked 3 inline comments as done.
  • feedback
lld/wasm/SymbolTable.cpp
45

Good point. I modified the implementation so that the dependencies are dictated only by the first stub object that contains that symbol.

dschuff added inline comments.Mar 6 2023, 5:01 PM
lld/docs/WebAssembly.rst
78

Can this be split out from the stub support? or is it tightly coupled somehow?

217

Can you clarify this a little? which module must export malloc and free? The module that imports foo? Any module loaded before foo is loaded at runtime?

Why do we need to have this per-symbol dependence rather than just having a list of symbols that the dylib imports and exports? Is our linking model actually that granular at runtime?

lld/test/wasm/Inputs/libstub.so
5
lld/wasm/Driver.cpp
283

Any particular reason not to just make #STUB the file magic for a new kind of file? or is that not done for text files?

lld/wasm/InputFiles.cpp
684

is there a more generic isspace-type function that would be better here?

sbc100 updated this revision to Diff 502868.Mar 6 2023, 5:15 PM

Feedback

sbc100 added inline comments.Mar 6 2023, 5:16 PM
lld/docs/WebAssembly.rst
78

I just noticed it was missing from the docs. Want me to split it out?

217

Yes it needs to be on a per-symbol basis. For example, imagine I just use one symbol from libemscripten.stub.so... and that symbol has no dependencies. But there are 1000 other symbols in there with various dependencies, including malloc and free.

If we just did this on a per-shared-object bases any program that uses any symbol from libemscripten.stub.so would be forced to define and export the superset of all possible dependencies. In that case of my tiny program I don't want to be forces to export anything, just to use emscripten_get_now, for example.

217

I tried to clarify a little. I use the term "output module" to refer to the module being linked. Perhaps you can think of a more clear way to express that?

sbc100 updated this revision to Diff 502880.Mar 6 2023, 5:57 PM
  • rebase
dschuff added inline comments.Mar 7 2023, 12:11 PM
lld/docs/WebAssembly.rst
217

But isn't it the case that a stub library represents a real DSO that will be loaded at runtime? (I think that's the case with ELF stubs). In which case the real DSO has whatever symbols and dependencies it has.

I guess the answer is actually "no", we are depending on the behavior that libemscripten isn't really a DSO, but it acts more like an archive with one function per member, i.e. each function will only be linked if it is needed.

I think in that sense, this isn't really a "stub object" at all, and has kind of weird linking behavior.

Is there any way we can improve this, to make stub objects behave more like regular objects or DSOs? I don't think we want to create a stub object for every JS function, that would get unwieldy very fast. Maybe a "stub archive" that can define multiple objects (and we'd just have one such object for every JS function).
Am I making sense here? I would be happier if this functionality were of more general use that only emscripten's JS use case.

sbc100 added inline comments.Mar 7 2023, 1:38 PM
lld/docs/WebAssembly.rst
217

Yes, I agree. What we really want here is something closer the behaviour of the library. So we should call this "Stub library" or "Stub archive" I guess. SGTM

I think this functionality will be useful to anyone wanting to express external runtime imports that come with required exports (malloc and free being the canonical such requirements).

sbc100 updated this revision to Diff 503598.Mar 8 2023, 7:04 PM
  • stub libraries

I think this feature could also be useful in non-emscripten environments, e.g. you could have a stub library for each WASI API provided by the embedder. Does wasi-sdk just use --allow-undefined for this now? Stub libraries seem nicer than that.
@sunfish WDYT, does this look useful?

sbc100 added a comment.EditedMar 9 2023, 9:51 AM

I think this feature could also be useful in non-emscripten environments, e.g. you could have a stub library for each WASI API provided by the embedder. Does wasi-sdk just use --allow-undefined for this now? Stub libraries seem nicer than that.
@sunfish WDYT, does this look useful?

I think wasi sdk currently relies on __attribute__(import_name) for all the WASI syscalls, but users who want to add more/custom imports often do --allow-undefined or --allow-undefined-file=.

I don't think there is any way to express the reverse dependencies (such as malloc/free), that imports have, and this would allow that. Having said that WASI itself doesn't have such reverse dependencies (at least not that I know of). I believe the component model bakes in the code dependency on malloc/free, so this could be useful for making process towards supporting the the component module in wasm-ld.

sbc100 retitled this revision from [lld][WebAssembly] Initial support for stub objects to [lld][WebAssembly] Initial support for stub libraries.Mar 13 2023, 3:12 PM
sbc100 added a comment.EditedMar 13 2023, 3:27 PM

WDYT @sunfish ? My guess is that wasi-sdk doesn't currently need to express these reverse dependencies, so its not currently relevant.

sbc100 updated this revision to Diff 504874.Mar 13 2023, 3:39 PM
  • error -> assert
sbc100 updated this revision to Diff 504876.Mar 13 2023, 3:42 PM
  • remove unused method

Otherwise this LGTM, but I'd still like to hear Dan's opinion if he's got one.

lld/wasm/Driver.cpp
822

can we rename the methods and variables/comments to match the renaming of the feature?

sbc100 updated this revision to Diff 504888.Mar 13 2023, 4:28 PM
  • feedback

Any objections to landing this?

dschuff accepted this revision.Mar 23 2023, 1:55 PM

I think we can go ahead. We can always adjust emscripten's behavior as needed to synchronize with wasi-sdk

This revision is now accepted and ready to land.Mar 23 2023, 1:55 PM
This revision was landed with ongoing or failed builds.Mar 23 2023, 2:26 PM
This revision was automatically updated to reflect the committed changes.