This is an archive of the discontinued LLVM Phabricator instance.

[clang][WebAssembly] support wasm32-wasi shared libraries
ClosedPublic

Authored by dicej on Jun 19 2023, 9:11 AM.

Details

Summary

This adds support for Emscripten-style shared libraries [1] to the wasm32-wasi
target. Previously, only static linking was supported, and the -shared and
-fPIC flags were simply ignored. Now both flags are honored.

Since WASI runtimes do not necessarily include JavaScript support, we cannot
rely on the JS-based Emscripten linker to link shared libraries. Instead, we
link them using the Component Model proposal [2].

We have prototyped shared library support in wasi-sdk [3] and put together a
demo [4] which uses a patched version of wit-component [5] to link libraries
using the Component Model. We plan to submit the required changes upstream to
the respective repositories in the next week or two.

[1] https://github.com/WebAssembly/tool-conventions/blob/main/DynamicLinking.md
[2] https://github.com/WebAssembly/component-model/blob/main/design/mvp/examples/SharedEverythingDynamicLinking.md
[3] https://github.com/dicej/wasi-sdk/tree/dynamic-linking
[4] https://github.com/dicej/component-linking-demo
[5] https://github.com/bytecodealliance/wasm-tools/tree/main/crates/wit-component

Signed-off-by: Joel Dice <joel.dice@fermyon.com>

Diff Detail

Event Timeline

dicej created this revision.Jun 19 2023, 9:11 AM
Herald added a project: Restricted Project. · View Herald TranscriptJun 19 2023, 9:11 AM
dicej requested review of this revision.Jun 19 2023, 9:11 AM
Herald added projects: Restricted Project, Restricted Project. · View Herald TranscriptJun 19 2023, 9:11 AM

Nice! lgtm with a couple of nits

clang/docs/ReleaseNotes.rst
705

Would it not make more sense to simply say that all WebAssembly targets now support shared libraries and PIC code generation? Perhaps we could phrase it something like this:

"Shared library support (and PIC code generation) for is no longer limited to the Emscripten target OS and now works with other targets such as wasm32-wasi."

clang/test/Driver/wasm-toolchain.c
38

Perhaps this part could be enerat

63

The test above say with known OS.. which is perhaps more appropriate here.

Also, shouldn't all these also work (and be tested with) unknown OS?

llvm/utils/lit/lit/llvm/config.py
347 ↗(On Diff #532688)

Are the the changes to this file meant to be part of this CL?

Thanks for the review. I'll post an update shortly.

clang/docs/ReleaseNotes.rst
705

Agreed! I'll update it.

clang/test/Driver/wasm-toolchain.c
38

Sorry, I'm having trouble parsing this; did your comment get cut off?

63

Sounds reasonable. I'll edit the comment and add tests for the unknown case.

llvm/utils/lit/lit/llvm/config.py
347 ↗(On Diff #532688)

The check-clang target doesn't work at all if the target "triple" is a double, e.g.:

llvm-lit: /Users/dicej/p/wasi-sdk/src/llvm-project/llvm/utils/lit/lit/llvm/config.py:459: note: using clang: /Users/dicej/p/wasi-sdk/build/llvm/bin/clang
llvm-lit: /Users/dicej/p/wasi-sdk/src/llvm-project/llvm/utils/lit/lit/llvm/config.py:324: fatal: Could not turn 'wasm32-wasi' into Itanium ABI triple

@sunfish had been using this patch locally to work around the issue, so I figured I'd include the patch here so it stops being a stumbling block. Perhaps there's a better way to address it?

dicej updated this revision to Diff 532705.Jun 19 2023, 10:32 AM

generalize Wasm tests and release notes

sbc100 accepted this revision.Jun 19 2023, 7:00 PM

lgtm.

This revision is now accepted and ready to land.Jun 19 2023, 7:00 PM
dicej added a comment.Jun 20 2023, 8:05 PM

@sbc100 Thanks for the "lgtm". What's the next step to get this merged?

I'm happy to merge it, but perhaps we should get @sunfish to lg too.

Also, should we remove the -experimental-pic linker flag, or are you OK with that warning from the linker? If yes, I wonder if we should do that as part of this CL or a followup?

This patch appears to be just the release notes and the test; is the actual driver change missing?

clang/docs/ReleaseNotes.rst
706

We should mention that this format is not yet stable and may change between LLVM versions, and that WASI does not yet have any facilities for loading dynamic libraries.

dicej added a comment.Jun 21 2023, 6:24 AM

This patch appears to be just the release notes and the test; is the actual driver change missing?

Sorry -- this is the first time I've used arc, and I apparently messed up. I meant to add an extra commit on top of the original, but it just replaced the original. I'll combine them and expand the release notes per your comment.

dicej added a comment.Jun 21 2023, 6:30 AM

I'm happy to merge it, but perhaps we should get @sunfish to lg too.

Also, should we remove the -experimental-pic linker flag, or are you OK with that warning from the linker? If yes, I wonder if we should do that as part of this CL or a followup?

Do emscripten users see that warning, too? I'd like to match whatever they see (or don't see).

dicej updated this revision to Diff 533243.Jun 21 2023, 6:45 AM

update release notes

I'm happy to merge it, but perhaps we should get @sunfish to lg too.

Also, should we remove the -experimental-pic linker flag, or are you OK with that warning from the linker? If yes, I wonder if we should do that as part of this CL or a followup?

Do emscripten users see that warning, too? I'd like to match whatever they see (or don't see).

No, the emscripten driver (emcc) currently adds --experimental-pic to the linker flags to avoid this warning. We could gladly remove that if it becomes the default.

dicej added a comment.EditedJun 21 2023, 12:10 PM

I'm happy to merge it, but perhaps we should get @sunfish to lg too.

Also, should we remove the -experimental-pic linker flag, or are you OK with that warning from the linker? If yes, I wonder if we should do that as part of this CL or a followup?

Do emscripten users see that warning, too? I'd like to match whatever they see (or don't see).

No, the emscripten driver (emcc) currently adds --experimental-pic to the linker flags to avoid this warning. We could gladly remove that if it becomes the default.

OK; I'd say let's leave the warning there since this feature is so new for non-emscripten targets. We can come back and remove it later on once things have settled.

sbc100 added inline comments.Jun 21 2023, 12:30 PM
llvm/utils/lit/lit/llvm/config.py
347 ↗(On Diff #532688)

Can you split this out into a separate change.. its seem unrelated.

dicej updated this revision to Diff 533375.Jun 21 2023, 1:29 PM

remove config.py changes per review feedback

dicej added a comment.Jun 23 2023, 6:12 AM

@sbc100 @sunfish I believe I've addressed all your feedback so far. Anything else you'd like me to do before we call this ready?

sbc100 accepted this revision.Jun 23 2023, 9:17 AM
This revision was landed with ongoing or failed builds.Jun 26 2023, 10:32 AM
This revision was automatically updated to reflect the committed changes.