Page MenuHomePhabricator

Please use GitHub pull requests for new patches. Avoid migrating existing patches. Phabricator shutdown timeline

Patch StandalonePlugin CMake for MacOS

Authored by makslevental on Apr 11 2023, 3:03 PM.



This is a story about man against machine (me vs. dyld)...

This is a story about two worlds separated by seemingly arbitrary and capricious differences (Linux and Mac OS)...

This is the greatest love story ever told (MLIR and extensibility)...

Okay in all seriousness this took an inordinate amount of time so I'm going to belabor a little what it took to figure it out. The TL;DR (if you don't enjoy dramatic retellings of bug squashing) is that we can use add_llvm_library with MODULE to make it work on Linux, Mac, and possibly Windows (with PLUGIN_TOOL).

Opening scene (the problem)

The previous patch (D147053) seemingly sometimes worked and sometimes didn't. In particular, on Mac it worked but you had to fudge things - adding -undefined dynamic_lookup to StandalonePlugin got some of the tests to pass but failed others, then removing the statically linked MLIR libs in MLIRStandalone got more to pass but that's hacky^2. In totum it seemed like a really weird asymmetry that implied linking on Mac worked dramatically differently from linking on Linux. It would've been plausible if they were only subtley different but dropping statically linked libs from MLIRStandalone just made it too suspicious.

Hero's journey (the debugging process)

So the suspicion was that at runtime dyld on Mac was doing something different from ld on Linux. On Linux you can use LD_DEBUG_OUTPUT=ldlog LD_DEBUG=all LD_BIND_NOT=1 to see all of the bindings as they happen (i.e., where the code implementing the symbol is fetched from):

binding file [0] to cmake-build-release/bin/mlir-opt [0]: normal symbol `_ZN4mlir19PassInstrumentation17runBeforePipelineESt8optionalINS_13OperationNameEERKNS0_18PipelineParentInfoE'
binding file [0] to /home/mlevental/dev_projects/llvm/cmake-build-release/bin/mlir-opt [0]: normal symbol `_ZN4mlir19PassInstrumentation16runAfterPipelineESt8optionalINS_13OperationNameEERKNS0_18PipelineParentInfoE'
binding file [0] to ./tools/mlir/test/Examples/standalone/lib/ [0]: normal symbol `_ZNK4mlir10standalone17StandaloneDialect9parseTypeERNS_16DialectAsmParserE'
binding file [0] to ./tools/mlir/test/Examples/standalone/lib/ [0]: normal symbol `_ZNK4mlir10standalone17StandaloneDialect9printTypeENS_4TypeERNS_17DialectAsmPrinterE'

Inspecting this showed that indeed (as expected) almost all symbols are bound to mlir-opt rather than on Linux. On Mac you can do a similar thing with DYLD_PRINT_SEARCHING=1 DYLD_PRINT_BINDINGS=1 DYLD_PRINT_LIBRARIES=1 but it doesn't show all the bindings. Ultimately it took setting breakpoints and printfing to reveal that what was happening was mlir::registerPass was being bound to libStandalonePlugin.dylib instead of mlir-opt and thus two separate static ... passRegistrys were being instantiated. But why???

Victory (solution)

At some point I got the bright (sarcasm - should've been the first) idea to look at how LLVM plugins work, in particular what linker flags are used; inspecting my I found the auspicious bit:

# Link the shared module unittests/Analysis/InlineOrderPlugin.dylib

build unittests/Analysis/InlineOrderPlugin.dylib: ...
  LINK_FLAGS = -L/opt/homebrew/opt/llvm/lib -Wl,-flat_namespace -Wl,-undefined -Wl,suppress

What is -Wl,-flat_namespace you might be wondering?

All references to interposable symbols can be redirected at runtime to
point to a different symbol definition (with the same name). For
example, if both dylib A and B define symbol _foo, and we load A before
B at runtime, then all references to _foo within dylib B will point to
the definition in dylib A.

[dyld] makes all extern symbols interposable when linking with -flat_namespace.

from none other than our very own D119294. These flags are applied on APPLE when linking MODULEs. Et voila - using LLVM's add_llvm_library with MODULE basically worked without a hitch (see inline comments for small hitch).


Maybe we want our own macro instead of add_llvm_library? Or an extension of add_mlir_library? I gave it a shot but you need to thread a few more things than just MODULE.


At some point I had a solution that left symbols to be resolved at runtime in MLIRStandalone for both Linux and Mac (i.e., no static linking at all). On Mac, as already mentioned, this requires -undefined dynamic_lookup> but on Linux I also had to pass -Wl,--unresolved-symbols=ignore-in-object-files and that was very strange because everywhere you read that on Linux, deferring symbol to resolution to runtime is the default behavior. Well it turns out we don't like defaults and force -z,defs. Probably wise but definitely confusing.

Btw, if you want to try just the test that intersects with this patch you can filter lit like this:

-DLLVM_LIT_ARGS="-v --show-unsupported --filter=Standalone.*"

Diff Detail

Event Timeline

makslevental created this revision.Apr 11 2023, 3:03 PM
Herald added a project: Restricted Project. · View Herald TranscriptApr 11 2023, 3:03 PM
makslevental edited the summary of this revision. (Show Details)Apr 12 2023, 8:28 AM

get it right (use add_llvm_library)

makslevental edited the summary of this revision. (Show Details)Apr 12 2023, 10:46 PM
makslevental added inline comments.Apr 12 2023, 10:52 PM

Will add a comment here that BUILDTREE_ONLY is only for the sake testing.


If I'm understanding this correctly then PLUGIN_TOOL makes this work on Windows as well.


I'm guessing that somewhere in add_mlir_library the directory for the generated headers and cpps is included.


lib is dropped because MODULE doesn't build with a lib prefix.

makslevental published this revision for review.Apr 12 2023, 11:02 PM
makslevental edited the summary of this revision. (Show Details)Apr 12 2023, 11:16 PM

Hey great explanation and patch, it's great that you figured out the right flags for MacOS!

mehdi_amini accepted this revision.Apr 17 2023, 2:03 PM
This revision is now accepted and ready to land.Apr 17 2023, 2:03 PM
This revision was automatically updated to reflect the committed changes.