Patch StandalonePlugin CMake for MacOS

[makslevental]

Summary

This is a story about man against machine (me vs. dyld)...

This is a story about two worlds separated by seemingly arbitrary and capricious differences (Linux and Mac OS)...

This is the greatest love story ever told (MLIR and extensibility)...

Okay in all seriousness this took an inordinate amount of time so I'm going to belabor a little what it took to figure it out. The TL;DR (if you don't enjoy dramatic retellings of bug squashing) is that we can use add_llvm_library with MODULE to make it work on Linux, Mac, and possibly Windows (with PLUGIN_TOOL).

Opening scene (the problem)

The previous patch (D147053) seemingly sometimes worked and sometimes didn't. In particular, on Mac it worked but you had to fudge things - adding -undefined dynamic_lookup to StandalonePlugin got some of the tests to pass but failed others, then removing the statically linked MLIR libs in MLIRStandalone got more to pass but that's hacky^2. In totum it seemed like a really weird asymmetry that implied linking on Mac worked dramatically differently from linking on Linux. It would've been plausible if they were only subtley different but dropping statically linked libs from MLIRStandalone just made it too suspicious.

Hero's journey (the debugging process)

So the suspicion was that at runtime dyld on Mac was doing something different from ld on Linux. On Linux you can use LD_DEBUG_OUTPUT=ldlog LD_DEBUG=all LD_BIND_NOT=1 to see all of the bindings as they happen (i.e., where the code implementing the symbol is fetched from):

binding file libStandalonePlugin.so [0] to cmake-build-release/bin/mlir-opt [0]: normal symbol `_ZN4mlir19PassInstrumentation17runBeforePipelineESt8optionalINS_13OperationNameEERKNS0_18PipelineParentInfoE'
binding file libStandalonePlugin.so [0] to /home/mlevental/dev_projects/llvm/cmake-build-release/bin/mlir-opt [0]: normal symbol `_ZN4mlir19PassInstrumentation16runAfterPipelineESt8optionalINS_13OperationNameEERKNS0_18PipelineParentInfoE'
binding file libStandalonePlugin.so [0] to ./tools/mlir/test/Examples/standalone/lib/libStandalonePlugin.so [0]: normal symbol `_ZNK4mlir10standalone17StandaloneDialect9parseTypeERNS_16DialectAsmParserE'
binding file libStandalonePlugin.so [0] to ./tools/mlir/test/Examples/standalone/lib/libStandalonePlugin.so [0]: normal symbol `_ZNK4mlir10standalone17StandaloneDialect9printTypeENS_4TypeERNS_17DialectAsmPrinterE'

Inspecting this showed that indeed (as expected) almost all symbols are bound to mlir-opt rather than libStandalonePlugin.so on Linux. On Mac you can do a similar thing with DYLD_PRINT_SEARCHING=1 DYLD_PRINT_BINDINGS=1 DYLD_PRINT_LIBRARIES=1 but it doesn't show all the bindings. Ultimately it took setting breakpoints and printfing to reveal that what was happening was mlir::registerPass was being bound to libStandalonePlugin.dylib instead of mlir-opt and thus two separate static ... passRegistrys were being instantiated. But why???

Victory (solution)

At some point I got the bright (sarcasm - should've been the first) idea to look at how LLVM plugins work, in particular what linker flags are used; inspecting my build.ninja I found the auspicious bit:

#############################################
# Link the shared module unittests/Analysis/InlineOrderPlugin.dylib

build unittests/Analysis/InlineOrderPlugin.dylib: ...
  LINK_FLAGS = -L/opt/homebrew/opt/llvm/lib -Wl,-flat_namespace -Wl,-undefined -Wl,suppress
  ...

What is -Wl,-flat_namespace you might be wondering?

All references to interposable symbols can be redirected at runtime to
point to a different symbol definition (with the same name). For
example, if both dylib A and B define symbol _foo, and we load A before
B at runtime, then all references to _foo within dylib B will point to
the definition in dylib A.

[dyld] makes all extern symbols interposable when linking with -flat_namespace.

from none other than our very own D119294. These flags are applied on APPLE when linking MODULEs. Et voila - using LLVM's add_llvm_library with MODULE basically worked without a hitch (see inline comments for small hitch).

Epilogue

Maybe we want our own macro instead of add_llvm_library? Or an extension of add_mlir_library? I gave it a shot but you need to thread a few more things than just MODULE.

Tidbit

At some point I had a solution that left symbols to be resolved at runtime in MLIRStandalone for both Linux and Mac (i.e., no static linking at all). On Mac, as already mentioned, this requires -undefined dynamic_lookup> but on Linux I also had to pass -Wl,--unresolved-symbols=ignore-in-object-files and that was very strange because everywhere you read that on Linux, deferring symbol to resolution to runtime is the default behavior. Well it turns out we don't like defaults and force -z,defs. Probably wise but definitely confusing.

Btw, if you want to try just the test that intersects with this patch you can filter lit like this:

-DLLVM_LIT_ARGS="-v --show-unsupported --filter=Standalone.*"

[makslevental]

get it right (use add_llvm_library)

[makslevental]

mlir/examples/standalone/standalone-plugin/CMakeLists.txt

2

Will add a comment here that BUILDTREE_ONLY is only for the sake testing.

8–9

If I'm understanding this correctly then PLUGIN_TOOL makes this work on Windows as well.

17

I'm guessing that somewhere in add_mlir_library the directory for the generated headers and cpps is included.

mlir/examples/standalone/test/Standalone/standalone-pass-plugin.mlir

1

lib is dropped because MODULE doesn't build with a lib prefix.

[makslevental]

Ping?

[fmorac]

Hey great explanation and patch, it's great that you figured out the right flags for MacOS!

mlir/examples/standalone/lib/Standalone/CMakeLists.txt

 add_mlir_dialect_library(MLIRStandalone
         StandaloneTypes.cpp
         StandaloneDialect.cpp
         StandaloneOps.cpp
         StandalonePasses.cpp
 
         ADDITIONAL_HEADER_DIRS
         ${PROJECT_SOURCE_DIR}/include/Standalone
 
         DEPENDS
         MLIRStandaloneOpsIncGen
         MLIRStandalonePassesIncGen
 
-  LINK_LIBS PUBLIC
-  MLIRIR
+        LINK_LIBS PUBLIC
+        MLIRIR
         MLIRInferTypeOpInterface
         MLIRFuncDialect
-  )
+        )

mlir/examples/standalone/standalone-plugin/CMakeLists.txt

-get_property(dialect_libs GLOBAL PROPERTY MLIR_DIALECT_LIBS)
-get_property(conversion_libs GLOBAL PROPERTY MLIR_CONVERSION_LIBS)
-set(LIBS
-        MLIRIR
-        MLIRPass
-        MLIRPluginsLib
-        MLIRStandalone
-        MLIRTransformUtils
-        )
-
-add_mlir_dialect_library(StandalonePlugin
-        SHARED
+add_llvm_library(StandalonePlugin
+        # BUILDTREE_ONLY is only for testing purposes
+        MODULE BUILDTREE_ONLY
         standalone-plugin.cpp
 
         DEPENDS
         MLIRStandalone
-        )
+        PLUGIN_TOOL
+        mlir-opt
 
-llvm_update_compile_flags(StandalonePlugin)
-target_link_libraries(StandalonePlugin PRIVATE ${LIBS})
-
-mlir_check_all_link_libraries(StandalonePlugin)
+        LINK_LIBS
+        MLIRStandalone
+        )
+target_include_directories(
+        StandalonePlugin
+        PRIVATE
+        "${STANDALONE_BINARY_DIR}/include"
+)

mlir/examples/standalone/test/CMakeLists.txt

 llvm_canonicalize_cmake_booleans(
         MLIR_ENABLE_BINDINGS_PYTHON
 )
 
 configure_lit_site_cfg(
         ${CMAKE_CURRENT_SOURCE_DIR}/lit.site.cfg.py.in
         ${CMAKE_CURRENT_BINARY_DIR}/lit.site.cfg.py
         MAIN_CONFIG
         ${CMAKE_CURRENT_SOURCE_DIR}/lit.cfg.py
 )
 
 set(STANDALONE_TEST_DEPENDS
         FileCheck count not
         standalone-capi-test
         standalone-opt
         standalone-translate
+        StandalonePlugin
         )
 if(MLIR_ENABLE_BINDINGS_PYTHON)
   list(APPEND STANDALONE_TEST_DEPENDS StandalonePythonModules)
 endif()
 
 add_lit_testsuite(check-standalone "Running the standalone regression tests"
         ${CMAKE_CURRENT_BINARY_DIR}
         DEPENDS ${STANDALONE_TEST_DEPENDS}
         )
 set_target_properties(check-standalone PROPERTIES FOLDER "Tests")
 
 add_lit_testsuites(STANDALONE ${CMAKE_CURRENT_SOURCE_DIR} DEPENDS ${STANDALONE_TEST_DEPENDS})
 
 add_subdirectory(CAPI)

mlir/examples/standalone/test/Standalone/standalone-pass-plugin.mlir

-// RUN: mlir-opt %s --load-pass-plugin=%standalone_libs/libStandalonePlugin.so --pass-pipeline="builtin.module(standalone-switch-bar-foo)" | FileCheck %s
+// RUN: mlir-opt %s --load-pass-plugin=%standalone_libs/StandalonePlugin%shlibext --pass-pipeline="builtin.module(standalone-switch-bar-foo)" | FileCheck %s
 
 module {
   // CHECK-LABEL: func @foo()
   func.func @bar() {
     return
   }
 
   // CHECK-LABEL: func @abar()
   func.func @abar() {
     return
   }
 }

mlir/examples/standalone/test/Standalone/standalone-plugin.mlir

-// RUN: mlir-opt %s --load-dialect-plugin=%standalone_libs/libStandalonePlugin.so --pass-pipeline="builtin.module(standalone-switch-bar-foo)" | FileCheck %s
+// RUN: mlir-opt %s --load-dialect-plugin=%standalone_libs/StandalonePlugin%shlibext --pass-pipeline="builtin.module(standalone-switch-bar-foo)" | FileCheck %s
 
 module {
   // CHECK-LABEL: func @foo()
   func.func @bar() {
     return
   }
 
   // CHECK-LABEL: func @standalone_types(%arg0: !standalone.custom<"10">)
   func.func @standalone_types(%arg0: !standalone.custom<"10">) {
     return
   }
 }

mlir/examples/standalone/test/lit.cfg.py

 
 # test_source_root: The root path where tests are located.
 config.test_source_root = os.path.dirname(__file__)
 
 # test_exec_root: The root path where tests should be run.
 config.test_exec_root = os.path.join(config.standalone_obj_root, 'test')
 
 config.substitutions.append(('%PATH%', config.environment['PATH']))
+config.substitutions.append(('%shlibext', config.llvm_shlib_ext))
 
 llvm_config.with_system_environment(
     ['HOME', 'INCLUDE', 'LIB', 'TMP', 'TEMP'])
 
 llvm_config.use_default_substitutions()
 
 # excludes: A list of directories to exclude from the testsuite. The 'Inputs'
 # subdirectories contain auxiliary inputs for various tests in their parent

mlir/examples/standalone/test/lit.site.cfg.py.in

 @LIT_SITE_CFG_IN_HEADER@
 
 config.llvm_tools_dir = lit_config.substitute("@LLVM_TOOLS_DIR@")
 config.mlir_obj_dir = "@MLIR_BINARY_DIR@"
 config.enable_bindings_python = @MLIR_ENABLE_BINDINGS_PYTHON@
 config.standalone_obj_root = "@STANDALONE_BINARY_DIR@"
+config.llvm_shlib_ext = "@SHLIBEXT@"
 
 import lit.llvm
 lit.llvm.initialize(lit_config, config)
 
 # Let the main config do the real work.
 lit_config.load_config(config, "@STANDALONE_SOURCE_DIR@/test/lit.cfg.py")

mlir/test/Examples/standalone/test.toy

-# RUN: %cmake_exe %mlir_src_root/examples/standalone -G "%cmake_generator" \
+# RUN: "%cmake_exe" "%mlir_src_root/examples/standalone" -G "%cmake_generator" \
 # RUN: -DCMAKE_CXX_COMPILER=%host_cxx -DCMAKE_C_COMPILER=%host_cc \
 # RUN: -DLLVM_ENABLE_LIBCXX=%enable_libcxx -DMLIR_DIR=%mlir_cmake_dir \
 # RUN: -DLLVM_USE_LINKER=%llvm_use_linker \
 # RUN: -DPython3_EXECUTABLE=%python
-# RUN: %cmake_exe --build . --target check-standalone | tee %t | FileCheck %s
+# RUN: "%cmake_exe" --build . --target check-standalone | tee %t | FileCheck %s
 
 # Note: The number of checked tests is not important. The command will fail
 # if any fail.
 # CHECK: Passed
 # UNSUPPORTED: target={{.*(windows|android).*}}