[MLIR] introduce and use add_mlir_{tool,executable} variants

[dtzWill]

Summary

• Identify MLIR targets
• Optionally don't build tools
• per-project installation directory (useful for standalone)

[dtzWill]

FWIW this isn't quite enough for MLIR standalone but helps get things there. One issue is that the use of LLVM's TableGen.cmake instead of our own means that mlir-tblgen wants to be installed to LLVM_TOOLS_INSTALL_DIR which isn't great when building MLIR separately from LLVM (and keeping installation paths separate, such as for packaging purposes).

The cmake bits are certainly copies of equivalents in LLVM, which I don't love but seems to be what other projects do as well.

(IIRC, similarly, downstream projects using MLIR that try using a standalone-built MLIR will have trouble finding its mlir-tblgen if not among the LLVM tools? Maybe that's just the way I've temporarily resolved the above issue)

[rriddle]

I am not really a competent CMake reviewer (just know enough to be dangerous), @mehdi_amini do you have an idea of someone who could competently review this?

[dtzWill]

Thanks!

I wasn't sure who to ask either.

FWIW https://reviews.llvm.org/D124208 is also in need of a reviewer and is of a similar nature (CMake-only changes, helps build behavior building standalone).

[stellaraccident]

In D124207#3470937, @rriddle wrote:

I am not really a competent CMake reviewer (just know enough to be dangerous), @mehdi_amini do you have an idea of someone who could competently review this?

Cmake is dangerous. Knowing more or less makes no difference :)

This looks good to me and has needed to be done for a while. We may want to tweak the macro more but having it is step one. Thank you.

Are you able to land or do you need assistance with that?

[stellaraccident]

In D124207#3465918, @dtzWill wrote:

FWIW this isn't quite enough for MLIR standalone but helps get things there. One issue is that the use of LLVM's TableGen.cmake instead of our own means that mlir-tblgen wants to be installed to LLVM_TOOLS_INSTALL_DIR which isn't great when building MLIR separately from LLVM (and keeping installation paths separate, such as for packaging purposes).

The cmake bits are certainly copies of equivalents in LLVM, which I don't love but seems to be what other projects do as well.

(IIRC, similarly, downstream projects using MLIR that try using a standalone-built MLIR will have trouble finding its mlir-tblgen if not among the LLVM tools? Maybe that's just the way I've temporarily resolved the above issue)

TableGen is special ina few ways that I would need to take a closer look at to identify issues. Happy to do so, but can we summarize problems on the discussion to look more at of needed?

[mehdi_amini]

per-project installation directory (useful for standalone)

Can you help me figuring out what this means? (which part of the code change does something about this here?)

[stellaraccident]

In D124207#3490221, @mehdi_amini wrote:

per-project installation directory (useful for standalone)

Can you help me figuring out what this means? (which part of the code change does something about this here?)

Doesn't look like it applies to this patch. Maybe leaking from a related patch.

[dtzWill]

In D124207#3490324, @stellaraccident wrote:

In D124207#3490221, @mehdi_amini wrote:

per-project installation directory (useful for standalone)

Can you help me figuring out what this means? (which part of the code change does something about this here?)

Doesn't look like it applies to this patch. Maybe leaking from a related patch.

Thanks for the review!

Honestly, I thought you might be right :)...

However I believe that this does apply here: using add_llvm_tool/add_llvm_executable from an already installed LLVM causes those targets to try to be installed to LLVM's tool bin directory, which when packaging isn't always appropriate (or writable).

See: https://github.com/llvm/llvm-project/blob/cbfa85734632ad7f961b18539464e7e870f80fd6/llvm/cmake/modules/AddLLVM.cmake#L1255
and
LLVM_TOOLS_INSTALL_DIR is set in the generated LLVMConfig.cmake using this template:
https://github.com/llvm/llvm-project/blob/cbfa85734632ad7f961b18539464e7e870f80fd6/llvm/cmake/modules/LLVMConfig.cmake.in#L124 .
The code added in this change uses CMAKE_INSTALL_BINDIR which is what that variable defaults to in LLVM's build: https://github.com/llvm/llvm-project/blob/cbfa85734632ad7f961b18539464e7e870f80fd6/llvm/CMakeLists.txt#L307 .

I think this also means downstream projects using MLIR's version will work standalone more easily too, but I forget if I tested that.

As for TableGen I'll get back to you. Basically more bits where it expects/searches/generates targets using LLVM paths or so, but I didn't find a good way to fix that served all use cases. For now I think I copy the installed cmake modules, sed-fu them, and point projects at that. I'll try to bring it up for discussion with more details soon!

[stellaraccident]

Thanks. I'm fine landing this as a starting point. Having a project specific macro, even if it doesn't do everything we want yet, is a step forward.

mlir/CMakeLists.txt

 set(MLIR_INSTALL_AGGREGATE_OBJECTS 1 CACHE BOOL
     "Installs object files needed for add_mlir_aggregate to work out of \
     tree. Package maintainers can disable this to exclude these assets if \
     not desired. Enabling this will result in object files being written \
     under lib/objects-{CMAKE_BUILD_TYPE}.")
 
 set(MLIR_BUILD_MLIR_C_DYLIB 0 CACHE BOOL "Builds libMLIR-C shared library.")
 
+option(MLIR_BUILD_TOOLS
+  "Build the MLIR tools. If OFF, just generate build targets." ON)
+
 #-------------------------------------------------------------------------------
 # Python Bindings Configuration
 # Requires:
 #   The pybind11 library can be found (set with -DPYBIND_DIR=...)
 #   The python executable is correct (set with -DPython3_EXECUTABLE=...)
 #-------------------------------------------------------------------------------
 
 set(MLIR_ENABLE_BINDINGS_PYTHON 0 CACHE BOOL

mlir/cmake/modules/AddMLIR.cmake

function(mlir_check_all_link_libraries name)
   if(TARGET ${name})
     get_target_property(libs ${name} LINK_LIBRARIES)
     # message("${name} libs are: ${libs}")
     foreach(lib ${libs})
       mlir_check_link_libraries(${lib})
     endforeach()
   endif()
 endfunction(mlir_check_all_link_libraries)
+
+macro(add_mlir_executable name)
+  add_llvm_executable(${name} ${ARGN})
+  set_target_properties(${name} PROPERTIES FOLDER "MLIR executables")
+endmacro(add_mlir_executable)
+
+macro(add_mlir_tool name)
+  if (NOT MLIR_BUILD_TOOLS)
+    set(EXCLUDE_FROM_ALL ON)
+  endif()
+
+  add_mlir_executable(${name} ${ARGN})
+
+  if (MLIR_BUILD_TOOLS)
+    get_target_export_arg(${name} MLIR export_to_mlirtargets)
+    install(TARGETS ${name}
+      ${export_to_mlirtargets}
+      RUNTIME DESTINATION "${CMAKE_INSTALL_BINDIR}"
+      COMPONENT ${name})
+
+    if(NOT LLVM_ENABLE_IDE)
+      add_llvm_install_targets(install-${name}
+                               DEPENDS ${name}
+                               COMPONENT ${name})
+    endif()
+    set_property(GLOBAL APPEND PROPERTY MLIR_EXPORTS ${name})
+  endif()
+endmacro()

mlir/examples/standalone/standalone-opt/CMakeLists.txt

 get_property(dialect_libs GLOBAL PROPERTY MLIR_DIALECT_LIBS)
 get_property(conversion_libs GLOBAL PROPERTY MLIR_CONVERSION_LIBS)
 set(LIBS
         ${dialect_libs}
         ${conversion_libs}
         MLIRArithmetic
         MLIROptLib
         MLIRStandalone
         )
-add_llvm_executable(standalone-opt standalone-opt.cpp)
+add_mlir_executable(standalone-opt standalone-opt.cpp)
 
 llvm_update_compile_flags(standalone-opt)
 target_link_libraries(standalone-opt PRIVATE ${LIBS})
 
 mlir_check_all_link_libraries(standalone-opt)

mlir/examples/standalone/standalone-translate/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   Support
   )
 
 get_property(dialect_libs GLOBAL PROPERTY MLIR_DIALECT_LIBS)
 get_property(translation_libs GLOBAL PROPERTY MLIR_TRANSLATION_LIBS)
 
-add_llvm_executable(standalone-translate
+add_mlir_executable(standalone-translate
   standalone-translate.cpp
   )
 llvm_update_compile_flags(standalone-translate)
 target_link_libraries(standalone-translate
   PRIVATE
   ${dialect_libs}
   ${translation_libs}
   MLIRIR
   MLIRParser
   MLIRPass
   MLIRSPIRV
   MLIRTranslateLib
   MLIRSupport
   )
 
 mlir_check_link_libraries(standalone-translate)

mlir/examples/standalone/test/CAPI/CMakeLists.txt

 # Here we create a single aggregate shared library with the parts of the CAPI
 # that we want to bundle together. Then we link a simple C executable
 # against it to demonstrate that it does have the fully self contained
 # core MLIR library and our own standalone dialect.
 add_mlir_aggregate(StandaloneCAPITestLib
   SHARED
   EMBED_LIBS
   MLIRCAPIIR
   MLIRCAPIRegistration
   StandaloneCAPI
 )
 
-add_llvm_executable(standalone-capi-test
+add_mlir_executable(standalone-capi-test
   standalone-capi-test.c
 )
 llvm_update_compile_flags(standalone-capi-test)
 target_link_libraries(standalone-capi-test
   PRIVATE StandaloneCAPITestLib)

mlir/test/CAPI/CMakeLists.txt

 function(_add_capi_test_executable name)
   cmake_parse_arguments(ARG
     ""
     ""
     "LINK_LIBS"
     ${ARGN})
   set(LLVM_LINK_COMPONENTS
     )
-  add_llvm_executable(${name}
+  add_mlir_executable(${name}
     PARTIAL_SOURCES_INTENDED
     ${ARG_UNPARSED_ARGUMENTS})
   llvm_update_compile_flags(${name})
   if(MLIR_BUILD_MLIR_C_DYLIB)
     target_link_libraries(${name} PRIVATE
       MLIR-C)
   else()
     target_link_libraries(${name} PRIVATE

mlir/tools/mlir-cpu-runner/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   Core
   Support
   nativecodegen
   native
   )
 
-add_llvm_tool(mlir-cpu-runner
+add_mlir_tool(mlir-cpu-runner
   mlir-cpu-runner.cpp
   )
 llvm_update_compile_flags(mlir-cpu-runner)
 target_link_libraries(mlir-cpu-runner PRIVATE
   MLIRAnalysis
   MLIRExecutionEngine
   MLIRIR
   MLIRJitRunner
   MLIRLLVMIR
   MLIRLLVMToLLVMIRTranslation
   MLIRToLLVMIRTranslationRegistration
   MLIRParser
   MLIRTargetLLVMIRExport
   MLIRSupport
   )

mlir/tools/mlir-linalg-ods-gen/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   Core
   Support
   )
 
 # New mlir-linalg-ods-yaml-gen.
-add_llvm_tool(mlir-linalg-ods-yaml-gen
+add_mlir_tool(mlir-linalg-ods-yaml-gen
   mlir-linalg-ods-yaml-gen.cpp
 )
 llvm_update_compile_flags(mlir-linalg-ods-yaml-gen)
 target_link_libraries(mlir-linalg-ods-yaml-gen PRIVATE
   MLIRIR
   MLIRSupport
   MLIRParser
   )

mlir/tools/mlir-lsp-server/CMakeLists.txt

set(LIBS
   MLIRParser
   MLIRPass
   MLIRTransforms
   MLIRTransformUtils
   MLIRSupport
   MLIRIR
   )
 
-add_llvm_tool(mlir-lsp-server
+add_mlir_tool(mlir-lsp-server
   mlir-lsp-server.cpp
 
   DEPENDS
   ${LIBS}
   )
 target_link_libraries(mlir-lsp-server PRIVATE ${LIBS})
 llvm_update_compile_flags(mlir-lsp-server)
 
 mlir_check_all_link_libraries(mlir-lsp-server)

mlir/tools/mlir-opt/CMakeLists.txt

add_mlir_library(MLIRMlirOptMain
   mlir-opt.cpp
 
   EXCLUDE_FROM_LIBMLIR
 
   LINK_LIBS PUBLIC
   ${LIBS}
   )
 
-add_llvm_tool(mlir-opt
+add_mlir_tool(mlir-opt
   mlir-opt.cpp
 
   DEPENDS
   ${LIBS}
   )
 target_link_libraries(mlir-opt PRIVATE ${LIBS})
 llvm_update_compile_flags(mlir-opt)
 
 mlir_check_all_link_libraries(mlir-opt)

mlir/tools/mlir-pdll-lsp-server/CMakeLists.txt

 set(LIBS
   MLIRPdllLspServerLib
   )
 
-add_llvm_tool(mlir-pdll-lsp-server
+add_mlir_tool(mlir-pdll-lsp-server
   mlir-pdll-lsp-server.cpp
 
   DEPENDS
   ${LIBS}
   )
 
 target_link_libraries(mlir-pdll-lsp-server PRIVATE ${LIBS})
 llvm_update_compile_flags(mlir-pdll-lsp-server)
 
 mlir_check_all_link_libraries(mlir-pdll-lsp-server)

mlir/tools/mlir-pdll/CMakeLists.txt

 set(LIBS
   MLIRPDLLAST
   MLIRPDLLCodeGen
   MLIRPDLLParser
   )
 
-add_llvm_tool(mlir-pdll
+add_mlir_tool(mlir-pdll
   mlir-pdll.cpp
 
   DEPENDS
   ${LIBS}
   )
 
 target_link_libraries(mlir-pdll PRIVATE ${LIBS})
 llvm_update_compile_flags(mlir-pdll)
 
 mlir_check_all_link_libraries(mlir-pdll)

mlir/tools/mlir-reduce/CMakeLists.txt

set(LIBS
   ${dialect_libs}
   ${test_libs}
   MLIRDialect
   MLIRIR
   MLIRPass
   MLIRReduceLib
   )
 
-add_llvm_tool(mlir-reduce
+add_mlir_tool(mlir-reduce
   mlir-reduce.cpp
 
   DEPENDS
   ${LIBS}
   )
 
 target_link_libraries(mlir-reduce PRIVATE ${LIBS})
 llvm_update_compile_flags(mlir-reduce)
 
 mlir_check_all_link_libraries(mlir-reduce)

mlir/tools/mlir-spirv-cpu-runner/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   Linker
 )
 
 if (MLIR_ENABLE_SPIRV_CPU_RUNNER)
   message(STATUS "Building SPIR-V CPU runner")
 
-  add_llvm_tool(mlir-spirv-cpu-runner
+  add_mlir_tool(mlir-spirv-cpu-runner
     mlir-spirv-cpu-runner.cpp
   )
 
   llvm_update_compile_flags(mlir-spirv-cpu-runner)
 
   get_property(conversion_libs GLOBAL PROPERTY MLIR_CONVERSION_LIBS)
 
   target_link_libraries(mlir-spirv-cpu-runner PRIVATE

mlir/tools/mlir-translate/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   Support
   )
 
 get_property(dialect_libs GLOBAL PROPERTY MLIR_DIALECT_LIBS)
 get_property(translation_libs GLOBAL PROPERTY MLIR_TRANSLATION_LIBS)
 
-add_llvm_tool(mlir-translate
+add_mlir_tool(mlir-translate
   mlir-translate.cpp
   )
 llvm_update_compile_flags(mlir-translate)
 target_link_libraries(mlir-translate
   PRIVATE
   ${dialect_libs}
   ${translation_libs}
   ${test_libs}

mlir/tools/mlir-vulkan-runner/CMakeLists.txt

if (MLIR_ENABLE_VULKAN_RUNNER)
   llvm_expand_pseudo_components(TARGET_LIBS AllTargetsCodeGens)
   # Prepend LLVM in front of every target, this is how the library
   # are named with CMake
   SET(targets_to_link)
   FOREACH(t ${TARGET_LIBS})
     LIST(APPEND targets_to_link "LLVM${t}")
   ENDFOREACH(t)
 
-  add_llvm_tool(mlir-vulkan-runner
+  add_mlir_tool(mlir-vulkan-runner
     mlir-vulkan-runner.cpp
 
     DEPENDS
     vulkan-runtime-wrappers
   )
   llvm_update_compile_flags(mlir-vulkan-runner)
   target_link_libraries(mlir-vulkan-runner PRIVATE ${LIBS})
 
 endif()