[MLIR] Add support for out of tree external projects using MLIR

[stephenneuendorffer]

Summary

LLVM has a documented mechanism for passing configuration information
to an out of tree project using cmake. See
https://llvm.org/docs/CMake.html#embedding-llvm-in-your-project. This
patch adds similar support for MLIR.

Using this requires something like:

cmake_minimum_required(VERSION 3.4.3)
project(SimpleProject)

find_package(MLIR REQUIRED CONFIG)

include_directories(${LLVM_INCLUDE_DIRS})
include_directories(${MLIR_INCLUDE_DIRS})
link_directories(${LLVM_BUILD_LIBRARY_DIR})
add_definitions(${LLVM_DEFINITIONS})

set(CMAKE_MODULE_PATH

${LLVM_CMAKE_DIR}
${MLIR_CMAKE_DIR}
)

include(AddLLVM)
include(TableGen)
include(AddMLIR)

add_executable(test-opt test-opt.cpp)
llvm_update_compile_flags(test-opt)

get_property(dialect_libs GLOBAL PROPERTY MLIR_DIALECT_LIBS)
get_property(conversion_libs GLOBAL PROPERTY MLIR_CONVERSION_LIBS)
message(dialects=${dialect_libs})
set(LIBS

${dialect_libs}
${conversion_libs}
MLIRLoopOpsTransforms
MLIRLoopAnalysis
MLIRAnalysis
MLIRDialect
MLIREDSC
MLIROptLib
MLIRParser
MLIRPass
MLIRQuantizerFxpMathConfig
MLIRQuantizerSupport
MLIRQuantizerTransforms
MLIRSPIRV
MLIRSPIRVTestPasses
MLIRSPIRVTransforms
MLIRTransforms
MLIRTransformUtils
MLIRTestDialect
MLIRTestIR
MLIRTestPass
MLIRTestTransforms
MLIRSupport
MLIRIR
MLIROptLib
LLVMSupport
LLVMCore
LLVMAsmParser
)

target_link_libraries(test-opt ${LIBS})

[mehdi_amini]

mlir/cmake/modules/CMakeLists.txt

46

I don't really get what this is doing and what is the removal path?

[stephenneuendorffer]

mlir/cmake/modules/CMakeLists.txt

46

It's copy-pasted from llvm. This delivers all the cmake files so that they are available for use in out-of-tree code. I don't
particularly understand the comment either, which appear to date back to this:

commit 81b580b455426dd885fe4858e5bd3e2f22c00f5f
Author: NAKAMURA Takumi <[e-mail]>
Date: Sun Feb 9 16:35:40 2014 +0000

Teach LLVMConfig to avoid modifying CMAKE_MODULE_PATH

Do not modify this value on the application's behalf and just ensure API
modules are always available next to the LLVMConfig module.  This is
already the case in the install tree so use file(COPY) to make it so in
the build tree.  Include the LLVM-Config API module from next to the
LLVMConfig location.

Contributed by Brad King.

llvm-svn: 201047

[stephenneuendorffer]

After looking at this again, I think that delivering the .td files isn't necessary for what I'm trying to do (build against a build area, where we still have the full source). It might be necessary for building against an install area, but we can deal with that later, I think. @rriddle do you have any concerns about the current state?

[marbre]

mlir/cmake/modules/CMakeLists.txt

50

Seems like this line might be a leftover from the pre-git times. Can it be dropped?

68–69

Outdated due to the updated definition in line 57?

[stephenneuendorffer]

mlir/cmake/modules/CMakeLists.txt

68–69

It appears that the intention here is to unset the corresponding variables which are used to pass information into the configure_file call. There is an error which is that not all of the *CONFIG* variables are unset.

[jeanPerier]

Hi, thanks for improving the way one can plugin MLIR ! While trying to rebase our f18 project that uses MLIR after this patch, I had a cmake issue using installed LLVMs out-of-tree, I tracked it to one of the line in this patch (see in-lined comment).

mlir/cmake/modules/MLIRConfig.cmake.in

23

I am now getting a build error here when building out-of-tree projects that use AddMLIR (in my case f18) and are provided an installed LLVM directory path (as opposed to a build directory path):

CMake Error at <my-llvm-install-dir>/lib/cmake/mlir/MLIRConfig.cmake:27 (include):
  include could not find load file:

    <my-llvm-install-dir>/lib/cmake/mlir/MLIRTargets.cmake
Call Stack (most recent call first):
  CMakeLists.txt:93 (find_package)

Providing an llvm build directory path to f18 cmake command, I do not have that issue (<my-llvm-build-dir>/lib/cmake/mlir/MLIRTargets.cmake always exists for me, but not <my-llvm-install-dir>/lib/cmake/mlir/MLIRTargets.cmake).

The issue seems to be that MLIRTargets.cmake is installed conditionally (depending on MLIR_HAS_EXPORTS), but it is expected to exists unconditionally here.

Would it make sens to make this line something like :

if (EXISTS "@MLIR_CONFIG_EXPORTS_FILE@")
  include("@MLIR_CONFIG_EXPORTS_FILE@")
endif()

or to always install MLIRTargets ?

[stephenneuendorffer]

In D76047#1931722, @jeanPerier wrote:

Hi, thanks for improving the way one can plugin MLIR ! While trying to rebase our f18 project that uses MLIR after this patch, I had a cmake issue using installed LLVMs out-of-tree, I tracked it to one of the line in this patch (see in-lined comment).

This code is partially copied from clang., and it looks like clang has the same issue, if CLANG_HAS_EXPORTS is not set. The big difference is that there is currently no mechanism to set MLIR_HAS_EXPORTS, so this will always get triggered. I think your EXISTS check is probably the best fix. I'll put together a patch.

[stephenneuendorffer]

In D76047#1932393, @stephenneuendorffer wrote:

In D76047#1931722, @jeanPerier wrote:

Hi, thanks for improving the way one can plugin MLIR ! While trying to rebase our f18 project that uses MLIR after this patch, I had a cmake issue using installed LLVMs out-of-tree, I tracked it to one of the line in this patch (see in-lined comment).

This code is partially copied from clang., and it looks like clang has the same issue, if CLANG_HAS_EXPORTS is not set. The big difference is that there is currently no mechanism to set MLIR_HAS_EXPORTS, so this will always get triggered. I think your EXISTS check is probably the best fix. I'll put together a patch.

I've checked in your suggested fix as a short-term workaround I'll need to revisit this longer to determine whether everything is getting exported/installed correctly.

[jeanPerier]

In D76047#1932607, @stephenneuendorffer wrote:

In D76047#1932393, @stephenneuendorffer wrote:

In D76047#1931722, @jeanPerier wrote:

Hi, thanks for improving the way one can plugin MLIR ! While trying to rebase our f18 project that uses MLIR after this patch, I had a cmake issue using installed LLVMs out-of-tree, I tracked it to one of the line in this patch (see in-lined comment).

This code is partially copied from clang., and it looks like clang has the same issue, if CLANG_HAS_EXPORTS is not set. The big difference is that there is currently no mechanism to set MLIR_HAS_EXPORTS, so this will always get triggered. I think your EXISTS check is probably the best fix. I'll put together a patch.

I've checked in your suggested fix as a short-term workaround I'll need to revisit this longer to determine whether everything is getting exported/installed correctly.

Thanks, that fixed the issue for me.

mlir/cmake/modules/CMakeLists.txt

 # Generate a list of CMake library targets so that other CMake projects can
 # link against them. LLVM calls its version of this file LLVMExports.cmake, but
 # the usual CMake convention seems to be ${Project}Targets.cmake.
-set(MLIR_INSTALL_PACKAGE_DIR lib${LLVM_LIBDIR_SUFFIX}/cmake/llvm)
+set(MLIR_INSTALL_PACKAGE_DIR lib${LLVM_LIBDIR_SUFFIX}/cmake/mlir)
 set(mlir_cmake_builddir "${CMAKE_BINARY_DIR}/${MLIR_INSTALL_PACKAGE_DIR}")
 
 # Keep this in sync with llvm/cmake/CMakeLists.txt!
 set(LLVM_INSTALL_PACKAGE_DIR lib${LLVM_LIBDIR_SUFFIX}/cmake/llvm)
 set(llvm_cmake_builddir "${LLVM_BINARY_DIR}/${LLVM_INSTALL_PACKAGE_DIR}")
 
 get_property(MLIR_EXPORTS GLOBAL PROPERTY MLIR_EXPORTS)
-export(TARGETS ${MLIR_EXPORTS} FILE ${mlir_cmake_builddir}/MlirTargets.cmake)
+export(TARGETS ${MLIR_EXPORTS} FILE ${mlir_cmake_builddir}/MLIRTargets.cmake)
+
+get_property(MLIR_ALL_LIBS GLOBAL PROPERTY MLIR_ALL_LIBS)
+get_property(MLIR_DIALECT_LIBS GLOBAL PROPERTY MLIR_DIALECT_LIBS)
+get_property(MLIR_CONVERSION_LIBS GLOBAL PROPERTY MLIR_CONVERSION_LIBS)
 
 # Generate MlirConfig.cmake for the build tree.
 set(MLIR_CONFIG_CMAKE_DIR "${mlir_cmake_builddir}")
 set(MLIR_CONFIG_LLVM_CMAKE_DIR "${llvm_cmake_builddir}")
-set(MLIR_CONFIG_EXPORTS_FILE "${mlir_cmake_builddir}/MlirTargets.cmake")
+set(MLIR_CONFIG_EXPORTS_FILE "\${MLIR_CMAKE_DIR}/MLIRTargets.cmake")
 set(MLIR_CONFIG_INCLUDE_DIRS
   "${MLIR_SOURCE_DIR}/include"
   "${MLIR_BINARY_DIR}/include"
   )
+configure_file(
+  ${CMAKE_CURRENT_SOURCE_DIR}/MLIRConfig.cmake.in
+  ${mlir_cmake_builddir}/MLIRConfig.cmake
+  @ONLY)
 set(MLIR_CONFIG_CMAKE_DIR)
 set(MLIR_CONFIG_LLVM_CMAKE_DIR)
 set(MLIR_CONFIG_EXPORTS_FILE)
+set(MLIR_CONFIG_INCLUDE_DIRS)
+
+# For compatibility with projects that include(MLIRConfig)
+# via CMAKE_MODULE_PATH, place API modules next to it.
+# This should be removed in the future.
+file(COPY .
+  DESTINATION ${mlir_cmake_builddir}
+  FILES_MATCHING PATTERN *.cmake
+  PATTERN CMakeFiles EXCLUDE
+  )
 
-# Generate MlirConfig.cmake for the install tree.
+# Generate MLIRConfig.cmake for the install tree.
 set(MLIR_CONFIG_CODE "
-# Compute the installation prefix from this LLVMConfig.cmake file location.
+# Compute the installation prefix from this MLIRConfig.cmake file location.
 get_filename_component(MLIR_INSTALL_PREFIX \"\${CMAKE_CURRENT_LIST_FILE}\" PATH)")
 # Construct the proper number of get_filename_component(... PATH)
 # calls to compute the installation prefix.
 string(REGEX REPLACE "/" ";" _count "${MLIR_INSTALL_PACKAGE_DIR}")
 foreach(p ${_count})
   set(MLIR_CONFIG_CODE "${MLIR_CONFIG_CODE}
 get_filename_component(MLIR_INSTALL_PREFIX \"\${MLIR_INSTALL_PREFIX}\" PATH)")
 endforeach(p)
 set(MLIR_CONFIG_CMAKE_DIR "\${MLIR_INSTALL_PREFIX}/${MLIR_INSTALL_PACKAGE_DIR}")
 set(MLIR_CONFIG_LLVM_CMAKE_DIR "\${MLIR_INSTALL_PREFIX}/${LLVM_INSTALL_PACKAGE_DIR}")
-set(MLIR_CONFIG_EXPORTS_FILE "\${MLIR_CMAKE_DIR}/MlirTargets.cmake")
+set(MLIR_CONFIG_EXPORTS_FILE "\${MLIR_CMAKE_DIR}/MLIRTargets.cmake")
 set(MLIR_CONFIG_INCLUDE_DIRS
   "\${MLIR_INSTALL_PREFIX}/include"
   )
+configure_file(
+  ${CMAKE_CURRENT_SOURCE_DIR}/MLIRConfig.cmake.in
+  ${CMAKE_CURRENT_BINARY_DIR}/CMakeFiles/MLIRConfig.cmake
+  @ONLY)
 set(MLIR_CONFIG_CODE)
 set(MLIR_CONFIG_CMAKE_DIR)
+set(MLIR_CONFIG_LLVM_CMAKE_DIR)
 set(MLIR_CONFIG_EXPORTS_FILE)
+set(MLIR_CONFIG_INCLUDE_DIRS)
 
 if (NOT LLVM_INSTALL_TOOLCHAIN_ONLY)
   # Not TOOLCHAIN ONLY, so install the MLIR parts as well
   # Include the cmake files so other tools can use mlir-tblgen, etc.
   get_property(mlir_has_exports GLOBAL PROPERTY MLIR_HAS_EXPORTS)
   if(mlir_has_exports)
-    install(EXPORT MlirTargets DESTINATION ${MLIR_INSTALL_PACKAGE_DIR}
+    install(EXPORT MLIRTargets DESTINATION ${MLIR_INSTALL_PACKAGE_DIR}
             COMPONENT mlir-cmake-exports)
   endif()
 
-  install(FILES #${CMAKE_CURRENT_BINARY_DIR}/CMakeFiles/MlirConfig.cmake
+  install(FILES
+    ${CMAKE_CURRENT_BINARY_DIR}/CMakeFiles/MLIRConfig.cmake
     ${CMAKE_CURRENT_SOURCE_DIR}/AddMLIR.cmake
     DESTINATION ${MLIR_INSTALL_PACKAGE_DIR}
     COMPONENT mlir-cmake-exports)
 
   if(NOT LLVM_ENABLE_IDE)
     # Add a dummy target so this can be used with LLVM_DISTRIBUTION_COMPONENTS
     add_custom_target(mlir-cmake-exports)
     add_llvm_install_targets(install-mlir-cmake-exports
                              COMPONENT mlir-cmake-exports)
   endif()
 endif()

mlir/cmake/modules/MLIRConfig.cmake.in

+# This file allows users to call find_package(MLIR) and pick up our targets.
+
+@MLIR_CONFIG_CODE@
+
+find_package(LLVM REQUIRED CONFIG
+             HINTS "@MLIR_CONFIG_LLVM_CMAKE_DIR@")
+
+set(MLIR_EXPORTED_TARGETS "@MLIR_EXPORTS@")
+set(MLIR_CMAKE_DIR "@MLIR_CONFIG_CMAKE_DIR@")
+set(MLIR_INCLUDE_DIRS "@MLIR_CONFIG_INCLUDE_DIRS@")
+set(MLIR_TABLEGEN_EXE "@MLIR_TABLEGEN_EXE@")
+
+# For mlir_tablegen()
+set(MLIR_INCLUDE_DIR "@MLIR_INCLUDE_DIR@")
+set(MLIR_MAIN_SRC_DIR "@MLIR_MAIN_SRC_DIR@")
+
+set_property(GLOBAL PROPERTY MLIR_ALL_LIBS "@MLIR_ALL_LIBS@")
+set_property(GLOBAL PROPERTY MLIR_DIALECT_LIBS "@MLIR_DIALECT_LIBS@")
+set_property(GLOBAL PROPERTY MLIR_CONVERSION_LIBS "@MLIR_CONVERSION_LIBS@")
+
+# Provide all our library targets to users.
+include("@MLIR_CONFIG_EXPORTS_FILE@")
+
+# By creating these targets here, subprojects that depend on MLIR's
+# tablegen-generated headers can always depend on these targets whether building
+# in-tree with MLIR or not.
+if(NOT TARGET mlir-tablegen-targets)
+  add_custom_target(mlir-tablegen-targets)
+endif()
+if(NOT TARGET mlir-headers)
+  add_custom_target(mlir-headers)
+endif()
+if(NOT TARGET mlir-doc)
+  add_custom_target(mlir-doc)
+endif()