[MLIR] Split off MLIRExecutionEngineUtils to fix libMLIR.so build (PR54242)

[nikic]

Summary

Building libMLIR.so currently fails with:

/usr/bin/ld: /tmp/ccNzulEA.ltrans39.ltrans.o: in function `(anonymous namespace)::SerializeToHsacoPass::optimizeLlvm(llvm::Module&, llvm::TargetMachine&)':
/builddir/build/BUILD/llvm-project-15.0.0.src/mlir/lib/Dialect/GPU/Transforms/SerializeToHsaco.cpp:328: undefined reference to `mlir::makeOptimizingTransformer(unsigned int, unsigned int, llvm::TargetMachine*)'

This is because MLIRGPUTransforms depends on MLIRExecutionEngine in https://github.com/llvm/llvm-project/blob/61bb2e4ea82fc5499a271d70d4537383d1942208/mlir/lib/Dialect/GPU/Transforms/SerializeToHsaco.cpp#L328, but MLIRExecutionEngine is marked as excluded from libMLIR.so.

However, this code doesn't require the full execution engine: It only performs middle-end optimization, and does not need any of the JIT/codegen infrastructure. As such, split off a separate library MLIRExecutionEngineUtils, which only contains that part and is not excluded from libMLIR.so.

Fixes https://github.com/llvm/llvm-project/issues/54242.

[mehdi_amini]

We should resolve the discussion in the tracking bug first I think: https://github.com/llvm/llvm-project/issues/54242#issuecomment-1063520117

[nikic]

Split off MLIRExecutionEngineUtils and only include that in libMLIR.so. This avoids embedding the JIT/MC/etc layer into libMLIR.so.

[nikic]

Does this general approach seem reasonable? I based this off what bazel does.

If we don't want to include this in libMLIR.so either, then I think the only alternative is to revert https://github.com/llvm/llvm-project/commit/bd22554af06e1f16dc9ff12eac8987f0ceebe8c1 due to the layering violation.

[krzysz00]

Thanks someone for having the bandwidth to factor makeOptimizingTransformer into a separate library so that "the utility function that lets us invoke LLVM" doesn't require pulling in the entire JIT.

[mehdi_amini]

LGTM, but please update the title and description.

mlir/lib/Dialect/GPU/CMakeLists.txt

add_mlir_dialect_library(MLIRGPUTransforms
   MLIRParallelLoopMapperAttrGen
   MLIRParallelLoopMapperEnumsGen
 
   LINK_LIBS PUBLIC
   MLIRAffineUtils
   MLIRArithmetic
   MLIRAsync
   MLIRDataLayoutInterfaces
+  MLIRExecutionEngineUtils
   MLIRGPUOps
   MLIRIR
   MLIRLLVMIR
   MLIRLLVMToLLVMIRTranslation
   MLIRMemRef
   MLIRPass
   MLIRSCF
   MLIRSupport

mlir/lib/ExecutionEngine/CMakeLists.txt

set(LLVM_JIT_LISTENER_LIB
       IntelJITEvents)
 endif(LLVM_USE_INTEL_JITEVENTS)
 
 if(LLVM_USE_PERF)
   set(LLVM_JIT_LISTENER_LIB
       PerfJITEvents)
 endif(LLVM_USE_PERF)
 
+# Use a separate library for OptUtils, to avoid pulling in the entire JIT and
+# codegen infrastructure. Unlike MLIRExecutionEngine, this is part of
+# libMLIR.so.
+add_mlir_library(MLIRExecutionEngineUtils
+  OptUtils.cpp
+
+  ADDITIONAL_HEADER_DIRS
+  ${MLIR_MAIN_INCLUDE_DIR}/mlir/ExecutionEngine
+
+  DEPENDS
+  intrinsics_gen
+
+  LINK_COMPONENTS
+  Analysis
+  Core
+  Coroutines
+  AggressiveInstCombine
+  InstCombine
+  ScalarOpts
+  Vectorize
+  TransformUtils
+  IPO
+  Passes
+  )
+
 add_mlir_library(MLIRExecutionEngine
   ExecutionEngine.cpp
-  OptUtils.cpp
 
   EXCLUDE_FROM_LIBMLIR
 
   ADDITIONAL_HEADER_DIRS
   ${MLIR_MAIN_INCLUDE_DIR}/mlir/ExecutionEngine
 
   DEPENDS
   intrinsics_gen
add_mlir_library(MLIRExecutionEngine
   Vectorize
   TransformUtils
   nativecodegen
   IPO
   Passes
   ${LLVM_JIT_LISTENER_LIB}
 
   LINK_LIBS PUBLIC
+  MLIRExecutionEngineUtils
   MLIRLLVMIR
   MLIRLLVMToLLVMIRTranslation
   MLIRTargetLLVMIRExport
   )
 
 get_property(dialect_libs GLOBAL PROPERTY MLIR_DIALECT_LIBS)
 add_mlir_library(MLIRJitRunner
   JitRunner.cpp