[mlir] Allow out-of-tree python building from installed MLIR.

[stellaraccident]

Summary

• Depends on D111504, which provides the boilerplate for building aggregate shared libraries from installed MLIR.
• Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled).
• Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing.
• Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API.
• Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan).
• Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time).
• Need to validate against a downstream or two and adjust, prior to submitting.

Downstreams will need to adapt by:

• Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory.
• To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension.

[stephenneuendorffer]

mlir/examples/standalone/CMakeLists.txt

26–30

So in my build (and I'm not quite sure why), this fails to find Python3, even though it's found at the toplevel.

mlir/examples/standalone/test/python/lit.local.cfg

2

I think the python tests need to be disabled if python is not found.

[stellaraccident]

mlir/examples/standalone/CMakeLists.txt

26–30

Likely cmake version dependent. Once Mike lands these fixes in the main project, I'd like to put them in a macro for all to use. Consider that these bits will be replaced before this patch is ready to land.

[stephenneuendorffer]

mlir/examples/standalone/CMakeLists.txt

39

I'm a little concerned about the TARGET check here. seems like this would be better off relying on an explicit configuration variable like MLIR_ENABLE_PYTHON_BINDINGS exported from MLIRConfig.cmake?

[stellaraccident]

mlir/examples/standalone/CMakeLists.txt

39

Agreed. Let me rework the entire standalone top level cmake file. I honestly forgot about it before sending the patch (and had just worked on it early to get everything building).

[mikeurbach]

mlir/examples/standalone/CMakeLists.txt

26–30

I think I'm seeing the same thing actually, which we also saw in the failures on this PR: https://github.com/llvm/torch-mlir/pull/357. I didn't see this when having MLIR and the downstream (CIRCT in my case) look for Development, but I do when looking for Development.Module. I noticed this on my laptop when I tried adopting the add_mlir_python_extension part of this patch and removed MLIR_BINDINGS_PYTHON_LOCK_VERSION.

In the container I'm testing for building manylinux wheels, I have no problem if MLIR and CIRCT both look for Development.Module.

[mikeurbach]

mlir/examples/standalone/CMakeLists.txt

26–30

I realized in the CIRCT project, if I remove REQUIRED from the find_package command, it doesn't fail, and the CMake command succeeds in general. It does say: -- Could NOT find Python3 (missing: Development.Module) (found suitable version "3.8.8", minimum required is "3.6") in the CIRCT CMake output, but I can still build the extension just fine.

[mikeurbach]

mlir/examples/standalone/CMakeLists.txt

26–30

Just to close the loop on this, I found what I was doing wrong. In my local setup, another external project other than CIRCT was searching for Development while both MLIR and CIRCT searched for Development.Module. I would have expected this to work, but in my CMake debug output, I see first MLIR finds Development.Module, then my other project finds Development, then CIRCT fails to find Development.Module. Perhaps there is a bug in CMake's FindPython caching or something. Regardless, as long as all project agree to find Development.Module, which is what I think we always want, there shouldn't be a problem. Does that help your situation @stephenneuendorffer?

[stellaraccident]

Use MLIR_ENABLE_BINDINGS_PYTHON to gate python config.

[stellaraccident]

mlir/examples/standalone/CMakeLists.txt

26–30

In https://reviews.llvm.org/D111689 I converted all of this in the main project to a macro and then just use that here. Should be exactly the same now.

39

I updated to export the MLIR_ENABLE_BINDINGS_PYTHON var in the MLIR installation and then use that here and elsewhere to gate. Downstreams can have their own property if they want but we should have the option of knowing whether MLIR was built with this support.

[stellaraccident]

Rebase.

[stellaraccident]

Rebase and fix test.

[stellaraccident]

Update test check.

[mehdi_amini]

Reverter, build was broken with:

CMake Error at /usr/share/cmake-3.20/Modules/FindPython/Support.cmake:3315 (add_library):
  Cannot find source file:
    /vol/worker/mlir-nvidia/mlir-nvidia/llvm.src/mlir/python/vol/worker/mlir-nvidia/mlir-nvidia/llvm.src/mlir/test/python/lib/PythonTestModule.cpp

https://lab.llvm.org/buildbot/#/builders/61/builds/17448

mlir/cmake/modules/AddMLIRPython.cmake

cmake_parse_arguments(ARG
     ""
     "ROOT_DIR;ADD_TO_PARENT;DEST_PREFIX"
     "SOURCES;SOURCES_GLOB"
     ${ARGN})
 
   if(NOT ARG_ROOT_DIR)
     set(ARG_ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}")
   endif()
+  set(_install_destination "src/python/${name}")
 
   # Process the glob.
   set(_glob_sources)
   if(ARG_SOURCES_GLOB)
     set(_glob_spec ${ARG_SOURCES_GLOB})
     list(TRANSFORM _glob_spec PREPEND "${ARG_ROOT_DIR}/")
     file(GLOB_RECURSE _glob_sources
       RELATIVE "${ARG_ROOT_DIR}"
       ${_glob_spec}
     )
     list(APPEND ARG_SOURCES ${_glob_sources})
   endif()
 
   # We create a custom target to carry properties and dependencies for
   # generated sources.
-  add_custom_target(${name})
+  add_library(${name} INTERFACE)
   set(_file_depends "${ARG_SOURCES}")
   list(TRANSFORM _file_depends PREPEND "${ARG_ROOT_DIR}/")
   set_target_properties(${name} PROPERTIES
-    PYTHON_SOURCES_TYPE pure
-    PYTHON_ROOT_DIR "${ARG_ROOT_DIR}"
-    PYTHON_DEST_PREFIX "${ARG_DEST_PREFIX}"
-    PYTHON_SOURCES "${ARG_SOURCES}"
-    PYTHON_FILE_DEPENDS "${_file_depends}"
-    PYTHON_DEPENDS ""
+    # Yes: Leading-lowercase property names are load bearing and the recommended
+    # way to do this: https://gitlab.kitware.com/cmake/cmake/-/issues/19261
+    # Note that ROOT_DIR and FILE_DEPENDS are not exported because they are
+    # only relevant to in-tree uses.
+    EXPORT_PROPERTIES "mlir_python_SOURCES_TYPE;mlir_python_DEST_PREFIX;mlir_python_DEST_PREFIX;mlir_python_SOURCES;mlir_python_DEPENDS"
+    mlir_python_SOURCES_TYPE pure
+    mlir_python_ROOT_DIR "${ARG_ROOT_DIR}"
+    mlir_python_DEST_PREFIX "${ARG_DEST_PREFIX}"
+    mlir_python_SOURCES "${ARG_SOURCES}"
+    mlir_python_FILE_DEPENDS "${_file_depends}"
+    mlir_python_DEPENDS ""
+  )
+  # Note that an "include" directory has no meaning to such faux targets,
+  # but it is a CMake supported way to specify a directory search list in a
+  # way that works both in-tree and out. It has some super powers which are
+  # not possible to emulate with custom properties (because of the prohibition
+  # on using generator expressions in exported custom properties and the
+  # special dispensation for $<INSTALL_PREFIX>).
+  target_include_directories(${name} INTERFACE
+    "$<BUILD_INTERFACE:${ARG_ROOT_DIR}>"
+    "$<INSTALL_INTERFACE:${_install_destination}>"
   )
 
   # Add to parent.
   if(ARG_ADD_TO_PARENT)
-    set_property(TARGET ${ARG_ADD_TO_PARENT} APPEND PROPERTY PYTHON_DEPENDS ${name})
+    set_property(TARGET ${ARG_ADD_TO_PARENT} APPEND PROPERTY mlir_python_DEPENDS ${name})
+  endif()
+
+  # Install.
+  if(NOT LLVM_INSTALL_TOOLCHAIN_ONLY)
+    _mlir_python_install_sources(
+      ${name} "${ARG_ROOT_DIR}" "${_install_destination}"
+      ${ARG_SOURCES}
+    )
   endif()
 endfunction()
 
 # Function: declare_mlir_python_extension
 # Declares a buildable python extension from C++ source files. The built
 # module is considered a python source file and included as everything else.
 # Arguments:
+#   ROOT_DIR: Root directory where sources are interpreted relative to.
+#     Defaults to CMAKE_CURRENT_SOURCE_DIR.
 #   MODULE_NAME: Local import name of the module (i.e. "_mlir").
 #   ADD_TO_PARENT: Same as for declare_mlir_python_sources.
 #   SOURCES: C++ sources making up the module.
 #   PRIVATE_LINK_LIBS: List of libraries to link in privately to the module
 #     regardless of how it is included in the project (generally should be
 #     static libraries that can be included with hidden visibility).
 #   EMBED_CAPI_LINK_LIBS: Dependent CAPI libraries that this extension depends
 #     on. These will be collected for all extensions and put into an
 #     aggregate dylib that is linked against.
 function(declare_mlir_python_extension name)
   cmake_parse_arguments(ARG
     ""
-    "MODULE_NAME;ADD_TO_PARENT"
+    "ROOT_DIR;MODULE_NAME;ADD_TO_PARENT"
     "SOURCES;PRIVATE_LINK_LIBS;EMBED_CAPI_LINK_LIBS"
     ${ARGN})
 
-  add_custom_target(${name})
+  if(NOT ARG_ROOT_DIR)
+    set(ARG_ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}")
+  endif()
+  set(_install_destination "src/python/${name}")
+
+  add_library(${name} INTERFACE)
   set_target_properties(${name} PROPERTIES
-    PYTHON_SOURCES_TYPE extension
-    PYTHON_EXTENSION_MODULE_NAME "${ARG_MODULE_NAME}"
-    PYTHON_CPP_SOURCES "${ARG_SOURCES}"
-    PYTHON_PRIVATE_LINK_LIBS "${ARG_PRIVATE_LINK_LIBS}"
-    PYTHON_EMBED_CAPI_LINK_LIBS "${ARG_EMBED_CAPI_LINK_LIBS}"
-    PYTHON_FILE_DEPENDS ""
-    PYTHON_DEPENDS ""
+    # Yes: Leading-lowercase property names are load bearing and the recommended
+    # way to do this: https://gitlab.kitware.com/cmake/cmake/-/issues/19261
+    # Note that ROOT_DIR and FILE_DEPENDS are not exported because they are
+    # only relevant to in-tree uses.
+    EXPORT_PROPERTIES "mlir_python_SOURCES_TYPE;mlir_python_EXTENSION_MODULE_NAME;mlir_python_CPP_SOURCES;mlir_python_PRIVATE_LINK_LIBS;mlir_python_EMBED_CAPI_LINK_LIBS;mlir_python_DEPENDS"
+    mlir_python_SOURCES_TYPE extension
+    mlir_python_ROOT_DIR "${ARG_ROOT_DIR}"
+    mlir_python_EXTENSION_MODULE_NAME "${ARG_MODULE_NAME}"
+    mlir_python_CPP_SOURCES "${ARG_SOURCES}"
+    mlir_python_PRIVATE_LINK_LIBS "${ARG_PRIVATE_LINK_LIBS}"
+    mlir_python_EMBED_CAPI_LINK_LIBS "${ARG_EMBED_CAPI_LINK_LIBS}"
+    mlir_python_FILE_DEPENDS ""
+    mlir_python_DEPENDS ""
+  )
+  # Note that an "include" directory has no meaning to such faux targets,
+  # but it is a CMake supported way to specify an install-prefix relative
+  # directory. It has some super powers which are not possible to emulate
+  # with custom properties (because of the prohibition on using generator
+  # expressions in exported custom properties and the special dispensation
+  # for $<INSTALL_PREFIX> and $<INSTALL_INTERFACE>). On imported targets,
+  # this is used as a single value, not as a list, so it must only have one
+  # item in it.
+  target_include_directories(${name} INTERFACE
+    "$<INSTALL_INTERFACE:${_install_destination}>"
   )
 
   # Add to parent.
   if(ARG_ADD_TO_PARENT)
-    set_property(TARGET ${ARG_ADD_TO_PARENT} APPEND PROPERTY PYTHON_DEPENDS ${name})
+    set_property(TARGET ${ARG_ADD_TO_PARENT} APPEND PROPERTY mlir_python_DEPENDS ${name})
+  endif()
+
+  # Install.
+  if(NOT LLVM_INSTALL_TOOLCHAIN_ONLY)
+    _mlir_python_install_sources(
+      ${name} "${ARG_ROOT_DIR}" "src/python/${name}"
+      ${ARG_SOURCES}
+    )
   endif()
 endfunction()
 
+function(_mlir_python_install_sources name source_root_dir destination)
+  foreach(source_relative_path ${ARGN})
+    # Transform "a/b/c.py" -> "${install_prefix}/a/b" for installation.
+    get_filename_component(
+      dest_relative_path "${source_relative_path}" DIRECTORY
+      BASE_DIR "${source_root_dir}"
+    )
+    install(
+      FILES "${source_root_dir}/${source_relative_path}"
+      DESTINATION "${destination}/${dest_relative_path}"
+      COMPONENT "${name}"
+    )
+  endforeach()
+  get_target_export_arg(${name} MLIR export_to_mlirtargets UMBRELLA mlir-libraries)
+  install(TARGETS ${name}
+    COMPONENT ${name}
+    ${export_to_mlirtargets}
+  )
+endfunction()
+
 # Function: add_mlir_python_modules
 # Adds python modules to a project, building them from a list of declared
 # source groupings (see declare_mlir_python_sources and
 # declare_mlir_python_extension). One of these must be called for each
 # packaging root in use.
 # Arguments:
 #   ROOT_PREFIX: The directory in the build tree to emit sources. This will
 #     typically be something like ${MY_BINARY_DIR}/python_packages/foobar
 #     for non-relocatable modules or a deeper directory tree for relocatable.
 #   INSTALL_PREFIX: Prefix into the install tree for installing the package.
 #     Typically mirrors the path above but without an absolute path.
 #   DECLARED_SOURCES: List of declared source groups to include. The entire
 #     DAG of source modules is included.
 #   COMMON_CAPI_LINK_LIBS: List of dylibs (typically one) to make every
 #     extension depend on (see mlir_python_add_common_capi_library).
 function(add_mlir_python_modules name)
   cmake_parse_arguments(ARG
     ""
     "ROOT_PREFIX;INSTALL_PREFIX;COMMON_CAPI_LINK_LIBS"
     "DECLARED_SOURCES"
     ${ARGN})
   # Helper to process an individual target.
   function(_process_target modules_target sources_target)
-    get_target_property(_source_type ${sources_target} PYTHON_SOURCES_TYPE)
+    get_target_property(_source_type ${sources_target} mlir_python_SOURCES_TYPE)
+
+    # The root directory differs based on whether it is IMPORTED (installed
+    # dep).
+    get_target_property(_is_imported ${sources_target} IMPORTED)
+    if(NOT _is_imported)
+      # In-tree.
+      get_target_property(_python_root_dir ${sources_target} mlir_python_ROOT_DIR)
+    else()
+      # Imported.
+      # Note: We only populate a single directory in
+      # INTERFACE_INCLUDE_DIRECTORIES, so we can get away with just using it
+      # as a single value.
+      get_target_property(_python_root_dir ${sources_target} INTERFACE_INCLUDE_DIRECTORIES)
+    endif()
+
     if(_source_type STREQUAL "pure")
       # Pure python sources to link into the tree.
-      get_target_property(_python_root_dir ${sources_target} PYTHON_ROOT_DIR)
-      get_target_property(_python_sources ${sources_target} PYTHON_SOURCES)
-      get_target_property(_specified_dest_prefix ${sources_target} PYTHON_DEST_PREFIX)
+      get_target_property(_python_sources ${sources_target} mlir_python_SOURCES)
+      get_target_property(_specified_dest_prefix ${sources_target} mlir_python_DEST_PREFIX)
       foreach(_source_relative_path ${_python_sources})
         set(_dest_relative_path "${_source_relative_path}")
         if(_specified_dest_prefix)
           set(_dest_relative_path "${_specified_dest_prefix}/${_dest_relative_path}")
         endif()
         set(_src_path "${_python_root_dir}/${_source_relative_path}")
         set(_dest_path "${ARG_ROOT_PREFIX}/${_dest_relative_path}")
 
if(_source_type STREQUAL "pure")
         install(
           FILES "${_src_path}"
           DESTINATION "${_install_path}"
           COMPONENT ${modules_target}
         )
       endforeach()
     elseif(_source_type STREQUAL "extension")
       # Native CPP extension.
-      get_target_property(_module_name ${sources_target} PYTHON_EXTENSION_MODULE_NAME)
-      get_target_property(_cpp_sources ${sources_target} PYTHON_CPP_SOURCES)
-      get_target_property(_private_link_libs ${sources_target} PYTHON_PRIVATE_LINK_LIBS)
+      get_target_property(_module_name ${sources_target} mlir_python_EXTENSION_MODULE_NAME)
+      get_target_property(_cpp_sources ${sources_target} mlir_python_CPP_SOURCES)
+      get_target_property(_private_link_libs ${sources_target} mlir_python_PRIVATE_LINK_LIBS)
+      # Transform relative source to based on root dir.
+      list(TRANSFORM _cpp_sources PREPEND "${_python_root_dir}/")
       set(_extension_target "${name}.extension.${_module_name}.dso")
       add_mlir_python_extension(${_extension_target} "${_module_name}"
         INSTALL_COMPONENT ${modules_target}
         INSTALL_DIR "${ARG_INSTALL_PREFIX}/_mlir_libs"
         OUTPUT_DIRECTORY "${ARG_ROOT_PREFIX}/_mlir_libs"
         SOURCES ${_cpp_sources}
         LINK_LIBS PRIVATE
           ${_private_link_libs}
           ${ARG_COMMON_CAPI_LINK_LIBS}
       )
       add_dependencies(${name} ${_extension_target})
       mlir_python_setup_extension_rpath(${_extension_target})
     else()
       message(SEND_ERROR "Unrecognized source type '${_source_type}' for python source target ${sources_target}")
       return()
     endif()
   endfunction()
 
   _flatten_mlir_python_targets(_flat_targets ${ARG_DECLARED_SOURCES})
   # Collect dependencies.
   set(_depends)
   foreach(sources_target ${_flat_targets})
-    get_target_property(_local_depends ${sources_target} PYTHON_FILE_DEPENDS)
+    get_target_property(_local_depends ${sources_target} mlir_python_FILE_DEPENDS)
+    if(_local_depends)
       list(APPEND _depends ${_local_depends})
+    endif()
   endforeach()
 
   # Build the modules target.
   add_custom_target(${name} ALL DEPENDS ${_depends})
   foreach(sources_target ${_flat_targets})
     _process_target(${name} ${sources_target})
   endforeach()
 
cmake_parse_arguments(ARG
     ""
     "INSTALL_COMPONENT;INSTALL_DESTINATION;OUTPUT_DIRECTORY;RELATIVE_INSTALL_ROOT"
     "DECLARED_SOURCES;EMBED_LIBS"
     ${ARGN})
   # Collect all explicit and transitive embed libs.
   set(_embed_libs ${ARG_EMBED_LIBS})
   _flatten_mlir_python_targets(_all_source_targets ${ARG_DECLARED_SOURCES})
   foreach(t ${_all_source_targets})
-    get_target_property(_local_embed_libs ${t} PYTHON_EMBED_CAPI_LINK_LIBS)
+    get_target_property(_local_embed_libs ${t} mlir_python_EMBED_CAPI_LINK_LIBS)
     if(_local_embed_libs)
       list(APPEND _embed_libs ${_local_embed_libs})
     endif()
   endforeach()
   list(REMOVE_DUPLICATES _embed_libs)
 
   # Generate the aggregate .so that everything depends on.
   add_mlir_aggregate(${name}
install(TARGETS ${name}
     RUNTIME DESTINATION "${ARG_INSTALL_DESTINATION}"
   )
 
 endfunction()
 
 function(_flatten_mlir_python_targets output_var)
   set(_flattened)
   foreach(t ${ARGN})
-    get_target_property(_source_type ${t} PYTHON_SOURCES_TYPE)
-    get_target_property(_depends ${t} PYTHON_DEPENDS)
+    get_target_property(_source_type ${t} mlir_python_SOURCES_TYPE)
+    get_target_property(_depends ${t} mlir_python_DEPENDS)
     if(_source_type)
       list(APPEND _flattened "${t}")
       if(_depends)
         _flatten_mlir_python_targets(_local_flattened ${_depends})
         list(APPEND _flattened ${_local_flattened})
       endif()
     endif()
   endforeach()

mlir/cmake/modules/MLIRConfig.cmake.in

 # This file allows users to call find_package(MLIR) and pick up our targets.
 
 @MLIR_CONFIG_CODE@
 
 find_package(LLVM REQUIRED CONFIG
              HINTS "@MLIR_CONFIG_LLVM_CMAKE_DIR@")
 
 set(MLIR_EXPORTED_TARGETS "@MLIR_EXPORTS@")
 set(MLIR_CMAKE_DIR "@MLIR_CONFIG_CMAKE_DIR@")
 set(MLIR_INCLUDE_DIRS "@MLIR_CONFIG_INCLUDE_DIRS@")
 set(MLIR_TABLEGEN_EXE "@MLIR_TABLEGEN_EXE@")
 set(MLIR_INSTALL_AGGREGATE_OBJECTS "@MLIR_INSTALL_AGGREGATE_OBJECTS@")
+set(MLIR_ENABLE_BINDINGS_PYTHON "@MLIR_ENABLE_BINDINGS_PYTHON@")
 
 # For mlir_tablegen()
 set(MLIR_INCLUDE_DIR "@MLIR_INCLUDE_DIR@")
 set(MLIR_MAIN_SRC_DIR "@MLIR_MAIN_SRC_DIR@")
 
 set_property(GLOBAL PROPERTY MLIR_ALL_LIBS "@MLIR_ALL_LIBS@")
 set_property(GLOBAL PROPERTY MLIR_DIALECT_LIBS "@MLIR_DIALECT_LIBS@")
 set_property(GLOBAL PROPERTY MLIR_CONVERSION_LIBS "@MLIR_CONVERSION_LIBS@")

mlir/examples/standalone/CMakeLists.txt

 set(MLIR_BINARY_DIR ${CMAKE_BINARY_DIR})
 
 list(APPEND CMAKE_MODULE_PATH "${MLIR_CMAKE_DIR}")
 list(APPEND CMAKE_MODULE_PATH "${LLVM_CMAKE_DIR}")
 include(TableGen)
 include(AddLLVM)
 include(AddMLIR)
 include(HandleLLVMOptions)
+include(MLIRDetectPythonEnv)
+
+if(MLIR_ENABLE_BINDINGS_PYTHON)
+  include(MLIRDetectPythonEnv)
+  mlir_configure_python_dev_packages()
+endif()
 
 include_directories(${LLVM_INCLUDE_DIRS})
 include_directories(${MLIR_INCLUDE_DIRS})
 include_directories(${PROJECT_SOURCE_DIR}/include)
 include_directories(${PROJECT_BINARY_DIR}/include)
 link_directories(${LLVM_BUILD_LIBRARY_DIR})
 add_definitions(${LLVM_DEFINITIONS})
 
 add_subdirectory(include)
 add_subdirectory(lib)
+if(MLIR_ENABLE_BINDINGS_PYTHON)
+  message(STATUS "Enabling Python API")
+  add_subdirectory(python)
+endif()
 add_subdirectory(test)
 add_subdirectory(standalone-opt)
 add_subdirectory(standalone-translate)

mlir/examples/standalone/include/Standalone/StandaloneOps.td

 //===- StandaloneOps.td - Standalone dialect ops -----------*- tablegen -*-===//
 //
 // This file is licensed under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef STANDALONE_OPS
 #define STANDALONE_OPS
 
-include "StandaloneDialect.td"
+include "Standalone/StandaloneDialect.td"
 include "mlir/Interfaces/SideEffectInterfaces.td"
 
 def Standalone_FooOp : Standalone_Op<"foo", [NoSideEffect,
                                              SameOperandsAndResultType]> {
     let summary = "Illustrates how to define an operation.";
     let description = [{
         The `standalone.foo` operation illustrates how to define a new
         operation in a dialect. It uses an operation trait to declare that it

mlir/examples/standalone/python/CMakeLists.txt

+include(AddMLIRPython)
+
+# Specifies that all MLIR packages are co-located under the `mlir_standalone`
+# top level package (the API has been embedded in a relocatable way).
+# TODO: Add an upstream cmake param for this vs having a global here.
+add_compile_definitions("MLIR_PYTHON_PACKAGE_PREFIX=mlir_standalone.")
+
+
+################################################################################
+# Sources
+################################################################################
+
+declare_mlir_python_sources(StandalonePythonSources)
+
+declare_mlir_dialect_python_bindings(
+  ADD_TO_PARENT StandalonePythonSources
+  ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir_standalone"
+  TD_FILE dialects/StandaloneOps.td
+  SOURCES
+    dialects/standalone.py
+  DIALECT_NAME standalone)
+
+declare_mlir_python_extension(StandalonePythonSources.Extension
+  MODULE_NAME _standaloneDialects
+  ADD_TO_PARENT StandalonePythonSources
+  SOURCES
+    StandaloneExtension.cpp
+  EMBED_CAPI_LINK_LIBS
+    StandaloneCAPI
+)
+
+################################################################################
+# Common CAPI
+################################################################################
+
+add_mlir_python_common_capi_library(StandalonePythonCAPI
+  INSTALL_COMPONENT StandalonePythonModules
+  INSTALL_DESTINATION python_packages/standalone/mlir_standalone/_mlir_libs
+  OUTPUT_DIRECTORY "${MLIR_BINARY_DIR}/python_packages/standalone/mlir_standalone/_mlir_libs"
+  RELATIVE_INSTALL_ROOT "../../../.."
+  DECLARED_SOURCES
+    StandalonePythonSources
+    MLIRPythonSources.Core
+)
+
+################################################################################
+# Instantiation of all Python modules
+################################################################################
+
+add_mlir_python_modules(StandalonePythonModules
+  ROOT_PREFIX "${MLIR_BINARY_DIR}/python_packages/standalone/mlir_standalone"
+  INSTALL_PREFIX "python_packages/standalone/mlir_standalone"
+  DECLARED_SOURCES
+    StandalonePythonSources
+    MLIRPythonSources
+  COMMON_CAPI_LINK_LIBS
+    StandalonePythonCAPI
+  )

mlir/examples/standalone/python/StandaloneExtension.cpp

+//===- StandaloneExtension.cpp - Extension module -------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "Standalone-c/Dialects.h"
+#include "mlir/Bindings/Python/PybindAdaptors.h"
+
+namespace py = pybind11;
+using namespace mlir::python::adaptors;
+
+PYBIND11_MODULE(_standaloneDialects, m) {
+  //===--------------------------------------------------------------------===//
+  // standalone dialect
+  //===--------------------------------------------------------------------===//
+  auto standalone_m = m.def_submodule("standalone");
+
+  standalone_m.def(
+      "register_dialect",
+      [](MlirContext context, bool load) {
+        MlirDialectHandle handle = mlirGetDialectHandle__standalone__();
+        mlirDialectHandleRegisterDialect(handle, context);
+        if (load) {
+          mlirDialectHandleLoadDialect(handle, context);
+        }
+      },
+      py::arg("context") = py::none(), py::arg("load") = true);
+}

mlir/examples/standalone/python/mlir_standalone/dialects/StandaloneOps.td

+//===-- StandaloneOps.td - Python bindings for standalone --*- tablegen -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===---------------------------------------------------------------------===//
+
+#ifndef PYTHON_BINDINGS_STANDALONE_OPS
+#define PYTHON_BINDINGS_STANDALONE_OPS
+
+include "mlir/Bindings/Python/Attributes.td"
+include "Standalone/StandaloneOps.td"
+
+#endif

mlir/examples/standalone/python/mlir_standalone/dialects/standalone.py

+#  Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+#  See https://llvm.org/LICENSE.txt for license information.
+#  SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+from ._standalone_ops_gen import *
+from .._mlir_libs._standaloneDialects.standalone import *

mlir/examples/standalone/test/CMakeLists.txt

+llvm_canonicalize_cmake_booleans(
+        MLIR_ENABLE_BINDINGS_PYTHON
+)
+
 configure_lit_site_cfg(
         ${CMAKE_CURRENT_SOURCE_DIR}/lit.site.cfg.py.in
         ${CMAKE_CURRENT_BINARY_DIR}/lit.site.cfg.py
         MAIN_CONFIG
         ${CMAKE_CURRENT_SOURCE_DIR}/lit.cfg.py
 )
 
 set(STANDALONE_TEST_DEPENDS

mlir/examples/standalone/test/lit.cfg.py

 
 # Tweak the PATH to include the tools dir.
 llvm_config.with_environment('PATH', config.llvm_tools_dir, append_path=True)
 
 tool_dirs = [config.standalone_tools_dir, config.llvm_tools_dir]
 tools = [
     'standalone-capi-test',
     'standalone-opt',
-    'standalone-translate'
+    'standalone-translate',
+    ToolSubst('%PYTHON', config.python_executable, unresolved='ignore'),
 ]
 
 llvm_config.add_tool_substitutions(tools, tool_dirs)
+
+llvm_config.with_environment('PYTHONPATH', [
+    os.path.join(config.mlir_binary_dir, 'python_packages', 'standalone'),
+], append_path=True)

mlir/examples/standalone/test/lit.site.cfg.py.in

 @LIT_SITE_CFG_IN_HEADER@
 
 import sys
 
 config.host_triple = "@LLVM_HOST_TRIPLE@"
 config.target_triple = "@TARGET_TRIPLE@"
 config.llvm_src_root = "@LLVM_SOURCE_DIR@"
 config.llvm_obj_root = "@LLVM_BINARY_DIR@"
 config.llvm_tools_dir = "@LLVM_TOOLS_DIR@"
 config.llvm_lib_dir = "@LLVM_LIBS_DIR@"
 config.llvm_shlib_dir = "@SHLIBDIR@"
 config.llvm_shlib_ext = "@SHLIBEXT@"
 config.llvm_exe_ext = "@EXEEXT@"
 config.lit_tools_dir = "@LLVM_LIT_TOOLS_DIR@"
-config.python_executable = "@PYTHON_EXECUTABLE@"
+config.mlir_binary_dir = "@MLIR_BINARY_DIR@"
+config.python_executable = "@Python3_EXECUTABLE@"
+config.enable_bindings_python = @MLIR_ENABLE_BINDINGS_PYTHON@
 config.gold_executable = "@GOLD_EXECUTABLE@"
 config.ld64_executable = "@LD64_EXECUTABLE@"
 config.enable_shared = @ENABLE_SHARED@
 config.enable_assertions = @ENABLE_ASSERTIONS@
 config.targets_to_build = "@TARGETS_TO_BUILD@"
 config.native_target = "@LLVM_NATIVE_ARCH@"
 config.llvm_bindings = "@LLVM_BINDINGS@".split(' ')
 config.host_os = "@HOST_OS@"

mlir/examples/standalone/test/python/lit.local.cfg

+config.suffixes.add('.py')
+
+if not config.enable_bindings_python:
+  config.unsupported = True

mlir/examples/standalone/test/python/smoketest.py

+# RUN: %PYTHON %s | FileCheck %s
+
+from mlir_standalone.ir import *
+from mlir_standalone.dialects import (
+  builtin as builtin_d,
+  standalone as standalone_d
+)
+
+with Context():
+  standalone_d.register_dialect()
+  module = Module.parse("""
+    %0 = constant 2 : i32
+    %1 = standalone.foo %0 : i32
+    """)
+  # CHECK: %[[C:.*]] = constant 2 : i32
+  # CHECK: standalone.foo %[[C]] : i32
+  print(str(module))

mlir/python/CMakeLists.txt

 # just reference them here instead of having ordered, cross package target
 # dependencies.
 ################################################################################
 
 set(PYTHON_SOURCE_DIR "${MLIR_SOURCE_DIR}/lib/Bindings/Python")
 declare_mlir_python_extension(MLIRPythonExtension.Core
   MODULE_NAME _mlir
   ADD_TO_PARENT MLIRPythonSources.Core
+  ROOT_DIR "${PYTHON_SOURCE_DIR}"
   SOURCES
-    ${PYTHON_SOURCE_DIR}/DialectLinalg.cpp  # TODO: Break this out.
-    ${PYTHON_SOURCE_DIR}/DialectSparseTensor.cpp  # TODO: Break this out.
-    ${PYTHON_SOURCE_DIR}/MainModule.cpp
-    ${PYTHON_SOURCE_DIR}/IRAffine.cpp
-    ${PYTHON_SOURCE_DIR}/IRAttributes.cpp
-    ${PYTHON_SOURCE_DIR}/IRCore.cpp
-    ${PYTHON_SOURCE_DIR}/IRInterfaces.cpp
-    ${PYTHON_SOURCE_DIR}/IRModule.cpp
-    ${PYTHON_SOURCE_DIR}/IRTypes.cpp
-    ${PYTHON_SOURCE_DIR}/PybindUtils.cpp
-    ${PYTHON_SOURCE_DIR}/Pass.cpp
+    DialectLinalg.cpp  # TODO: Break this out.
+    DialectSparseTensor.cpp  # TODO: Break this out.
+    MainModule.cpp
+    IRAffine.cpp
+    IRAttributes.cpp
+    IRCore.cpp
+    IRInterfaces.cpp
+    IRModule.cpp
+    IRTypes.cpp
+    PybindUtils.cpp
+    Pass.cpp
+
+    # Headers must be included explicitly so they are installed.
+    Dialects.h
+    Globals.h
+    IRModule.h
+    Pass.h
+    PybindUtils.h
   PRIVATE_LINK_LIBS
     LLVMSupport
   EMBED_CAPI_LINK_LIBS
     MLIRCAPIDebug
     MLIRCAPIIR
     MLIRCAPIInterfaces
     MLIRCAPIRegistration  # TODO: See about dis-aggregating
 
     # Dialects
     MLIRCAPILinalg  # TODO: Remove when above is removed.
     MLIRCAPISparseTensor  # TODO: Remove when above is removed.
     MLIRCAPIStandard
 )
 
 declare_mlir_python_extension(MLIRPythonExtension.AllPassesRegistration
   MODULE_NAME _mlirAllPassesRegistration
+  ROOT_DIR "${PYTHON_SOURCE_DIR}"
   SOURCES
-    ${PYTHON_SOURCE_DIR}/AllPassesRegistration.cpp
+    AllPassesRegistration.cpp
   PRIVATE_LINK_LIBS
     LLVMSupport
   EMBED_CAPI_LINK_LIBS
     MLIRCAPIConversion
     MLIRCAPITransforms
 )
 
 declare_mlir_python_extension(MLIRPythonExtension.AsyncDialectPasses
   MODULE_NAME _mlirAsyncPasses
   ADD_TO_PARENT MLIRPythonSources.Dialects.async_dialect
+  ROOT_DIR "${PYTHON_SOURCE_DIR}"
   SOURCES
-    ${PYTHON_SOURCE_DIR}/AsyncPasses.cpp
+    AsyncPasses.cpp
   PRIVATE_LINK_LIBS
     LLVMSupport
   EMBED_CAPI_LINK_LIBS
     MLIRCAPIAsync
 )
 
 declare_mlir_python_extension(MLIRPythonExtension.Conversions
   MODULE_NAME _mlirConversions
   ADD_TO_PARENT MLIRPythonSources.Passes
+  ROOT_DIR "${PYTHON_SOURCE_DIR}"
   SOURCES
-    ${PYTHON_SOURCE_DIR}/Conversions/Conversions.cpp
+    Conversions/Conversions.cpp
   PRIVATE_LINK_LIBS
     LLVMSupport
   EMBED_CAPI_LINK_LIBS
   MLIRCAPIConversion
 )
 
 declare_mlir_python_extension(MLIRPythonExtension.ExecutionEngine
   MODULE_NAME _mlirExecutionEngine
   ADD_TO_PARENT MLIRPythonSources.ExecutionEngine
+  ROOT_DIR "${PYTHON_SOURCE_DIR}"
   SOURCES
-    ${PYTHON_SOURCE_DIR}/ExecutionEngineModule.cpp
+    ExecutionEngineModule.cpp
   PRIVATE_LINK_LIBS
     LLVMSupport
   EMBED_CAPI_LINK_LIBS
     MLIRCEXECUTIONENGINE
 )
 
 declare_mlir_python_extension(MLIRPythonExtension.GPUDialectPasses
   MODULE_NAME _mlirGPUPasses
   ADD_TO_PARENT MLIRPythonSources.Dialects.gpu
+  ROOT_DIR "${PYTHON_SOURCE_DIR}"
   SOURCES
-    ${PYTHON_SOURCE_DIR}/GPUPasses.cpp
+    GPUPasses.cpp
   PRIVATE_LINK_LIBS
     LLVMSupport
   EMBED_CAPI_LINK_LIBS
     MLIRCAPIGPU
 )
 
 declare_mlir_python_extension(MLIRPythonExtension.LinalgPasses
   MODULE_NAME _mlirLinalgPasses
   ADD_TO_PARENT MLIRPythonSources.Dialects.linalg
+  ROOT_DIR "${PYTHON_SOURCE_DIR}"
   SOURCES
-    ${PYTHON_SOURCE_DIR}/LinalgPasses.cpp
+    LinalgPasses.cpp
   PRIVATE_LINK_LIBS
     LLVMSupport
   EMBED_CAPI_LINK_LIBS
     MLIRCAPILinalg
 )
 
 declare_mlir_python_extension(MLIRPythonExtension.SparseTensorDialectPasses
   MODULE_NAME _mlirSparseTensorPasses
   ADD_TO_PARENT MLIRPythonSources.Dialects.sparse_tensor
+  ROOT_DIR "${PYTHON_SOURCE_DIR}"
   SOURCES
-    ${PYTHON_SOURCE_DIR}/SparseTensorPasses.cpp
+    SparseTensorPasses.cpp
   PRIVATE_LINK_LIBS
     LLVMSupport
   EMBED_CAPI_LINK_LIBS
     MLIRCAPISparseTensor
 )
 
 declare_mlir_python_extension(MLIRPythonExtension.Transforms
   MODULE_NAME _mlirTransforms
   ADD_TO_PARENT MLIRPythonSources.Passes
+  ROOT_DIR "${PYTHON_SOURCE_DIR}"
   SOURCES
-    ${PYTHON_SOURCE_DIR}/Transforms/Transforms.cpp
+    Transforms/Transforms.cpp
   PRIVATE_LINK_LIBS
     LLVMSupport
   EMBED_CAPI_LINK_LIBS
     MLIRCAPITransforms
 )
 
 # TODO: Figure out how to put this in the test tree.
 # This should not be included in the main Python extension. However,

mlir/test/Examples/standalone/test.toy

 # RUN: %cmake %mlir_src_root/examples/standalone -DCMAKE_CXX_COMPILER=%host_cxx -DCMAKE_C_COMPILER=%host_cc -DLLVM_ENABLE_LIBCXX=%enable_libcxx -DMLIR_DIR=%mlir_cmake_dir ; %cmake --build . --target check-standalone | tee %t | FileCheck %s
 
-# CHECK: Passed: 4
+# Note: The number of checked tests is not important. The command will fail
+# if any fail.
+# CHECK: Passed
 # UNSUPPORTED: windows, android