[mlir][Python] Support finding pybind11 from the python environment.

[stellaraccident]

Summary

• Makes pip install pybind11 do the right thing with no further config.
• Since we now require a version of pybind11 greater than many LTS OS installs (>=2.6), a more convenient way to get a recent version is preferable.
• Also adds the version spec to find_package so it will skip older versions that may be lying around.
• Tested the full matrix of old system install, no system install, pip install and no pip install.

[zhanghb97]

I'm wondering can we use the MLIRDetectPythonEnv.cmake or names like this (detect all the required Python environments), instead of MLIRDetectPybind11.cmake (only responsible for checking pybind11). In this case, we can define all the detecting functions (like mlir_detect_numpy_install) in one cmake file to avoid fragmentation.

[stellaraccident]

Address comments and rebase

[stellaraccident]

In D91903#2409384, @zhanghb97 wrote:

I'm wondering can we use the MLIRDetectPythonEnv.cmake or names like this (detect all the required Python environments), instead of MLIRDetectPybind11.cmake (only responsible for checking pybind11). In this case, we can define all the detecting functions (like mlir_detect_numpy_install) in one cmake file to avoid fragmentation.

Good idea. Done.

mlir/CMakeLists.txt

 #-------------------------------------------------------------------------------
 
 set(MLIR_BINDINGS_PYTHON_ENABLED 0 CACHE BOOL
        "Enables building of Python bindings.")
 set(MLIR_PYTHON_BINDINGS_VERSION_LOCKED 1 CACHE BOOL
        "Links to specific python libraries, resolving all symbols.")
 
 if(MLIR_BINDINGS_PYTHON_ENABLED)
+  include(MLIRDetectPythonEnv)
   find_package(Python3 COMPONENTS Interpreter Development REQUIRED)
   message(STATUS "Found python include dirs: ${Python3_INCLUDE_DIRS}")
   message(STATUS "Found python libraries: ${Python3_LIBRARIES}")
-  find_package(pybind11 CONFIG REQUIRED)
-  # TODO: pybind11 v2.6 switched from pybind11_INCLUDE_DIRS (plural) to
-  # pybind11_INCLUDE_DIR (singular). A lot has changed in this area since this
-  # was written and overall python config and pybind11 should be modernized.
-  set(pybind11_INCLUDE_DIR ${pybind11_INCLUDE_DIR} ${pybind11_INCLUDE_DIRS})
+  mlir_detect_pybind11_install()
+  find_package(pybind11 2.6 CONFIG REQUIRED)
   message(STATUS "Found pybind11 v${pybind11_VERSION}: ${pybind11_INCLUDE_DIR}")
   message(STATUS "Python prefix = '${PYTHON_MODULE_PREFIX}', "
                  "suffix = '${PYTHON_MODULE_SUFFIX}', "
                  "extension = '${PYTHON_MODULE_EXTENSION}")
 endif()
 
 include_directories( "include")
 include_directories( ${MLIR_INCLUDE_DIR})

mlir/cmake/modules/MLIRDetectPythonEnv.cmake

+# Macros and functions related to detecting details of the Python environment.
+
+# Detects a pybind11 package installed in the current python environment
+# and sets variables to allow it to be found. This allows pybind11 to be
+# installed via pip, which typically yields a much more recent version than
+# the OS install, which will be available otherwise.
+function(mlir_detect_pybind11_install)
+  if(pybind11_DIR)
+    message(STATUS "Using explicit pybind11 cmake directory: ${pybind11_DIR} (-Dpybind11_DIR to change)")
+  else()
+    message(CHECK_START "Checking for pybind11 in python path...")
+    execute_process(
+      COMMAND "${PYTHON_EXECUTABLE}"
+      -c "import pybind11;print(pybind11.get_cmake_dir(), end='')"
+      WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
+      RESULT_VARIABLE STATUS
+      OUTPUT_VARIABLE PACKAGE_DIR
+      ERROR_QUIET)
+    if(NOT STATUS EQUAL "0")
+      message(CHECK_FAIL "not found (install via 'pip install pybind11' or set pybind11_DIR)")
+      return()
+    endif()
+    message(CHECK_PASS "found (${PACKAGE_DIR})")
+    set(pybind11_DIR "${PACKAGE_DIR}" PARENT_SCOPE)
+  endif()
+endfunction()

mlir/docs/Bindings/Python.md

 # MLIR Python Bindings
 
 Current status: Under development and not enabled by default
 
 ## Building
 
 ### Pre-requisites
 
-* [`pybind11`](https://github.com/pybind/pybind11) must be installed and able to
-  be located by CMake. Note: minimum version required: :2.6.0
 * A relatively recent Python3 installation
+* [`pybind11`](https://github.com/pybind/pybind11) must be installed and able to
+  be located by CMake (auto-detected if installed via
+  `python -m pip install pybind11`). Note: minimum version required: :2.6.0.
 
 ### CMake variables
 
 * **`MLIR_BINDINGS_PYTHON_ENABLED`**`:BOOL`
 
   Enables building the Python bindings. Defaults to `OFF`.
 
 * **`MLIR_PYTHON_BINDINGS_VERSION_LOCKED`**`:BOOL`