[MLIR] Workaround for python detection problems.

[stephenneuendorffer]

Summary

Although cmake should be platform-independent, we've observed
that some aspects of Python detection don't work on all platforms,
even with recent versions of cmake. This appears to be due to bugs
in the python detection logic, especially when the NumPy component
is required and not located within the python installation.
As a workaround, this patch first searches for "Development" before
searching for "Development.Module", which seems to workaround the
issue.

[stellaraccident]

There have been so many permutations of this that I can't remember my expectation for issues here, but I feel like this was causing other issues. Have you tried this on a manylinux image? Iirc, that was where things went bad when module detection didn't work. (I can give this a shot if you haven't)

[stephenneuendorffer]

In D118148#3269641, @stellaraccident wrote:

There have been so many permutations of this that I can't remember my expectation for issues here, but I feel like this was causing other issues. Have you tried this on a manylinux image? Iirc, that was where things went bad when module detection didn't work. (I can give this a shot if you haven't)

If you can, that would be great!

[stellaraccident]

In D118148#3270268, @stephenneuendorffer wrote:

In D118148#3269641, @stellaraccident wrote:

There have been so many permutations of this that I can't remember my expectation for issues here, but I feel like this was causing other issues. Have you tried this on a manylinux image? Iirc, that was where things went bad when module detection didn't work. (I can give this a shot if you haven't)

If you can, that would be great!

Let me give it a shot tomorrow.

[stellaraccident]

So this appears to work. Congratulations on finding a novel hack.

Here is how I checked (uses the dockcross launcher, which I find slightly easier for interactive use vs the official manylinux images):

# manylinux launcher created via:
host$ docker run --rm dockcross/manylinux2014-x64 > ~/bin/manylinux2014-x64
host$ ~/bin/manylinux2014-x64 bash
# In container.
container$ export PATH=/opt/python/cp38-cp38/bin:$PATH
container$ cmake -G Ninja ../llvm    -DLLVM_ENABLE_PROJECTS=mlir    -DLLVM_BUILD_EXAMPLES=ON    -DLLVM_TARGETS_TO_BUILD="X86;NVPTX;AMDGPU"    -DCMAKE_BUILD_TYPE=Release    -DLLVM_ENABLE_ASSERTIONS=ON -DMLIR_ENABLE_BINDINGS_PYTHON:BOOL=ON -DPython3_EXECUTABLE=/opt/python/cp38-cp38/bin/python

Here is the relevant output running a clean cmake invocation (after pip install numpy pybind11):

-- Found Python3: /opt/python/cp38-cp38/bin/python (found suitable version "3.8.11", minimum required is "3.6") found components: Interpreter 
-- Linker detection: GNU ld        
-- Performing Test HAS_WERROR_GLOBAL_CTORS    
-- Performing Test HAS_WERROR_GLOBAL_CTORS - Failed
-- Found Git: /usr/local/bin/git (found version "2.32.0") 
-- Targeting X86                               
-- Targeting NVPTX 
-- Targeting AMDGPU
-- Performing Test C_SUPPORTS_WERROR_IMPLICIT_FUNCTION_DECLARATION
-- Performing Test C_SUPPORTS_WERROR_IMPLICIT_FUNCTION_DECLARATION - Success
-- Performing Test C_SUPPORTS_WERROR_MISMATCHED_TAGS
-- Performing Test C_SUPPORTS_WERROR_MISMATCHED_TAGS - Failed
-- Found Python3: /opt/python/cp38-cp38/bin/python (found suitable version "3.8.11", minimum required is "3.6") found components: Interpreter Development.Module NumPy 
-- Found python include dirs: /opt/python/cp38-cp38/include/python3.8
-- Found python libraries: 
-- Found numpy v1.22.1: /opt/python/cp38-cp38/lib/python3.8/site-packages/numpy/core/include
-- Checking for pybind11 in python path...
-- found (/opt/python/cp38-cp38/lib/python3.8/site-packages/pybind11/share/cmake/pybind11)

[stellaraccident]

(note that the manylinux images do not have the Development feature, only Development.Module)

[rkayaith]

This change seems to have broken the python build on my system. I think this might be due to me having multiple python versions installed (managed through pyenv + virtualenvs). With this build command:

$ cmake -G Ninja ../llvm    -DLLVM_ENABLE_PROJECTS=mlir    -DLLVM_BUILD_EXAMPLES=ON    -DLLVM_TARGETS_TO_BUILD="X86;NVPTX;AMDGPU"    -DCMAKE_BUILD_TYPE=Release    -DLLVM_ENABLE_ASSERTIONS=ON -DMLIR_ENABLE_BINDINGS_PYTHON:BOOL=ON -DPython3_EXECUTABLE=$(pyenv which python)

This is the tail end of the CMake output:

-- Found Python3: /home/rk/.pyenv/versions/mlir-test/bin/python (found suitable version "3.7.5", minimum required is "3.6") found components: Interpreter 
-- Linker detection: GNU ld
-- Performing Test HAS_WERROR_GLOBAL_CTORS
-- Performing Test HAS_WERROR_GLOBAL_CTORS - Failed
-- Found Git: /usr/bin/git (found version "2.25.1") 
-- Targeting X86
-- Targeting NVPTX
-- Targeting AMDGPU
-- Performing Test C_SUPPORTS_WERROR_IMPLICIT_FUNCTION_DECLARATION
-- Performing Test C_SUPPORTS_WERROR_IMPLICIT_FUNCTION_DECLARATION - Success
-- Performing Test C_SUPPORTS_WERROR_MISMATCHED_TAGS
-- Performing Test C_SUPPORTS_WERROR_MISMATCHED_TAGS - Failed
-- Could NOT find Python3 (missing: Python3_INCLUDE_DIRS Development Development.Module Development.Embed) (found suitable version "3.8", minimum required is "3.6")
CMake Error at /usr/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find Python3 (missing: Development.Module NumPy) (found suitable
  version "3.7.5", minimum required is "3.6")
Call Stack (most recent call first):
  /usr/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-3.22/Modules/FindPython/Support.cmake:3166 (find_package_handle_standard_args)
  /usr/share/cmake-3.22/Modules/FindPython3.cmake:490 (include)
  /home/rk/repos/imaigine/llvm-project/mlir/cmake/modules/MLIRDetectPythonEnv.cmake:33 (find_package)
  /home/rk/repos/imaigine/llvm-project/mlir/CMakeLists.txt:135 (mlir_configure_python_dev_packages)


-- Configuring incomplete, errors occurred!

The first line found the python binary that I wanted (version 3.7.5), but at the end it seems like it found the system python (version 3.8) for some reason, and gotten itself confused? There's some messages about not being able to find certain components, but if I revert this change everything works.

[stellaraccident]

Can you provide some more system information? I don't have a solid repro for this on any of my systems.

cmake --version
pyenv which python
# "--version" for the above
# "-m pip freeze" for the above

[stellaraccident]

The error messages that cmake prints here leave a lot to be desired. Particularly, when it reports that components cannot be found, it could be that any of the listed ones are missing out don't meet requirements. In this case, it could bea missing numpy in the version selected.

[rkayaith]

$ cmake --version
cmake version 3.22.1
$ pyenv which python
/home/rk/.pyenv/versions/mlir-test/bin/python
$ $(pyenv which python) --version
Python 3.7.5
$ $(pyenv which python) -m pip freeze
dataclasses==0.6
numpy==1.21.5
pybind11==2.9.1
PyYAML==6.0

This is on Ubuntu 20.04, and in this venv the only python packages I've installed are the ones from mlir/python/requirements.txt

[stellaraccident]

Ok, thanks. And can you confirm that this happens on a fresh configure without an existing CMakeCache.txt? Cmake gets annoyingly "latchy" and I want to make sure we're not fighting a ghost.

[rkayaith]

yep, I nuked the build/ directory before every run

[stellaraccident]

In D118148#3335844, @rkayaith wrote:

yep, I nuked the build/ directory before every run

Thanks. Let me try to repro your setup.

I suspect that we are just inching up on needing to do very explicit python detection and stop relying on the flaky cmake logic as the only way through. But I've wanted to have a solid failure case in view before going that way.

[stellaraccident]

I was unfortunately not able to repro your issue with a fresh install of everything on a ubuntu-20.04 docker.

What I did:

$ mkdir -p ~/tmp/docker_work
# Create an llvm-project worktree under the above

$ docker run --rm -it -v ~/tmp/docker_work:/work stellaraccident/ubuntu-20.04-llvmdev:latest
# From inside container.
$ cmake --version
cmake version 3.22.2

$ $(pyenv which python) --version
Python 3.7.12

$ $(pyenv which python) -m pip freeze
dataclasses==0.6
numpy==1.21.5
pybind11==2.9.1
PyYAML==6.0

$ cd /work/llvm-project
$ mkdir build
$ cd build
$ cmake -G Ninja ../llvm    -DLLVM_ENABLE_PROJECTS=mlir    -DLLVM_BUILD_EXAMPLES=ON    -DLLVM_TARGETS_TO_BUILD="X86;NVPTX;AMDGPU"    -DCMAKE_BUILD_TYPE=Release    -DLLVM_ENABLE_ASSERTIONS=ON -DMLIR_ENABLE_BINDINGS_PYTHON:BOOL=ON -DPython3_EXECUTABLE=$(pyenv which python)

Relevant configure output:

-- Found Python3: /root/.pyenv/versions/3.7.12/bin/python (found suitable version "3.7.12", minimum required is "3.6") found components: Interpreter
-- Could NOT find Python3 (missing: Python3_INCLUDE_DIRS Python3_LIBRARIES Development Development.Module Development.Embed) (Required is at least version "3.6")
-- Found Python3: /root/.pyenv/versions/3.7.12/bin/python (found suitable version "3.7.12", minimum required is "3.6") found components: Interpreter Development.Module NumPy
-- Found python include dirs: /root/.pyenv/versions/3.7.12/include/python3.7m
-- Found python libraries: Python3_LIBRARY-NOTFOUND
-- Found numpy v1.21.5: /root/.pyenv/versions/3.7.12/lib/python3.7/site-packages/numpy/core/include
-- Checking for pybind11 in python path...
-- found (/root/.pyenv/versions/3.7.12/lib/python3.7/site-packages/pybind11/share/cmake/pybind11)
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Found pybind11: /root/.pyenv/versions/3.7.12/lib/python3.7/site-packages/pybind11/include (found version "2.9.1")
-- Found pybind11 v2.9.1: /root/.pyenv/versions/3.7.12/lib/python3.7/site-packages/pybind11/include
-- Python prefix = '', suffix = '', extension = '.cpython-37m-x86_64-linux-gnu.so...

This matches what I would expect from a pyenv install, which by default does not build a shared-library based Python interpreter (i.e. it will only find Development.Module, not Development). The first "Could NOT find Python3" signals the lack of the Development component. The subsequence "Found Python3" indicates that Development.Module and NumPy were found, and the rest matches what I would expect for a static-build of Python.

CMake debugging is not loads of fun, but adding --trace-expand --trace-redirect=cmake.trace can generate a haystack worth of information that sometimes contains the needle.

Opening the resulting cmake.trace file and searching for MLIRDetectPythonEnv.cmake can get you to the right point. If you put that (or a subset starting at this point) somewhere, I can squint at it. Also feel free to run my docker image above. I pushed it to docker hub. It is a fresh ubuntu-20.04 install, with build tools, modern cmake, system-python with MLIR required deps, and pyenv with 3.7.12 installed/defaulted.

[rkayaith]

Here's the output: https://gist.githubusercontent.com/rkayaith/3768481d385907cbcdc05d788679eed2/raw/aa05f598ea57011c362441eac8886c48ddf784e8/cmake.trace
I'll see if I can reproduce this in your docker image

[rkayaith]

I wasn't able to reproduce this, and I tried from a clean 20.04 image as well. Mysterious...

[rkayaith]

I started rebuilding my dev environment to get around this, and it just ended up happening again.
Here's the steps to reproduce: https://gist.github.com/rkayaith/1368238ceabf5c0c255528a0eba2e0c0
I ran that script on a live Ubuntu 20.04 USB:

$ sudo add-apt-repository universe
$ sudo apt install -y curl
$ sudo -s
$ curl https://gist.githubusercontent.com/rkayaith/1368238ceabf5c0c255528a0eba2e0c0/raw/mlir_build_2004_pyenv.sh > build.sh
$ source build.sh

Running the same script in docker (ubuntu:20.04) works fine... It also seems to be specific to pyenv-virtualenv; on the Ubuntu live image using pyenv shell 3.7.10 or regular python -m venv worked fine.

mlir/cmake/modules/MLIRDetectPythonEnv.cmake

if(CMAKE_VERSION VERSION_LESS "3.18.0")
     message(WARNING
         "This version of CMake is not compatible with statically built Python "
         "installations. If Python fails to detect below this may apply to you. "
         "Recommend upgrading to at least CMake 3.18. "
         "Detected current version: ${CMAKE_VERSION}"
     )
     set(_python_development_component Development)
   else()
+    # Prime the search for python to see if there is a full
+    # development package. This seems to work around cmake bugs
+    # searching only for Development.Module in some environments.
+    find_package(Python3 ${LLVM_MINIMUM_PYTHON_VERSION}
+      COMPONENTS Development)
     set(_python_development_component Development.Module)
   endif()
   find_package(Python3 ${LLVM_MINIMUM_PYTHON_VERSION}
     COMPONENTS Interpreter ${_python_development_component} NumPy REQUIRED)
   unset(_python_development_component)
   message(STATUS "Found python include dirs: ${Python3_INCLUDE_DIRS}")
   message(STATUS "Found python libraries: ${Python3_LIBRARIES}")
   message(STATUS "Found numpy v${Python3_NumPy_VERSION}: ${Python3_NumPy_INCLUDE_DIRS}")