Page MenuHomePhabricator

[CMake] Fail when Python interpreter doesn't match Python libraries version
ClosedPublic

Authored by JDevlieghere on Jul 16 2019, 11:19 AM.

Details

Summary

Because of how CMake finds the Python libraries and interpreter, it's possible to end up with a discrepancy between the two. For example, you'd end up using a Python 3 interpreter to run the test suite while LLDB was build and linked against Python 2. This patch adds a CMake warning so we find out at configuration time, instead of finding out at test time, after having build LLDB.

Diff Detail

Repository
rL LLVM

Event Timeline

JDevlieghere created this revision.Jul 16 2019, 11:19 AM
xiaobai accepted this revision.Jul 16 2019, 11:20 AM

This definitely caused me some pain a few months ago. Thanks for adding this!

This revision is now accepted and ready to land.Jul 16 2019, 11:20 AM

Hmm, I've just looked through CMake docs, and I think a better solution might be:

find_package(Python COMPONENTS Interpreter Development)

Apparently using separate modules has been deprecated in 3.12, and using a single FindPython call guarantees version match.

Of course, this would require changing all the stuff that relies on Python. if you don't have time to fight it, I suppose this is good enough as interim solution.

This revision was automatically updated to reflect the committed changes.
Herald added a project: Restricted Project. · View Herald TranscriptJul 16 2019, 11:27 AM

Hmm, I've just looked through CMake docs, and I think a better solution might be:

find_package(Python COMPONENTS Interpreter Development)

Apparently using separate modules has been deprecated in 3.12, and using a single FindPython call guarantees version match.

Of course, this would require changing all the stuff that relies on Python. if you don't have time to fight it, I suppose this is good enough as interim solution.

If that works in 3.4 I'm happy to go that way!

rnk added a subscriber: rnk.Jul 16 2019, 11:38 AM

You broke my build. =/ I got this output:

CMake Error at C:/src/llvm-project/lldb/cmake/modules/LLDBConfig.cmake:201 (message):
  Found incompatible Python interpreter (3.7.3) and Python libraries ()

I'll mess with it a bit I guess.

If that works in 3.4 I'm happy to go that way!

FWICS it's been added in 3.12, so no.

rnk added a comment.Jul 16 2019, 11:47 AM
In D64812#1588055, @rnk wrote:

You broke my build. =/ I got this output:

CMake Error at C:/src/llvm-project/lldb/cmake/modules/LLDBConfig.cmake:201 (message):
  Found incompatible Python interpreter (3.7.3) and Python libraries ()

I'll mess with it a bit I guess.

Fixed that in r366247. Apparently Windows has totally custom Python version detection logic.

mgorny added inline comments.Jul 16 2019, 11:53 AM
lldb/trunk/cmake/modules/LLDBConfig.cmake
197

As a small optimization, I think you can prepend major+minor version into Python_ADDITIONAL_VERSIONS. Then it will prefer finding the same libs as interpreter.

JDevlieghere marked an inline comment as done.Jul 16 2019, 2:40 PM
JDevlieghere added inline comments.
lldb/trunk/cmake/modules/LLDBConfig.cmake
197

The implementation of PythonLibs should already honor the minor/major version it gets from the interpreter: https://github.com/Kitware/CMake/blob/master/Modules/FindPythonLibs.cmake#L116

In D64812#1588061, @rnk wrote:
In D64812#1588055, @rnk wrote:

You broke my build. =/ I got this output:

CMake Error at C:/src/llvm-project/lldb/cmake/modules/LLDBConfig.cmake:201 (message):
  Found incompatible Python interpreter (3.7.3) and Python libraries ()

I'll mess with it a bit I guess.

Fixed that in r366247. Apparently Windows has totally custom Python version detection logic.

Still busted for me, even with rnk's fix:

  • LLDB Found PythonExecutable: C:/Python36/python_d.exe
  • LLDB Found PythonLibs: C:/Python36/libs/python36_d.lib
  • LLDB Found PythonDLL: C:/Python36/python36_d.dll
  • LLDB Found PythonIncludeDirs: C:/Python36/Include CMake Error at D:/src/llvm/mono/llvm-project/lldb/cmake/modules/LLDBConfig.cmake:204 (message): Found incompatible Python interpreter (2.7.15) and Python libraries (3.6.8)

Note that it found all the Python things, but somehow the Python interpreter is still set incorrectly. I haven't been able to find where PYTHON_VERSION_STRING gets set, but it seems to happen long before the Windows-specific detection logic and it seems to ignore the PYTHON_HOME setting.

OK, the only way I was able to make this work was to remove all traces of Python 2.x from my machine. As long as the older Python existed on my machine CMake would find that one, regardless of which one was in my PATH or indicated by PYTHON_HOME_DIR. Of course, it required killing the cmake cache a couple times, too.

Interestingly, this suggests that I've been in this allegedly "incompatible" state for a while (Python 2.7 interpreter with Python 3.6 for everything else), and there were no obvious errors. I was able to build and test lldb, including all the Python-based tests. This makes me think this FATAL_ERROR message might not actually be necessary.

mgorny added inline comments.Jul 16 2019, 8:28 PM
lldb/trunk/cmake/modules/LLDBConfig.cmake
197

But note that it is appended *after* Python_ADDITIONAL_VERSIONS, so in our case full version list takes precedence over that. If I explicitly set PYTHON_EXECUTABLE to 2.7, it will still prefer libs from 3.x.