This is an archive of the discontinued LLVM Phabricator instance.

Making linking against Python simpler on Windows
ClosedPublic

Authored by zturner on Apr 10 2015, 11:03 AM.

Details

Summary

Until now, on Windows in order to link against Python the user must
manually specify a value for 3 different CMake variables. PYTHON_EXECUTABLE,
PYTHON_LIBRARY, and PYTHON_INCLUDE_DIR. And the user must
manage 4 different sets of values for these variables, depending on whether
he is building debug, release, x86, or x64.

On top of that, the user must update the value of PYTHONHOME and
PYTHONPATH for each of these 4 configurations.

This patch addresses all of these issues. To start with, it deprecates the
three Python CMake variables in favor of a single variable PYTHON_HOME
which points to the root of a python installation. Since building Python from
source doesn't output the files in a structure that is compatible with the
PYTHONHOME environment variable (or the Py_SetPythonHome API), we
also provide a script install_custom_python.py which will copy the output of a
custom python build to the correct directory structure.

Second, it automatically uses the value of PYTHON_HOME as a parameter to
Py_SetPythonHome() on interpreter initialization. Previously this API was not
being called, so Python was attempting to figure out a valid Python home at
runtime, and it was falling back to the system installed Python, which was
leading to errors on x64 builds since it was finding an x86 Python as the system
installed python.

The supported workflow after this patch will be to build python once for each
configuration and architecture {Debug,Release} x {x86,x64} and then run the script.
Then run CMake specifying -DPYTHON_HOME=<path>. The first time you do
this will probably require you to delete your CMake cache first.

The old workflow is still supported during a transitionary period, but a warning is
printed at CMake time, and this will eventually be removed.

Diff Detail

Repository
rL LLVM

Event Timeline

zturner updated this revision to Diff 23611.Apr 10 2015, 11:03 AM
zturner retitled this revision from to Making linking against Python simpler on Windows.
zturner updated this object.
zturner edited the test plan for this revision. (Show Details)
zturner added a subscriber: Unknown Object (MLST).
ted added a subscriber: ted.Apr 10 2015, 1:00 PM

So if I specify -DPYTHON_HOME:STRING=c:\pythonfoo, it will pick up the python includes and library from c:\pythonfoo, and if LLDB_RELOCATABLE_PYTHON is set to 0, lldb will use c:\pythonfoo as PYTHONHOME?

This will work if you can guarantee that the machines you run lldb on have the python you want; in my case, I can't. Right now I'm shipping python alongside lldb, and in lldb setting PYTHONHOME and PYTHONPATH based on CMAKE variables I've set up. With your PYTHONPATH fix I can stop doing that, and I think taking my PYTHONHOME CMAKE changes and combining them with this will give us a complete solution - explicitly specify it if I want to, or use the value from PYTHON_HOME, or let the user set it manually.

I do this for Linux, too, because we run many different flavors of Linux, from SLES 10 and 11 to Ubuntu 10, 12 and 14, and python does very bad things when you try to mix and match versions of the .so and versions of the modules.

In D8979#154928, @ted wrote:

So if I specify -DPYTHON_HOME:STRING=c:\pythonfoo, it will pick up the python includes and library from c:\pythonfoo, and if LLDB_RELOCATABLE_PYTHON is set to 0, lldb will use c:\pythonfoo as PYTHONHOME?

If LLDB_RELOCATABLE_PYTHON is set to 0 (the default), then yes LLDB will use c:\pythonfoo as PYTHONHOME. If LLDB_RELOCATABLE_PYTHON is set to 1, then it will be up to you to set PYTHONHOME in the environment before running LLDB.

This will work if you can guarantee that the machines you run lldb on have the python you want; in my case, I can't. Right now I'm shipping python alongside lldb, and in lldb setting PYTHONHOME and PYTHONPATH based on CMAKE variables I've set up. With your PYTHONPATH fix I can stop doing that, and I think taking my PYTHONHOME CMAKE changes and combining them with this will give us a complete solution - explicitly specify it if I want to, or use the value from PYTHON_HOME, or let the user set it manually.

So if you ship the correct version of Python to your users, then all you have to do is build with LLDB_RELOCATABLE_PYTHON=1, and then LLDB will use whatever the PYTHONHOME environment variable is set to. I think that should continue to work for your case right? If you want to avoid environment variables entirely, you could also consider adding a command line option to LLDB so that you can run LLDB like "lldb --python-home=C:\lldb\python"

I do this for Linux, too, because we run many different flavors of Linux, from SLES 10 and 11 to Ubuntu 10, 12 and 14, and python does very bad things when you try to mix and match versions of the .so and versions of the modules.

If you want to do this for Linux, the general structure of the CMake should be set up to make this easy. You would need to fill out the non-MSVC codepath in the CMake and then everything else should work. I didn't do this because the non-Windows build doesn't use the same output directories, so I figured I would leave that to someone who needed it and could test it appropriately.

zturner updated this revision to Diff 23629.Apr 10 2015, 2:52 PM

Updated website information.

I'm going to go ahead and commit this. I will be watching for any issues that arise, please respond to this thread or the thread on lldb-dev if anyone has any issues.

This revision was automatically updated to reflect the committed changes.