Page MenuHomePhabricator

Fix swig python package path
AbandonedPublic

Authored by hhb on Sep 13 2019, 11:00 PM.

Details

Summary

The path defined in CMakeLists.txt doesn't match the path generated in
our python script. This change fixes that.

LLVM_LIBRARY_OUTPUT_INTDIR is defined as:

${CMAKE_BINARY_DIR}/${CMAKE_CFG_INTDIR}/lib${LLVM_LIBDIR_SUFFIX})

On the other hand, the path of site-package is generaged in
get_framework_python_dir_windows() in finishSwigPythonLLDB.py as:
(Dispite its name, the function is used for everything other than xcode)

prefix/cmakeBuildConfiguration/distutils.sysconfig.get_python_lib()

From lldb/CMakeLists.txt, we can see that:
prefix=${CMAKE_BINARY_DIR},
cmakeBuildConfiguration=${CMAKE_CFG_INTDIR}

And from python source code, we can see get_python_lib() always returns
lib/pythonx.y/site-packages for posix, or Lib/site-packages for windows:
https://github.com/python/cpython/blob/3.8/Lib/distutils/sysconfig.py#L128

We should make them match each other.

Diff Detail

Repository
rL LLVM

Event Timeline

hhb created this revision.Sep 13 2019, 11:00 PM
Herald added a project: Restricted Project. · View Herald TranscriptSep 13 2019, 11:00 PM
hhb updated this revision to Diff 220206.Sep 13 2019, 11:02 PM

Update comment

Harbormaster completed remote builds in B38129: Diff 220207.
hhb updated this revision to Diff 220392.Sep 16 2019, 2:37 PM

Use CMAKE_HOST_SYSTEM_NAME instead. Since the python script is run on host.

JDevlieghere accepted this revision.Sep 16 2019, 3:20 PM

LGTM. Thank you!

This revision is now accepted and ready to land.Sep 16 2019, 3:20 PM
Closed by commit rL372047: Fix swig python package path (authored by hhb, committed by ). · Explain WhySep 16 2019, 4:33 PM
This revision was automatically updated to reflect the committed changes.
Herald added a project: Restricted Project. · View Herald TranscriptSep 16 2019, 4:33 PM
ZeGentzy reopened this revision.EditedSep 17 2019, 5:35 PM
ZeGentzy added a subscriber: ZeGentzy.

Hello folks,

These changes break running ninja install when building with -D LLVM_LIBDIR_SUFFIX=32.

[...]
-- Up-to-date: /build/llvm-git-gentz/pkg/lib32-llvm-git-gentz/opt/llvm32/include
-- Up-to-date: /build/llvm-git-gentz/pkg/lib32-llvm-git-gentz/opt/llvm32/include/lldb
-- Up-to-date: /build/llvm-git-gentz/pkg/lib32-llvm-git-gentz/opt/llvm32/include/lldb/Host
-- Installing: /build/llvm-git-gentz/pkg/lib32-llvm-git-gentz/opt/llvm32/include/lldb/Host/Config.h
CMake Error at tools/lldb/scripts/cmake_install.cmake:41 (file):
  file INSTALL cannot find
  "/build/llvm-git-gentz/src/_build32_final/./lib/python3.7".
Call Stack (most recent call first):
  tools/lldb/cmake_install.cmake:50 (include)
  tools/cmake_install.cmake:50 (include)
  cmake_install.cmake:68 (include)


FAILED: CMakeFiles/install.util 
cd /build/llvm-git-gentz/src/_build32_final && /usr/bin/cmake -P cmake_install.cmake
ninja: build stopped: subcommand failed.

After some investigation, I found that there is no /build/llvm-git-gentz/src/_build32_final/./lib/python3.7 but there is an /build/llvm-git-gentz/src/_build32_final/./lib32/python3.7.

I've temporary solved this problem by applying the following patch:

diff --git a/lldb/scripts/CMakeLists.txt b/lldb/scripts/CMakeLists.txt
index 9de96ef5565..7d346deee7f 100644
--- a/lldb/scripts/CMakeLists.txt
+++ b/lldb/scripts/CMakeLists.txt
@@ -47,7 +47,7 @@ if(NOT LLDB_BUILD_FRAMEWORK)
   if(CMAKE_HOST_SYSTEM_NAME MATCHES "Windows")
     set(swig_python_subdir Lib/site-packages)
   else()
-    set(swig_python_subdir lib/python${PYTHON_VERSION_MAJOR}.${PYTHON_VERSION_MINOR})
+    set(swig_python_subdir lib${LLVM_LIBDIR_SUFFIX}/python${PYTHON_VERSION_MAJOR}.${PYTHON_VERSION_MINOR})
   endif()
 
   set(SWIG_PYTHON_DIR ${CMAKE_BINARY_DIR}/${CMAKE_CFG_INTDIR}/${swig_python_subdir})

I believe solving this problem correctly would also involve updating https://github.com/python/cpython/blob/3.8/Lib/distutils/sysconfig.py#L150 to account for ${LLVM_LIBDIR_SUFFIX}, however, I've never used the script before and am unfamiliar with it's purpose.

@hhb Could you please revisit this revision so that it doesn't break builds with a modified LLVM_LIBDIR_SUFFIX and revert this revision until then?

This revision is now accepted and ready to land.Sep 17 2019, 5:35 PM
ZeGentzy requested changes to this revision.Sep 17 2019, 5:37 PM
This revision now requires changes to proceed.Sep 17 2019, 5:37 PM
hhb added a comment.EditedSep 18 2019, 12:43 PM

Hello folks,

These changes break running ninja install when building with -D LLVM_LIBDIR_SUFFIX=32.

[...]
-- Up-to-date: /build/llvm-git-gentz/pkg/lib32-llvm-git-gentz/opt/llvm32/include
-- Up-to-date: /build/llvm-git-gentz/pkg/lib32-llvm-git-gentz/opt/llvm32/include/lldb
-- Up-to-date: /build/llvm-git-gentz/pkg/lib32-llvm-git-gentz/opt/llvm32/include/lldb/Host
-- Installing: /build/llvm-git-gentz/pkg/lib32-llvm-git-gentz/opt/llvm32/include/lldb/Host/Config.h
CMake Error at tools/lldb/scripts/cmake_install.cmake:41 (file):
  file INSTALL cannot find
  "/build/llvm-git-gentz/src/_build32_final/./lib/python3.7".
Call Stack (most recent call first):
  tools/lldb/cmake_install.cmake:50 (include)
  tools/cmake_install.cmake:50 (include)
  cmake_install.cmake:68 (include)


FAILED: CMakeFiles/install.util 
cd /build/llvm-git-gentz/src/_build32_final && /usr/bin/cmake -P cmake_install.cmake
ninja: build stopped: subcommand failed.

After some investigation, I found that there is no /build/llvm-git-gentz/src/_build32_final/./lib/python3.7 but there is an /build/llvm-git-gentz/src/_build32_final/./lib32/python3.7.

I've temporary solved this problem by applying the following patch:

diff --git a/lldb/scripts/CMakeLists.txt b/lldb/scripts/CMakeLists.txt
index 9de96ef5565..7d346deee7f 100644
--- a/lldb/scripts/CMakeLists.txt
+++ b/lldb/scripts/CMakeLists.txt
@@ -47,7 +47,7 @@ if(NOT LLDB_BUILD_FRAMEWORK)
   if(CMAKE_HOST_SYSTEM_NAME MATCHES "Windows")
     set(swig_python_subdir Lib/site-packages)
   else()
-    set(swig_python_subdir lib/python${PYTHON_VERSION_MAJOR}.${PYTHON_VERSION_MINOR})
+    set(swig_python_subdir lib${LLVM_LIBDIR_SUFFIX}/python${PYTHON_VERSION_MAJOR}.${PYTHON_VERSION_MINOR})
   endif()
 
   set(SWIG_PYTHON_DIR ${CMAKE_BINARY_DIR}/${CMAKE_CFG_INTDIR}/${swig_python_subdir})

I believe solving this problem correctly would also involve updating https://github.com/python/cpython/blob/3.8/Lib/distutils/sysconfig.py#L150 to account for ${LLVM_LIBDIR_SUFFIX}, however, I've never used the script before and am unfamiliar with it's purpose.

@hhb Could you please revisit this revision so that it doesn't break builds with a modified LLVM_LIBDIR_SUFFIX and revert this revision until then?

Hmm that's interesting... In my case the issue is opposite: LLVM_LIBDIR_SUFFIX=64 but the file is in lib/ .

I don't think we can change python to fix this. But I'm curious how the suffix is applied. Let me try LLVM_LIBDIR_SUFFIX=32...

hhb added a comment.EditedSep 18 2019, 2:12 PM
In D67583#1674430, @hhb wrote:

Hello folks,

These changes break running ninja install when building with -D LLVM_LIBDIR_SUFFIX=32.

[...]
-- Up-to-date: /build/llvm-git-gentz/pkg/lib32-llvm-git-gentz/opt/llvm32/include
-- Up-to-date: /build/llvm-git-gentz/pkg/lib32-llvm-git-gentz/opt/llvm32/include/lldb
-- Up-to-date: /build/llvm-git-gentz/pkg/lib32-llvm-git-gentz/opt/llvm32/include/lldb/Host
-- Installing: /build/llvm-git-gentz/pkg/lib32-llvm-git-gentz/opt/llvm32/include/lldb/Host/Config.h
CMake Error at tools/lldb/scripts/cmake_install.cmake:41 (file):
  file INSTALL cannot find
  "/build/llvm-git-gentz/src/_build32_final/./lib/python3.7".
Call Stack (most recent call first):
  tools/lldb/cmake_install.cmake:50 (include)
  tools/cmake_install.cmake:50 (include)
  cmake_install.cmake:68 (include)


FAILED: CMakeFiles/install.util 
cd /build/llvm-git-gentz/src/_build32_final && /usr/bin/cmake -P cmake_install.cmake
ninja: build stopped: subcommand failed.

After some investigation, I found that there is no /build/llvm-git-gentz/src/_build32_final/./lib/python3.7 but there is an /build/llvm-git-gentz/src/_build32_final/./lib32/python3.7.

I've temporary solved this problem by applying the following patch:

diff --git a/lldb/scripts/CMakeLists.txt b/lldb/scripts/CMakeLists.txt
index 9de96ef5565..7d346deee7f 100644
--- a/lldb/scripts/CMakeLists.txt
+++ b/lldb/scripts/CMakeLists.txt
@@ -47,7 +47,7 @@ if(NOT LLDB_BUILD_FRAMEWORK)
   if(CMAKE_HOST_SYSTEM_NAME MATCHES "Windows")
     set(swig_python_subdir Lib/site-packages)
   else()
-    set(swig_python_subdir lib/python${PYTHON_VERSION_MAJOR}.${PYTHON_VERSION_MINOR})
+    set(swig_python_subdir lib${LLVM_LIBDIR_SUFFIX}/python${PYTHON_VERSION_MAJOR}.${PYTHON_VERSION_MINOR})
   endif()
 
   set(SWIG_PYTHON_DIR ${CMAKE_BINARY_DIR}/${CMAKE_CFG_INTDIR}/${swig_python_subdir})

I believe solving this problem correctly would also involve updating https://github.com/python/cpython/blob/3.8/Lib/distutils/sysconfig.py#L150 to account for ${LLVM_LIBDIR_SUFFIX}, however, I've never used the script before and am unfamiliar with it's purpose.

@hhb Could you please revisit this revision so that it doesn't break builds with a modified LLVM_LIBDIR_SUFFIX and revert this revision until then?

Hmm that's interesting... In my case the issue is opposite: LLVM_LIBDIR_SUFFIX=64 but the file is in lib/ .

I don't think we can change python to fix this. But I'm curious how the suffix is applied. Let me try LLVM_LIBDIR_SUFFIX=32...

Can you share your full compile command and environment? I tried to build on Ubuntu (Python 2.7.16) with the latest llvm source code:

$ cmake /src/llvm-project/llvm -DLLVM_LIBDIR_SUFFIX=32 "-DLLVM_ENABLE_PROJECTS=clang;lldb"  -DCMAKE_INSTALL_PREFIX=/src/llvm-install -DLLDB_DISABLE_LIBEDIT=ON -GNinja
$ ninja
...
$ find -name site-packages
./lib/python2.7/site-packages

Or with python 3.6.8:

$ cmake /src/llvm-project/llvm -DLLVM_LIBDIR_SUFFIX=32 "-DLLVM_ENABLE_PROJECTS=clang;lldb"  -DCMAKE_INSTALL_PREFIX=/src/llvm-install -DLLDB_DISABLE_LIBEDIT=ON -GNinja -DPYTHON_EXECUTABLE=/usr/bin/python3.6 -DPYTHON_INCLUDE_DIR=/usr/include/python3.6 -DPYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython3.6m.so
$ ninja
...
$ find -name site-packages
./lib/python3.6/site-packages
$ ls lib32/python*
zsh: no matches found: lib32/python*

That's odd....

Can you share your full compile command and environment?

I'm using this PKGBUILD: https://github.com/ZeGentzy/aur-buildscripts/blob/master/llvm-git-gentz/PKGBUILD#L376

The commands being run are these:

export PKG_CONFIG_PATH="/usr/lib32/pkgconfig"
export CFLAGS="$CFLAGS -m32"
export CXXFLAGS="$CXXFLAGS -m32"
export LDFLAGS="$LDFLAGS -m32"

cmake /build/llvm-git-gentz/src/llvm-project32/llvm -G Ninja \
    -D CMAKE_BUILD_TYPE="Release" \
    -D LLVM_DEFAULT_TARGET_TRIPLE="i686-pc-linux-gnu" \
    -D LLVM_LIBDIR_SUFFIX=32 \
    -D LLVM_TARGET_ARCH:STRING=i686 \
    -D LLVM_CONFIG="/usr/bin/llvm-config32" \
    -D COMPILER_RT_BUILD_LIBFUZZER=OFF \
    -D LLVM_ENABLE_BINDINGS=OFF \
    -D PYTHON_EXECUTABLE=/usr/bin/python-32 \
    -D CMAKE_CXX_COMPILER="/usr/bin/clang++" \
    -D CMAKE_C_COMPILER="/usr/bin/clang" \
    -D LLVM_USE_LINKER="/usr/bin/ld.lld" \
    -D CMAKE_RANLIB="/usr/bin/llvm-ranlib" \
    -D CMAKE_AR="/usr/bin/llvm-ar" \
    -D LLVM_ENABLE_LTO=OFF \
    -D LLVM_BUILD_LLVM_DYLIB=ON \
    -D LLVM_LINK_LLVM_DYLIB=ON \
    -D LLVM_ENABLE_PROJECTS="clang;compiler-rt;lld;lldb;clang-tools-extra" \
    -D LLVM_APPEND_VC_REV=ON \
    -D LLVM_HOST_TRIPLE="x86_64-pc-linux-gnu" \
    -D LLVM_ENABLE_RTTI=ON \
    -D LLVM_ENABLE_FFI=ON \
    -D FFI_INCLUDE_DIR="/usr/lib32/libffi-3.2.1/include" \
    -D LLVM_BUILD_TESTS=ON \
    -D LLVM_VERSION_SUFFIX= \
    -D CMAKE_POLICY_DEFAULT_CMP0075=NEW \
    -D LLVM_OPTIMIZED_TABLEGEN=ON \
    -D LLVM_CCACHE_BUILD=ON \
    -D CMAKE_POLICY_DEFAULT_CMP0075=NEW \
    -D CMAKE_C_FLAGS="-march=native -O3 -pipe -ffast-math -funroll-loops -fstack-protector-strong -fno-plt -mno-xop -mno-fma4 -mno-tbm -m32" \
    -D CMAKE_CXX_FLAGS="-march=native -O3 -pipe -ffast-math -funroll-loops -fstack-protector-strong -fno-plt -mno-xop -mno-fma4 -mno-tbm -m32" \
    -D LLVM_TARGETS_TO_BUILD="AMDGPU;X86" \
    -D LLVM_BINUTILS_INCDIR=/usr/include \
    -D LLVM_EXTERNAL_LIT="/usr/bin/lit" \
    -D LLVM_BUILD_DOCS=OFF \
    -D LLVM_INCLUDE_DOCS=OFF \
    -D LLVM_ENABLE_SPHINX=OFF \
    -D LLVM_ENABLE_DOXYGEN=OFF \
    -D LLVM_INSTALL_UTILS=ON \
    -D LLVM_PARALLEL_LINK_JOBS=1 \
    -D CMAKE_INSTALL_PREFIX="/opt/llvm32"
ninja -j4 all
DESTDIR=/build/llvm-git-gentz/pkg/lib32-llvm-git-gentz ninja -j4 install

A lot... err, most... of the things I cargo culted from other sources.

hhb added a comment.EditedSep 18 2019, 11:26 PM

I don't have the right environment to try your command. But I didn't see anything obviously problematic.

Can you try the following:

cd /src/llvm-project/lldb/scripts/Python

Then run the following python program:

import sys
sys.path.append('/src/llvm-project/lldb/scripts/Python')
import finishSwigPythonLLDB
vDictArgs = {"--srcRoot": "/src/llvm-project/lldb",
             "--targetDir": "/src/out/./lib32",
             "--cfgBldDir": "/src/out/tools/lldb/scripts",
             "--prefix": "/src/out",
             "--cmakeBuildConfiguration": ".",
             "--lldbLibDir": "lib32",
             "-m": None,
}
finishSwigPythonLLDB.get_framework_python_dir(vDictArgs)

For me the output is:

(True, '/src/out/./lib/python2.7/site-packages/lldb', '')

Notice that the lib part don't have 32 suffix, even we specified that in parameters.

I kind of feeling that we should change this script to simply return targetDir. There's no good reason to use get_python_lib() and depend on python's implementation. But I don't know enough to make this decision...

In D67583#1674912, @hhb wrote:

I don't have the right environment to try your command. But I didn't see anything obviously problematic.

Can you try the following:

cd /src/llvm-project/lldb/scripts/Python

Then run the following python program:

import sys
sys.path.append('/src/llvm-project/lldb/scripts/Python')
import finishSwigPythonLLDB
vDictArgs = {"--srcRoot": "/src/llvm-project/lldb",
             "--targetDir": "/src/out/./lib32",
             "--cfgBldDir": "/src/out/tools/lldb/scripts",
             "--prefix": "/src/out",
             "--cmakeBuildConfiguration": ".",
             "--lldbLibDir": "lib32",
             "-m": None,
}
finishSwigPythonLLDB.get_framework_python_dir(vDictArgs)

For me the output is:

(True, '/src/out/./lib/python2.7/site-packages/lldb', '')

Notice that the lib part don't have 32 suffix, even we specified that in parameters.

I kind of feeling that we should change this script to simply return targetDir. There's no good reason to use get_python_lib() and depend on python's implementation. But I don't know enough to make this decision...

Unfortunately, the script does not seam to be functional:

$ python a.py         
Traceback (most recent call last):
  File "a.py", line 3, in <module>
    import finishSwigPythonLLDB
  File "/home/gentz/Documents/aur/gfx/backups/second/llvm-git-gentz/llvm-git-gentz/src/llvm-project32/lldb/scripts/Python/finishSwigPythonLLDB.py", line 47, in <module>
    import utilsOsType      # Determine the OS type this script is running on
ModuleNotFoundError: No module named 'utilsOsType'

$ python2 a.py                  
Traceback (most recent call last):
  File "a.py", line 3, in <module>
    import finishSwigPythonLLDB
  File "/home/gentz/Documents/aur/gfx/backups/second/llvm-git-gentz/llvm-git-gentz/src/llvm-project32/lldb/scripts/Python/finishSwigPythonLLDB.py", line 47, in <module>
    import utilsOsType      # Determine the OS type this script is running on
ImportError: No module named utilsOsType

$ pwd         
/home/gentz/Documents/aur/gfx/backups/second/llvm-git-gentz/llvm-git-gentz/src/llvm-project32/lldb/scripts/Python
hhb added a comment.Sep 19 2019, 9:47 AM

Oh sorry my mistake. The current directory should be llvm-project/lldb/scripts. I.e.

cd /src/llvm-project/lldb/scripts
In D67583#1675415, @hhb wrote:

Oh sorry my mistake. The current directory should be llvm-project/lldb/scripts. I.e.

cd /src/llvm-project/lldb/scripts

Maybe I'm doing something wrong, but that does not work either.

$ python a.py               
Traceback (most recent call last):
  File "a.py", line 3, in <module>
    import finishSwigPythonLLDB
ModuleNotFoundError: No module named 'finishSwigPythonLLDB'

$ python2 a.py
Traceback (most recent call last):
  File "a.py", line 3, in <module>
    import finishSwigPythonLLDB
ImportError: No module named finishSwigPythonLLDB

$ pwd         
/home/gentz/Documents/aur/gfx/backups/second/llvm-git-gentz/llvm-git-gentz/src/llvm-project32/lldb/scripts

$ cat a.py                        
import sys
sys.path.append('/src/llvm-project/lldb/scripts/Python')
import finishSwigPythonLLDB
vDictArgs = {"--srcRoot": "/src/llvm-project/lldb",
             "--targetDir": "/src/out/./lib32",
             "--cfgBldDir": "/src/out/tools/lldb/scripts",
             "--prefix": "/src/out",
             "--cmakeBuildConfiguration": ".",
             "--lldbLibDir": "lib32",
             "-m": None,
}
finishSwigPythonLLDB.get_framework_python_dir(vDictArgs)
hhb added a comment.Sep 19 2019, 4:05 PM

I'm making a change in finishSwigPythonLLDB.py to write directly to targetDir. This way we don't have the problem to guess where the files are written.

hhb added a comment.Sep 19 2019, 4:11 PM
In D67583#1675415, @hhb wrote:

Oh sorry my mistake. The current directory should be llvm-project/lldb/scripts. I.e.

cd /src/llvm-project/lldb/scripts

Maybe I'm doing something wrong, but that does not work either.

$ python a.py               
Traceback (most recent call last):
  File "a.py", line 3, in <module>
    import finishSwigPythonLLDB
ModuleNotFoundError: No module named 'finishSwigPythonLLDB'

$ python2 a.py
Traceback (most recent call last):
  File "a.py", line 3, in <module>
    import finishSwigPythonLLDB
ImportError: No module named finishSwigPythonLLDB

$ pwd         
/home/gentz/Documents/aur/gfx/backups/second/llvm-git-gentz/llvm-git-gentz/src/llvm-project32/lldb/scripts

$ cat a.py                        
import sys
sys.path.append('/src/llvm-project/lldb/scripts/Python')
import finishSwigPythonLLDB
vDictArgs = {"--srcRoot": "/src/llvm-project/lldb",
             "--targetDir": "/src/out/./lib32",
             "--cfgBldDir": "/src/out/tools/lldb/scripts",
             "--prefix": "/src/out",
             "--cmakeBuildConfiguration": ".",
             "--lldbLibDir": "lib32",
             "-m": None,
}
finishSwigPythonLLDB.get_framework_python_dir(vDictArgs)

My theory now is that your python has a different implementation of get_python_lib(). After all we should not guess what python would do. I'm making a change.
If you still interested in testing that, you can fix the path in a.py...

ZeGentzy added a comment.EditedSep 19 2019, 8:20 PM
In D67583#1675714, @hhb wrote:

My theory now is that your python has a different implementation of get_python_lib(). After all we should not guess what python would do. I'm making a change.
If you still interested in testing that, you can fix the path in a.py...

Oh, derp, I completely missed that. I get (True, '/src/out/./lib/python3.7/site-packages/lldb', '') for python and (True, '/src/out/./lib/python2.7/site-packages/lldb', '') for python2.

EDIT:

$ cat a.py                       
import sys
sys.path.append('/home/gentz/Documents/aur/gfx/backups/second/llvm-git-gentz/llvm-git-gentz/src/llvm-project32/lldb/scripts/Python')
import finishSwigPythonLLDB
vDictArgs = {"--srcRoot": "/src/llvm-project/lldb",
             "--targetDir": "/src/out/./lib32",
             "--cfgBldDir": "/src/out/tools/lldb/scripts",
             "--prefix": "/src/out",
             "--cmakeBuildConfiguration": ".",
             "--lldbLibDir": "lib32",
             "-m": None,
}
print(finishSwigPythonLLDB.get_framework_python_dir(vDictArgs))
labath added a subscriber: labath.Sep 20 2019, 5:17 AM

I could be off mark here, but I do recall some people trying to get this code to work in environments where python and lldb(llvm) are configured with different libdirs (so LLVM_LIBDIR_SUFFIX in llvm, and configure --prefix=X in python). Is that the case you're running into here?

If that is something we want to support (personally, I would very much like to just declare that "unsupported", but maybe we don't have that luxury). Then we'll need to be super careful about distinguishing the _llvm_ libdir name (i.e. the place where liblldb and friends should go to) from the _python_ libdir (the destination for the site-packages stuff). I think the code already partially tries to do that, but I am not sure if it is really correct.

Since I'm already complaining about python paths, I'll add that the way we currently compute the python path is wrong for cross-compilation. (Because it will pick up the path from the host python, not the target one.) If we're going to be changing something here, it would be nice to fix that too (which, I hope, will also allow us avoid needing to keep multiple places in sync).

hhb added a comment.Sep 20 2019, 10:59 AM
In D67583#1675714, @hhb wrote:

My theory now is that your python has a different implementation of get_python_lib(). After all we should not guess what python would do. I'm making a change.
If you still interested in testing that, you can fix the path in a.py...

Oh, derp, I completely missed that. I get (True, '/src/out/./lib/python3.7/site-packages/lldb', '') for python and (True, '/src/out/./lib/python2.7/site-packages/lldb', '') for python2.

EDIT:

$ cat a.py                       
import sys
sys.path.append('/home/gentz/Documents/aur/gfx/backups/second/llvm-git-gentz/llvm-git-gentz/src/llvm-project32/lldb/scripts/Python')
import finishSwigPythonLLDB
vDictArgs = {"--srcRoot": "/src/llvm-project/lldb",
             "--targetDir": "/src/out/./lib32",
             "--cfgBldDir": "/src/out/tools/lldb/scripts",
             "--prefix": "/src/out",
             "--cmakeBuildConfiguration": ".",
             "--lldbLibDir": "lib32",
             "-m": None,
}
print(finishSwigPythonLLDB.get_framework_python_dir(vDictArgs))

That's really... weird... I don't know how your files end up in the lib32 directory. Anyway this change is reverted in https://reviews.llvm.org/D67781.

hhb added a comment.EditedSep 20 2019, 11:05 AM

I could be off mark here, but I do recall some people trying to get this code to work in environments where python and lldb(llvm) are configured with different libdirs (so LLVM_LIBDIR_SUFFIX in llvm, and configure --prefix=X in python). Is that the case you're running into here?

If that is something we want to support (personally, I would very much like to just declare that "unsupported", but maybe we don't have that luxury). Then we'll need to be super careful about distinguishing the _llvm_ libdir name (i.e. the place where liblldb and friends should go to) from the _python_ libdir (the destination for the site-packages stuff). I think the code already partially tries to do that, but I am not sure if it is really correct.

Since I'm already complaining about python paths, I'll add that the way we currently compute the python path is wrong for cross-compilation. (Because it will pick up the path from the host python, not the target one.) If we're going to be changing something here, it would be nice to fix that too (which, I hope, will also allow us avoid needing to keep multiple places in sync).

I also realized that this is totally wrong for cross-compilation. We are doing cross compilation to windows with MinGW so I need to fix that.

Actually I don't know why we need get_python_lib(). From what I can see now, the 3 paths should match:

  1. finishSwigPythonLLDB.py write necessary files to ${CMAKE_BINARY_DIR}/${CMAKE_CFG_INTDIR}/distutils.sysconfig.get_python_lib()
  1. lldb/scripts/CMakeLists.txt copies them to lib${LLVM_LIBDIR_SUFFIX} (if not xcode)
  1. ScriptInterpreterPython.cpp adds the right path into PYTHONPATH.

As long as these 3 paths matches, we are good. Then why not simply put everything into ${liblldb path}/site-packages for all platforms?

(I'm gonna make a change to test that. But let me know if anything is obviously wrong.

ZeGentzy added a comment.EditedSep 20 2019, 1:45 PM

I could be off mark here, but I do recall some people trying to get this code to work in environments where python and lldb(llvm) are configured with different libdirs (so LLVM_LIBDIR_SUFFIX in llvm, and configure --prefix=X in python). Is that the case you're running into here?

I think not. I'm cross compiling llvm, getting it to build a 32-bit llvm for my 64-bit host, so I can install it along side my 64-bit llvm and use it with my 32-bit mesa.

I'm building two pythons, python2s, llvms and mesas, one for 32-bits and one for 64-bits.

Here is my 32-bit python2's configured options:

--prefix=/usr 
--enable-shared
--with-threads
--enable-optimizations
--with-lto
--enable-ipv6
--with-system-expat
--with-dbmliborder=gdbm:ndbm
--with-system-ffi
--without-ensurepip
--libdir=/usr/lib32
--libexecdir=/usr/lib32
--includedir="/usr/lib32/python${_pybasever}/include"
--exec_prefix="/usr/lib32/python${_pybasever}/"
--bindir=/usr/bin
--sbindir=/usr/bin
--with-suffix='-32'

My 32-bit python, the one in use by llvm for it's 32-bit builds, was configured with the following options:

--prefix=/usr 
--enable-shared
--with-threads
--with-computed-gotos
--enable-optimizations
--with-lto
--enable-ipv6
--with-system-expat
--with-dbmliborder=gdbm:ndbm
--with-system-ffi
--with-system-libmpdec
--enable-loadable-sqlite-extensions
--without-ensurepip
--libdir=/usr/lib32
--libexecdir=/usr/lib32
--includedir=/usr/lib32/python${_pybasever}m/include
--exec_prefix=/usr/lib32/python${_pybasever}/
--bindir=/usr/bin
--sbindir=/usr/bin
--with-suffix='-32'

This is done so my 32-bit python stuff does not conflict with the 64-bit python stuff.


Actually, thinking about it, I should have probably run your scripts with the 32-bit pythons, as those are what llvm uses:

[root@GENTZ-PC-NEW2 scripts]# python a.py
(True, '/src/out/./lib/python3.7/site-packages/lldb', '')
[root@GENTZ-PC-NEW2 scripts]# python-32 a.py
(True, '/src/out/./lib32/python3.7/site-packages/lldb', '')
[root@GENTZ-PC-NEW2 scripts]# python2-32 a.py
(True, '/src/out/./lib32/python2.7/site-packages/lldb', '')
[root@GENTZ-PC-NEW2 scripts]# python2 a.py
(True, '/src/out/./lib/python2.7/site-packages/lldb', '')
[root@GENTZ-PC-NEW2 scripts]# file /usr/lib[0-9]*/python*/ 
/usr/lib32/python2.7/:  directory
/usr/lib32/python3.7/:  directory
/usr/lib32/python3.7m/: directory
/usr/lib64/python2.7/:  directory
/usr/lib64/python3.7/:  directory
In D67583#1677141, @hhb wrote:

Since I'm already complaining about python paths, I'll add that the way we currently compute the python path is wrong for cross-compilation. (Because it will pick up the path from the host python, not the target one.) If we're going to be changing something here, it would be nice to fix that too (which, I hope, will also allow us avoid needing to keep multiple places in sync).

I also realized that this is totally wrong for cross-compilation. We are doing cross compilation to windows with MinGW so I need to fix that.

Actually I don't know why we need get_python_lib(). From what I can see now, the 3 paths should match:

  1. finishSwigPythonLLDB.py write necessary files to ${CMAKE_BINARY_DIR}/${CMAKE_CFG_INTDIR}/distutils.sysconfig.get_python_lib()
  2. lldb/scripts/CMakeLists.txt copies them to lib${LLVM_LIBDIR_SUFFIX} (if not xcode)
  3. ScriptInterpreterPython.cpp adds the right path into PYTHONPATH.

    As long as these 3 paths matches, we are good. Then why not simply put everything into ${liblldb path}/site-packages for all platforms?

    (I'm gonna make a change to test that. But let me know if anything is obviously wrong.

This will work for the python-in-lldb case. However, the idea is lldb can also be used as python package, from any python application (by just typing import lldb). For that to work, we need to install lldb python stuff to a location that will be searched for by python in its default configuration.

This will work for the python-in-lldb case. However, the idea is lldb can also be used as python package, from any python application (by just typing import lldb). For that to work, we need to install lldb python stuff to a location that will be searched for by python in its default configuration.

I agree. Hence, I sent D67890 yesterday.

hhb abandoned this revision.Oct 3 2019, 4:20 PM