Fix swig python package path

[hhb]

Summary

The path defined in CMakeLists.txt doesn't match the path generated in
our python script. This change fixes that.

LLVM_LIBRARY_OUTPUT_INTDIR is defined as:

${CMAKE_BINARY_DIR}/${CMAKE_CFG_INTDIR}/lib${LLVM_LIBDIR_SUFFIX})

On the other hand, the path of site-package is generaged in
get_framework_python_dir_windows() in finishSwigPythonLLDB.py as:
(Dispite its name, the function is used for everything other than xcode)

prefix/cmakeBuildConfiguration/distutils.sysconfig.get_python_lib()

From lldb/CMakeLists.txt, we can see that:
prefix=${CMAKE_BINARY_DIR},
cmakeBuildConfiguration=${CMAKE_CFG_INTDIR}

And from python source code, we can see get_python_lib() always returns
lib/pythonx.y/site-packages for posix, or Lib/site-packages for windows:
https://github.com/python/cpython/blob/3.8/Lib/distutils/sysconfig.py#L128

We should make them match each other.

[hhb]

Update comment

[hhb]

Rebase

[hhb]

Use CMAKE_HOST_SYSTEM_NAME instead. Since the python script is run on host.

[JDevlieghere]

LGTM. Thank you!

[ZeGentzy]

Hello folks,

These changes break running ninja install when building with -D LLVM_LIBDIR_SUFFIX=32.

[...]
-- Up-to-date: /build/llvm-git-gentz/pkg/lib32-llvm-git-gentz/opt/llvm32/include
-- Up-to-date: /build/llvm-git-gentz/pkg/lib32-llvm-git-gentz/opt/llvm32/include/lldb
-- Up-to-date: /build/llvm-git-gentz/pkg/lib32-llvm-git-gentz/opt/llvm32/include/lldb/Host
-- Installing: /build/llvm-git-gentz/pkg/lib32-llvm-git-gentz/opt/llvm32/include/lldb/Host/Config.h
CMake Error at tools/lldb/scripts/cmake_install.cmake:41 (file):
  file INSTALL cannot find
  "/build/llvm-git-gentz/src/_build32_final/./lib/python3.7".
Call Stack (most recent call first):
  tools/lldb/cmake_install.cmake:50 (include)
  tools/cmake_install.cmake:50 (include)
  cmake_install.cmake:68 (include)


FAILED: CMakeFiles/install.util 
cd /build/llvm-git-gentz/src/_build32_final && /usr/bin/cmake -P cmake_install.cmake
ninja: build stopped: subcommand failed.

After some investigation, I found that there is no /build/llvm-git-gentz/src/_build32_final/./lib/python3.7 but there is an /build/llvm-git-gentz/src/_build32_final/./lib32/python3.7.

I've temporary solved this problem by applying the following patch:

diff --git a/lldb/scripts/CMakeLists.txt b/lldb/scripts/CMakeLists.txt
index 9de96ef5565..7d346deee7f 100644
--- a/lldb/scripts/CMakeLists.txt
+++ b/lldb/scripts/CMakeLists.txt
@@ -47,7 +47,7 @@ if(NOT LLDB_BUILD_FRAMEWORK)
   if(CMAKE_HOST_SYSTEM_NAME MATCHES "Windows")
     set(swig_python_subdir Lib/site-packages)
   else()
-    set(swig_python_subdir lib/python${PYTHON_VERSION_MAJOR}.${PYTHON_VERSION_MINOR})
+    set(swig_python_subdir lib${LLVM_LIBDIR_SUFFIX}/python${PYTHON_VERSION_MAJOR}.${PYTHON_VERSION_MINOR})
   endif()
 
   set(SWIG_PYTHON_DIR ${CMAKE_BINARY_DIR}/${CMAKE_CFG_INTDIR}/${swig_python_subdir})

I believe solving this problem correctly would also involve updating https://github.com/python/cpython/blob/3.8/Lib/distutils/sysconfig.py#L150 to account for ${LLVM_LIBDIR_SUFFIX}, however, I've never used the script before and am unfamiliar with it's purpose.

@hhb Could you please revisit this revision so that it doesn't break builds with a modified LLVM_LIBDIR_SUFFIX and revert this revision until then?

[hhb]

In D67583#1673356, @ZeGentzy wrote:

Hello folks,

These changes break running ninja install when building with -D LLVM_LIBDIR_SUFFIX=32.

[...]
-- Up-to-date: /build/llvm-git-gentz/pkg/lib32-llvm-git-gentz/opt/llvm32/include
-- Up-to-date: /build/llvm-git-gentz/pkg/lib32-llvm-git-gentz/opt/llvm32/include/lldb
-- Up-to-date: /build/llvm-git-gentz/pkg/lib32-llvm-git-gentz/opt/llvm32/include/lldb/Host
-- Installing: /build/llvm-git-gentz/pkg/lib32-llvm-git-gentz/opt/llvm32/include/lldb/Host/Config.h
CMake Error at tools/lldb/scripts/cmake_install.cmake:41 (file):
  file INSTALL cannot find
  "/build/llvm-git-gentz/src/_build32_final/./lib/python3.7".
Call Stack (most recent call first):
  tools/lldb/cmake_install.cmake:50 (include)
  tools/cmake_install.cmake:50 (include)
  cmake_install.cmake:68 (include)


FAILED: CMakeFiles/install.util 
cd /build/llvm-git-gentz/src/_build32_final && /usr/bin/cmake -P cmake_install.cmake
ninja: build stopped: subcommand failed.

After some investigation, I found that there is no /build/llvm-git-gentz/src/_build32_final/./lib/python3.7 but there is an /build/llvm-git-gentz/src/_build32_final/./lib32/python3.7.

I've temporary solved this problem by applying the following patch:

diff --git a/lldb/scripts/CMakeLists.txt b/lldb/scripts/CMakeLists.txt
index 9de96ef5565..7d346deee7f 100644
--- a/lldb/scripts/CMakeLists.txt
+++ b/lldb/scripts/CMakeLists.txt
@@ -47,7 +47,7 @@ if(NOT LLDB_BUILD_FRAMEWORK)
   if(CMAKE_HOST_SYSTEM_NAME MATCHES "Windows")
     set(swig_python_subdir Lib/site-packages)
   else()
-    set(swig_python_subdir lib/python${PYTHON_VERSION_MAJOR}.${PYTHON_VERSION_MINOR})
+    set(swig_python_subdir lib${LLVM_LIBDIR_SUFFIX}/python${PYTHON_VERSION_MAJOR}.${PYTHON_VERSION_MINOR})
   endif()
 
   set(SWIG_PYTHON_DIR ${CMAKE_BINARY_DIR}/${CMAKE_CFG_INTDIR}/${swig_python_subdir})

I believe solving this problem correctly would also involve updating https://github.com/python/cpython/blob/3.8/Lib/distutils/sysconfig.py#L150 to account for ${LLVM_LIBDIR_SUFFIX}, however, I've never used the script before and am unfamiliar with it's purpose.

@hhb Could you please revisit this revision so that it doesn't break builds with a modified LLVM_LIBDIR_SUFFIX and revert this revision until then?

Hmm that's interesting... In my case the issue is opposite: LLVM_LIBDIR_SUFFIX=64 but the file is in lib/ .

I don't think we can change python to fix this. But I'm curious how the suffix is applied. Let me try LLVM_LIBDIR_SUFFIX=32...

[hhb]

In D67583#1674430, @hhb wrote:

In D67583#1673356, @ZeGentzy wrote:

Hello folks,

These changes break running ninja install when building with -D LLVM_LIBDIR_SUFFIX=32.

[...]
-- Up-to-date: /build/llvm-git-gentz/pkg/lib32-llvm-git-gentz/opt/llvm32/include
-- Up-to-date: /build/llvm-git-gentz/pkg/lib32-llvm-git-gentz/opt/llvm32/include/lldb
-- Up-to-date: /build/llvm-git-gentz/pkg/lib32-llvm-git-gentz/opt/llvm32/include/lldb/Host
-- Installing: /build/llvm-git-gentz/pkg/lib32-llvm-git-gentz/opt/llvm32/include/lldb/Host/Config.h
CMake Error at tools/lldb/scripts/cmake_install.cmake:41 (file):
  file INSTALL cannot find
  "/build/llvm-git-gentz/src/_build32_final/./lib/python3.7".
Call Stack (most recent call first):
  tools/lldb/cmake_install.cmake:50 (include)
  tools/cmake_install.cmake:50 (include)
  cmake_install.cmake:68 (include)


FAILED: CMakeFiles/install.util 
cd /build/llvm-git-gentz/src/_build32_final && /usr/bin/cmake -P cmake_install.cmake
ninja: build stopped: subcommand failed.

After some investigation, I found that there is no /build/llvm-git-gentz/src/_build32_final/./lib/python3.7 but there is an /build/llvm-git-gentz/src/_build32_final/./lib32/python3.7.

I've temporary solved this problem by applying the following patch:

diff --git a/lldb/scripts/CMakeLists.txt b/lldb/scripts/CMakeLists.txt
index 9de96ef5565..7d346deee7f 100644
--- a/lldb/scripts/CMakeLists.txt
+++ b/lldb/scripts/CMakeLists.txt
@@ -47,7 +47,7 @@ if(NOT LLDB_BUILD_FRAMEWORK)
   if(CMAKE_HOST_SYSTEM_NAME MATCHES "Windows")
     set(swig_python_subdir Lib/site-packages)
   else()
-    set(swig_python_subdir lib/python${PYTHON_VERSION_MAJOR}.${PYTHON_VERSION_MINOR})
+    set(swig_python_subdir lib${LLVM_LIBDIR_SUFFIX}/python${PYTHON_VERSION_MAJOR}.${PYTHON_VERSION_MINOR})
   endif()
 
   set(SWIG_PYTHON_DIR ${CMAKE_BINARY_DIR}/${CMAKE_CFG_INTDIR}/${swig_python_subdir})

I believe solving this problem correctly would also involve updating https://github.com/python/cpython/blob/3.8/Lib/distutils/sysconfig.py#L150 to account for ${LLVM_LIBDIR_SUFFIX}, however, I've never used the script before and am unfamiliar with it's purpose.

@hhb Could you please revisit this revision so that it doesn't break builds with a modified LLVM_LIBDIR_SUFFIX and revert this revision until then?

Hmm that's interesting... In my case the issue is opposite: LLVM_LIBDIR_SUFFIX=64 but the file is in lib/ .

I don't think we can change python to fix this. But I'm curious how the suffix is applied. Let me try LLVM_LIBDIR_SUFFIX=32...

Can you share your full compile command and environment? I tried to build on Ubuntu (Python 2.7.16) with the latest llvm source code:

$ cmake /src/llvm-project/llvm -DLLVM_LIBDIR_SUFFIX=32 "-DLLVM_ENABLE_PROJECTS=clang;lldb"  -DCMAKE_INSTALL_PREFIX=/src/llvm-install -DLLDB_DISABLE_LIBEDIT=ON -GNinja
$ ninja
...
$ find -name site-packages
./lib/python2.7/site-packages

Or with python 3.6.8:

$ cmake /src/llvm-project/llvm -DLLVM_LIBDIR_SUFFIX=32 "-DLLVM_ENABLE_PROJECTS=clang;lldb"  -DCMAKE_INSTALL_PREFIX=/src/llvm-install -DLLDB_DISABLE_LIBEDIT=ON -GNinja -DPYTHON_EXECUTABLE=/usr/bin/python3.6 -DPYTHON_INCLUDE_DIR=/usr/include/python3.6 -DPYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython3.6m.so
$ ninja
...
$ find -name site-packages
./lib/python3.6/site-packages
$ ls lib32/python*
zsh: no matches found: lib32/python*

[ZeGentzy]

That's odd....

Can you share your full compile command and environment?

I'm using this PKGBUILD: https://github.com/ZeGentzy/aur-buildscripts/blob/master/llvm-git-gentz/PKGBUILD#L376

The commands being run are these:

export PKG_CONFIG_PATH="/usr/lib32/pkgconfig"
export CFLAGS="$CFLAGS -m32"
export CXXFLAGS="$CXXFLAGS -m32"
export LDFLAGS="$LDFLAGS -m32"

cmake /build/llvm-git-gentz/src/llvm-project32/llvm -G Ninja \
    -D CMAKE_BUILD_TYPE="Release" \
    -D LLVM_DEFAULT_TARGET_TRIPLE="i686-pc-linux-gnu" \
    -D LLVM_LIBDIR_SUFFIX=32 \
    -D LLVM_TARGET_ARCH:STRING=i686 \
    -D LLVM_CONFIG="/usr/bin/llvm-config32" \
    -D COMPILER_RT_BUILD_LIBFUZZER=OFF \
    -D LLVM_ENABLE_BINDINGS=OFF \
    -D PYTHON_EXECUTABLE=/usr/bin/python-32 \
    -D CMAKE_CXX_COMPILER="/usr/bin/clang++" \
    -D CMAKE_C_COMPILER="/usr/bin/clang" \
    -D LLVM_USE_LINKER="/usr/bin/ld.lld" \
    -D CMAKE_RANLIB="/usr/bin/llvm-ranlib" \
    -D CMAKE_AR="/usr/bin/llvm-ar" \
    -D LLVM_ENABLE_LTO=OFF \
    -D LLVM_BUILD_LLVM_DYLIB=ON \
    -D LLVM_LINK_LLVM_DYLIB=ON \
    -D LLVM_ENABLE_PROJECTS="clang;compiler-rt;lld;lldb;clang-tools-extra" \
    -D LLVM_APPEND_VC_REV=ON \
    -D LLVM_HOST_TRIPLE="x86_64-pc-linux-gnu" \
    -D LLVM_ENABLE_RTTI=ON \
    -D LLVM_ENABLE_FFI=ON \
    -D FFI_INCLUDE_DIR="/usr/lib32/libffi-3.2.1/include" \
    -D LLVM_BUILD_TESTS=ON \
    -D LLVM_VERSION_SUFFIX= \
    -D CMAKE_POLICY_DEFAULT_CMP0075=NEW \
    -D LLVM_OPTIMIZED_TABLEGEN=ON \
    -D LLVM_CCACHE_BUILD=ON \
    -D CMAKE_POLICY_DEFAULT_CMP0075=NEW \
    -D CMAKE_C_FLAGS="-march=native -O3 -pipe -ffast-math -funroll-loops -fstack-protector-strong -fno-plt -mno-xop -mno-fma4 -mno-tbm -m32" \
    -D CMAKE_CXX_FLAGS="-march=native -O3 -pipe -ffast-math -funroll-loops -fstack-protector-strong -fno-plt -mno-xop -mno-fma4 -mno-tbm -m32" \
    -D LLVM_TARGETS_TO_BUILD="AMDGPU;X86" \
    -D LLVM_BINUTILS_INCDIR=/usr/include \
    -D LLVM_EXTERNAL_LIT="/usr/bin/lit" \
    -D LLVM_BUILD_DOCS=OFF \
    -D LLVM_INCLUDE_DOCS=OFF \
    -D LLVM_ENABLE_SPHINX=OFF \
    -D LLVM_ENABLE_DOXYGEN=OFF \
    -D LLVM_INSTALL_UTILS=ON \
    -D LLVM_PARALLEL_LINK_JOBS=1 \
    -D CMAKE_INSTALL_PREFIX="/opt/llvm32"
ninja -j4 all
DESTDIR=/build/llvm-git-gentz/pkg/lib32-llvm-git-gentz ninja -j4 install

A lot... err, most... of the things I cargo culted from other sources.

[hhb]

I don't have the right environment to try your command. But I didn't see anything obviously problematic.

Can you try the following:

cd /src/llvm-project/lldb/scripts/Python

Then run the following python program:

import sys
sys.path.append('/src/llvm-project/lldb/scripts/Python')
import finishSwigPythonLLDB
vDictArgs = {"--srcRoot": "/src/llvm-project/lldb",
             "--targetDir": "/src/out/./lib32",
             "--cfgBldDir": "/src/out/tools/lldb/scripts",
             "--prefix": "/src/out",
             "--cmakeBuildConfiguration": ".",
             "--lldbLibDir": "lib32",
             "-m": None,
}
finishSwigPythonLLDB.get_framework_python_dir(vDictArgs)

For me the output is:

(True, '/src/out/./lib/python2.7/site-packages/lldb', '')

Notice that the lib part don't have 32 suffix, even we specified that in parameters.

I kind of feeling that we should change this script to simply return targetDir. There's no good reason to use get_python_lib() and depend on python's implementation. But I don't know enough to make this decision...

[hhb]

+deepak2427 since this behavior is introduced in https://github.com/llvm/llvm-project/commit/9b35cf52d2a88cda5167c9638696adce5b152720

[ZeGentzy]

In D67583#1674912, @hhb wrote:

I don't have the right environment to try your command. But I didn't see anything obviously problematic.

Can you try the following:

cd /src/llvm-project/lldb/scripts/Python

Then run the following python program:

import sys
sys.path.append('/src/llvm-project/lldb/scripts/Python')
import finishSwigPythonLLDB
vDictArgs = {"--srcRoot": "/src/llvm-project/lldb",
             "--targetDir": "/src/out/./lib32",
             "--cfgBldDir": "/src/out/tools/lldb/scripts",
             "--prefix": "/src/out",
             "--cmakeBuildConfiguration": ".",
             "--lldbLibDir": "lib32",
             "-m": None,
}
finishSwigPythonLLDB.get_framework_python_dir(vDictArgs)

For me the output is:

(True, '/src/out/./lib/python2.7/site-packages/lldb', '')

Notice that the lib part don't have 32 suffix, even we specified that in parameters.

I kind of feeling that we should change this script to simply return targetDir. There's no good reason to use get_python_lib() and depend on python's implementation. But I don't know enough to make this decision...

Unfortunately, the script does not seam to be functional:

$ python a.py         
Traceback (most recent call last):
  File "a.py", line 3, in <module>
    import finishSwigPythonLLDB
  File "/home/gentz/Documents/aur/gfx/backups/second/llvm-git-gentz/llvm-git-gentz/src/llvm-project32/lldb/scripts/Python/finishSwigPythonLLDB.py", line 47, in <module>
    import utilsOsType      # Determine the OS type this script is running on
ModuleNotFoundError: No module named 'utilsOsType'

$ python2 a.py                  
Traceback (most recent call last):
  File "a.py", line 3, in <module>
    import finishSwigPythonLLDB
  File "/home/gentz/Documents/aur/gfx/backups/second/llvm-git-gentz/llvm-git-gentz/src/llvm-project32/lldb/scripts/Python/finishSwigPythonLLDB.py", line 47, in <module>
    import utilsOsType      # Determine the OS type this script is running on
ImportError: No module named utilsOsType

$ pwd         
/home/gentz/Documents/aur/gfx/backups/second/llvm-git-gentz/llvm-git-gentz/src/llvm-project32/lldb/scripts/Python

[hhb]

Oh sorry my mistake. The current directory should be llvm-project/lldb/scripts. I.e.

cd /src/llvm-project/lldb/scripts

[ZeGentzy]

In D67583#1675415, @hhb wrote:

Oh sorry my mistake. The current directory should be llvm-project/lldb/scripts. I.e.

cd /src/llvm-project/lldb/scripts

Maybe I'm doing something wrong, but that does not work either.

$ python a.py               
Traceback (most recent call last):
  File "a.py", line 3, in <module>
    import finishSwigPythonLLDB
ModuleNotFoundError: No module named 'finishSwigPythonLLDB'

$ python2 a.py
Traceback (most recent call last):
  File "a.py", line 3, in <module>
    import finishSwigPythonLLDB
ImportError: No module named finishSwigPythonLLDB

$ pwd         
/home/gentz/Documents/aur/gfx/backups/second/llvm-git-gentz/llvm-git-gentz/src/llvm-project32/lldb/scripts

$ cat a.py                        
import sys
sys.path.append('/src/llvm-project/lldb/scripts/Python')
import finishSwigPythonLLDB
vDictArgs = {"--srcRoot": "/src/llvm-project/lldb",
             "--targetDir": "/src/out/./lib32",
             "--cfgBldDir": "/src/out/tools/lldb/scripts",
             "--prefix": "/src/out",
             "--cmakeBuildConfiguration": ".",
             "--lldbLibDir": "lib32",
             "-m": None,
}
finishSwigPythonLLDB.get_framework_python_dir(vDictArgs)

[hhb]

I'm making a change in finishSwigPythonLLDB.py to write directly to targetDir. This way we don't have the problem to guess where the files are written.

[hhb]

In D67583#1675697, @ZeGentzy wrote:

In D67583#1675415, @hhb wrote:

Oh sorry my mistake. The current directory should be llvm-project/lldb/scripts. I.e.

cd /src/llvm-project/lldb/scripts

Maybe I'm doing something wrong, but that does not work either.

$ python a.py               
Traceback (most recent call last):
  File "a.py", line 3, in <module>
    import finishSwigPythonLLDB
ModuleNotFoundError: No module named 'finishSwigPythonLLDB'

$ python2 a.py
Traceback (most recent call last):
  File "a.py", line 3, in <module>
    import finishSwigPythonLLDB
ImportError: No module named finishSwigPythonLLDB

$ pwd         
/home/gentz/Documents/aur/gfx/backups/second/llvm-git-gentz/llvm-git-gentz/src/llvm-project32/lldb/scripts

$ cat a.py                        
import sys
sys.path.append('/src/llvm-project/lldb/scripts/Python')
import finishSwigPythonLLDB
vDictArgs = {"--srcRoot": "/src/llvm-project/lldb",
             "--targetDir": "/src/out/./lib32",
             "--cfgBldDir": "/src/out/tools/lldb/scripts",
             "--prefix": "/src/out",
             "--cmakeBuildConfiguration": ".",
             "--lldbLibDir": "lib32",
             "-m": None,
}
finishSwigPythonLLDB.get_framework_python_dir(vDictArgs)

My theory now is that your python has a different implementation of get_python_lib(). After all we should not guess what python would do. I'm making a change.
If you still interested in testing that, you can fix the path in a.py...

[ZeGentzy]

In D67583#1675714, @hhb wrote:

My theory now is that your python has a different implementation of get_python_lib(). After all we should not guess what python would do. I'm making a change.
If you still interested in testing that, you can fix the path in a.py...

Oh, derp, I completely missed that. I get (True, '/src/out/./lib/python3.7/site-packages/lldb', '') for python and (True, '/src/out/./lib/python2.7/site-packages/lldb', '') for python2.

EDIT:

$ cat a.py                       
import sys
sys.path.append('/home/gentz/Documents/aur/gfx/backups/second/llvm-git-gentz/llvm-git-gentz/src/llvm-project32/lldb/scripts/Python')
import finishSwigPythonLLDB
vDictArgs = {"--srcRoot": "/src/llvm-project/lldb",
             "--targetDir": "/src/out/./lib32",
             "--cfgBldDir": "/src/out/tools/lldb/scripts",
             "--prefix": "/src/out",
             "--cmakeBuildConfiguration": ".",
             "--lldbLibDir": "lib32",
             "-m": None,
}
print(finishSwigPythonLLDB.get_framework_python_dir(vDictArgs))

[labath]

I could be off mark here, but I do recall some people trying to get this code to work in environments where python and lldb(llvm) are configured with different libdirs (so LLVM_LIBDIR_SUFFIX in llvm, and configure --prefix=X in python). Is that the case you're running into here?

If that is something we want to support (personally, I would very much like to just declare that "unsupported", but maybe we don't have that luxury). Then we'll need to be super careful about distinguishing the _llvm_ libdir name (i.e. the place where liblldb and friends should go to) from the _python_ libdir (the destination for the site-packages stuff). I think the code already partially tries to do that, but I am not sure if it is really correct.

Since I'm already complaining about python paths, I'll add that the way we currently compute the python path is wrong for cross-compilation. (Because it will pick up the path from the host python, not the target one.) If we're going to be changing something here, it would be nice to fix that too (which, I hope, will also allow us avoid needing to keep multiple places in sync).

[hhb]

In D67583#1675889, @ZeGentzy wrote:

In D67583#1675714, @hhb wrote:

My theory now is that your python has a different implementation of get_python_lib(). After all we should not guess what python would do. I'm making a change.
If you still interested in testing that, you can fix the path in a.py...

Oh, derp, I completely missed that. I get (True, '/src/out/./lib/python3.7/site-packages/lldb', '') for python and (True, '/src/out/./lib/python2.7/site-packages/lldb', '') for python2.

EDIT:

$ cat a.py                       
import sys
sys.path.append('/home/gentz/Documents/aur/gfx/backups/second/llvm-git-gentz/llvm-git-gentz/src/llvm-project32/lldb/scripts/Python')
import finishSwigPythonLLDB
vDictArgs = {"--srcRoot": "/src/llvm-project/lldb",
             "--targetDir": "/src/out/./lib32",
             "--cfgBldDir": "/src/out/tools/lldb/scripts",
             "--prefix": "/src/out",
             "--cmakeBuildConfiguration": ".",
             "--lldbLibDir": "lib32",
             "-m": None,
}
print(finishSwigPythonLLDB.get_framework_python_dir(vDictArgs))

That's really... weird... I don't know how your files end up in the lib32 directory. Anyway this change is reverted in https://reviews.llvm.org/D67781.

[hhb]

In D67583#1676531, @labath wrote:

I could be off mark here, but I do recall some people trying to get this code to work in environments where python and lldb(llvm) are configured with different libdirs (so LLVM_LIBDIR_SUFFIX in llvm, and configure --prefix=X in python). Is that the case you're running into here?

If that is something we want to support (personally, I would very much like to just declare that "unsupported", but maybe we don't have that luxury). Then we'll need to be super careful about distinguishing the _llvm_ libdir name (i.e. the place where liblldb and friends should go to) from the _python_ libdir (the destination for the site-packages stuff). I think the code already partially tries to do that, but I am not sure if it is really correct.

Since I'm already complaining about python paths, I'll add that the way we currently compute the python path is wrong for cross-compilation. (Because it will pick up the path from the host python, not the target one.) If we're going to be changing something here, it would be nice to fix that too (which, I hope, will also allow us avoid needing to keep multiple places in sync).

I also realized that this is totally wrong for cross-compilation. We are doing cross compilation to windows with MinGW so I need to fix that.

Actually I don't know why we need get_python_lib(). From what I can see now, the 3 paths should match:

1. finishSwigPythonLLDB.py write necessary files to ${CMAKE_BINARY_DIR}/${CMAKE_CFG_INTDIR}/distutils.sysconfig.get_python_lib()

2. lldb/scripts/CMakeLists.txt copies them to lib${LLVM_LIBDIR_SUFFIX} (if not xcode)

3. ScriptInterpreterPython.cpp adds the right path into PYTHONPATH.

As long as these 3 paths matches, we are good. Then why not simply put everything into ${liblldb path}/site-packages for all platforms?

(I'm gonna make a change to test that. But let me know if anything is obviously wrong.

[ZeGentzy]

In D67583#1676531, @labath wrote:

I could be off mark here, but I do recall some people trying to get this code to work in environments where python and lldb(llvm) are configured with different libdirs (so LLVM_LIBDIR_SUFFIX in llvm, and configure --prefix=X in python). Is that the case you're running into here?

I think not. I'm cross compiling llvm, getting it to build a 32-bit llvm for my 64-bit host, so I can install it along side my 64-bit llvm and use it with my 32-bit mesa.

I'm building two pythons, python2s, llvms and mesas, one for 32-bits and one for 64-bits.

Here is my 32-bit python2's configured options:

--prefix=/usr 
--enable-shared
--with-threads
--enable-optimizations
--with-lto
--enable-ipv6
--with-system-expat
--with-dbmliborder=gdbm:ndbm
--with-system-ffi
--without-ensurepip
--libdir=/usr/lib32
--libexecdir=/usr/lib32
--includedir="/usr/lib32/python${_pybasever}/include"
--exec_prefix="/usr/lib32/python${_pybasever}/"
--bindir=/usr/bin
--sbindir=/usr/bin
--with-suffix='-32'

My 32-bit python, the one in use by llvm for it's 32-bit builds, was configured with the following options:

--prefix=/usr 
--enable-shared
--with-threads
--with-computed-gotos
--enable-optimizations
--with-lto
--enable-ipv6
--with-system-expat
--with-dbmliborder=gdbm:ndbm
--with-system-ffi
--with-system-libmpdec
--enable-loadable-sqlite-extensions
--without-ensurepip
--libdir=/usr/lib32
--libexecdir=/usr/lib32
--includedir=/usr/lib32/python${_pybasever}m/include
--exec_prefix=/usr/lib32/python${_pybasever}/
--bindir=/usr/bin
--sbindir=/usr/bin
--with-suffix='-32'

This is done so my 32-bit python stuff does not conflict with the 64-bit python stuff.

---

Actually, thinking about it, I should have probably run your scripts with the 32-bit pythons, as those are what llvm uses:

[root@GENTZ-PC-NEW2 scripts]# python a.py
(True, '/src/out/./lib/python3.7/site-packages/lldb', '')
[root@GENTZ-PC-NEW2 scripts]# python-32 a.py
(True, '/src/out/./lib32/python3.7/site-packages/lldb', '')
[root@GENTZ-PC-NEW2 scripts]# python2-32 a.py
(True, '/src/out/./lib32/python2.7/site-packages/lldb', '')
[root@GENTZ-PC-NEW2 scripts]# python2 a.py
(True, '/src/out/./lib/python2.7/site-packages/lldb', '')
[root@GENTZ-PC-NEW2 scripts]# file /usr/lib[0-9]*/python*/ 
/usr/lib32/python2.7/:  directory
/usr/lib32/python3.7/:  directory
/usr/lib32/python3.7m/: directory
/usr/lib64/python2.7/:  directory
/usr/lib64/python3.7/:  directory

[labath]

In D67583#1677141, @hhb wrote:

In D67583#1676531, @labath wrote:

Since I'm already complaining about python paths, I'll add that the way we currently compute the python path is wrong for cross-compilation. (Because it will pick up the path from the host python, not the target one.) If we're going to be changing something here, it would be nice to fix that too (which, I hope, will also allow us avoid needing to keep multiple places in sync).

I also realized that this is totally wrong for cross-compilation. We are doing cross compilation to windows with MinGW so I need to fix that.

Actually I don't know why we need get_python_lib(). From what I can see now, the 3 paths should match:

1. finishSwigPythonLLDB.py write necessary files to ${CMAKE_BINARY_DIR}/${CMAKE_CFG_INTDIR}/distutils.sysconfig.get_python_lib()

2. lldb/scripts/CMakeLists.txt copies them to lib${LLVM_LIBDIR_SUFFIX} (if not xcode)

3. ScriptInterpreterPython.cpp adds the right path into PYTHONPATH.

As long as these 3 paths matches, we are good. Then why not simply put everything into ${liblldb path}/site-packages for all platforms?

(I'm gonna make a change to test that. But let me know if anything is obviously wrong.

This will work for the python-in-lldb case. However, the idea is lldb can also be used as python package, from any python application (by just typing import lldb). For that to work, we need to install lldb python stuff to a location that will be searched for by python in its default configuration.

[mgorny]

In D67583#1678505, @labath wrote:

This will work for the python-in-lldb case. However, the idea is lldb can also be used as python package, from any python application (by just typing import lldb). For that to work, we need to install lldb python stuff to a location that will be searched for by python in its default configuration.

I agree. Hence, I sent D67890 yesterday.

lldb/trunk/scripts/CMakeLists.txt

add_custom_command(
   COMMENT "Python script building LLDB Python wrapper")
 
 add_custom_target(swig_wrapper ALL DEPENDS
   ${CMAKE_CURRENT_BINARY_DIR}/LLDBWrapPython.cpp
   ${CMAKE_CURRENT_BINARY_DIR}/lldb.py
 )
 
 if(NOT LLDB_BUILD_FRAMEWORK)
-  if(CMAKE_SYSTEM_NAME MATCHES "Windows")
-    set(swig_python_subdir site-packages)
+  # The path here should match the result of python function
+  # distutils.sysconfig.get_python_lib().
+  if(CMAKE_HOST_SYSTEM_NAME MATCHES "Windows")
+    set(swig_python_subdir Lib/site-packages)
   else()
-    set(swig_python_subdir python${PYTHON_VERSION_MAJOR}.${PYTHON_VERSION_MINOR})
+    set(swig_python_subdir lib/python${PYTHON_VERSION_MAJOR}.${PYTHON_VERSION_MINOR})
   endif()
 
-  set(SWIG_PYTHON_DIR ${LLVM_LIBRARY_OUTPUT_INTDIR}/${swig_python_subdir})
+  set(SWIG_PYTHON_DIR ${CMAKE_BINARY_DIR}/${CMAKE_CFG_INTDIR}/${swig_python_subdir})
   set(SWIG_INSTALL_DIR lib${LLVM_LIBDIR_SUFFIX})
 
   # Install the LLDB python module
   install(DIRECTORY ${SWIG_PYTHON_DIR} DESTINATION ${SWIG_INSTALL_DIR})
 endif()