[lldb] fix -print-script-interpreter-info on windows

[lawrence_danna]

Summary

Apparently "{sys.prefix}/bin/python3" isn't where you find the
python interpreter on windows, so the test I wrote for
-print-script-interpreter-info is failing.

We can't rely on sys.executable at runtime, because that will point
to lldb.exe not python.exe.

We can't just record sys.executable from build time, because python
could have been moved to a different location.

But it should be OK to apply relative path from sys.prefix to sys.executable
from build-time to the sys.prefix at runtime.

[stella.stamenova]

lldb/source/Plugins/ScriptInterpreter/Python/ScriptInterpreterPython.cpp

413

Does sys.executable not work for some reason?

432

Python on windows could have a version too (e.g. python3.exe)

[lawrence_danna]

lldb/source/Plugins/ScriptInterpreter/Python/ScriptInterpreterPython.cpp

413

Yes, unfortunately it will point to lldb.exe, not python.exe

432

Under what circumstances? How do we determine if it does? The one from python.org doesn't.

[lawrence_danna]

comments

[stella.stamenova]

lldb/source/Plugins/ScriptInterpreter/Python/ScriptInterpreterPython.cpp

432

I am not sure. I *think* it depends on whether you have python 2 and python 3 installed at the same time.

[lawrence_danna]

lldb/source/Plugins/ScriptInterpreter/Python/ScriptInterpreterPython.cpp

432

yikes, OK I think I have a better idea.....

[lawrence_danna]

maybe this approach is better

[lawrence_danna]

OK this should work in all configs I think, as long as sys.executable ultimately points at something under sys.prefix

[lawrence_danna]

fix cmake error message condition for cross-compile

[lawrence_danna]

cleanup cmake usage

[stella.stamenova]

Can you please disable the test on Windows or back your change out until there's a fix? There are already other failures that are getting missed because the bot is red.

[lawrence_danna]

In D113650#3128674, @stella.stamenova wrote:

Can you please disable the test on Windows or back your change out until there's a fix? There are already other failures that are getting missed because the bot is red.

done.

[lawrence_danna]

revert change that disabled the test

[lawrence_danna]

cmake fix

[JDevlieghere]

LGTM if the Windows bot is happy :-)

[aprantl]

This seems to have broken the green dragon incremental bot: https://green.lab.llvm.org/green/view/LLDB/job/lldb-cmake/38387/

[lawrence_danna]

In D113650#3136407, @aprantl wrote:

This seems to have broken the green dragon incremental bot: https://green.lab.llvm.org/green/view/LLDB/job/lldb-cmake/38387/

(edited)

ah dang you’re not cross compiling you shouldn’t get that error. i’ll fix it.

[lawrence_danna]

In D113650#3136407, @aprantl wrote:

This seems to have broken the green dragon incremental bot: https://green.lab.llvm.org/green/view/LLDB/job/lldb-cmake/38387/

looks like its fixed now.

[clayborg]

I can't build on macOS now. I checked out the source code and tried to do:

cmake -G Ninja -DCMAKE_BUILD_TYPE:STRING=Debug -DLLVM_ENABLE_ASSERTIONS:BOOL=TRUE -DLLVM_ENABLE_PROJECTS='clang;libcxx;libcxxabi;lldb' -DLLDB_BUILD_FRAMEWORK:BOOL=TRUE -DLLDB_USE_SYSTEM_DEBUGSERVER=ON -DLLDB_EDITLINE_USE_WCHAR=0 -DLLDB_ENABLE_LIBEDIT:BOOL=TRUE -DLLDB_ENABLE_CURSES:BOOL=TRUE -DLLDB_ENABLE_PYTHON:BOOL=TRUE -DLLDB_ENABLE_LIBXML2:BOOL=TRUE -DLLDB_ENABLE_LUA:BOOL=FALSE ../llvm-project/llvm

The error is:

Traceback (most recent call last):
  File "/Users/gclayton/Documents/src/lldb/clean/llvm-project/lldb/bindings/python/get-python-config.py", line 60, in <module>
    main()
  File "/Users/gclayton/Documents/src/lldb/clean/llvm-project/lldb/bindings/python/get-python-config.py", line 33, in main
    print(relpath_nodots(sysconfig.get_path("platlib"), sys.prefix))
  File "/Users/gclayton/Documents/src/lldb/clean/llvm-project/lldb/bindings/python/get-python-config.py", line 14, in relpath_nodots
    raise ValueError(f"{path} is not under {base}")
ValueError: /Library/Python/3.8/site-packages is not under /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8
CMake Error at /Users/gclayton/Documents/src/lldb/clean/llvm-project/lldb/source/Plugins/ScriptInterpreter/Python/CMakeLists.txt:2 (message):
  LLDB_PYTHON_RELATIVE_PATH is not set.

[clayborg]

if I try to manually set the python3 executable with:

-DPYTHON_EXECUTABLE=/Applications/Xcode.app/Contents/Developer/usr/bin/python3

I get the following error:

[cmake] CMake Error at /Users/gclayton/Documents/src/lldb/clean/llvm-project/lldb/CMakeLists.txt:53 (message):
[cmake]   Crosscompiling LLDB with Python requires manually setting
[cmake]   LLDB_PYTHON_RELATIVE_PATH.

[clayborg]

I am not cross compiling in this case.

[lawrence_danna]

@clayborg

Does this fix it for you?

https://reviews.llvm.org/D114973

[clayborg]

In D113650#3168812, @lawrence_danna wrote:

@clayborg

Does this fix it for you?

https://reviews.llvm.org/D114973

Yes it does! Thanks

[nealsid]

In D113650#3168622, @clayborg wrote:

I can't build on macOS now. I checked out the source code and tried to do:

cmake -G Ninja -DCMAKE_BUILD_TYPE:STRING=Debug -DLLVM_ENABLE_ASSERTIONS:BOOL=TRUE -DLLVM_ENABLE_PROJECTS='clang;libcxx;libcxxabi;lldb' -DLLDB_BUILD_FRAMEWORK:BOOL=TRUE -DLLDB_USE_SYSTEM_DEBUGSERVER=ON -DLLDB_EDITLINE_USE_WCHAR=0 -DLLDB_ENABLE_LIBEDIT:BOOL=TRUE -DLLDB_ENABLE_CURSES:BOOL=TRUE -DLLDB_ENABLE_PYTHON:BOOL=TRUE -DLLDB_ENABLE_LIBXML2:BOOL=TRUE -DLLDB_ENABLE_LUA:BOOL=FALSE ../llvm-project/llvm

Random drive-by comment - any particular reason you compile without WCHAR support in libedit? I have a patch that removes the conditional support from our Editline class (actually it was committed and had to be reverted :P). But I was under the impression that nobody was using the non-WCHAR support anymore. Thanks.

[clayborg]

In D113650#3173914, @nealsid wrote:

In D113650#3168622, @clayborg wrote:

I can't build on macOS now. I checked out the source code and tried to do:

cmake -G Ninja -DCMAKE_BUILD_TYPE:STRING=Debug -DLLVM_ENABLE_ASSERTIONS:BOOL=TRUE -DLLVM_ENABLE_PROJECTS='clang;libcxx;libcxxabi;lldb' -DLLDB_BUILD_FRAMEWORK:BOOL=TRUE -DLLDB_USE_SYSTEM_DEBUGSERVER=ON -DLLDB_EDITLINE_USE_WCHAR=0 -DLLDB_ENABLE_LIBEDIT:BOOL=TRUE -DLLDB_ENABLE_CURSES:BOOL=TRUE -DLLDB_ENABLE_PYTHON:BOOL=TRUE -DLLDB_ENABLE_LIBXML2:BOOL=TRUE -DLLDB_ENABLE_LUA:BOOL=FALSE ../llvm-project/llvm

Random drive-by comment - any particular reason you compile without WCHAR support in libedit? I have a patch that removes the conditional support from our Editline class (actually it was committed and had to be reverted :P). But I was under the impression that nobody was using the non-WCHAR support anymore. Thanks.

No real reason. I would be fine if this went away. Not sure if there are any platforms that have an older libedit shared library that need to avoid bugs in the wide character implementation, but I don't work on any. So feel free to remove this and see how things go!

lldb/CMakeLists.txt

if(uppercase_CMAKE_BUILD_TYPE STREQUAL "DEBUG" )
   add_definitions(-DLLDB_CONFIGURATION_DEBUG)
 endif()
 
 if (WIN32)
   add_definitions(-D_ENABLE_EXTENDED_ALIGNED_STORAGE)
 endif()
 
 if (LLDB_ENABLE_PYTHON)
-  if (NOT CMAKE_CROSSCOMPILING)
+  set(cachestring_LLDB_PYTHON_RELATIVE_PATH
+    "Path where Python modules are installed, relative to install prefix")
+  set(cachestring_LLDB_PYTHON_EXE_RELATIVE_PATH
+    "Path to python interpreter exectuable, relative to install prefix")
+
+  foreach(var LLDB_PYTHON_RELATIVE_PATH LLDB_PYTHON_EXE_RELATIVE_PATH)
+    if(NOT DEFINED ${var} AND NOT CMAKE_CROSSCOMPILING)
       execute_process(
         COMMAND ${Python3_EXECUTABLE}
-          -c "import distutils.sysconfig; print(distutils.sysconfig.get_python_lib(True, False, ''))"
-      OUTPUT_VARIABLE LLDB_PYTHON_DEFAULT_RELATIVE_PATH
+          ${CMAKE_CURRENT_SOURCE_DIR}/bindings/python/get-python-config.py
+          ${var}
+        OUTPUT_VARIABLE value
         OUTPUT_STRIP_TRAILING_WHITESPACE)
-
-    file(TO_CMAKE_PATH ${LLDB_PYTHON_DEFAULT_RELATIVE_PATH} LLDB_PYTHON_DEFAULT_RELATIVE_PATH)
-  else ()
-    if ("${LLDB_PYTHON_RELATIVE_PATH}" STREQUAL "")
+      file(TO_CMAKE_PATH "${value}" value)
+      set(${var} ${value} CACHE STRING ${cachestring_${var}})
+    else()
+      if ("${${var}}" STREQUAL "")
         message(FATAL_ERROR
-        "Crosscompiling LLDB with Python requires manually setting
-        LLDB_PYTHON_RELATIVE_PATH.")
-    endif ()
-  endif ()
-
-  set(LLDB_PYTHON_RELATIVE_PATH ${LLDB_PYTHON_DEFAULT_RELATIVE_PATH}
-    CACHE STRING "Path where Python modules are installed, relative to install prefix")
+          "Crosscompiling LLDB with Python requires manually setting ${var}.")
+      endif()
+    endif()
+  endforeach()
 endif ()
 
 if (LLDB_ENABLE_LUA)
   find_program(Lua_EXECUTABLE lua5.3)
   set(LLDB_LUA_DEFAULT_RELATIVE_PATH "lib/lua/5.3")
   set(LLDB_LUA_RELATIVE_PATH ${LLDB_LUA_DEFAULT_RELATIVE_PATH}
     CACHE STRING "Path where Lua modules are installed, relative to install prefix")
 endif ()

lldb/bindings/python/get-python-config.py

+#!/usr/bin/env python3
+
+import os
+import sys
+import argparse
+import sysconfig
+import distutils.sysconfig
+
+
+def relpath_nodots(path, base):
+    rel = os.path.normpath(os.path.relpath(path, base))
+    assert not os.path.isabs(rel)
+    parts = rel.split(os.path.sep)
+    if parts and parts[0] == '..':
+        raise ValueError(f"{path} is not under {base}")
+    return rel
+
+def main():
+    parser = argparse.ArgumentParser(description="extract cmake variables from python")
+    parser.add_argument("variable_name")
+    args = parser.parse_args()
+    if args.variable_name == "LLDB_PYTHON_RELATIVE_PATH":
+        print(distutils.sysconfig.get_python_lib(True, False, ''))
+    elif args.variable_name == "LLDB_PYTHON_EXE_RELATIVE_PATH":
+        tried = list()
+        exe = sys.executable
+        while True:
+            try:
+                print(relpath_nodots(exe, sys.prefix))
+                break
+            except ValueError:
+                tried.append(exe)
+                if os.path.islink(exe):
+                    exe = os.path.join(os.path.dirname(exe), os.readlink(exe))
+                    continue
+                else:
+                    print("Could not find a relative path to sys.executable under sys.prefix", file=sys.stderr)
+                    for e in tried:
+                        print("tried:", e, file=sys.stderr)
+                    print("sys.prefix:", sys.prefix, file=sys.stderr)
+                    sys.exit(1)
+
+    else:
+        parser.error(f"unknown variable {args.variable_name}")
+
+if __name__ == '__main__':
+    main()
 No newline at end of file

lldb/source/Plugins/ScriptInterpreter/Python/CMakeLists.txt

 if(NOT LLDB_PYTHON_RELATIVE_PATH)
   message(FATAL_ERROR "LLDB_PYTHON_RELATIVE_PATH is not set.")
 endif()
 add_definitions(-DLLDB_PYTHON_RELATIVE_LIBDIR="${LLDB_PYTHON_RELATIVE_PATH}")
 
+if(NOT LLDB_PYTHON_EXE_RELATIVE_PATH)
+  message(FATAL_ERROR "LLDB_PYTHON_EXE_RELATIVE_PATH is not set.")
+endif()
+add_definitions(-DLLDB_PYTHON_EXE_RELATIVE_PATH="${LLDB_PYTHON_EXE_RELATIVE_PATH}")
+
+
 if (LLDB_ENABLE_LIBEDIT)
   list(APPEND LLDB_LIBEDIT_LIBS ${LibEdit_LIBRARIES})
 endif()
 
 add_lldb_library(lldbPluginScriptInterpreterPython PLUGIN
   PythonDataObjects.cpp
   PythonReadline.cpp
   ScriptInterpreterPython.cpp

lldb/source/Plugins/ScriptInterpreter/Python/ScriptInterpreterPython.cpp

#else
     ComputePythonDir(path);
 #endif
     spec.GetDirectory().SetString(path);
     return spec;
   }();
   return g_spec;
 }
 
-StructuredData::DictionarySP ScriptInterpreterPython::GetInterpreterInfo() {
-  GIL gil;
-  FileSpec python_dir_spec = GetPythonDir();
-  if (!python_dir_spec)
-    return nullptr;
-  PythonString python_dir(python_dir_spec.GetPath());
-  PythonDictionary info(PyInitialValue::Empty);
-  llvm::Error error = info.SetItem("lldb-pythonpath", python_dir);
-  if (error)
-    return nullptr;
-  static const char script[] = R"(
-def main(info):
-  import sys
-  import os
-  name = 'python' + str(sys.version_info.major)
-  info.update({
+static const char GetInterpreterInfoScript[] = R"(
+import sys
+
+def main(lldb_python_dir, python_exe_relative_path):
+  info = {
+    "lldb-pythonpath": lldb_python_dir,
     "language": "python",
     "prefix": sys.prefix,
-    "executable": os.path.join(sys.prefix, "bin", name),
-  })
+    "executable": os.path.join(sys.prefix, python_exe_relative_path)
+  }
   return info
 )";
-  PythonScript get_info(script);
-  auto info_json = unwrapIgnoringErrors(As<PythonDictionary>(get_info(info)));
+
+static const char python_exe_relative_path[] = LLDB_PYTHON_EXE_RELATIVE_PATH;
+
+StructuredData::DictionarySP ScriptInterpreterPython::GetInterpreterInfo() {
+  GIL gil;
+  FileSpec python_dir_spec = GetPythonDir();
+  if (!python_dir_spec)
+    return nullptr;
+  PythonScript get_info(GetInterpreterInfoScript);
+  auto info_json = unwrapIgnoringErrors(
+      As<PythonDictionary>(get_info(PythonString(python_dir_spec.GetPath()),
+                                    PythonString(python_exe_relative_path))));
   if (!info_json)
     return nullptr;
   return info_json.CreateStructuredDictionary();
 }
 
 void ScriptInterpreterPython::SharedLibraryDirectoryHelper(
     FileSpec &this_file) {
   // When we're loaded from python, this_file will point to the file inside the

lldb/test/API/functionalities/paths/TestPaths.py

class TestPaths(TestBase):
 
     @no_debug_info_test
     def test_interpreter_info(self):
         info_sd = self.dbg.GetScriptInterpreterInfo(self.dbg.GetScriptingLanguage("python"))
         self.assertTrue(info_sd.IsValid())
         stream = lldb.SBStream()
         self.assertTrue(info_sd.GetAsJSON(stream).Success())
         info = json.loads(stream.GetData())
-        if os.name == 'nt': #FIXME
-            return
         prefix = info['prefix']
         self.assertEqual(os.path.realpath(sys.prefix), os.path.realpath(prefix))
         self.assertEqual(
             os.path.realpath(os.path.join(info['lldb-pythonpath'], 'lldb')),
             os.path.realpath(os.path.dirname(lldb.__file__)))
         self.assertTrue(os.path.exists(info['executable']))
         self.assertEqual(info['language'], 'python')