Install six.py conditionally

[krytarowski]

Summary

The current version of LLDB installs six.py into global python library directory. This approach produces conflicts downstream with distribution's py-six copy.

Introduce new configure option LLDB_USE_SYSTEM_SIX (disabled by default). Once specified as TRUE, six.py won't be installed to Python's directory.

Add new option in finishSwigWrapperClasses.py, namely --useSystemSix.

Sponsored by <The NetBSD Foundation>

[krytarowski]

Please help to test this patch for regressions, at the moment I don't have all the test suits passing and I might have miss something.

At least scripts/Python/modify-python-lldb.py had to retain 'import six'.

[labath]

It does not seem to work for me. I get this when I run ninja check-lldb in the build directory:

Traceback (most recent call last):
  File "/usr/local/google/home/labath/ll/llvm/tools/lldb/test/dotest.py", line 6, in <module>
    import lldbsuite.test
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/__init__.py", line 5, in <module>
    from . import dotest
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/dotest.py", line 36, in <module>
    from lldb import six
ImportError: No module named lldb

Any idea what could be the cause?

I have no issue with this approach in general (assuming it can be made to work), but I want to throw an alternative solution out there:

• Add a LLDB_USE_SYSTEM_SIX cmake option. If this option is set, LLDB will not install or use the built-in copy of six.py, but will rely on the system one instead.

Thoughts?

[krytarowski]

In D29405#663566, @labath wrote:

It does not seem to work for me. I get this when I run ninja check-lldb in the build directory:

Traceback (most recent call last):
  File "/usr/local/google/home/labath/ll/llvm/tools/lldb/test/dotest.py", line 6, in <module>
    import lldbsuite.test
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/__init__.py", line 5, in <module>
    from . import dotest
  File "/usr/local/google/home/labath/ll/lldb/packages/Python/lldbsuite/test/dotest.py", line 36, in <module>
    from lldb import six
ImportError: No module named lldb

Any idea what could be the cause?

I've installed lldb before running tests, I'm not sure if it has the impact on it.

I have no issue with this approach in general (assuming it can be made to work), but I want to throw an alternative solution out there:

• Add a LLDB_USE_SYSTEM_SIX cmake option. If this option is set, LLDB will not install or use the built-in copy of six.py, but will rely on the system one instead.

Thoughts?

It's less invasive, I will go for it.

[krytarowski]

LLDB_USE_SYSTEM_SIX approach

[krytarowski]

@labath patch done.

I'm specifying -DLLDB_USE_SYSTEM_SIX:BOOL=TRUE in pkgsrc.

It works for me.

[labath]

Unfortunately, this prevents six.py being copied into the build directory, so a non-installed lldb will still not work (if you don't have the system six.py).

It looks like you will have to pass this flag into finishSwigPythonLLDB.py and check the condition there.

[krytarowski]

In D29405#664957, @labath wrote:

Unfortunately, this prevents six.py being copied into the build directory, so a non-installed lldb will still not work (if you don't have the system six.py).

It looks like you will have to pass this flag into finishSwigPythonLLDB.py and check the condition there.

I would do it but I understand how to pass a flag to this script. What and when is calling it? I guess that swig executable?

Is there an option to install this file to a build directory with CMake? This part is clearer.

[krytarowski]

I will try to reverse engineer it..

[labath]

Look at top level CmakeLists.txt file
you will find an add_custom_target rule that runs it.

[krytarowski]

Pass option to --useSystemSix to finishSwigWrapperClasses.py

[labath]

looks good, thank you.

CMakeLists.txt

 set(LLDB_DISABLE_LIBEDIT ${LLDB_DEFAULT_DISABLE_LIBEDIT} CACHE BOOL "Disables the use of editline.")
 if (LLDB_DISABLE_LIBEDIT)
   add_definitions( -DLLDB_DISABLE_LIBEDIT )
 endif()
 
 # add_subdirectory(include)
 add_subdirectory(docs)
 if (NOT LLDB_DISABLE_PYTHON)
+  if(LLDB_USE_SYSTEM_SIX)
+    set(SIX_EXTRA_ARGS "--useSystemSix")
+  endif()
+
   set(LLDB_PYTHON_TARGET_DIR ${LLDB_BINARY_DIR}/scripts)
   if(LLDB_BUILD_FRAMEWORK)
     set(LLDB_PYTHON_TARGET_DIR
       ${CMAKE_BINARY_DIR}/${CMAKE_CFG_INTDIR}/${LLDB_FRAMEWORK_INSTALL_DIR})
   else()
     # Don't set -m when building the framework.
     set(FINISH_EXTRA_ARGS "-m")
   endif()
add_custom_target(finish_swig ALL
         COMMAND
            ${PYTHON_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/scripts/finishSwigWrapperClasses.py
                --srcRoot=${LLDB_SOURCE_DIR}
                --targetDir=${LLDB_PYTHON_TARGET_DIR}
                --cfgBldDir=${CMAKE_CURRENT_BINARY_DIR}/scripts
                --prefix=${CMAKE_BINARY_DIR}
                --cmakeBuildConfiguration=${CMAKE_CFG_INTDIR}
                --lldbLibDir=lib${LLVM_LIBDIR_SUFFIX}
+               ${SIX_EXTRA_ARGS}
                ${FINISH_EXTRA_ARGS}
         VERBATIM
         DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/scripts/finishSwigWrapperClasses.py
         DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/scripts/lldb.py
         COMMENT "Python script sym-linking LLDB Python API")
     # We depend on liblldb being built before we can do this step.
     add_dependencies(finish_swig liblldb lldb-argdumper)
 

cmake/modules/LLDBConfig.cmake

 set(LLDB_DISABLE_PYTHON ${LLDB_DEFAULT_DISABLE_PYTHON} CACHE BOOL
   "Disables the Python scripting integration.")
 set(LLDB_DISABLE_CURSES ${LLDB_DEFAULT_DISABLE_CURSES} CACHE BOOL
   "Disables the Curses integration.")
 
 set(LLDB_RELOCATABLE_PYTHON 0 CACHE BOOL
   "Causes LLDB to use the PYTHONHOME environment variable to locate Python.")
 
+set(LLDB_USE_SYSTEM_SIX 0 CACHE BOOL
+  "Use six.py shipped with system and do not install a copy of it")
+
 if (NOT CMAKE_SYSTEM_NAME MATCHES "Windows")
   set(LLDB_EXPORT_ALL_SYMBOLS 0 CACHE BOOL
     "Causes lldb to export all symbols when building liblldb.")
 else()
   # Windows doesn't support toggling this, so don't bother making it a
   # cache variable.
   set(LLDB_EXPORT_ALL_SYMBOLS 0)
 endif()

scripts/Python/finishSwigPythonLLDB.py

def main(vDictArgs):
     if bOk:
         bOk, strMsg = find_or_create_python_dir(
             vDictArgs, strFrameworkPythonDir)
 
     if bOk:
         bOk, strMsg = create_symlinks(
             vDictArgs, strFrameworkPythonDir, strLldbLibDir)
 
-    if bOk:
+    bUseSystemSix = "--useSystemSix" in vDictArgs
+
+    if not bUseSystemSix and bOk:
         bOk, strMsg = copy_six(vDictArgs, strFrameworkPythonDir)
 
     if bOk:
         bOk, strMsg = copy_lldbpy_file_to_lldb_pkg_dir(vDictArgs,
                                                        strFrameworkPythonDir,
                                                        strCfgBldDir)
     strRoot = os.path.normpath(vDictArgs["--srcRoot"])
     if bOk:

scripts/finishSwigWrapperClasses.py

--prefix=           (optional) Is the root directory used to determine where\n\
                     automatically. Python install directory.\n\
     --lldbLibDir    (optional) The name of the directory containing liblldb.so.\n\
                     \"lib\" by default.\n\
     --cmakeBuildConfiguration=  (optional) Is the build configuration(Debug, Release, RelWithDebugInfo)\n\
                     used to determine where the bin and lib directories are \n\
                     created for a Windows build.\n\
     --argsFile=         The args are read from a file instead of the\n\
                     command line. Other command line args are ignored.\n\
+    --useSystemSix  Use system six.py version.\n\
 \n\
 Usage:\n\
     finishSwigWrapperClasses.py --srcRoot=ADirPath --targetDir=ADirPath\n\
     --cfgBldDir=ADirPath --prefix=ADirPath --lldbLibDir=ADirPath -m -d\n\
 \n\
 "  # TAG_PROGRAM_HELP_INFO
 
 #++---------------------------------------------------------------------------
def validate_arguments(vArgv):
     strListArgs = "hdm"  # Format "hiox:" = -h -i -o -x <arg>
     listLongArgs = [
         "srcRoot=",
         "targetDir=",
         "cfgBldDir=",
         "prefix=",
         "cmakeBuildConfiguration=",
         "lldbLibDir=",
-        "argsFile"]
+        "argsFile",
+        "useSystemSix"]
     dictArgReq = {"-h": "o",          # o = optional, m = mandatory
                   "-d": "o",
                   "-m": "o",
                   "--srcRoot": "m",
                   "--targetDir": "m",
                   "--cfgBldDir": "o",
                   "--prefix": "o",
                   "--cmakeBuildConfiguration": "o",
                   "--lldbLibDir": "o",
-                  "--argsFile": "o"}
+                  "--argsFile": "o",
+                  "--useSystemSix": "o"}
 
     # Check for mandatory parameters
     nResult, dictArgs, strMsg = utilsArgsParse.parse(vArgv, strListArgs,
                                                      listLongArgs,
                                                      dictArgReq,
                                                      strHelpInfo)
     if nResult < 0:
         program_exit_on_failure(nResult, strMsg)
Args:   -h (optional)   Print help information on this program.
                             automatically. Python install directory.
             --cmakeBuildConfiguration=  (optional) Is the build configuration(Debug, Release, RelWithDebugInfo)\n\
                             used to determine where the bin and lib directories are \n\
                             created for a Windows build.\n\
             --lldbLibDir=   The name of the directory containing liblldb.so.
             (optional)      "lib" by default.
             --argsFile=     The args are read from a file instead of the
                             command line. Other command line args are ignored.
+            --useSystemSix  Use system six.py version.
     Usage:
             finishSwigWrapperClasses.py --srcRoot=ADirPath --targetDir=ADirPath
             --cfgBldDir=ADirPath --prefix=ADirPath --lldbLibDir=ADirPath -m -d
+            --useSystemSix
 
     Results:    0 Success
                 -1 Error - invalid parameters passed.
                 -2 Error - incorrect number of mandatory parameters passed.
 
                 -4 Error - unable to determine OS type.
                 -5 Error - program not run with name of "__main__".
                 -8 Error - unable to locate the scripts folder.

scripts/utilsArgsParse.py

for opt, val in dictOptsNeeded:
             if opt == argg:
                 if "m" == vDictArgReq[opt]:
                     countMandatoryOpts = countMandatoryOpts + 1
                 dictArgs[opt] = val
                 match = 1
                 break
         if match == 0:
             for arg in vListLongArgs:
-                argg = "--" + arg[:arg.__len__() - 1]
+                argg = "--" + arg.rstrip('=')
                 if opt == argg:
                     if "m" == vDictArgReq[opt]:
                         countMandatoryOpts = countMandatoryOpts + 1
                     dictArgs[opt] = val
-                    if val.__len__() == 0:
+                    if arg[-1:] == '=' and val.__len__() == 0:
                         bFoundNoInputValue = True
                     break
 
     # Do any of the long arguments not have a value attached
     if bFoundNoInputValue:
         strMsg = strMsgErrorInvalidParameters
         strMsg += str(vArgv)
         return (-1, dictDummy, strMsg)