[test] Fix LLDB tests with just-built libcxx when using a target directory.

[rupprecht]

Summary

In certain configurations, libc++ headers all exist in the same directory, and libc++ binaries exist in the same directory as lldb libs. When LLVM_ENABLE_PER_TARGET_RUNTIME_DIR is enabled (*and* the host is not Apple, which is why I assume this wasn't caught by others?), this is not the case: most headers will exist in the usual include/c++/v1 directory, but __config_site exists in include/$TRIPLE/c++/v1. Likewise, the libc++.so binary exists in lib/$TRIPLE/, not lib/ (where LLDB libraries reside).

This also adds the just-built-libcxx functionality to the lldb-dotest tool.

The LIBCXX_ cmake config is borrowed from libcxx/CMakeLists.txt. I could not figure out a way to share the cmake config; ideally we would reuse the same config instead of copy/paste.

[JDevlieghere]

Do we need both LIBCPP_INCLUDE_DIR and LIBCPP_INCLUDE_TARGET_DIR? I see we pass another -cxx-isystem flag, does this result in another search path or does that replace the former? If we only need one, we could set LIBCPP_INCLUDE_DIR to libcxx_include_target_dir or libcxx_include_dir respectively.

[rupprecht]

In D133973#3793398, @JDevlieghere wrote:

Do we need both LIBCPP_INCLUDE_DIR and LIBCPP_INCLUDE_TARGET_DIR? I see we pass another -cxx-isystem flag, does this result in another search path or does that replace the former? If we only need one, we could set LIBCPP_INCLUDE_DIR to libcxx_include_target_dir or libcxx_include_dir respectively.

We need both. --libcxx_include_dir sets the regular include/c++/v1 dir, which contains almost all of libc++ headers. But __config_site is separately installed to include/$TRIPLE/c++/v1, so without it, all the build steps fail because it can't find #include <__config_site>. And include/$TRIPLE/c++/v1 only contains the __config_site file, so we can't use the target dir alone.

[fdeazeve]

In D133973#3793398, @JDevlieghere wrote:

does this result in another search path or does that replace the former

I did some local testing and it seems to add another path.

[fdeazeve]

lldb/test/API/lit.cfg.py

176–180

Isn't this always going to return true? (same for is_configured (libcxx_include_target_dir)).

is_configured returns the empty string when the CMake variables are empty, and the empty string converts to True

[rupprecht]

lldb/test/API/lit.cfg.py

176–180

Since this is a python file, it should follow the python rules where an empty string is treated as false:

$ python3
Python 3.10.6 (main, Aug 10 2022, 11:19:32) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> x = ''
>>> 'foo' if x else 'bar'
'bar'

[fdeazeve]

lldb/test/API/lit.cfg.py

176–180

Oh, my bad. I had been bitten by if('0') in a similar context recently, so I was a bit paranoid here.

[fdeazeve]

This LGTM and it worked without issues when I ran some tests on my machine, but please let @JDevlieghere have another look

[JDevlieghere]

LGTM

lldb/test/API/lit.cfg.py

178–179

Any reason you put this before adding config.libcxx_libs_dir below? Same question for lldb-dotest.

[rupprecht]

lldb/test/API/lit.cfg.py

178–179

I did this to group together the include-related args passed to dotest.py. It shouldn't be necessary, it just helps eyeballing the args to see if dotest.py is being invoked correctly.

lldb/packages/Python/lldbsuite/test/builders/builder.py

def getModuleCacheSpec(self):
         """
         if configuration.clang_module_cache_dir:
             return ["CLANG_MODULE_CACHE_DIR={}".format(
                 configuration.clang_module_cache_dir)]
         return []
 
     def getLibCxxArgs(self):
         if configuration.libcxx_include_dir and configuration.libcxx_library_dir:
-            return ["LIBCPP_INCLUDE_DIR={}".format(configuration.libcxx_include_dir),
+            libcpp_args = ["LIBCPP_INCLUDE_DIR={}".format(configuration.libcxx_include_dir),
                            "LIBCPP_LIBRARY_DIR={}".format(configuration.libcxx_library_dir)]
+            if configuration.libcxx_include_target_dir:
+                libcpp_args.append("LIBCPP_INCLUDE_TARGET_DIR={}".format(
+                    configuration.libcxx_include_target_dir))
+            return libcpp_args
         return []
 
     def _getDebugInfoArgs(self, debug_info):
         if debug_info is None:
             return []
         if debug_info == "dwarf":
             return ["MAKE_DSYM=NO"]
         if debug_info == "dwo":

lldb/packages/Python/lldbsuite/test/configuration.py

 # The names of all tests. Used to assert we don't have two tests with the
 # same base name.
 all_tests = set()
 
 # LLDB library directory.
 lldb_libs_dir = None
 
 libcxx_include_dir = None
+libcxx_include_target_dir = None
 libcxx_library_dir = None
 
 # A plugin whose tests will be enabled, like intel-pt.
 enabled_plugins = []
 
 
 def shouldSkipBecauseOfCategories(test_categories):
     if use_categories:

lldb/packages/Python/lldbsuite/test/dotest.py

def parseOptionsAndInitTestdirs():
     if args.llvm_tools_dir:
         configuration.filecheck = shutil.which("FileCheck", path=args.llvm_tools_dir)
         configuration.yaml2obj = shutil.which("yaml2obj", path=args.llvm_tools_dir)
 
     if not configuration.get_filecheck_path():
         logging.warning('No valid FileCheck executable; some tests may fail...')
         logging.warning('(Double-check the --llvm-tools-dir argument to dotest.py)')
 
-    configuration.libcxx_include_dir = args.libcxx_include_dir
-    configuration.libcxx_library_dir = args.libcxx_library_dir
     if args.libcxx_include_dir or args.libcxx_library_dir:
         if args.lldb_platform_name:
             logging.warning('Custom libc++ is not supported for remote runs: ignoring --libcxx arguments')
-        elif args.libcxx_include_dir and args.libcxx_library_dir:
-            configuration.libcxx_include_dir = args.libcxx_include_dir
-            configuration.libcxx_library_dir = args.libcxx_library_dir
-        else:
+        elif not (args.libcxx_include_dir and args.libcxx_library_dir):
             logging.error('Custom libc++ requires both --libcxx-include-dir and --libcxx-library-dir')
             sys.exit(-1)
+    configuration.libcxx_include_dir = args.libcxx_include_dir
+    configuration.libcxx_include_target_dir = args.libcxx_include_target_dir
+    configuration.libcxx_library_dir = args.libcxx_library_dir
 
     if args.channels:
         lldbtest_config.channels = args.channels
 
     if args.log_success:
         lldbtest_config.log_success = args.log_success
 
     if args.out_of_tree_debugserver:

lldb/packages/Python/lldbsuite/test/dotest_args.py

group.add_argument(
         metavar='arch',
         dest='arch',
         help=textwrap.dedent('''Specify the architecture(s) to test. This option can be specified more than once'''))
     group.add_argument('-C', '--compiler', metavar='compiler', dest='compiler', help=textwrap.dedent(
         '''Specify the compiler(s) used to build the inferior executables. The compiler path can be an executable basename or a full path to a compiler executable. This option can be specified multiple times.'''))
     if sys.platform == 'darwin':
         group.add_argument('--apple-sdk', metavar='apple_sdk', dest='apple_sdk', default="", help=textwrap.dedent(
             '''Specify the name of the Apple SDK (macosx, macosx.internal, iphoneos, iphoneos.internal, or path to SDK) and use the appropriate tools from that SDK's toolchain.'''))
-    group.add_argument('--libcxx-include-dir', help=textwrap.dedent('Specify the path to a custom libc++ include directory. Must be used in conjunction with --libcxx-library-dir.'))
-    group.add_argument('--libcxx-library-dir', help=textwrap.dedent('Specify the path to a custom libc++ library directory. Must be used in conjunction with --libcxx-include-dir.'))
+    group.add_argument('--libcxx-include-dir', help=textwrap.dedent(
+        'Specify the path to a custom libc++ include directory. Must be used in conjunction with --libcxx-library-dir.'))
+    group.add_argument('--libcxx-include-target-dir', help=textwrap.dedent(
+        'Specify the path to a custom libc++ include target directory to use in addition to --libcxx-include-dir. Optional.'))
+    group.add_argument('--libcxx-library-dir', help=textwrap.dedent(
+        'Specify the path to a custom libc++ library directory. Must be used in conjunction with --libcxx-include-dir.'))
     # FIXME? This won't work for different extra flags according to each arch.
     group.add_argument(
         '-E',
         metavar='extra-flags',
         help=textwrap.dedent('''Specify the extra flags to be passed to the toolchain when building the inferior programs to be debugged
                                                            suggestions: do not lump the "-A arch1 -A arch2" together such that the -E option applies to only one of the architectures'''))
 
     group.add_argument('--dsymutil', metavar='dsymutil', dest='dsymutil', help=textwrap.dedent('Specify which dsymutil to use.'))

lldb/packages/Python/lldbsuite/test/make/Makefile.rules

ifneq (,$(findstring clang,$(CC)))
     CXXFLAGS += -stdlib=libstdc++
     LDFLAGS += -stdlib=libstdc++
   endif
 endif
 
 ifeq (1,$(USE_LIBCPP))
   ifneq ($(and $(LIBCPP_INCLUDE_DIR), $(LIBCPP_LIBRARY_DIR)),)
     CXXFLAGS += -nostdlib++ -nostdinc++ -cxx-isystem $(LIBCPP_INCLUDE_DIR)
-    LDFLAGS += -L$(LLVM_LIBS_DIR) -Wl,-rpath,$(LIBCPP_LIBRARY_DIR) -lc++
+    ifneq "$(LIBCPP_INCLUDE_TARGET_DIR)" ""
+        CXXFLAGS += -cxx-isystem $(LIBCPP_INCLUDE_TARGET_DIR)
+    endif
+    LDFLAGS += -L$(LIBCPP_LIBRARY_DIR) -Wl,-rpath,$(LIBCPP_LIBRARY_DIR) -lc++
   else
     ifeq "$(OS)" "Android"
         # Nothing to do, this is already handled in
         # Android.rules.
     else
         CXXFLAGS += -stdlib=libc++
         LDFLAGS += -stdlib=libc++
     endif
     ifneq (,$(filter $(OS), FreeBSD Linux NetBSD))
         ifneq (,$(LLVM_LIBS_DIR))
         LDFLAGS += -Wl,-rpath,$(LLVM_LIBS_DIR)
         endif
     endif
   endif
 endif
 
 # If no explicit request was made, but we have paths to a custom libcxx, use
 # them.
 ifeq ($(or $(USE_LIBSTDCPP), $(USE_LIBCPP), $(USE_SYSTEM_STDLIB)),)
   ifneq ($(and $(LIBCPP_INCLUDE_DIR), $(LIBCPP_LIBRARY_DIR)),)
     CXXFLAGS += -nostdlib++ -nostdinc++ -cxx-isystem $(LIBCPP_INCLUDE_DIR)
-    LDFLAGS += -L$(LLVM_LIBS_DIR) -Wl,-rpath,$(LIBCPP_LIBRARY_DIR) -lc++
+    ifneq "$(LIBCPP_INCLUDE_TARGET_DIR)" ""
+        CXXFLAGS += -cxx-isystem $(LIBCPP_INCLUDE_TARGET_DIR)
+    endif
+    LDFLAGS += -L$(LIBCPP_LIBRARY_DIR) -Wl,-rpath,$(LIBCPP_LIBRARY_DIR) -lc++
   endif
 endif
 
 #----------------------------------------------------------------------
 # Additional system libraries
 #----------------------------------------------------------------------
 ifeq (1,$(USE_LIBDL))
   ifeq (,$(filter $(OS), NetBSD Windows_NT))

lldb/test/API/lit.cfg.py

 
 # This path may be needed to locate required llvm tools
 if is_configured('llvm_tools_dir'):
   dotest_cmd += ['--env', 'LLVM_TOOLS_DIR=' + config.llvm_tools_dir]
 
 # If we have a just-built libcxx, prefer it over the system one.
 if is_configured('has_libcxx') and config.has_libcxx:
   if platform.system() != 'Windows':
-    if is_configured('llvm_include_dir') and is_configured('llvm_libs_dir'):
-      dotest_cmd += ['--libcxx-include-dir', os.path.join(config.llvm_include_dir, 'c++', 'v1')]
-      dotest_cmd += ['--libcxx-library-dir', config.llvm_libs_dir]
+    if is_configured('libcxx_include_dir') and is_configured('libcxx_libs_dir'):
+      dotest_cmd += ['--libcxx-include-dir', config.libcxx_include_dir]
+      if is_configured('libcxx_include_target_dir'):
+        dotest_cmd += ['--libcxx-include-target-dir', config.libcxx_include_target_dir]
+      dotest_cmd += ['--libcxx-library-dir', config.libcxx_libs_dir]
 
 # Forward ASan-specific environment variables to tests, as a test may load an
 # ASan-ified dylib.
 for env_var in ('ASAN_OPTIONS', 'DYLD_INSERT_LIBRARIES'):
   if env_var in config.environment:
     dotest_cmd += ['--inferior-env', env_var + '=' + config.environment[env_var]]
 
 if is_configured('test_arch'):

lldb/test/API/lit.site.cfg.py.in

 config.lldb_enable_python = @LLDB_ENABLE_PYTHON@
 config.dotest_lit_args_str = None
 config.enabled_plugins = []
 config.lldb_executable = lit_config.substitute('@LLDB_TEST_EXECUTABLE@')
 config.test_arch = '@LLDB_TEST_ARCH@'
 config.test_compiler = lit_config.substitute('@LLDB_TEST_COMPILER@')
 config.dsymutil = lit_config.substitute('@LLDB_TEST_DSYMUTIL@')
 config.has_libcxx = @LLDB_HAS_LIBCXX@
+config.libcxx_libs_dir = "@LIBCXX_LIBRARY_DIR@"
+config.libcxx_include_dir = "@LIBCXX_GENERATED_INCLUDE_DIR@"
+config.libcxx_include_target_dir = "@LIBCXX_GENERATED_INCLUDE_TARGET_DIR@"
 # The API tests use their own module caches.
 config.lldb_module_cache = os.path.join("@LLDB_TEST_MODULE_CACHE_LLDB@", "lldb-api")
 config.clang_module_cache = os.path.join("@LLDB_TEST_MODULE_CACHE_CLANG@", "lldb-api")
 
 # Plugins
 lldb_build_intel_pt = '@LLDB_BUILD_INTEL_PT@'
 if lldb_build_intel_pt == '1':
     config.enabled_plugins.append('intel-pt')

lldb/test/CMakeLists.txt

 
 # Add dependencies if we test with the in-tree clang.
 # This works with standalone builds as they import the clang target.
 if(TARGET clang)
   add_lldb_test_dependency(clang)
 
   if (TARGET libcxx OR ("libcxx" IN_LIST LLVM_ENABLE_RUNTIMES))
     set(LLDB_HAS_LIBCXX ON)
+    if(LLVM_ENABLE_PER_TARGET_RUNTIME_DIR AND NOT APPLE)
+      set(LIBCXX_LIBRARY_DIR ${LLVM_LIBRARY_OUTPUT_INTDIR}/${LLVM_DEFAULT_TARGET_TRIPLE})
+      set(LIBCXX_GENERATED_INCLUDE_DIR "${LLVM_BINARY_DIR}/include/c++/v1")
+      set(LIBCXX_GENERATED_INCLUDE_TARGET_DIR "${LLVM_BINARY_DIR}/include/${LLVM_DEFAULT_TARGET_TRIPLE}/c++/v1")
+    else()
+      set(LIBCXX_LIBRARY_DIR ${CMAKE_BINARY_DIR}/lib${LIBCXX_LIBDIR_SUFFIX})
+      set(LIBCXX_GENERATED_INCLUDE_DIR "${CMAKE_BINARY_DIR}/include/c++/v1")
+    endif()
     add_lldb_test_dependency(cxx)
   endif()
 
   if (TARGET compiler-rt OR "compiler-rt" IN_LIST LLVM_ENABLE_RUNTIMES)
     set(LLDB_HAS_COMPILER_RT ON)
     add_lldb_test_dependency(compiler-rt)
   endif()
 

lldb/utils/lldb-dotest/CMakeLists.txt

 # Make lldb-dotest a custom target.
 add_custom_target(lldb-dotest)
 add_dependencies(lldb-dotest lldb-test-depends)
 set_target_properties(lldb-dotest PROPERTIES FOLDER "lldb utils")
 
 get_property(LLDB_TEST_USER_ARGS GLOBAL PROPERTY LLDB_TEST_USER_ARGS_PROPERTY)
 get_property(LLDB_TEST_COMMON_ARGS GLOBAL PROPERTY LLDB_TEST_COMMON_ARGS_PROPERTY)
 set(LLDB_LIBS_DIR "${LLVM_LIBRARY_OUTPUT_INTDIR}")
 
 llvm_canonicalize_cmake_booleans(
   LLDB_BUILD_INTEL_PT
+  LLDB_HAS_LIBCXX
 )
 
+if ("libcxx" IN_LIST LLVM_ENABLE_RUNTIMES)
+  set(LLDB_HAS_LIBCXX ON)
+  if(LLVM_ENABLE_PER_TARGET_RUNTIME_DIR AND NOT APPLE)
+    set(LIBCXX_LIBRARY_DIR ${LLVM_LIBRARY_OUTPUT_INTDIR}/${LLVM_DEFAULT_TARGET_TRIPLE})
+    set(LIBCXX_GENERATED_INCLUDE_DIR "${LLVM_BINARY_DIR}/include/c++/v1")
+    set(LIBCXX_GENERATED_INCLUDE_TARGET_DIR "${LLVM_BINARY_DIR}/include/${LLVM_DEFAULT_TARGET_TRIPLE}/c++/v1")
+  else()
+    set(LIBCXX_LIBRARY_DIR ${CMAKE_BINARY_DIR}/lib${LIBCXX_LIBDIR_SUFFIX})
+    set(LIBCXX_GENERATED_INCLUDE_DIR "${CMAKE_BINARY_DIR}/include/c++/v1")
+  endif()
+endif()
+
 set(LLVM_TOOLS_DIR "${LLVM_TOOLS_BINARY_DIR}")
 set(vars
   LLDB_TEST_COMMON_ARGS
   LLDB_TEST_USER_ARGS
   LLDB_SOURCE_DIR
   LLDB_FRAMEWORK_DIR
   LLDB_TEST_BUILD_DIRECTORY
   LLDB_TEST_EXECUTABLE
   LLDB_TEST_COMPILER
   LLDB_TEST_DSYMUTIL
   LLDB_LIBS_DIR
   LLVM_TOOLS_DIR
+  LIBCXX_LIBRARY_DIR
+  LIBCXX_GENERATED_INCLUDE_DIR
+  LIBCXX_GENERATED_INCLUDE_TARGET_DIR
   )
 
+llvm_canonicalize_cmake_booleans(LLDB_HAS_LIBCXX)
+
 # Generate lldb-dotest Python driver script for each build mode.
 if(LLDB_BUILT_STANDALONE)
   set(config_types ".")
   if(CMAKE_CONFIGURATION_TYPES)
     set(config_types ${CMAKE_CONFIGURATION_TYPES})
   endif()
   foreach(config_type ${config_types})
     # In paths to our build-tree, replace CMAKE_CFG_INTDIR with our actual configuration names.

lldb/utils/lldb-dotest/lldb-dotest.in

 #!@Python3_EXECUTABLE@
 import subprocess
 import sys
 
 dotest_path = '@LLDB_SOURCE_DIR_CONFIGURED@/test/API/dotest.py'
 dotest_args_str = '@LLDB_DOTEST_ARGS_CONFIGURED@'
 arch = '@LLDB_TEST_ARCH@'
 executable = '@LLDB_TEST_EXECUTABLE_CONFIGURED@'
 compiler = '@LLDB_TEST_COMPILER_CONFIGURED@'
 dsymutil = '@LLDB_TEST_DSYMUTIL_CONFIGURED@'
 lldb_build_dir = '@LLDB_TEST_BUILD_DIRECTORY_CONFIGURED@'
 lldb_build_intel_pt = "@LLDB_BUILD_INTEL_PT@"
 lldb_framework_dir = "@LLDB_FRAMEWORK_DIR_CONFIGURED@"
 lldb_libs_dir = "@LLDB_LIBS_DIR_CONFIGURED@"
 llvm_tools_dir = "@LLVM_TOOLS_DIR_CONFIGURED@"
+has_libcxx = @LLDB_HAS_LIBCXX@
+libcxx_libs_dir = "@LIBCXX_LIBRARY_DIR@"
+libcxx_include_dir = "@LIBCXX_GENERATED_INCLUDE_DIR@"
+libcxx_include_target_dir = "@LIBCXX_GENERATED_INCLUDE_TARGET_DIR@"
 
 if __name__ == '__main__':
     wrapper_args = sys.argv[1:]
     dotest_args = []
     # split on an empty string will produce [''] and if you
     # add that to the command, it will be treated as a directory...
     if len(dotest_args_str) > 0:
         dotest_args = dotest_args_str.split(';')
     # Build dotest.py command.
     cmd = [sys.executable, dotest_path]
     cmd.extend(['--arch', arch])
     cmd.extend(dotest_args)
     cmd.extend(['--build-dir', lldb_build_dir])
     cmd.extend(['--executable', executable])
     cmd.extend(['--compiler', compiler])
     cmd.extend(['--dsymutil', dsymutil])
     cmd.extend(['--lldb-libs-dir', lldb_libs_dir])
     cmd.extend(['--llvm-tools-dir', llvm_tools_dir])
+    if has_libcxx:
+        cmd.extend(['--libcxx-include-dir', libcxx_include_dir])
+        if libcxx_include_target_dir:
+            cmd.extend(['--libcxx-include-target-dir', libcxx_include_target_dir])
+        cmd.extend(['--libcxx-library-dir', libcxx_libs_dir])
     if lldb_framework_dir:
         cmd.extend(['--framework', lldb_framework_dir])
     if lldb_build_intel_pt == "1":
         cmd.extend(['--enable-plugin', 'intel-pt'])
     cmd.extend(wrapper_args)
     # Invoke dotest.py and return exit code.
     print(' '.join(cmd))
     sys.exit(subprocess.call(cmd))