[lldb] Fix missing dependency on libc++ from LLDB test suite on non-Darwin platforms

[teemperor]

Summary

Right now we only set a dependency on libc++ when the host is Darwin, which means that
libc++ in the build directory is in some undefined state when running the test suite
(it can be fully built, out-of-date or missing). Depending on whether we have a system libc++
(which LLDB also supports running the libc++ tests against), the outcome is that we sometimes
skip the libc++ tests or we run the tests against a mix of ToT-libc++/system-libc++
(e.g., we compile against the ToT-libc++ headers and link against the system libc++ library).

This can be demonstrated via export LIT_FILTER=TestDataFormatterLibcxxSet ninja check-lldb-api
(or any other libc++ test) and then gradually building parts of libc++ in the same build
(which will slowly change the test behaviour from UNSUPPORTED to various failures to passing
depending on how much of libcxx is built at test time).

[labath]

Perhaps even more important than the system-libc++ use case (which does not exactly work right now), is that one should be able to run the test suite without any libc++ whatsoever (the libc++-dependant tests should skip themselves, something which also does not completely work right now).

But in any case, this is the right thing to do.

[JDevlieghere]

Thanks!

[dblaikie]

So this'll add the right test dependency, if the libcxx project is enabled in the build? (& if the build hasn't enabled libcxx, the tests will run, but against the system compiler - and if that system compiler happens to default to libc++ ( https://github.com/llvm/llvm-project/blob/e9e4fc0fd3e0780207c731a1f2b8f6aacd24e8f8/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/list/TestDataFormatterLibcxxList.py#L49 )) that should remove the "different results if you run ninja check-lldb before or after ninja check-libcxx" issue? Great, thanks!

Hmm, it doesn't appear to be working for me, running this (in a clean build directly, synced to 74c4d44d47b282769f6584153e9b433e8e5fa671 which includes 366fb539485a9753e4a8167fe5140bf4fb00a098 (& validated by inspecting lldb/test/CMakeLists.txt) still shows the tests as unsupported:

CC=clang CXX=clang++ cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_WERROR=true -DLLVM_ENABLE_PROJECTS='llvm;clang;clang-tools-extra;lld;lldb;cross-project-tests' -DLLVM_ENABLE_RUNTIMES='compiler-rt;libcxx;libcxxabi;libunwind' -DCMAKE_INSTALL_PREFIX=$HOME/install ~/dev/llvm/src/llvm
ninja check-lldb-api-functionalities-data-formatter-data-formatter-stl-libcxx

And then running ninja cxx and the same check again:

********************
FAIL: lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py (23 of 23)
******************** TEST 'lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py' FAILED ********************
Script:
--
/usr/bin/python3.9 /usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/dotest.py -u CXXFLAGS -u CFLAGS --env ARCHIVER=/usr/bin/ar --env OBJCOPY=/usr/bin/objcopy --env LLVM_LIBS_DIR=/usr/local/google/home/blaikie/dev/llvm/build/release/./lib --arch x86_64 --build-dir /usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex --lldb-module-cache-dir /usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/module-cache-lldb/lldb-api --clang-module-cache-dir /usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/module-cache-clang/lldb-api --executable /usr/local/google/home/blaikie/dev/llvm/build/release/./bin/lldb --compiler /usr/local/google/home/blaikie/dev/llvm/build/release/./bin/clang --dsymutil /usr/local/google/home/blaikie/dev/llvm/build/release/./bin/dsymutil --llvm-tools-dir /usr/local/google/home/blaikie/dev/llvm/build/release/./bin --lldb-libs-dir /usr/local/google/home/blaikie/dev/llvm/build/release/./lib /usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set -p TestDataFormatterLibcxxSet.py
--
Exit Code: 1

Command Output (stdout):
--
lldb version 14.0.0 ([e-mail]:llvm/llvm-project.git revision 74c4d44d47b282769f6584153e9b433e8e5fa671)
  clang revision 74c4d44d47b282769f6584153e9b433e8e5fa671
  llvm revision 74c4d44d47b282769f6584153e9b433e8e5fa671
Skipping the following test categories: ['dsym', 'gmodules', 'debugserver', 'objc']

--
Command Output (stderr):
--
UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_dsym (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run) 
FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_gmodules (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run) 
UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_dsym (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run) 
FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_gmodules (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run) 
======================================================================
FAIL: test_ref_and_ptr_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
   Test that the data formatters work on ref and ptr.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
    return attrvalue(self)
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 128, in test_ref_and_ptr
    (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
    return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
    test.assertEqual(process.GetState(), lldb.eStateStopped)
AssertionError: 10 != 5
Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
======================================================================
FAIL: test_ref_and_ptr_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
   Test that the data formatters work on ref and ptr.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
    return attrvalue(self)
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 128, in test_ref_and_ptr
    (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
    return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
    test.assertEqual(process.GetState(), lldb.eStateStopped)
AssertionError: 10 != 5
Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
======================================================================
FAIL: test_with_run_command_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
   Test that that file and class static variables display correctly.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
    return attrvalue(self)
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 49, in test_with_run_command
    (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
    return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
    test.assertEqual(process.GetState(), lldb.eStateStopped)
AssertionError: 10 != 5
Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
======================================================================
FAIL: test_with_run_command_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
   Test that that file and class static variables display correctly.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
    return attrvalue(self)
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 49, in test_with_run_command
    (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
    return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
    test.assertEqual(process.GetState(), lldb.eStateStopped)
AssertionError: 10 != 5
Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
----------------------------------------------------------------------
Ran 8 tests in 3.109s

RESULT: FAILED (0 passes, 4 failures, 0 errors, 4 skipped, 0 expected failures, 0 unexpected successes)

--

Any ideas?

[teemperor]

In D111981#3071349, @dblaikie wrote:

So this'll add the right test dependency, if the libcxx project is enabled in the build? (& if the build hasn't enabled libcxx, the tests will run, but against the system compiler - and if that system compiler happens to default to libc++ ( https://github.com/llvm/llvm-project/blob/e9e4fc0fd3e0780207c731a1f2b8f6aacd24e8f8/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/list/TestDataFormatterLibcxxList.py#L49 )) that should remove the "different results if you run ninja check-lldb before or after ninja check-libcxx" issue? Great, thanks!

Hmm, it doesn't appear to be working for me, running this (in a clean build directly, synced to 74c4d44d47b282769f6584153e9b433e8e5fa671 which includes 366fb539485a9753e4a8167fe5140bf4fb00a098 (& validated by inspecting lldb/test/CMakeLists.txt) still shows the tests as unsupported:

CC=clang CXX=clang++ cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_WERROR=true -DLLVM_ENABLE_PROJECTS='llvm;clang;clang-tools-extra;lld;lldb;cross-project-tests' -DLLVM_ENABLE_RUNTIMES='compiler-rt;libcxx;libcxxabi;libunwind' -DCMAKE_INSTALL_PREFIX=$HOME/install ~/dev/llvm/src/llvm
ninja check-lldb-api-functionalities-data-formatter-data-formatter-stl-libcxx

And then running ninja cxx and the same check again:

********************
FAIL: lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py (23 of 23)
******************** TEST 'lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py' FAILED ********************
Script:
--
/usr/bin/python3.9 /usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/dotest.py -u CXXFLAGS -u CFLAGS --env ARCHIVER=/usr/bin/ar --env OBJCOPY=/usr/bin/objcopy --env LLVM_LIBS_DIR=/usr/local/google/home/blaikie/dev/llvm/build/release/./lib --arch x86_64 --build-dir /usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex --lldb-module-cache-dir /usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/module-cache-lldb/lldb-api --clang-module-cache-dir /usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/module-cache-clang/lldb-api --executable /usr/local/google/home/blaikie/dev/llvm/build/release/./bin/lldb --compiler /usr/local/google/home/blaikie/dev/llvm/build/release/./bin/clang --dsymutil /usr/local/google/home/blaikie/dev/llvm/build/release/./bin/dsymutil --llvm-tools-dir /usr/local/google/home/blaikie/dev/llvm/build/release/./bin --lldb-libs-dir /usr/local/google/home/blaikie/dev/llvm/build/release/./lib /usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set -p TestDataFormatterLibcxxSet.py
--
Exit Code: 1

Command Output (stdout):
--
lldb version 14.0.0 ([e-mail]:llvm/llvm-project.git revision 74c4d44d47b282769f6584153e9b433e8e5fa671)
  clang revision 74c4d44d47b282769f6584153e9b433e8e5fa671
  llvm revision 74c4d44d47b282769f6584153e9b433e8e5fa671
Skipping the following test categories: ['dsym', 'gmodules', 'debugserver', 'objc']

--
Command Output (stderr):
--
UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_dsym (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run) 
FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_gmodules (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run) 
UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_dsym (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run) 
FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_gmodules (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run) 
======================================================================
FAIL: test_ref_and_ptr_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
   Test that the data formatters work on ref and ptr.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
    return attrvalue(self)
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 128, in test_ref_and_ptr
    (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
    return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
    test.assertEqual(process.GetState(), lldb.eStateStopped)
AssertionError: 10 != 5
Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
======================================================================
FAIL: test_ref_and_ptr_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
   Test that the data formatters work on ref and ptr.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
    return attrvalue(self)
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 128, in test_ref_and_ptr
    (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
    return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
    test.assertEqual(process.GetState(), lldb.eStateStopped)
AssertionError: 10 != 5
Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
======================================================================
FAIL: test_with_run_command_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
   Test that that file and class static variables display correctly.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
    return attrvalue(self)
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 49, in test_with_run_command
    (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
    return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
    test.assertEqual(process.GetState(), lldb.eStateStopped)
AssertionError: 10 != 5
Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
======================================================================
FAIL: test_with_run_command_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
   Test that that file and class static variables display correctly.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
    return attrvalue(self)
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 49, in test_with_run_command
    (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
    return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
    test.assertEqual(process.GetState(), lldb.eStateStopped)
AssertionError: 10 != 5
Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
----------------------------------------------------------------------
Ran 8 tests in 3.109s

RESULT: FAILED (0 passes, 4 failures, 0 errors, 4 skipped, 0 expected failures, 0 unexpected successes)

--

Any ideas?

I think this is more of a speculative fix (the assert message doesn't have that much information beside that the process crashes). Can you apply https://reviews.llvm.org/D111978 and see what the error output with that patch is? It should give you exit status/description and stderr.

[dblaikie]

In D111981#3071378, @teemperor wrote:

In D111981#3071349, @dblaikie wrote:

So this'll add the right test dependency, if the libcxx project is enabled in the build? (& if the build hasn't enabled libcxx, the tests will run, but against the system compiler - and if that system compiler happens to default to libc++ ( https://github.com/llvm/llvm-project/blob/e9e4fc0fd3e0780207c731a1f2b8f6aacd24e8f8/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/list/TestDataFormatterLibcxxList.py#L49 )) that should remove the "different results if you run ninja check-lldb before or after ninja check-libcxx" issue? Great, thanks!

Hmm, it doesn't appear to be working for me, running this (in a clean build directly, synced to 74c4d44d47b282769f6584153e9b433e8e5fa671 which includes 366fb539485a9753e4a8167fe5140bf4fb00a098 (& validated by inspecting lldb/test/CMakeLists.txt) still shows the tests as unsupported:

CC=clang CXX=clang++ cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_WERROR=true -DLLVM_ENABLE_PROJECTS='llvm;clang;clang-tools-extra;lld;lldb;cross-project-tests' -DLLVM_ENABLE_RUNTIMES='compiler-rt;libcxx;libcxxabi;libunwind' -DCMAKE_INSTALL_PREFIX=$HOME/install ~/dev/llvm/src/llvm
ninja check-lldb-api-functionalities-data-formatter-data-formatter-stl-libcxx

And then running ninja cxx and the same check again:

********************
FAIL: lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py (23 of 23)
******************** TEST 'lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py' FAILED ********************
Script:
--
/usr/bin/python3.9 /usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/dotest.py -u CXXFLAGS -u CFLAGS --env ARCHIVER=/usr/bin/ar --env OBJCOPY=/usr/bin/objcopy --env LLVM_LIBS_DIR=/usr/local/google/home/blaikie/dev/llvm/build/release/./lib --arch x86_64 --build-dir /usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex --lldb-module-cache-dir /usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/module-cache-lldb/lldb-api --clang-module-cache-dir /usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/module-cache-clang/lldb-api --executable /usr/local/google/home/blaikie/dev/llvm/build/release/./bin/lldb --compiler /usr/local/google/home/blaikie/dev/llvm/build/release/./bin/clang --dsymutil /usr/local/google/home/blaikie/dev/llvm/build/release/./bin/dsymutil --llvm-tools-dir /usr/local/google/home/blaikie/dev/llvm/build/release/./bin --lldb-libs-dir /usr/local/google/home/blaikie/dev/llvm/build/release/./lib /usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set -p TestDataFormatterLibcxxSet.py
--
Exit Code: 1

Command Output (stdout):
--
lldb version 14.0.0 ([e-mail]:llvm/llvm-project.git revision 74c4d44d47b282769f6584153e9b433e8e5fa671)
  clang revision 74c4d44d47b282769f6584153e9b433e8e5fa671
  llvm revision 74c4d44d47b282769f6584153e9b433e8e5fa671
Skipping the following test categories: ['dsym', 'gmodules', 'debugserver', 'objc']

--
Command Output (stderr):
--
UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_dsym (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run) 
FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_gmodules (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run) 
UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_dsym (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run) 
FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_gmodules (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run) 
======================================================================
FAIL: test_ref_and_ptr_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
   Test that the data formatters work on ref and ptr.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
    return attrvalue(self)
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 128, in test_ref_and_ptr
    (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
    return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
    test.assertEqual(process.GetState(), lldb.eStateStopped)
AssertionError: 10 != 5
Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
======================================================================
FAIL: test_ref_and_ptr_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
   Test that the data formatters work on ref and ptr.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
    return attrvalue(self)
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 128, in test_ref_and_ptr
    (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
    return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
    test.assertEqual(process.GetState(), lldb.eStateStopped)
AssertionError: 10 != 5
Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
======================================================================
FAIL: test_with_run_command_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
   Test that that file and class static variables display correctly.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
    return attrvalue(self)
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 49, in test_with_run_command
    (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
    return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
    test.assertEqual(process.GetState(), lldb.eStateStopped)
AssertionError: 10 != 5
Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
======================================================================
FAIL: test_with_run_command_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
   Test that that file and class static variables display correctly.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
    return attrvalue(self)
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 49, in test_with_run_command
    (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
    return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
    test.assertEqual(process.GetState(), lldb.eStateStopped)
AssertionError: 10 != 5
Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
----------------------------------------------------------------------
Ran 8 tests in 3.109s

RESULT: FAILED (0 passes, 4 failures, 0 errors, 4 skipped, 0 expected failures, 0 unexpected successes)

--

Any ideas?

I think this is more of a speculative fix (the assert message doesn't have that much information beside that the process crashes).

Oh, mostly my point was "the output changes after I explicitly run ninja cxx" so it looks like the dependency didn't fully address the build consistency issue, regardless of what the failure actually is?

Can you apply https://reviews.llvm.org/D111978 and see what the error output with that patch is? It should give you exit status/description and stderr.

Sure, I'll give that a go (was going to test it once it was submitted), but emulating something similar to the test, by debugging the binary directly:

$ ./bin/lldb ./lldb-test-build.noindex/functionalities/data-formatter/data-formatter-stl/libcxx/map/TestDataFormatterLibccMap.test_with_run_command_dwarf/a.out
(lldb) target create "./lldb-test-build.noindex/functionalities/data-formatter/data-formatter-stl/libcxx/map/TestDataFormatterLibccMap.test_with_run_command_dwarf/a.out"
Current executable set to '/usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/functionalities/data-formatter/data-formatter-stl/libcxx/map/TestDataFormatterLibccMap.test_with_run_command_dwarf/a.out' (x86_64).
(lldb) start
error: 'start' is not a valid command.
(lldb) r
warning: (x86_64) /lib/x86_64-linux-gnu/ld-2.32.so Unable to initialize decompressor for section '.debug_abbrev': zlib is not available
warning: (x86_64) /lib/x86_64-linux-gnu/ld-2.32.so Unable to initialize decompressor for section '.debug_info': zlib is not available
Process 3723631 launched: '/usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/functionalities/data-formatter/data-formatter-stl/libcxx/map/TestDataFormatterLibccMap.test_with_run_command_dwarf/a.out' (x86_64)
warning: (x86_64) /lib/x86_64-linux-gnu/ld-2.32.so Unable to initialize decompressor for section '.debug_aranges': zlib is not available
warning: (x86_64) /lib64/ld-linux-x86-64.so.2 Unable to initialize decompressor for section '.debug_abbrev': zlib is not available
warning: (x86_64) /lib64/ld-linux-x86-64.so.2 Unable to initialize decompressor for section '.debug_info': zlib is not available
/usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/functionalities/data-formatter/data-formatter-stl/libcxx/map/TestDataFormatterLibccMap.test_with_run_command_dwarf/a.out: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory

I don't /think/ there's any reason (given the current Cmake configuration/code/etc) that the binary should be able to find libc++.so.1? In the libc++ tests (not the lldb libc++ tests, but the libc++ libc++ tests) they specify -rpath when compiling libc++ binaries against the just-built libc++ so they'll find the just-built libc++.so. I don't see anything like that in the lldb libc++ tests/build.

[dblaikie]

Yeah, applying that patch gets the expected cannot open shared object file issue for libc++.so.1:

======================================================================
FAIL: test_with_run_command_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
   Test that that file and class static variables display correctly.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
    return attrvalue(self)
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 49, in test_with_run_command
    (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 1007, in run_to_source_breakpoint
    return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 924, in run_to_breakpoint_do_run
    test.fail("Test process is not stopped at breakpoint, but instead in" +
AssertionError: Test process is not stopped at breakpoint, but instead in state 'exited'. Exit code/status: 127. Exit description: None.
stderr of inferior:
/usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.test_with_run_command_dwo/a.out: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory

Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
----------------------------------------------------------------------

[labath]

In D111981#3071414, @dblaikie wrote:

Oh, mostly my point was "the output changes after I explicitly run ninja cxx" so it looks like the dependency didn't fully address the build consistency issue, regardless of what the failure actually is?

This looks like a classic example of the reason why if(TARGET FOO) is discouraged in cmake.
The top level cmake file looks like this:

add_subdirectory(projects) # libcxx IN LLVM_ENABLE_PROJECTS processed here

if( LLVM_INCLUDE_TOOLS )
  add_subdirectory(tools) # lldb IN LLVM_ENABLE_PROJECTS processed here
endif()

if( LLVM_INCLUDE_RUNTIMES )
  add_subdirectory(runtimes) # libcxx in LLVM_ENABLE_RUNTIMES processed here
endif()

If one enables libcxx via LLVM_ENABLE_PROJECTS (which is I guess what the apple folks are using) then all is fine, as the cxx target will be defined by the time we get to lldb. OTOH, if one uses LLVM_ENABLE_RUNTIMES, then the check will return false (and not add the dependency), even though the target will be defined afterwards (and will be found by clang at runtime).
I think (but I haven't verified) that this could be fixed by replacing TARGET cxx with libcxx IN LLVM_ENABLE_PROJECTS OR libcxx IN LLVM_ENABLE_RUNTIMES.

Can you apply https://reviews.llvm.org/D111978 and see what the error output with that patch is? It should give you exit status/description and stderr.

Sure, I'll give that a go (was going to test it once it was submitted), but emulating something similar to the test, by debugging the binary directly:

$ ./bin/lldb ./lldb-test-build.noindex/functionalities/data-formatter/data-formatter-stl/libcxx/map/TestDataFormatterLibccMap.test_with_run_command_dwarf/a.out
(lldb) target create "./lldb-test-build.noindex/functionalities/data-formatter/data-formatter-stl/libcxx/map/TestDataFormatterLibccMap.test_with_run_command_dwarf/a.out"
Current executable set to '/usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/functionalities/data-formatter/data-formatter-stl/libcxx/map/TestDataFormatterLibccMap.test_with_run_command_dwarf/a.out' (x86_64).
(lldb) start
error: 'start' is not a valid command.
(lldb) r
warning: (x86_64) /lib/x86_64-linux-gnu/ld-2.32.so Unable to initialize decompressor for section '.debug_abbrev': zlib is not available
warning: (x86_64) /lib/x86_64-linux-gnu/ld-2.32.so Unable to initialize decompressor for section '.debug_info': zlib is not available
Process 3723631 launched: '/usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/functionalities/data-formatter/data-formatter-stl/libcxx/map/TestDataFormatterLibccMap.test_with_run_command_dwarf/a.out' (x86_64)
warning: (x86_64) /lib/x86_64-linux-gnu/ld-2.32.so Unable to initialize decompressor for section '.debug_aranges': zlib is not available
warning: (x86_64) /lib64/ld-linux-x86-64.so.2 Unable to initialize decompressor for section '.debug_abbrev': zlib is not available
warning: (x86_64) /lib64/ld-linux-x86-64.so.2 Unable to initialize decompressor for section '.debug_info': zlib is not available
/usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/functionalities/data-formatter/data-formatter-stl/libcxx/map/TestDataFormatterLibccMap.test_with_run_command_dwarf/a.out: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory

I don't /think/ there's any reason (given the current Cmake configuration/code/etc) that the binary should be able to find libc++.so.1? In the libc++ tests (not the lldb libc++ tests, but the libc++ libc++ tests) they specify -rpath when compiling libc++ binaries against the just-built libc++ so they'll find the just-built libc++.so. I don't see anything like that in the lldb libc++ tests/build.

Yeah, we'd have to take some positive action to make that hapen.
I think the best way to go about that is to call registerSharedLibrariesWithTarget under the right circumstances. That would ensure ((DY)LD_LIBRARY_)PATH is set, and also copy the library for remote tests. I'm just not entirely sure what are "the right circumstances".

lldb/test/CMakeLists.txt

else()
           # If `LLVM_ENABLE_PROJECTS` is being used, suggest adding it.
           message(FATAL_ERROR
             "LLDB test suite requires libc++, but it is currently disabled. "
             "Please add `libcxx` to `LLVM_ENABLE_PROJECTS` or "
             "`LLVM_ENABLE_RUNTIMES`, or disable tests via "
             "`LLDB_INCLUDE_TESTS=OFF`.")
         endif()
       endif()
-      add_lldb_test_dependency(cxx)
     endif()
   endif()
+
+  # Add libc++ dependency for libc++-specific tests. This is an optional
+  # dependency as it's also possible to run the libc++ tests against the libc++
+  # installed on the system.
+  if (TARGET cxx)
+    add_lldb_test_dependency(cxx)
+  endif()
+
 endif()
 
 if (LLDB_BUILT_STANDALONE)
   set(LLVM_HOST_TRIPLE ${TARGET_TRIPLE})
 endif()
 
 add_lldb_test_dependency(
   lit-cpuid