This is an archive of the discontinued LLVM Phabricator instance.

[lldb] Fix missing dependency on libc++ from LLDB test suite on non-Darwin platforms
ClosedPublic

Authored by teemperor on Oct 18 2021, 3:25 AM.

Details

Summary

Right now we only set a dependency on libc++ when the host is Darwin, which means that
libc++ in the build directory is in some undefined state when running the test suite
(it can be fully built, out-of-date or missing). Depending on whether we have a system libc++
(which LLDB also supports running the libc++ tests against), the outcome is that we sometimes
skip the libc++ tests or we run the tests against a mix of ToT-libc++/system-libc++
(e.g., we compile against the ToT-libc++ headers and link against the system libc++ library).

This can be demonstrated via export LIT_FILTER=TestDataFormatterLibcxxSet ninja check-lldb-api
(or any other libc++ test) and then gradually building parts of libc++ in the same build
(which will slowly change the test behaviour from UNSUPPORTED to various failures to passing
depending on how much of libcxx is built at test time).

Diff Detail

Event Timeline

teemperor created this revision.Oct 18 2021, 3:25 AM
teemperor requested review of this revision.Oct 18 2021, 3:25 AM
labath accepted this revision.Oct 18 2021, 3:30 AM

Perhaps even more important than the system-libc++ use case (which does not exactly work right now), is that one should be able to run the test suite without any libc++ whatsoever (the libc++-dependant tests should skip themselves, something which also does not completely work right now).

But in any case, this is the right thing to do.

This revision is now accepted and ready to land.Oct 18 2021, 3:30 AM

So this'll add the right test dependency, if the libcxx project is enabled in the build? (& if the build hasn't enabled libcxx, the tests will run, but against the system compiler - and if that system compiler happens to default to libc++ ( https://github.com/llvm/llvm-project/blob/e9e4fc0fd3e0780207c731a1f2b8f6aacd24e8f8/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/list/TestDataFormatterLibcxxList.py#L49 )) that should remove the "different results if you run ninja check-lldb before or after ninja check-libcxx" issue? Great, thanks!

Hmm, it doesn't appear to be working for me, running this (in a clean build directly, synced to 74c4d44d47b282769f6584153e9b433e8e5fa671 which includes 366fb539485a9753e4a8167fe5140bf4fb00a098 (& validated by inspecting lldb/test/CMakeLists.txt) still shows the tests as unsupported:

CC=clang CXX=clang++ cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_WERROR=true -DLLVM_ENABLE_PROJECTS='llvm;clang;clang-tools-extra;lld;lldb;cross-project-tests' -DLLVM_ENABLE_RUNTIMES='compiler-rt;libcxx;libcxxabi;libunwind' -DCMAKE_INSTALL_PREFIX=$HOME/install ~/dev/llvm/src/llvm
ninja check-lldb-api-functionalities-data-formatter-data-formatter-stl-libcxx

And then running ninja cxx and the same check again:

********************
FAIL: lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py (23 of 23)
******************** TEST 'lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py' FAILED ********************
Script:
--
/usr/bin/python3.9 /usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/dotest.py -u CXXFLAGS -u CFLAGS --env ARCHIVER=/usr/bin/ar --env OBJCOPY=/usr/bin/objcopy --env LLVM_LIBS_DIR=/usr/local/google/home/blaikie/dev/llvm/build/release/./lib --arch x86_64 --build-dir /usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex --lldb-module-cache-dir /usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/module-cache-lldb/lldb-api --clang-module-cache-dir /usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/module-cache-clang/lldb-api --executable /usr/local/google/home/blaikie/dev/llvm/build/release/./bin/lldb --compiler /usr/local/google/home/blaikie/dev/llvm/build/release/./bin/clang --dsymutil /usr/local/google/home/blaikie/dev/llvm/build/release/./bin/dsymutil --llvm-tools-dir /usr/local/google/home/blaikie/dev/llvm/build/release/./bin --lldb-libs-dir /usr/local/google/home/blaikie/dev/llvm/build/release/./lib /usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set -p TestDataFormatterLibcxxSet.py
--
Exit Code: 1

Command Output (stdout):
--
lldb version 14.0.0 (git@github.com:llvm/llvm-project.git revision 74c4d44d47b282769f6584153e9b433e8e5fa671)
  clang revision 74c4d44d47b282769f6584153e9b433e8e5fa671
  llvm revision 74c4d44d47b282769f6584153e9b433e8e5fa671
Skipping the following test categories: ['dsym', 'gmodules', 'debugserver', 'objc']

--
Command Output (stderr):
--
UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_dsym (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run) 
FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_gmodules (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run) 
UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_dsym (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run) 
FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_gmodules (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run) 
======================================================================
FAIL: test_ref_and_ptr_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
   Test that the data formatters work on ref and ptr.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
    return attrvalue(self)
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 128, in test_ref_and_ptr
    (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
    return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
    test.assertEqual(process.GetState(), lldb.eStateStopped)
AssertionError: 10 != 5
Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
======================================================================
FAIL: test_ref_and_ptr_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
   Test that the data formatters work on ref and ptr.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
    return attrvalue(self)
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 128, in test_ref_and_ptr
    (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
    return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
    test.assertEqual(process.GetState(), lldb.eStateStopped)
AssertionError: 10 != 5
Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
======================================================================
FAIL: test_with_run_command_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
   Test that that file and class static variables display correctly.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
    return attrvalue(self)
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 49, in test_with_run_command
    (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
    return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
    test.assertEqual(process.GetState(), lldb.eStateStopped)
AssertionError: 10 != 5
Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
======================================================================
FAIL: test_with_run_command_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
   Test that that file and class static variables display correctly.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
    return attrvalue(self)
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 49, in test_with_run_command
    (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
    return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
    test.assertEqual(process.GetState(), lldb.eStateStopped)
AssertionError: 10 != 5
Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
----------------------------------------------------------------------
Ran 8 tests in 3.109s

RESULT: FAILED (0 passes, 4 failures, 0 errors, 4 skipped, 0 expected failures, 0 unexpected successes)

--

Any ideas?

So this'll add the right test dependency, if the libcxx project is enabled in the build? (& if the build hasn't enabled libcxx, the tests will run, but against the system compiler - and if that system compiler happens to default to libc++ ( https://github.com/llvm/llvm-project/blob/e9e4fc0fd3e0780207c731a1f2b8f6aacd24e8f8/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/list/TestDataFormatterLibcxxList.py#L49 )) that should remove the "different results if you run ninja check-lldb before or after ninja check-libcxx" issue? Great, thanks!

Hmm, it doesn't appear to be working for me, running this (in a clean build directly, synced to 74c4d44d47b282769f6584153e9b433e8e5fa671 which includes 366fb539485a9753e4a8167fe5140bf4fb00a098 (& validated by inspecting lldb/test/CMakeLists.txt) still shows the tests as unsupported:

CC=clang CXX=clang++ cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_WERROR=true -DLLVM_ENABLE_PROJECTS='llvm;clang;clang-tools-extra;lld;lldb;cross-project-tests' -DLLVM_ENABLE_RUNTIMES='compiler-rt;libcxx;libcxxabi;libunwind' -DCMAKE_INSTALL_PREFIX=$HOME/install ~/dev/llvm/src/llvm
ninja check-lldb-api-functionalities-data-formatter-data-formatter-stl-libcxx

And then running ninja cxx and the same check again:

********************
FAIL: lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py (23 of 23)
******************** TEST 'lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py' FAILED ********************
Script:
--
/usr/bin/python3.9 /usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/dotest.py -u CXXFLAGS -u CFLAGS --env ARCHIVER=/usr/bin/ar --env OBJCOPY=/usr/bin/objcopy --env LLVM_LIBS_DIR=/usr/local/google/home/blaikie/dev/llvm/build/release/./lib --arch x86_64 --build-dir /usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex --lldb-module-cache-dir /usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/module-cache-lldb/lldb-api --clang-module-cache-dir /usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/module-cache-clang/lldb-api --executable /usr/local/google/home/blaikie/dev/llvm/build/release/./bin/lldb --compiler /usr/local/google/home/blaikie/dev/llvm/build/release/./bin/clang --dsymutil /usr/local/google/home/blaikie/dev/llvm/build/release/./bin/dsymutil --llvm-tools-dir /usr/local/google/home/blaikie/dev/llvm/build/release/./bin --lldb-libs-dir /usr/local/google/home/blaikie/dev/llvm/build/release/./lib /usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set -p TestDataFormatterLibcxxSet.py
--
Exit Code: 1

Command Output (stdout):
--
lldb version 14.0.0 (git@github.com:llvm/llvm-project.git revision 74c4d44d47b282769f6584153e9b433e8e5fa671)
  clang revision 74c4d44d47b282769f6584153e9b433e8e5fa671
  llvm revision 74c4d44d47b282769f6584153e9b433e8e5fa671
Skipping the following test categories: ['dsym', 'gmodules', 'debugserver', 'objc']

--
Command Output (stderr):
--
UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_dsym (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run) 
FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_gmodules (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run) 
UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_dsym (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run) 
FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_gmodules (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run) 
======================================================================
FAIL: test_ref_and_ptr_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
   Test that the data formatters work on ref and ptr.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
    return attrvalue(self)
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 128, in test_ref_and_ptr
    (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
    return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
    test.assertEqual(process.GetState(), lldb.eStateStopped)
AssertionError: 10 != 5
Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
======================================================================
FAIL: test_ref_and_ptr_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
   Test that the data formatters work on ref and ptr.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
    return attrvalue(self)
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 128, in test_ref_and_ptr
    (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
    return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
    test.assertEqual(process.GetState(), lldb.eStateStopped)
AssertionError: 10 != 5
Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
======================================================================
FAIL: test_with_run_command_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
   Test that that file and class static variables display correctly.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
    return attrvalue(self)
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 49, in test_with_run_command
    (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
    return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
    test.assertEqual(process.GetState(), lldb.eStateStopped)
AssertionError: 10 != 5
Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
======================================================================
FAIL: test_with_run_command_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
   Test that that file and class static variables display correctly.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
    return attrvalue(self)
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 49, in test_with_run_command
    (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
    return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
    test.assertEqual(process.GetState(), lldb.eStateStopped)
AssertionError: 10 != 5
Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
----------------------------------------------------------------------
Ran 8 tests in 3.109s

RESULT: FAILED (0 passes, 4 failures, 0 errors, 4 skipped, 0 expected failures, 0 unexpected successes)

--

Any ideas?

I think this is more of a speculative fix (the assert message doesn't have that much information beside that the process crashes). Can you apply https://reviews.llvm.org/D111978 and see what the error output with that patch is? It should give you exit status/description and stderr.

So this'll add the right test dependency, if the libcxx project is enabled in the build? (& if the build hasn't enabled libcxx, the tests will run, but against the system compiler - and if that system compiler happens to default to libc++ ( https://github.com/llvm/llvm-project/blob/e9e4fc0fd3e0780207c731a1f2b8f6aacd24e8f8/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/list/TestDataFormatterLibcxxList.py#L49 )) that should remove the "different results if you run ninja check-lldb before or after ninja check-libcxx" issue? Great, thanks!

Hmm, it doesn't appear to be working for me, running this (in a clean build directly, synced to 74c4d44d47b282769f6584153e9b433e8e5fa671 which includes 366fb539485a9753e4a8167fe5140bf4fb00a098 (& validated by inspecting lldb/test/CMakeLists.txt) still shows the tests as unsupported:

CC=clang CXX=clang++ cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_WERROR=true -DLLVM_ENABLE_PROJECTS='llvm;clang;clang-tools-extra;lld;lldb;cross-project-tests' -DLLVM_ENABLE_RUNTIMES='compiler-rt;libcxx;libcxxabi;libunwind' -DCMAKE_INSTALL_PREFIX=$HOME/install ~/dev/llvm/src/llvm
ninja check-lldb-api-functionalities-data-formatter-data-formatter-stl-libcxx

And then running ninja cxx and the same check again:

********************
FAIL: lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py (23 of 23)
******************** TEST 'lldb-api :: functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py' FAILED ********************
Script:
--
/usr/bin/python3.9 /usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/dotest.py -u CXXFLAGS -u CFLAGS --env ARCHIVER=/usr/bin/ar --env OBJCOPY=/usr/bin/objcopy --env LLVM_LIBS_DIR=/usr/local/google/home/blaikie/dev/llvm/build/release/./lib --arch x86_64 --build-dir /usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex --lldb-module-cache-dir /usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/module-cache-lldb/lldb-api --clang-module-cache-dir /usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/module-cache-clang/lldb-api --executable /usr/local/google/home/blaikie/dev/llvm/build/release/./bin/lldb --compiler /usr/local/google/home/blaikie/dev/llvm/build/release/./bin/clang --dsymutil /usr/local/google/home/blaikie/dev/llvm/build/release/./bin/dsymutil --llvm-tools-dir /usr/local/google/home/blaikie/dev/llvm/build/release/./bin --lldb-libs-dir /usr/local/google/home/blaikie/dev/llvm/build/release/./lib /usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set -p TestDataFormatterLibcxxSet.py
--
Exit Code: 1

Command Output (stdout):
--
lldb version 14.0.0 (git@github.com:llvm/llvm-project.git revision 74c4d44d47b282769f6584153e9b433e8e5fa671)
  clang revision 74c4d44d47b282769f6584153e9b433e8e5fa671
  llvm revision 74c4d44d47b282769f6584153e9b433e8e5fa671
Skipping the following test categories: ['dsym', 'gmodules', 'debugserver', 'objc']

--
Command Output (stderr):
--
UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_dsym (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run) 
FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_ref_and_ptr_gmodules (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run) 
UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_dsym (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run) 
FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
FAIL: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
UNSUPPORTED: LLDB (/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang-x86_64) :: test_with_run_command_gmodules (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase) (test case does not fall in any category of interest for this run) 
======================================================================
FAIL: test_ref_and_ptr_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
   Test that the data formatters work on ref and ptr.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
    return attrvalue(self)
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 128, in test_ref_and_ptr
    (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
    return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
    test.assertEqual(process.GetState(), lldb.eStateStopped)
AssertionError: 10 != 5
Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
======================================================================
FAIL: test_ref_and_ptr_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
   Test that the data formatters work on ref and ptr.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
    return attrvalue(self)
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 128, in test_ref_and_ptr
    (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
    return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
    test.assertEqual(process.GetState(), lldb.eStateStopped)
AssertionError: 10 != 5
Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
======================================================================
FAIL: test_with_run_command_dwarf (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
   Test that that file and class static variables display correctly.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
    return attrvalue(self)
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 49, in test_with_run_command
    (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
    return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
    test.assertEqual(process.GetState(), lldb.eStateStopped)
AssertionError: 10 != 5
Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
======================================================================
FAIL: test_with_run_command_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
   Test that that file and class static variables display correctly.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
    return attrvalue(self)
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 49, in test_with_run_command
    (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 970, in run_to_source_breakpoint
    return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 892, in run_to_breakpoint_do_run
    test.assertEqual(process.GetState(), lldb.eStateStopped)
AssertionError: 10 != 5
Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
----------------------------------------------------------------------
Ran 8 tests in 3.109s

RESULT: FAILED (0 passes, 4 failures, 0 errors, 4 skipped, 0 expected failures, 0 unexpected successes)

--

Any ideas?

I think this is more of a speculative fix (the assert message doesn't have that much information beside that the process crashes).

Oh, mostly my point was "the output changes after I explicitly run ninja cxx" so it looks like the dependency didn't fully address the build consistency issue, regardless of what the failure actually is?

Can you apply https://reviews.llvm.org/D111978 and see what the error output with that patch is? It should give you exit status/description and stderr.

Sure, I'll give that a go (was going to test it once it was submitted), but emulating something similar to the test, by debugging the binary directly:

$ ./bin/lldb ./lldb-test-build.noindex/functionalities/data-formatter/data-formatter-stl/libcxx/map/TestDataFormatterLibccMap.test_with_run_command_dwarf/a.out
(lldb) target create "./lldb-test-build.noindex/functionalities/data-formatter/data-formatter-stl/libcxx/map/TestDataFormatterLibccMap.test_with_run_command_dwarf/a.out"
Current executable set to '/usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/functionalities/data-formatter/data-formatter-stl/libcxx/map/TestDataFormatterLibccMap.test_with_run_command_dwarf/a.out' (x86_64).
(lldb) start
error: 'start' is not a valid command.
(lldb) r
warning: (x86_64) /lib/x86_64-linux-gnu/ld-2.32.so Unable to initialize decompressor for section '.debug_abbrev': zlib is not available
warning: (x86_64) /lib/x86_64-linux-gnu/ld-2.32.so Unable to initialize decompressor for section '.debug_info': zlib is not available
Process 3723631 launched: '/usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/functionalities/data-formatter/data-formatter-stl/libcxx/map/TestDataFormatterLibccMap.test_with_run_command_dwarf/a.out' (x86_64)
warning: (x86_64) /lib/x86_64-linux-gnu/ld-2.32.so Unable to initialize decompressor for section '.debug_aranges': zlib is not available
warning: (x86_64) /lib64/ld-linux-x86-64.so.2 Unable to initialize decompressor for section '.debug_abbrev': zlib is not available
warning: (x86_64) /lib64/ld-linux-x86-64.so.2 Unable to initialize decompressor for section '.debug_info': zlib is not available
/usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/functionalities/data-formatter/data-formatter-stl/libcxx/map/TestDataFormatterLibccMap.test_with_run_command_dwarf/a.out: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory

I don't /think/ there's any reason (given the current Cmake configuration/code/etc) that the binary should be able to find libc++.so.1? In the libc++ tests (not the lldb libc++ tests, but the libc++ libc++ tests) they specify -rpath when compiling libc++ binaries against the just-built libc++ so they'll find the just-built libc++.so. I don't see anything like that in the lldb libc++ tests/build.

Yeah, applying that patch gets the expected cannot open shared object file issue for libc++.so.1:

======================================================================
FAIL: test_with_run_command_dwo (TestDataFormatterLibcxxSet.LibcxxSetDataFormatterTestCase)
   Test that that file and class static variables display correctly.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1823, in test_method
    return attrvalue(self)
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/test/API/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py", line 49, in test_with_run_command
    (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint(
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 1007, in run_to_source_breakpoint
    return run_to_breakpoint_do_run(test, target, breakpoint, launch_info,
  File "/usr/local/google/home/blaikie/dev/llvm/src/lldb/packages/Python/lldbsuite/test/lldbutil.py", line 924, in run_to_breakpoint_do_run
    test.fail("Test process is not stopped at breakpoint, but instead in" +
AssertionError: Test process is not stopped at breakpoint, but instead in state 'exited'. Exit code/status: 127. Exit description: None.
stderr of inferior:
/usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.test_with_run_command_dwo/a.out: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory

Config=x86_64-/usr/local/google/home/blaikie/dev/llvm/build/release/bin/clang
----------------------------------------------------------------------

Oh, mostly my point was "the output changes after I explicitly run ninja cxx" so it looks like the dependency didn't fully address the build consistency issue, regardless of what the failure actually is?

This looks like a classic example of the reason why if(TARGET FOO) is discouraged in cmake.
The top level cmake file looks like this:

add_subdirectory(projects) # libcxx IN LLVM_ENABLE_PROJECTS processed here

if( LLVM_INCLUDE_TOOLS )
  add_subdirectory(tools) # lldb IN LLVM_ENABLE_PROJECTS processed here
endif()

if( LLVM_INCLUDE_RUNTIMES )
  add_subdirectory(runtimes) # libcxx in LLVM_ENABLE_RUNTIMES processed here
endif()

If one enables libcxx via LLVM_ENABLE_PROJECTS (which is I guess what the apple folks are using) then all is fine, as the cxx target will be defined by the time we get to lldb. OTOH, if one uses LLVM_ENABLE_RUNTIMES, then the check will return false (and not add the dependency), even though the target will be defined afterwards (and will be found by clang at runtime).
I think (but I haven't verified) that this could be fixed by replacing TARGET cxx with libcxx IN LLVM_ENABLE_PROJECTS OR libcxx IN LLVM_ENABLE_RUNTIMES.

Can you apply https://reviews.llvm.org/D111978 and see what the error output with that patch is? It should give you exit status/description and stderr.

Sure, I'll give that a go (was going to test it once it was submitted), but emulating something similar to the test, by debugging the binary directly:

$ ./bin/lldb ./lldb-test-build.noindex/functionalities/data-formatter/data-formatter-stl/libcxx/map/TestDataFormatterLibccMap.test_with_run_command_dwarf/a.out
(lldb) target create "./lldb-test-build.noindex/functionalities/data-formatter/data-formatter-stl/libcxx/map/TestDataFormatterLibccMap.test_with_run_command_dwarf/a.out"
Current executable set to '/usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/functionalities/data-formatter/data-formatter-stl/libcxx/map/TestDataFormatterLibccMap.test_with_run_command_dwarf/a.out' (x86_64).
(lldb) start
error: 'start' is not a valid command.
(lldb) r
warning: (x86_64) /lib/x86_64-linux-gnu/ld-2.32.so Unable to initialize decompressor for section '.debug_abbrev': zlib is not available
warning: (x86_64) /lib/x86_64-linux-gnu/ld-2.32.so Unable to initialize decompressor for section '.debug_info': zlib is not available
Process 3723631 launched: '/usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/functionalities/data-formatter/data-formatter-stl/libcxx/map/TestDataFormatterLibccMap.test_with_run_command_dwarf/a.out' (x86_64)
warning: (x86_64) /lib/x86_64-linux-gnu/ld-2.32.so Unable to initialize decompressor for section '.debug_aranges': zlib is not available
warning: (x86_64) /lib64/ld-linux-x86-64.so.2 Unable to initialize decompressor for section '.debug_abbrev': zlib is not available
warning: (x86_64) /lib64/ld-linux-x86-64.so.2 Unable to initialize decompressor for section '.debug_info': zlib is not available
/usr/local/google/home/blaikie/dev/llvm/build/release/lldb-test-build.noindex/functionalities/data-formatter/data-formatter-stl/libcxx/map/TestDataFormatterLibccMap.test_with_run_command_dwarf/a.out: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory

I don't /think/ there's any reason (given the current Cmake configuration/code/etc) that the binary should be able to find libc++.so.1? In the libc++ tests (not the lldb libc++ tests, but the libc++ libc++ tests) they specify -rpath when compiling libc++ binaries against the just-built libc++ so they'll find the just-built libc++.so. I don't see anything like that in the lldb libc++ tests/build.

Yeah, we'd have to take some positive action to make that hapen.
I think the best way to go about that is to call registerSharedLibrariesWithTarget under the right circumstances. That would ensure ((DY)LD_LIBRARY_)PATH is set, and also copy the library for remote tests. I'm just not entirely sure what are "the right circumstances".