[WIP][libc++] Add an executor that bundles tests for deferred execution

[Mordante]

Summary

Instead of actually running tests, this executor instead bundles the
executable and its dependencies into a tarball, and then creates a
self-contained script that can be used to run that test later.

The script contains the tarball of dependencies embedded inside itself,
so one can simply copy the script anywhere and execute it, provided the
machine and the executable have compatible architectures. This is an
alternative to SSH executors, which run tests remotely as lit executes
instead of deferring that execution. Deferring execution allows doing
batch copies and executes, which can be much faster than SSHing to a
remote machine for each test.

However, there are still some problems with this approach as of right now:

1. Lit doesn't have a notion of "deferred execution", so tests that should actually run a command will be marked as PASS even though they might fail when the test is actually run. Simmilarly, if the test is expected to XFAIL at runtime, lit will think that the test XPASSed if the bundling step succeeds.
2. Similarly, when running the tests on the target machine, any XFAIL test that (correctly) fails will kind-of be reported as a failure, since the executable will return a non-zero code.
3. When running a ShTest that has multiple RUN commands (all prefixed by %{exec}), the bundle will be re-created for each line instead of having a single bundle for all RUN lines. This basically doesn't work at all.

I believe there are solutions to all of the above, but solving them
properly will require some thinking and some changes to lit. Here are
some ideas:

• Lit could have a notion of HOST-RUN: and TARGET-RUN: lines. The idea is that Lit wouldn't even try running TARGET-RUN: lines on the host, and if any such line exists, the test would be reported as having a new status HOST-PASS or TARGET-DEFERRED or whatever. That new test status would represent that Lit can't know for sure yet whether the test will pass on the target. That would solve (1).

• In addition to the above, we could solve (2) by running Lit on the target in a mode where only the TARGET-RUN: lines are executed. Unfortunately, this requires the target to support running Lit, which is not always reasonable (imagine small embedded devices).

• Lit could support multi-line RUN: lines, where it would basically create a small script and consider it like a single RUN line. We could then use the %{exec} substitution on such a multi-line RUN instead of having multiple calls to %{exec}. This would solve (3). Alternatively, Lit could have builtin support for running HOST-RUN and TARGET-RUN lines specially.

[ldionne]

This review is a WIP for solving remote execution of libc++ tests with lit better than we currently do. See the review summary for the initial discussion.

Adding a bunch of folks that might be interested by the discussion -- feel free to unsubscribe if you have no interest!

[ldionne]

Also, just to explain how I'm currently running the tests with this patch applied:

rsync -am --include='*.bundle' --include='*/' --exclude='*' $PWD/build $PWD/all-tests

for bundle in $(find all-tests -name '*.bundle'); do
    if ! sh ${bundle}; then
        echo ${bundle} failed
    fi
done

I'm currently running them locally, but if the tests were compiled for some device, I could just as easily copy the bundles over and execute them with a single ssh command.

[yln]

This is very exciting! I would definitely want to adopt this for compiler-rt tests. I am sure the Swift folks would be interested in this as well.

My unorganized thoughts:
I agree that we should teach lit the notion of "deferred execution" in some form or another to make this work. We essentially need to be able to break up tests into separate steps: setup (compile, generate inputs on host), execute (run on device), assert (inspect outputs on host) and then change execution to do thing stage-by-stage: setup, copy to device, run on device, copy back results, finish tests; instead of test-by-test. I think most of compiler-rt tests would be amendable to this without changes.

Lit could have a notion of HOST-RUN: and TARGET-RUN: lines.

Did you consider inferring this via the presence of %{exec}? In compiler-rt, we have %run, which is empty if host==target, but does simctl spawn for the iOS simulator.

In addition to the above, we could solve (2) by running Lit on the target in a mode where only the TARGET-RUN: lines are executed. Unfortunately, this requires the target to support running Lit, which is not always reasonable (imagine small embedded devices).

I think we should keep the execute step on the target as "dumb" as possible, i.e., no running lit on the target. I would restrict the target to do what's specified via %{exec}: execute the copied binary and produce outputs.

Lit could support multi-line RUN: lines, where it would basically create a small script and consider it like a single RUN line. We could then use the %{exec} substitution on such a multi-line RUN instead of having multiple calls to %{exec}. This would solve (3). Alternatively, Lit could have builtin support for running HOST-RUN and TARGET-RUN lines specially.

Did you consider to generate one bundle per test including all of the %{exec} lines, instead one bundle per %{exec} line?

[ldionne]

Rebase onto main and on top of the from-scratch config.

Dan Albert and I spoke today and he may be interested in picking up this work,
which I've had to put on the side for lack of time.

[danalbert]

Re problems 1 and 2 (xfail handling), my idea what that we'd add --xfail to the bundler interface and delegate to the test runner to handle that. For Android I need to fit these tests into tradefed, so our bundler would work by creating the test config files for tradefed and would let it handle xfails. We don't have much reason to interact with LIT again after the tests are built. This more or less matches how we've been running libc++ tests for years, though with some improvements (xfail info is currently manually tracked rather than being given by LIT). I've been carrying a local patch that adds a --build-only flag which just ignores xfail annotations and defers all the execution and interpretation to our test runner.

@ldionne do you have an example for 3? Shell tests were hardly a thing the last time I spent any real time here so I'm not sure why multiple run lines would be used in a test.

One other thought is that it would be beneficial for us if we could build one bundle rather than a bundle per test, but that's probably impractical.

[danalbert]

Initial pass at my suggestion of deferring XFAIL handling to the bundler. This is incomplete because we'll need to differentiate XFAIL-BUILD and XFAIL-RUN in tests (running this as-is makes any test that is XFAIL because it doesn't _build_ turn to FAIL, but things that fail to _run_ are handled correctly), but it's enough to get feedback on the idea.

[danalbert]

Adds XFAIL-BUILD to deal with the deferred-run cases that fail to build.

[danalbert]

Fix remaining tests for the deferred test configuration.

[Mordante]

This patch has been inactive for quite a while. We want to cleanup the libc++ review quite before Phabricator becomes read-only I'll close this revision now. Please open a GitHub PR if you want to pursue this patch.

libcxx/test/configs/libcxx-trunk-bundled.cfg.in

+@AUTO_GEN_COMMENT@
+
+LIBCXX_ROOT = "@LIBCXX_SOURCE_DIR@"
+INSTALL_ROOT = "@CMAKE_BINARY_DIR@"
+COMPILER = "@CMAKE_CXX_COMPILER@"
+EXEC_ROOT = "@LIBCXX_BINARY_DIR@"
+CMAKE_OSX_SYSROOT = "@CMAKE_OSX_SYSROOT@";
+
+import os
+import pipes
+import site
+import sys
+site.addsitedir(os.path.join(LIBCXX_ROOT, 'utils'))
+import libcxx.test.features
+import libcxx.test.format
+import libcxx.test.newconfig
+import libcxx.test.params
+
+# Configure basic properties of the test suite
+config.name = 'libcxx-trunk-bundled'
+config.target_triple = "@TARGET_TRIPLE@"
+config.test_source_root = os.path.join(LIBCXX_ROOT, 'test')
+config.test_format = libcxx.test.format.CxxStandardLibraryTest()
+config.recursiveExpansionLimit = 10
+config.test_exec_root = EXEC_ROOT
+
+# Configure basic substitutions
+bundlePy = os.path.join(LIBCXX_ROOT, 'utils', 'bundle.py')
+config.substitutions.append(('%{cxx}', COMPILER))
+config.substitutions.append(('%{flags}',
+    '-isysroot {}'.format(CMAKE_OSX_SYSROOT) if CMAKE_OSX_SYSROOT else ''
+))
+config.substitutions.append(('%{compile_flags}',
+    '-nostdinc++ -isystem {} -I {}'.format(
+        os.path.join(INSTALL_ROOT, 'include', 'c++', 'v1'),
+        os.path.join(LIBCXX_ROOT, 'test', 'support'))
+))
+config.substitutions.append(('%{link_flags}',
+    '-nostdlib++ {} {} -latomic -lpthread'.format(
+        os.path.join(INSTALL_ROOT, 'lib', 'libc++.a'),
+        os.path.join(INSTALL_ROOT, 'lib', 'libc++abi.a'))
+))
+config.substitutions.append(('%{exec}',
+    '{} {} %{{exec_args}} --bundle %t.bundle -- '.format(
+        pipes.quote(sys.executable),
+        pipes.quote(bundlePy))
+))
+
+config.available_features.add("run-deferred")
+
+# Add parameters and features to the config
+libcxx.test.newconfig.configure(
+    libcxx.test.params.DEFAULT_PARAMETERS,
+    libcxx.test.features.DEFAULT_FEATURES,
+    config,
+    lit_config
+)

libcxx/test/libcxx/debug/extern-templates.sh.cpp

 // for members of <locale> even when the debug mode is enabled, which is
 // necessary for correctness. See https://llvm.org/D94718 for details.
 
 // UNSUPPORTED: libcxx-no-debug-mode
 // UNSUPPORTED: libcpp-has-no-localization
 
 // XFAIL: LIBCXX-WINDOWS-FIXME
 
+// BUILT_DEPENDENCIES: %t.lib
 // RUN: %{cxx} %{flags} %{compile_flags} %s %{link_flags} -fPIC -DTU1 -D_LIBCPP_DEBUG=1 -fvisibility=hidden -shared -o %t.lib
 // RUN: cd %T && %{cxx} %{flags} %{compile_flags} %s ./%basename_t.tmp.lib %{link_flags} -fPIC -DTU2 -D_LIBCPP_DEBUG=1 -fvisibility=hidden -o %t.exe
 // RUN: %{exec} %t.exe
 
 #include <cassert>
 #include <cstdio>
 #include <sstream>
 #include <string>

libcxx/test/libcxx/selftest/dsl/dsl.sh.py

 #===----------------------------------------------------------------------===##
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
 #===----------------------------------------------------------------------===##
 
+# UNSUPPORTED: run-deferred
 # XFAIL: LIBCXX-WINDOWS-FIXME
 
 # Note: We prepend arguments with 'x' to avoid thinking there are too few
 #       arguments in case an argument is an empty string.
 # RUN: %{python} %s x%S \
 # RUN:              x%T \
 # RUN:              x%{escaped_exec} \
 # RUN:              x%{escaped_cxx} \

libcxx/test/libcxx/selftest/pass.cpp/compile-error.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
-// XFAIL: *
+// XFAIL-BUILD: *
 
 // Make sure the test DOES NOT pass if it fails at compile-time
 
 struct Foo { };
 typedef Foo::x x;
 
 int main(int, char**) { return 0; }

libcxx/test/libcxx/selftest/pass.cpp/link-error.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
-// XFAIL: *
+// XFAIL-BUILD: *
 
 // Make sure the test DOES NOT pass if it fails at link-time
 
 extern void this_is_an_undefined_symbol();
 
 int main(int, char**) {
     this_is_an_undefined_symbol();
     return 0;
 }

libcxx/test/libcxx/selftest/pass.cpp/werror.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
-// XFAIL: *
+// XFAIL-BUILD: *
 
 // Make sure that even a simple unused variable warning is treated as an
 // error in the test suite. This is to make sure the test suite always runs
 // with -Werror.
 
 // ADDITIONAL_COMPILE_FLAGS: -Wunused-variable
 
 // TODO: We don't enable -Werror on GCC right now, because too many tests fail.
 // UNSUPPORTED: gcc
 
 int main(int, char**) {
     int foo;
 }

libcxx/test/libcxx/selftest/pass.mm/compile-error.pass.mm

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // REQUIRES: objective-c++
 
-// XFAIL: *
+// XFAIL-BUILD: *
 
 // Make sure the test DOES NOT pass if it fails at compile-time
 
 struct Foo { };
 typedef Foo::x x;
 
 int main(int, char**) { return 0; }

libcxx/test/libcxx/selftest/pass.mm/link-error.pass.mm

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // REQUIRES: objective-c++
 
-// XFAIL: *
+// XFAIL-BUILD: *
 
 // Make sure the test DOES NOT pass if it fails at link-time
 
 extern void this_is_an_undefined_symbol();
 
 int main(int, char**) {
     this_is_an_undefined_symbol();
     return 0;
 }

libcxx/test/libcxx/strings/c.strings/version_cuchar.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
-// XFAIL: *
+// XFAIL-BUILD: *
 
 // Skip this test on windows. If built on top of the MSVC runtime, the
 // <cuchar> header actually does exist (although not provided by us).
 // This should be removed once D97870 has landed.
 // UNSUPPORTED: windows
 
 // <cuchar>
 

libcxx/test/std/input.output/filesystems/class.path/path.nonmember/path.io.unicode_bug.pass.cpp

 // template <class charT, class traits>
 // basic_istream<charT, traits>&
 // operator>>(basic_istream<charT, traits>& is, path& p)
 //
 
 // TODO(EricWF) This test fails because "std::quoted" fails to compile
 // for char16_t and char32_t types. Combine with path.io.pass.cpp when this
 // passes.
-// XFAIL: *
+// XFAIL-BUILD: *
 
 #include "filesystem_include.h"
 #include <type_traits>
 #include <sstream>
 #include <cassert>
 
 #include "test_macros.h"
 #include "test_iterators.h"

libcxx/test/std/strings/c.strings/cuchar.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
-// XFAIL: libc++
+// XFAIL-BUILD: libc++
 
 // Skip this test on windows. If built on top of the MSVC runtime, the
 // <cuchar> header actually does exist (although not provided by us).
 // This should be removed once D97870 has landed.
 // UNSUPPORTED: windows
 
 // <cuchar>
 

libcxx/test/std/utilities/function.objects/func.search/func.search.bm/default.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // UNSUPPORTED: c++03, c++11, c++14
-// XFAIL: libc++
+// XFAIL-BUILD: libc++
 
 // <functional>
 
 // boyer_moore searcher
 // template<class RandomAccessIterator1,
 //          class Hash = hash<typename iterator_traits<RandomAccessIterator1>::value_type>,
 //          class BinaryPredicate = equal_to<>>
 // class boyer_moore_searcher {

libcxx/test/std/utilities/function.objects/func.search/func.search.bm/hash.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // UNSUPPORTED: c++03, c++11, c++14
-// XFAIL: libc++
+// XFAIL-BUILD: libc++
 
 // <functional>
 
 // boyer_moore searcher
 // template<class RandomAccessIterator1,
 //          class Hash = hash<typename iterator_traits<RandomAccessIterator1>::value_type>,
 //          class BinaryPredicate = equal_to<>>
 // class boyer_moore_searcher {

libcxx/test/std/utilities/function.objects/func.search/func.search.bm/hash.pred.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // UNSUPPORTED: c++03, c++11, c++14
-// XFAIL: libc++
+// XFAIL-BUILD: libc++
 
 // <functional>
 
 // boyer_moore searcher
 // template<class RandomAccessIterator1,
 //          class Hash = hash<typename iterator_traits<RandomAccessIterator1>::value_type>,
 //          class BinaryPredicate = equal_to<>>
 // class boyer_moore_searcher {

libcxx/test/std/utilities/function.objects/func.search/func.search.bm/pred.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // UNSUPPORTED: c++03, c++11, c++14
-// XFAIL: libc++
+// XFAIL-BUILD: libc++
 
 // <functional>
 
 // boyer_moore searcher
 // template<class RandomAccessIterator1,
 //          class Hash = hash<typename iterator_traits<RandomAccessIterator1>::value_type>,
 //          class BinaryPredicate = equal_to<>>
 // class boyer_moore_searcher {

libcxx/test/std/utilities/function.objects/func.search/func.search.bmh/default.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // UNSUPPORTED: c++03, c++11, c++14
-// XFAIL: libc++
+// XFAIL-BUILD: libc++
 
 // <functional>
 
 // boyer_moore_horspool searcher
 // template<class RandomAccessIterator1,
 //          class Hash = hash<typename iterator_traits<RandomAccessIterator1>::value_type>,
 //          class BinaryPredicate = equal_to<>>
 // class boyer_moore_horspool_searcher {

libcxx/test/std/utilities/function.objects/func.search/func.search.bmh/hash.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // UNSUPPORTED: c++03, c++11, c++14
-// XFAIL: libc++
+// XFAIL-BUILD: libc++
 
 // <functional>
 
 // boyer_moore_horspool searcher
 // template<class RandomAccessIterator1,
 //          class Hash = hash<typename iterator_traits<RandomAccessIterator1>::value_type>,
 //          class BinaryPredicate = equal_to<>>
 // class boyer_moore_horspool_searcher {

libcxx/test/std/utilities/function.objects/func.search/func.search.bmh/hash.pred.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // UNSUPPORTED: c++03, c++11, c++14
-// XFAIL: libc++
+// XFAIL-BUILD: libc++
 
 // <functional>
 
 // boyer_moore_horspool searcher
 // template<class RandomAccessIterator1,
 //          class Hash = hash<typename iterator_traits<RandomAccessIterator1>::value_type>,
 //          class BinaryPredicate = equal_to<>>
 // class boyer_moore_horspool_searcher {

libcxx/test/std/utilities/function.objects/func.search/func.search.bmh/pred.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // UNSUPPORTED: c++03, c++11, c++14
-// XFAIL: libc++
+// XFAIL-BUILD: libc++
 
 // <functional>
 
 // boyer_moore_horspool searcher
 // template<class RandomAccessIterator1,
 //          class Hash = hash<typename iterator_traits<RandomAccessIterator1>::value_type>,
 //          class BinaryPredicate = equal_to<>>
 // class boyer_moore_horspool_searcher {

libcxx/test/std/utilities/time/time.cal/time.cal.day/time.cal.day.nonmembers/streaming.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // UNSUPPORTED: c++03, c++11, c++14, c++17
 // UNSUPPORTED: libcpp-has-no-localization
-// XFAIL: *
+// XFAIL-BUILD: *
 
 // <chrono>
 // class day;
 
 // template<class charT, class traits>
 //   basic_ostream<charT, traits>&
 //   operator<<(basic_ostream<charT, traits>& os, const day& d);
 //

libcxx/test/std/utilities/time/time.cal/time.cal.md/time.cal.md.nonmembers/streaming.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 // UNSUPPORTED: c++03, c++11, c++14, c++17
-// XFAIL: *
+// XFAIL-BUILD: *
 
 // <chrono>
 // class month_day;
 
 // template<class charT, class traits>
 //     basic_ostream<charT, traits>&
 //     operator<<(basic_ostream<charT, traits>& os, const month_day& md);
 //

libcxx/test/std/utilities/time/time.cal/time.cal.mdlast/streaming.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 // UNSUPPORTED: c++03, c++11, c++14, c++17
-// XFAIL: *
+// XFAIL-BUILD: *
 
 // <chrono>
 // class month_day_last;
 //
 // template<class charT, class traits>
 //     basic_ostream<charT, traits>&
 //     operator<<(basic_ostream<charT, traits>& os, const month_day_last& mdl);
 //

libcxx/test/std/utilities/time/time.cal/time.cal.month/time.cal.month.nonmembers/streaming.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 // UNSUPPORTED: c++03, c++11, c++14, c++17
-// XFAIL: *
+// XFAIL-BUILD: *
 
 // <chrono>
 // class month;
 
 // template<class charT, class traits>
 //   basic_ostream<charT, traits>&
 //   operator<<(basic_ostream<charT, traits>& os, const month& m);
 //

libcxx/test/std/utilities/time/time.cal/time.cal.mwd/time.cal.mwd.nonmembers/streaming.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 // UNSUPPORTED: c++03, c++11, c++14, c++17
-// XFAIL: *
+// XFAIL-BUILD: *
 
 // <chrono>
 // class month_weekday;
 
 // template<class charT, class traits>
 //     basic_ostream<charT, traits>&
 //     operator<<(basic_ostream<charT, traits>& os, const month_weekday& mwd);
 //

libcxx/test/std/utilities/time/time.cal/time.cal.mwdlast/time.cal.mwdlast.nonmembers/streaming.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 // UNSUPPORTED: c++03, c++11, c++14, c++17
-// XFAIL: *
+// XFAIL-BUILD: *
 
 // <chrono>
 // class month_weekday_last;
 //
 // template<class charT, class traits>
 //     basic_ostream<charT, traits>&
 //     operator<<(basic_ostream<charT, traits>& os, const month_weekday_last& mdl);
 //

libcxx/test/std/utilities/time/time.cal/time.cal.wdidx/time.cal.wdidx.nonmembers/streaming.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 // UNSUPPORTED: c++03, c++11, c++14, c++17
-// XFAIL: *
+// XFAIL-BUILD: *
 
 // <chrono>
 // class weekday_indexed;
 
 // template<class charT, class traits>
 //   basic_ostream<charT, traits>&
 //   operator<<(basic_ostream<charT, traits>& os, const weekday_indexed& wdi);
 //

libcxx/test/std/utilities/time/time.cal/time.cal.wdlast/time.cal.wdlast.nonmembers/streaming.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 // UNSUPPORTED: c++03, c++11, c++14, c++17
-// XFAIL: *
+// XFAIL-BUILD: *
 
 // <chrono>
 // class weekday_last;
 
 //   template<class charT, class traits>
 //     basic_ostream<charT, traits>&
 //     operator<<(basic_ostream<charT, traits>& os, const weekday_last& wdl);
 //

libcxx/test/std/utilities/time/time.cal/time.cal.weekday/time.cal.weekday.nonmembers/streaming.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 // UNSUPPORTED: c++03, c++11, c++14, c++17
-// XFAIL: *
+// XFAIL-BUILD: *
 
 // <chrono>
 // class weekday;
 
 // template<class charT, class traits>
 //   basic_ostream<charT, traits>&
 //   operator<<(basic_ostream<charT, traits>& os, const weekday& wd);
 //

libcxx/test/std/utilities/time/time.cal/time.cal.year/time.cal.year.nonmembers/streaming.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 // UNSUPPORTED: c++03, c++11, c++14, c++17
-// XFAIL: *
+// XFAIL-BUILD: *
 
 // <chrono>
 // class year;
 
 // template<class charT, class traits>
 //   basic_ostream<charT, traits>&
 //   operator<<(basic_ostream<charT, traits>& os, const year& y);
 //

libcxx/test/std/utilities/time/time.cal/time.cal.ym/time.cal.ym.nonmembers/streaming.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 // UNSUPPORTED: c++03, c++11, c++14, c++17
-// XFAIL: *
+// XFAIL-BUILD: *
 
 // <chrono>
 // class year_month;
 
 // template<class charT, class traits>
 //     basic_ostream<charT, traits>&
 //     operator<<(basic_ostream<charT, traits>& os, const year_month& ym);
 //

libcxx/test/std/utilities/time/time.cal/time.cal.ymd/time.cal.ymd.nonmembers/streaming.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 // UNSUPPORTED: c++03, c++11, c++14, c++17
-// XFAIL: *
+// XFAIL-BUILD: *
 
 // <chrono>
 // class year_month_day;
 
 // template<class charT, class traits>
 //     basic_ostream<charT, traits>&
 //     operator<<(basic_ostream<charT, traits>& os, const year_month_day& ym);
 //

libcxx/test/std/utilities/time/time.cal/time.cal.ymdlast/time.cal.ymdlast.nonmembers/streaming.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 // UNSUPPORTED: c++03, c++11, c++14, c++17
-// XFAIL: *
+// XFAIL-BUILD: *
 
 // <chrono>
 // class year_month_day_last;
 
 // template<class charT, class traits>
 //   basic_ostream<charT, traits>&
 //   operator<<(basic_ostream<charT, traits>& os, const year_month_day_last& ymdl);
 //

libcxx/test/std/utilities/time/time.cal/time.cal.ymwd/time.cal.ymwd.nonmembers/streaming.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 // UNSUPPORTED: c++03, c++11, c++14, c++17
-// XFAIL: *
+// XFAIL-BUILD: *
 
 // <chrono>
 // class year_month_weekday;
 
 // template<class charT, class traits>
 //     basic_ostream<charT, traits>&
 //     operator<<(basic_ostream<charT, traits>& os, const year_month_weekday& ym);
 //

libcxx/test/std/utilities/time/time.cal/time.cal.ymwdlast/time.cal.ymwdlast.nonmembers/streaming.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 // UNSUPPORTED: c++03, c++11, c++14, c++17
-// XFAIL: *
+// XFAIL-BUILD: *
 
 // <chrono>
 // class year_month_weekday_last;
 
 // template<class charT, class traits>
 //     basic_ostream<charT, traits>&
 //     operator<<(basic_ostream<charT, traits>& os, const year_month_weekday_last& ymwdl);
 //

libcxx/utils/bundle.py

+# ===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+# ===----------------------------------------------------------------------===##
+
+"""
+Bundles an executable along with its dependencies into a directory suitable
+for running an individual test at a later time.
+"""
+
+import argparse
+import base64
+from io import BytesIO, SEEK_SET
+from pathlib import Path
+import subprocess
+import tarfile
+from typing import Optional
+
+
+class Bundler:
+    def __init__(
+        self,
+        xfail: bool,
+        dependencies: list[Path],
+        env: list[str],
+        codesign_identity: Optional[str],
+    ) -> None:
+        self.xfail = xfail
+        self.dependencies = dependencies
+        self.env = env
+        self.codesign_identity = codesign_identity
+
+    @staticmethod
+    def is_test_exe(arg: str) -> bool:
+        # If an argument is a file that ends in `.tmp.exe`, assume it is the name of an
+        # executable generated by a test file. We call these test-executables below.
+        # This allows us to do custom processing like codesigning test-executables and
+        # changing their path when running from the bundle. It's also possible for there
+        # to be no such executable, for example in the case of a .sh.cpp test.
+        return arg.endswith(".tmp.exe") and Path(arg).exists()
+
+    def sign_if_needed(self, path: Path) -> None:
+        assert self.codesign_identity is not None
+        if self.is_test_exe(str(path)):
+            subprocess.run(
+                ["xcrun", "codesign", "-f", "-s", self.codesign_identity, str(path)],
+                check=True,
+                env={},
+            )
+
+    def sign_binaries(self) -> None:
+        if self.codesign_identity is None:
+            return
+        for dep in self.dependencies:
+            self.sign_if_needed(dep)
+
+    def create_bundle(self, output: Path, command: list[str]) -> None:
+        raise NotImplementedError
+
+
+class SelfExtractingTarballBundler(Bundler):
+    def create_bundle(self, output: Path, command: list[str]) -> None:
+        self.sign_binaries()
+        output.write_text("\n".join(self.create_script_lines(command)))
+        subprocess.run(["chmod", "+x", str(output)], check=True)
+
+    def bundle_path(self, path: Path) -> Path:
+        return self.bundle_var / path.name
+
+    @property
+    def bundle_var_name(self) -> str:
+        return "BUNDLE"
+
+    @property
+    def bundle_var(self) -> Path:
+        return Path(f"${{{self.bundle_var_name}}}")
+
+    def substitute_file_paths(self, arg: str) -> str:
+        for dep in self.dependencies:
+            if not dep.is_absolute:
+                continue
+            if str(dep) in arg:
+                arg = arg.replace(str(dep), str(self.bundle_path(dep)))
+        return f'"{arg}"'
+
+    def create_script_lines(self, command: list[str]) -> None:
+        lines = [
+            "#!/bin/sh",
+            "set -e",
+            "STATUS=0",
+            f'{self.bundle_var_name}="$(mktemp -d /tmp/libcxx.XXXXXXXXXX)"',
+        ]
+
+        # Get the tarball by decoding the end of the shell script, which embeds the
+        # tarball at the end of itself. Then, untar the dependencies in the temporary
+        # directory, and remove the intermediate tarball that was extracted from the
+        # script in the process.
+        tar_path = self.bundle_var / ".temporary.tar"
+        lines += [
+            f'tail -n +$[ $(grep -n "^BEGIN_TARBALL" "$0" | cut -d ":" -f 1) + 1 ] "${0}" | base64 --decode > "{tar_path}"',
+            f'tar -xf "{tar_path}" -C "{self.bundle_var}"',
+            f'rm "{tar_path}"',
+        ]
+
+        # Make sure all test-executables in the bundle have 'execute' permissions on the
+        # host where the bundle is run. The host that compiled the test-executable might
+        # not have a notion of 'executable' permissions.
+        for exe in (d for d in self.dependencies if self.is_test_exe(str(d))):
+            lines.append(f"chmod +x {self.bundle_path(exe)}")
+
+        # Execute the command in the temporary directory, with the correct environment.
+        # We tweak the command line to run it from the bundle by transforming the path
+        # of test-executables to their path in the temporary directory, where we know
+        # they have been untarred above.
+        lines.append(f'cd "{self.bundle_var}"')
+
+        if self.env:
+            lines.append(f"export {' '.join(self.env)}")
+
+        lines.append("set +e")
+
+        lines.append(" ".join(self.substitute_file_paths(x) for x in command))
+
+        comparison = "-eq" if self.xfail else "-ne"
+        lines.append(f"if [ $? {comparison} 0 ]; then STATUS=1; fi")
+
+        lines.append("set -e")
+
+        # Make sure the temporary directory is removed when we're done.
+        lines.append(f'rm -r "{self.bundle_var}"')
+
+        # The tarball itself is embedded at the end of the script so that it's
+        # sufficient to copy the script to get all dependencies. It's simpler than
+        # having to carry around a separate tarball.
+        lines += [
+            "exit $STATUS",
+            "BEGIN_TARBALL",
+            self.create_base64_tar().decode("utf-8"),
+        ]
+
+        return lines
+
+    def create_base64_tar(self) -> bytes:
+        f = BytesIO()
+        with tarfile.open(fileobj=f, mode="w") as tarball:
+            for dep in self.dependencies:
+                if not dep.exists():
+                    raise RuntimeError(f"Missing test dependency: {dep}")
+                tarball.add(dep, arcname=dep.name)
+        f.seek(0, SEEK_SET)
+        return base64.b64encode(f.read())
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--bundle", type=Path, required=True)
+    parser.add_argument("--codesign_identity")
+    parser.add_argument("--env", nargs="*", default=[])
+    parser.add_argument(
+        "--dependency", dest="dependencies", action="append", type=Path, default=[]
+    )
+    parser.add_argument("--xfail", action="store_true")
+    parser.add_argument("command", nargs="+")
+    args = parser.parse_args()
+
+    SelfExtractingTarballBundler(
+        args.xfail, args.dependencies, args.env, args.codesign_identity
+    ).create_bundle(args.bundle, args.command)
+
+
+if __name__ == "__main__":
+    main()

libcxx/utils/libcxx/test/features.py

return [
     AddFeature('with_system_cxx_lib={}-{}-{}{}'.format(arch, vendor, sysname, version)),
     AddFeature('with_system_cxx_lib={}{}'.format(sysname, version)),
     AddFeature('with_system_cxx_lib={}'.format(sysname)),
   ]
 DEFAULT_FEATURES += [
   Feature(name='__dummy_use_system_cxx_lib',
           when=lambda cfg: 'use_system_cxx_lib' in cfg.available_features,
           actions=_addSystemCxxLibDeclinations)
-]
+]
 No newline at end of file

libcxx/utils/libcxx/test/format.py

 #===----------------------------------------------------------------------===##
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
 #===----------------------------------------------------------------------===##
 
+from pathlib import Path
 import lit
 import lit.formats
 import os
-import pipes
 import re
-import shutil
 import subprocess
 
 def _supportsVerify(config):
     """
     Determine whether clang-verify is supported by the given configuration.
 
     This is done by checking whether the %{cxx} substitution in that
     configuration supports certain compiler flags.
def parseScript(test, preamble):
     # Check base substitutions and add the %{build} and %{run} convenience substitutions
     _checkBaseSubstitutions(substitutions)
     substitutions.append(('%{build}', '%{cxx} %s %{flags} %{compile_flags} %{link_flags} -o %t.exe'))
     substitutions.append(('%{run}', '%{exec} %t.exe'))
 
     # Parse the test file, including custom directives
     additionalCompileFlags = []
     fileDependencies = []
+    builtDependencies = ["%t.exe"]
+    test.build_xfails = []
     parsers = [
         lit.TestRunner.IntegratedTestKeywordParser('FILE_DEPENDENCIES:',
                                                    lit.TestRunner.ParserKind.LIST,
                                                    initial_value=fileDependencies),
+        lit.TestRunner.IntegratedTestKeywordParser('BUILT_DEPENDENCIES:',
+                                                   lit.TestRunner.ParserKind.LIST,
+                                                   initial_value=builtDependencies),
         lit.TestRunner.IntegratedTestKeywordParser('ADDITIONAL_COMPILE_FLAGS:',
                                                    lit.TestRunner.ParserKind.LIST,
-                                                   initial_value=additionalCompileFlags)
+                                                   initial_value=additionalCompileFlags),
+        lit.TestRunner.IntegratedTestKeywordParser('XFAIL-BUILD:',
+                                                   lit.TestRunner.ParserKind.LIST,
+                                                   initial_value=test.build_xfails),
     ]
 
     scriptInTest = lit.TestRunner.parseIntegratedTestScript(test, additional_parsers=parsers,
                                                             require_script=not preamble)
     if isinstance(scriptInTest, lit.Test.Result):
         return scriptInTest
 
+    # Copy any XFAIL-BUILD commands into the regular XFAIL set for the non-deferred case.
+    test.xfails.extend(test.build_xfails)
+
     script = []
 
+    exec_args = []
+    if test.isExpectedToFail():
+        exec_args.append("--xfail")
+
+    # TODO: Add --test-exe flag to clean up bundler implementation.
+    exec_args.extend(lit.TestRunner.applySubstitutions(
+        (f"--dependency={d}" for d in builtDependencies),
+        substitutions,
+        recursion_limit=test.config.recursiveExpansionLimit,
+    ))
+
     # For each file dependency in FILE_DEPENDENCIES, inject a command to copy
     # that file to the execution directory. Execute the copy from %S to allow
     # relative paths from the test directory.
+    test_dir = Path(test.getSourcePath()).parent
     for dep in fileDependencies:
         script += ['%dbg(SETUP) cd %S && cp {} %T'.format(dep)]
+        exec_args.append(f"--dependency={test_dir / dep}")
     script += preamble
     script += scriptInTest
 
     # Add compile flags specified with ADDITIONAL_COMPILE_FLAGS.
     substitutions = [(s, x + ' ' + ' '.join(additionalCompileFlags)) if s == '%{compile_flags}'
                             else (s, x) for (s, x) in substitutions]
 
+    substitutions.append(("%{exec_args}", " ".join(exec_args)))
+
     # Perform substitutions in the script itself.
     script = lit.TestRunner.applySubstitutions(script, substitutions,
                                                recursion_limit=test.config.recursiveExpansionLimit)
 
     return script
 
 
 class CxxStandardLibraryTest(lit.formats.TestFormat):
The test format will define the following substitutions for use inside tests:
             Expands to a command-line that builds the current source
             file with the %{flags}, %{compile_flags} and %{link_flags}
             substitutions, and that produces an executable named %t.exe.
 
         %{run}
             Equivalent to `%{exec} %t.exe`. This is intended to be used
             in conjunction with the %{build} substitution.
     """
+
     def getTestsInDirectory(self, testSuite, pathInSuite, litConfig, localConfig):
         SUPPORTED_SUFFIXES = ['[.]pass[.]cpp$', '[.]pass[.]mm$',
                               '[.]compile[.]pass[.]cpp$', '[.]compile[.]fail[.]cpp$',
                               '[.]link[.]pass[.]cpp$', '[.]link[.]fail[.]cpp$',
                               '[.]sh[.][^.]+$',
                               '[.]verify[.]cpp$',
                               '[.]fail[.]cpp$']
         sourcePath = testSuite.getSourcePath(pathInSuite)
def execute(self, test, litConfig):
             return self._executeShTest(test, litConfig, steps)
         # Make sure to check these ones last, since they will match other
         # suffixes above too.
         elif filename.endswith('.pass.cpp') or filename.endswith('.pass.mm'):
             steps = [
                 "%dbg(COMPILED WITH) %{cxx} %s %{flags} %{compile_flags} %{link_flags} -o %t.exe",
                 "%dbg(EXECUTED AS) %{exec} %t.exe"
             ]
-            return self._executeShTest(test, litConfig, steps)
+            return self._executeShTest(test, litConfig, steps, run_deferrable=True)
         # This is like a .verify.cpp test when clang-verify is supported,
         # otherwise it's like a .compile.fail.cpp test. This is only provided
         # for backwards compatibility with the test suite.
         elif filename.endswith('.fail.cpp'):
             if supportsVerify:
                 steps = [
                     "%dbg(COMPILED WITH) %{{cxx}} %s %{{flags}} %{{compile_flags}} -fsyntax-only -Wno-error {}".format(VERIFY_FLAGS)
                 ]
             else:
                 steps = [
                     "%dbg(COMPILED WITH) ! %{cxx} %s %{flags} %{compile_flags} -fsyntax-only"
                 ]
             return self._executeShTest(test, litConfig, steps)
         else:
             return lit.Test.Result(lit.Test.UNRESOLVED, "Unknown test suffix for '{}'".format(filename))
 
     # Utility function to add compile flags in lit.local.cfg files.
     def addCompileFlags(self, config, *flags):
         string = ' '.join(flags)
         config.substitutions = [(s, x + ' ' + string) if s == '%{compile_flags}' else (s, x) for (s, x) in config.substitutions]
 
-    def _executeShTest(self, test, litConfig, steps):
+    def _executeShTest(self, test, litConfig, steps, run_deferrable: bool = False):
         if test.config.unsupported:
             return lit.Test.Result(lit.Test.UNSUPPORTED, 'Test is unsupported')
 
         script = parseScript(test, steps)
         if isinstance(script, lit.Test.Result):
             return script
 
         if litConfig.noExecute:
             return lit.Test.Result(lit.Test.XFAIL if test.isExpectedToFail() else lit.Test.PASS)
-        else:
-            _, tmpBase = _getTempPaths(test)
-            useExternalSh = False
-            return lit.TestRunner._runShTest(test, litConfig, useExternalSh, script, tmpBase)
+
+        result = lit.TestRunner._runShTest(
+            test, litConfig, useExternalSh=False, script=script, tmpBase=tmpBase
+        )
+        if run_deferrable and "run-deferred" in test.config.available_features:
+            test.xfails = test.build_xfails
+        return result

libcxx/utils/run_bundles.py

+#!/usr/bin/env python
+from argparse import ArgumentParser
+import concurrent.futures
+from concurrent.futures import ProcessPoolExecutor
+from dataclasses import dataclass
+import logging
+from pathlib import Path
+import subprocess
+import sys
+
+
+@dataclass(frozen=True)
+class TestResult:
+    path: Path
+    passed: bool
+    output: str
+
+    @property
+    def failed(self) -> bool:
+        return not self.passed
+
+    def __str__(self) -> str:
+        status = "PASS" if self.passed else "FAIL"
+        output = f"\n{self.output}" if self.output else ""
+        return f"{status}: {self.path}{output}"
+
+
+def run_bundle(path: Path) -> TestResult:
+    result = subprocess.run(
+        [str(path)], stdout=subprocess.PIPE, stderr=subprocess.STDOUT
+    )
+    try:
+        output = result.stdout.decode("utf-8")
+    except UnicodeDecodeError:
+        output = repr(result.stdout)
+    return TestResult(path, result.returncode == 0, output)
+
+
+class BundleRunner:
+    def __init__(self, test_dir: Path) -> None:
+        self.test_dir = test_dir
+
+    def run_all(self) -> bool:
+        failed = False
+        with ProcessPoolExecutor() as pool:
+            futures = []
+            for bundle in self.test_dir.glob("**/*.bundle"):
+                logging.debug("Submitting work for %s", bundle)
+                futures.append(pool.submit(run_bundle, bundle))
+            for future in concurrent.futures.as_completed(futures):
+                result = future.result()
+                if result.passed:
+                    logging.info(result)
+                else:
+                    failed = True
+                    logging.error(result)
+        return failed
+
+
+def main() -> None:
+    parser = ArgumentParser()
+    parser.add_argument("-v", "--verbose", action="count", default=0)
+    parser.add_argument("test_dir", type=Path)
+    args = parser.parse_args()
+
+    if args.verbose >= 2:
+        level = logging.DEBUG
+    elif args.verbose == 1:
+        level = logging.INFO
+    else:
+        level = logging.WARNING
+    logging.basicConfig(level=level)
+    if BundleRunner(args.test_dir).run_all():
+        sys.exit("Tests failed")
+
+
+if __name__ == "__main__":
+    main()