[libc++] Use the new .gen tests to implement __verbose_abort tests

[ldionne]

Summary

This reduces the amount of boilerplate that we need to generate
for each commit. It also resolves a problem where the modular CI
would run extremely slow on this test because we'd define a macro
before including the standard library, defeating the module cache.

[ldionne]

libcxx/test/libcxx/lit.local.cfg

5

This stuff would have to be duplicated until all the tests are moved to this new format.

[aaron.ballman]

What's the expectation for folks on Windows (where .sh isn't likely to work too well)?

[philnik]

In D151156#4362491, @aaron.ballman wrote:

What's the expectation for folks on Windows (where .sh isn't likely to work too well)?

The tests are generated using python scripts, so I don't think that's a problem.

libcxx/test/libcxx/assertions/headers_declare_verbose_abort.gen.py

26–31

I haven't tested this, but I think something like this would be easier to follow.

libcxx/test/libcxx/lit.local.cfg

5

Could we move this to its own file? lit.local.cfg isn't exactly the first place I'd expect some data. Maybe just put something like header_test_info.py into libcxx/utils/data?

[ldionne]

In D151156#4362491, @aaron.ballman wrote:

What's the expectation for folks on Windows (where .sh isn't likely to work too well)?

I believe we're using Lit's internal shell which actually also works on Windows. In this case, the invocation is extremely simple -- we basically call python <script> so that should work on windows just as well. There really isn't much of a change compared to how we used to do things, but this shifts the moment at which the generation happens (from being explicit via developer running a command to being implicit when you run the test suite).

[aaron.ballman]

In D151156#4362994, @ldionne wrote:

In D151156#4362491, @aaron.ballman wrote:

What's the expectation for folks on Windows (where .sh isn't likely to work too well)?

I believe we're using Lit's internal shell which actually also works on Windows. In this case, the invocation is extremely simple -- we basically call python <script> so that should work on windows just as well. There really isn't much of a change compared to how we used to do things, but this shifts the moment at which the generation happens (from being explicit via developer running a command to being implicit when you run the test suite).

Ah, thank you! I hadn't realized this was using lit's internal shell, I thought these scripts were to be executed on the command line as well. If it's all going through python and lit, that sounds lovely to me. Thanks!

[Mordante]

Thanks a lot for working on this! I really think this is very useful. Just a few nits, but I would like to see it after a green CI run.

libcxx/test/libcxx/lit.local.cfg

5

+1

20

[ldionne]

Rebase and address comments.

[ldionne]

Update review description.

[philnik]

LGTM. Leaving final approval to @Mordante, since he had comments.

[ldionne]

Fix CI

[ldionne]

Rebase to fix CI

[ldionne]

Make sure we exclude CMakeLists.txt from generated header tests

libcxx/test/libcxx/assertions/headers_declare_verbose_abort.gen.py

+#===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===##
+
+# Test that all public C++ headers define the verbose termination function, which
+# is required for users to be able to include any public header and then override
+# the function using a strong definition.
+
+# XFAIL: availability-verbose_abort-missing
+
+# RUN: %{python} %s %{header-variables}
+
+import base64, pickle, sys
+vars = pickle.loads(base64.b64decode(sys.argv[1]))
+
+for header in vars['public_headers']:
+    # Skip C compatibility headers.
+    if header.endswith('.h'):
+        continue
+
+    print(f'//--- {header}.compile.pass.cpp')
+    if header in vars['header_restrictions']:
+        print(f'#if {vars["header_restrictions"][header]}')
+    print(f'#include <{header}>')
+    print(f'using HandlerType = decltype(std::__libcpp_verbose_abort);')
+    if header in vars['header_restrictions']:
+        print(f'#endif')

libcxx/test/libcxx/assertions/headers_declare_verbose_abort.sh.cpp

-//===----------------------------------------------------------------------===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-
-// Test that all public C++ headers define the verbose termination function, which
-// is required for users to be able to include any public header and then override
-// the function using a strong definition.
-
-// XFAIL: availability-verbose_abort-missing
-
-/*
-BEGIN-SCRIPT
-
-for i, header in enumerate(public_headers):
-    # Skip C compatibility headers.
-    if header.endswith('.h'):
-        continue
-
-    vars = {
-        'run': 'RUN',
-        'i': i,
-        'restrictions': ' && ' + header_restrictions[header] if header in header_restrictions else '',
-        'header': header
-    }
-
-    print("""\
-// {run}: %{{build}} -DTEST_{i}
-#if defined(TEST_{i}){restrictions}
-#   include <{header}>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-""".format(**vars))
-
-END-SCRIPT
-*/
-
-#include <__config>
-
-int main(int, char**) { return 0; }
-
-// DO NOT MANUALLY EDIT ANYTHING BETWEEN THE MARKERS BELOW
-// GENERATED-MARKER
-// RUN: %{build} -DTEST_0
-#if defined(TEST_0)
-#   include <algorithm>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_1
-#if defined(TEST_1)
-#   include <any>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_2
-#if defined(TEST_2)
-#   include <array>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_3
-#if defined(TEST_3)
-#   include <atomic>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_4
-#if defined(TEST_4) && !defined(_LIBCPP_HAS_NO_THREADS)
-#   include <barrier>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_5
-#if defined(TEST_5)
-#   include <bit>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_6
-#if defined(TEST_6)
-#   include <bitset>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_7
-#if defined(TEST_7)
-#   include <cassert>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_8
-#if defined(TEST_8)
-#   include <ccomplex>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_9
-#if defined(TEST_9)
-#   include <cctype>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_10
-#if defined(TEST_10)
-#   include <cerrno>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_11
-#if defined(TEST_11)
-#   include <cfenv>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_12
-#if defined(TEST_12)
-#   include <cfloat>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_13
-#if defined(TEST_13)
-#   include <charconv>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_14
-#if defined(TEST_14)
-#   include <chrono>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_15
-#if defined(TEST_15)
-#   include <cinttypes>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_16
-#if defined(TEST_16)
-#   include <ciso646>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_17
-#if defined(TEST_17)
-#   include <climits>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_18
-#if defined(TEST_18) && !defined(_LIBCPP_HAS_NO_LOCALIZATION)
-#   include <clocale>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_19
-#if defined(TEST_19)
-#   include <cmath>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_20
-#if defined(TEST_20) && !defined(_LIBCPP_HAS_NO_LOCALIZATION)
-#   include <codecvt>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_21
-#if defined(TEST_21)
-#   include <compare>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_22
-#if defined(TEST_22)
-#   include <complex>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_24
-#if defined(TEST_24)
-#   include <concepts>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_25
-#if defined(TEST_25)
-#   include <condition_variable>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_26
-#if defined(TEST_26) && (defined(__cpp_impl_coroutine) && __cpp_impl_coroutine >= 201902L) || (defined(__cpp_coroutines) && __cpp_coroutines >= 201703L)
-#   include <coroutine>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_27
-#if defined(TEST_27)
-#   include <csetjmp>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_28
-#if defined(TEST_28)
-#   include <csignal>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_29
-#if defined(TEST_29)
-#   include <cstdarg>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_30
-#if defined(TEST_30)
-#   include <cstdbool>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_31
-#if defined(TEST_31)
-#   include <cstddef>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_32
-#if defined(TEST_32)
-#   include <cstdint>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_33
-#if defined(TEST_33)
-#   include <cstdio>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_34
-#if defined(TEST_34)
-#   include <cstdlib>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_35
-#if defined(TEST_35)
-#   include <cstring>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_36
-#if defined(TEST_36)
-#   include <ctgmath>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_37
-#if defined(TEST_37)
-#   include <ctime>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_39
-#if defined(TEST_39)
-#   include <cuchar>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_40
-#if defined(TEST_40) && !defined(_LIBCPP_HAS_NO_WIDE_CHARACTERS)
-#   include <cwchar>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_41
-#if defined(TEST_41) && !defined(_LIBCPP_HAS_NO_WIDE_CHARACTERS)
-#   include <cwctype>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_42
-#if defined(TEST_42)
-#   include <deque>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_44
-#if defined(TEST_44)
-#   include <exception>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_45
-#if defined(TEST_45)
-#   include <execution>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_46
-#if defined(TEST_46)
-#   include <expected>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_48
-#if defined(TEST_48) && !defined(_LIBCPP_HAS_NO_FILESYSTEM_LIBRARY)
-#   include <filesystem>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_50
-#if defined(TEST_50)
-#   include <format>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_51
-#if defined(TEST_51)
-#   include <forward_list>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_52
-#if defined(TEST_52) && !defined(_LIBCPP_HAS_NO_LOCALIZATION) && !defined(_LIBCPP_HAS_NO_FSTREAM)
-#   include <fstream>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_53
-#if defined(TEST_53)
-#   include <functional>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_54
-#if defined(TEST_54) && !defined(_LIBCPP_HAS_NO_THREADS)
-#   include <future>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_55
-#if defined(TEST_55)
-#   include <initializer_list>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_57
-#if defined(TEST_57) && !defined(_LIBCPP_HAS_NO_LOCALIZATION)
-#   include <iomanip>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_58
-#if defined(TEST_58) && !defined(_LIBCPP_HAS_NO_LOCALIZATION)
-#   include <ios>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_59
-#if defined(TEST_59)
-#   include <iosfwd>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_60
-#if defined(TEST_60) && !defined(_LIBCPP_HAS_NO_LOCALIZATION)
-#   include <iostream>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_61
-#if defined(TEST_61) && !defined(_LIBCPP_HAS_NO_LOCALIZATION)
-#   include <istream>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_62
-#if defined(TEST_62)
-#   include <iterator>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_63
-#if defined(TEST_63) && !defined(_LIBCPP_HAS_NO_THREADS)
-#   include <latch>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_64
-#if defined(TEST_64)
-#   include <limits>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_66
-#if defined(TEST_66)
-#   include <list>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_67
-#if defined(TEST_67) && !defined(_LIBCPP_HAS_NO_LOCALIZATION)
-#   include <locale>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_69
-#if defined(TEST_69)
-#   include <map>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_71
-#if defined(TEST_71)
-#   include <mdspan>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_72
-#if defined(TEST_72)
-#   include <memory>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_73
-#if defined(TEST_73)
-#   include <memory_resource>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_74
-#if defined(TEST_74) && !defined(_LIBCPP_HAS_NO_THREADS)
-#   include <mutex>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_75
-#if defined(TEST_75)
-#   include <new>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_76
-#if defined(TEST_76)
-#   include <numbers>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_77
-#if defined(TEST_77)
-#   include <numeric>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_78
-#if defined(TEST_78)
-#   include <optional>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_79
-#if defined(TEST_79) && !defined(_LIBCPP_HAS_NO_LOCALIZATION)
-#   include <ostream>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_80
-#if defined(TEST_80)
-#   include <queue>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_81
-#if defined(TEST_81)
-#   include <random>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_82
-#if defined(TEST_82)
-#   include <ranges>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_83
-#if defined(TEST_83)
-#   include <ratio>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_84
-#if defined(TEST_84) && !defined(_LIBCPP_HAS_NO_LOCALIZATION)
-#   include <regex>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_85
-#if defined(TEST_85)
-#   include <scoped_allocator>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_86
-#if defined(TEST_86) && !defined(_LIBCPP_HAS_NO_THREADS)
-#   include <semaphore>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_87
-#if defined(TEST_87)
-#   include <set>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_89
-#if defined(TEST_89) && !defined(_LIBCPP_HAS_NO_THREADS)
-#   include <shared_mutex>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_90
-#if defined(TEST_90)
-#   include <source_location>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_91
-#if defined(TEST_91)
-#   include <span>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_92
-#if defined(TEST_92) && !defined(_LIBCPP_HAS_NO_LOCALIZATION)
-#   include <sstream>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_93
-#if defined(TEST_93)
-#   include <stack>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_97
-#if defined(TEST_97)
-#   include <stdexcept>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_101
-#if defined(TEST_101) && !defined(_LIBCPP_HAS_NO_LOCALIZATION)
-#   include <streambuf>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_102
-#if defined(TEST_102)
-#   include <string>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_104
-#if defined(TEST_104)
-#   include <string_view>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_105
-#if defined(TEST_105) && !defined(_LIBCPP_HAS_NO_LOCALIZATION)
-#   include <strstream>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_106
-#if defined(TEST_106)
-#   include <system_error>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_108
-#if defined(TEST_108) && !defined(_LIBCPP_HAS_NO_THREADS)
-#   include <thread>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_109
-#if defined(TEST_109)
-#   include <tuple>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_110
-#if defined(TEST_110)
-#   include <type_traits>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_111
-#if defined(TEST_111)
-#   include <typeindex>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_112
-#if defined(TEST_112)
-#   include <typeinfo>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_114
-#if defined(TEST_114)
-#   include <unordered_map>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_115
-#if defined(TEST_115)
-#   include <unordered_set>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_116
-#if defined(TEST_116)
-#   include <utility>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_117
-#if defined(TEST_117)
-#   include <valarray>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_118
-#if defined(TEST_118)
-#   include <variant>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_119
-#if defined(TEST_119)
-#   include <vector>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_120
-#if defined(TEST_120)
-#   include <version>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_123
-#if defined(TEST_123) && __cplusplus >= 201103L
-#   include <experimental/deque>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_124
-#if defined(TEST_124) && __cplusplus >= 201103L
-#   include <experimental/forward_list>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_125
-#if defined(TEST_125) && __cplusplus >= 201103L
-#   include <experimental/iterator>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_126
-#if defined(TEST_126) && __cplusplus >= 201103L
-#   include <experimental/list>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_127
-#if defined(TEST_127) && __cplusplus >= 201103L
-#   include <experimental/map>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_128
-#if defined(TEST_128) && __cplusplus >= 201103L
-#   include <experimental/memory_resource>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_129
-#if defined(TEST_129) && __cplusplus >= 201103L
-#   include <experimental/propagate_const>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_130
-#if defined(TEST_130) && !defined(_LIBCPP_HAS_NO_LOCALIZATION) && __cplusplus >= 201103L
-#   include <experimental/regex>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_131
-#if defined(TEST_131) && __cplusplus >= 201103L
-#   include <experimental/set>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_132
-#if defined(TEST_132) && __cplusplus >= 201103L
-#   include <experimental/simd>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_133
-#if defined(TEST_133) && __cplusplus >= 201103L
-#   include <experimental/string>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_134
-#if defined(TEST_134) && __cplusplus >= 201103L
-#   include <experimental/type_traits>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_135
-#if defined(TEST_135) && __cplusplus >= 201103L
-#   include <experimental/unordered_map>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_136
-#if defined(TEST_136) && __cplusplus >= 201103L
-#   include <experimental/unordered_set>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_137
-#if defined(TEST_137) && __cplusplus >= 201103L
-#   include <experimental/utility>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// RUN: %{build} -DTEST_138
-#if defined(TEST_138) && __cplusplus >= 201103L
-#   include <experimental/vector>
-    using HandlerType = decltype(std::__libcpp_verbose_abort);
-#endif
-
-// GENERATED-MARKER

libcxx/test/libcxx/lit.local.cfg

 # The tests in this directory need to run Python
 import pipes, sys
 config.substitutions.append(('%{python}', pipes.quote(sys.executable)))
+
+# Several tests in this directory require additional information about libc++ headers
+# to generate tests.
+import os, pathlib
+header_restrictions = {
+    "barrier": "!defined(_LIBCPP_HAS_NO_THREADS)",
+    "future": "!defined(_LIBCPP_HAS_NO_THREADS)",
+    "latch": "!defined(_LIBCPP_HAS_NO_THREADS)",
+    "mutex": "!defined(_LIBCPP_HAS_NO_THREADS)",
+    "semaphore": "!defined(_LIBCPP_HAS_NO_THREADS)",
+    "shared_mutex": "!defined(_LIBCPP_HAS_NO_THREADS)",
+    "stdatomic.h": "__cplusplus > 202002L && !defined(_LIBCPP_HAS_NO_THREADS)",
+    "thread": "!defined(_LIBCPP_HAS_NO_THREADS)",
+
+    "filesystem": "!defined(_LIBCPP_HAS_NO_FILESYSTEM_LIBRARY)",
+
+    # TODO LLVM17: simplify this to __cplusplus >= 202002L
+    "coroutine": "(defined(__cpp_impl_coroutine) && __cpp_impl_coroutine >= 201902L) || (defined(__cpp_coroutines) && __cpp_coroutines >= 201703L)",
+
+    "clocale": "!defined(_LIBCPP_HAS_NO_LOCALIZATION)",
+    "codecvt": "!defined(_LIBCPP_HAS_NO_LOCALIZATION)",
+    "fstream": "!defined(_LIBCPP_HAS_NO_LOCALIZATION) && !defined(_LIBCPP_HAS_NO_FSTREAM)",
+    "iomanip": "!defined(_LIBCPP_HAS_NO_LOCALIZATION)",
+    "ios": "!defined(_LIBCPP_HAS_NO_LOCALIZATION)",
+    "iostream": "!defined(_LIBCPP_HAS_NO_LOCALIZATION)",
+    "istream": "!defined(_LIBCPP_HAS_NO_LOCALIZATION)",
+    "locale.h": "!defined(_LIBCPP_HAS_NO_LOCALIZATION)",
+    "locale": "!defined(_LIBCPP_HAS_NO_LOCALIZATION)",
+    "ostream": "!defined(_LIBCPP_HAS_NO_LOCALIZATION)",
+    "regex": "!defined(_LIBCPP_HAS_NO_LOCALIZATION)",
+    "sstream": "!defined(_LIBCPP_HAS_NO_LOCALIZATION)",
+    "streambuf": "!defined(_LIBCPP_HAS_NO_LOCALIZATION)",
+    "strstream": "!defined(_LIBCPP_HAS_NO_LOCALIZATION)",
+
+    "wctype.h": "!defined(_LIBCPP_HAS_NO_WIDE_CHARACTERS)",
+    "cwctype": "!defined(_LIBCPP_HAS_NO_WIDE_CHARACTERS)",
+    "cwchar": "!defined(_LIBCPP_HAS_NO_WIDE_CHARACTERS)",
+    "wchar.h": "!defined(_LIBCPP_HAS_NO_WIDE_CHARACTERS)",
+
+    "experimental/algorithm": "__cplusplus >= 201103L",
+    "experimental/deque": "__cplusplus >= 201103L",
+    "experimental/forward_list": "__cplusplus >= 201103L",
+    "experimental/functional": "__cplusplus >= 201103L",
+    "experimental/iterator": "__cplusplus >= 201103L",
+    "experimental/list": "__cplusplus >= 201103L",
+    "experimental/map": "__cplusplus >= 201103L",
+    "experimental/memory_resource": "__cplusplus >= 201103L",
+    "experimental/propagate_const": "__cplusplus >= 201103L",
+    "experimental/regex": "!defined(_LIBCPP_HAS_NO_LOCALIZATION) && __cplusplus >= 201103L",
+    "experimental/set": "__cplusplus >= 201103L",
+    "experimental/simd": "__cplusplus >= 201103L",
+    "experimental/span": "__cplusplus >= 201103L",
+    "experimental/string": "__cplusplus >= 201103L",
+    "experimental/type_traits": "__cplusplus >= 201103L",
+    "experimental/unordered_map": "__cplusplus >= 201103L",
+    "experimental/unordered_set": "__cplusplus >= 201103L",
+    "experimental/utility": "__cplusplus >= 201103L",
+    "experimental/vector": "__cplusplus >= 201103L",
+}
+
+private_headers_still_public_in_modules = [
+    '__assert', '__config',
+    '__config_site.in', '__debug', '__hash_table',
+    '__threading_support', '__tree', '__undef_macros', '__verbose_abort'
+]
+
+def is_header(file):
+    """Returns whether the given file is a header (i.e. not a directory or the modulemap file)."""
+    return not file.is_dir() and not file.name == 'module.modulemap.in' and file.name != 'libcxx.imp'
+
+libcxx_root = pathlib.Path(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))
+include = pathlib.Path(os.path.join(libcxx_root, 'include'))
+test = pathlib.Path(os.path.join(libcxx_root, 'test'))
+assert(libcxx_root.exists())
+
+toplevel_headers     = sorted(str(p.relative_to(include)) for p in include.glob('[a-z]*') if is_header(p))
+experimental_headers = sorted(str(p.relative_to(include)) for p in include.glob('experimental/[a-z]*') if is_header(p))
+public_headers       = toplevel_headers + experimental_headers
+private_headers      = sorted(str(p.relative_to(include)) for p in include.rglob('*') if is_header(p) and str(p.relative_to(include)).startswith('__') and not p.name.startswith('pstl'))
+variables = {
+    'toplevel_headers': toplevel_headers,
+    'experimental_headers': experimental_headers,
+    'public_headers': public_headers,
+    'private_headers': private_headers,
+    'header_restrictions': header_restrictions,
+    'private_headers_still_public_in_modules': private_headers_still_public_in_modules
+}
+
+import base64, lit.util, pickle
+base64Encode = lambda s: lit.util.to_string(base64.b64encode(lit.util.to_bytes(s)))
+escapedVariables = base64Encode(pickle.dumps(variables))
+config.substitutions.append(('%{header-variables}', escapedVariables))

libcxx/utils/generate_header_tests.py

variables = {
         'toplevel_headers': toplevel_headers,
         'experimental_headers': experimental_headers,
         'public_headers': public_headers,
         'private_headers': private_headers,
         'header_restrictions': header_restrictions,
         'private_headers_still_public_in_modules': private_headers_still_public_in_modules
     }
 
-    produce(test.joinpath('libcxx/assertions/headers_declare_verbose_abort.sh.cpp'), variables)
     produce(test.joinpath('libcxx/clang_tidy.sh.cpp'), variables)
     produce(test.joinpath('libcxx/double_include.sh.cpp'), variables)
     produce(test.joinpath('libcxx/min_max_macros.compile.pass.cpp'), variables)
     produce(test.joinpath('libcxx/modules_include.sh.cpp'), variables)
     produce(test.joinpath('libcxx/nasty_macros.compile.pass.cpp'), variables)
     produce(test.joinpath('libcxx/no_assert_include.compile.pass.cpp'), variables)
     produce(test.joinpath('libcxx/private_headers.verify.cpp'), variables)
     produce(test.joinpath('libcxx/transitive_includes.sh.cpp'), variables)
 
 
 if __name__ == '__main__':
     main()

libcxx/utils/libcxx/test/dsl.py

def f(config, *args, **kwargs):
         cache[cacheKey] = function(config, *args, **kwargs)
         # Update the persistent cache so it knows about the new key
         with open(persistentCache, 'wb') as cacheFile:
           pickle.dump(cache, cacheFile)
       return cache[cacheKey]
     return f
   return decorator
 
-def _executeScriptInternal(test, commands):
+def _executeWithFakeConfig(test, commands):
   """
-  Returns (stdout, stderr, exitCode, timeoutInfo)
-
-  TODO: This really should be easier to access from Lit itself
+  Returns (stdout, stderr, exitCode, timeoutInfo, parsedCommands)
   """
-  parsedCommands = libcxx.test.format.parseScript(test, preamble=commands)
-
   litConfig = lit.LitConfig.LitConfig(
     progname='lit',
     path=[],
     quiet=False,
     useValgrind=False,
     valgrindLeakCheck=False,
     valgrindArgs=[],
     noExecute=False,
     debug=False,
     isWindows=platform.system() == 'Windows',
     order='smart',
     params={})
-  _, tmpBase = libcxx.test.format._getTempPaths(test)
-  execDir = os.path.dirname(test.getExecPath())
-  res = lit.TestRunner.executeScriptInternal(test, litConfig, tmpBase, parsedCommands, execDir)
-  if isinstance(res, lit.Test.Result): # Handle failure to parse the Lit test
-    res = ('', res.output, 127, None)
-  (out, err, exitCode, timeoutInfo) = res
-
-  # TODO: As a temporary workaround until https://reviews.llvm.org/D81892 lands, manually
-  #       split any stderr output that is included in stdout. It shouldn't be there, but
-  #       the Lit internal shell conflates stderr and stdout.
-  conflatedErrorOutput = re.search("(# command stderr:.+$)", out, flags=re.DOTALL)
-  if conflatedErrorOutput:
-    conflatedErrorOutput = conflatedErrorOutput.group(0)
-    out = out[:-len(conflatedErrorOutput)]
-    err += conflatedErrorOutput
-
-  return (out, err, exitCode, timeoutInfo, parsedCommands)
+  return libcxx.test.format._executeScriptInternal(test, litConfig, commands)
 
 def _makeConfigTest(config):
   # Make sure the support directories exist, which is needed to create
   # the temporary file %t below.
   sourceRoot = os.path.join(config.test_exec_root, '__config_src__')
   execRoot = os.path.join(config.test_exec_root, '__config_exec__')
   for supportDir in (sourceRoot, execRoot):
     if not os.path.exists(supportDir):
def sourceBuilds(config, source, additionalFlags=[]):
 
   This is done by compiling and linking a program that consists of the given
   source with the %{cxx} substitution, and seeing whether that succeeds. If
   any additional flags are passed, they are appended to the compiler invocation.
   """
   with _makeConfigTest(config) as test:
     with open(test.getSourcePath(), 'w') as sourceFile:
       sourceFile.write(source)
-    _, _, exitCode, _, _ = _executeScriptInternal(test, ['%{{build}} {}'.format(' '.join(additionalFlags))])
+    _, _, exitCode, _, _ = _executeWithFakeConfig(test, ['%{{build}} {}'.format(' '.join(additionalFlags))])
     return exitCode == 0
 
 @_memoizeExpensiveOperation(lambda c, p, args=None: (c.substitutions, c.environment, p, args))
 def programOutput(config, program, args=None):
   """
   Compiles a program for the test target, run it on the test target and return
   the output.
 
   Note that execution of the program is done through the %{exec} substitution,
   which means that the program may be run on a remote host depending on what
   %{exec} does.
   """
   if args is None:
     args = []
   with _makeConfigTest(config) as test:
     with open(test.getSourcePath(), 'w') as source:
       source.write(program)
-    _, err, exitCode, _, buildcmd = _executeScriptInternal(test, ['%{build}'])
+    _, err, exitCode, _, buildcmd = _executeWithFakeConfig(test, ['%{build}'])
     if exitCode != 0:
       raise ConfigurationCompilationError("Failed to build program, cmd:\n{}\nstderr is:\n{}".format(buildcmd, err))
 
-    out, err, exitCode, _, runcmd = _executeScriptInternal(test, ["%{{run}} {}".format(' '.join(args))])
+    out, err, exitCode, _, runcmd = _executeWithFakeConfig(test, ["%{{run}} {}".format(' '.join(args))])
     if exitCode != 0:
       raise ConfigurationRuntimeError("Failed to run program, cmd:\n{}\nstderr is:\n{}".format(runcmd, err))
 
     actualOut = re.search("# command output:\n(.+)\n$", out, flags=re.DOTALL)
     actualOut = actualOut.group(1) if actualOut else ""
     return actualOut
 
 @_memoizeExpensiveOperation(lambda c, p, args=None: (c.substitutions, c.environment, p, args))

 def hasCompileFlag(config, flag):
   """
   Return whether the compiler in the configuration supports a given compiler flag.
 
   This is done by executing the %{cxx} substitution with the given flag and
   checking whether that succeeds.
   """
   with _makeConfigTest(config) as test:
-    out, err, exitCode, timeoutInfo, _ = _executeScriptInternal(test, [
+    out, err, exitCode, timeoutInfo, _ = _executeWithFakeConfig(test, [
       "%{{cxx}} -xc++ {} -Werror -fsyntax-only %{{flags}} %{{compile_flags}} {}".format(os.devnull, flag)
     ])
     return exitCode == 0
 
 @_memoizeExpensiveOperation(lambda c, s: (c.substitutions, c.environment, s))
 def runScriptExitCode(config, script):
   """
   Runs the given script as a Lit test, and returns the exit code of the execution.
 
   The script must be a list of commands, each of which being something that
   could appear on the right-hand-side of a `RUN:` keyword.
   """
   with _makeConfigTest(config) as test:
-    _, _, exitCode, _, _ = _executeScriptInternal(test, script)
+    _, _, exitCode, _, _ = _executeWithFakeConfig(test, script)
     return exitCode
 
 @_memoizeExpensiveOperation(lambda c, s: (c.substitutions, c.environment, s))
 def commandOutput(config, command):
   """
   Runs the given script as a Lit test, and returns the output.
   If the exit code isn't 0 an exception is raised.
 
   The script must be a list of commands, each of which being something that
   could appear on the right-hand-side of a `RUN:` keyword.
   """
   with _makeConfigTest(config) as test:
-    out, err, exitCode, _, cmd = _executeScriptInternal(test, command)
+    out, err, exitCode, _, cmd = _executeWithFakeConfig(test, command)
     if exitCode != 0:
      raise ConfigurationRuntimeError("Failed to run command: {}\nstderr is:\n{}".format(cmd, err))
     return out
 
 @_memoizeExpensiveOperation(lambda c, l: (c.substitutions, c.environment, l))
 def hasAnyLocale(config, locales):
   """
   Return whether the runtime execution environment supports a given locale.
def compilerMacros(config, flags=''):
   """
   with _makeConfigTest(config) as test:
     with open(test.getSourcePath(), 'w') as sourceFile:
       sourceFile.write("""
       #if __has_include(<__config_site>)
       #  include <__config_site>
       #endif
       """)
-    unparsedOutput, err, exitCode, _, cmd = _executeScriptInternal(test, [
+    unparsedOutput, err, exitCode, _, cmd = _executeWithFakeConfig(test, [
       "%{{cxx}} %s -dM -E %{{flags}} %{{compile_flags}} {}".format(flags)
     ])
     if exitCode != 0:
       raise ConfigurationCompilationError("Failed to retrieve compiler macros, compiler invocation is:\n{}\nstderr is:\n{}".format(cmd, err))
     parsedMacros = dict()
     defines = (l.strip() for l in unparsedOutput.split('\n') if l.startswith('#define '))
     for line in defines:
       line = line[len('#define '):]

libcxx/utils/libcxx/test/format.py

 #===----------------------------------------------------------------------===##
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
 #===----------------------------------------------------------------------===##
 
+import contextlib
+import io
 import lit
 import lit.formats
 import os
 import pipes
 import re
 import shutil
 
 def _getTempPaths(test):
def _getTempPaths(test):
     tmpBase = os.path.join(tmpDir, 't')
     return tmpDir, tmpBase
 
 def _checkBaseSubstitutions(substitutions):
     substitutions = [s for (s, _) in substitutions]
     for s in ['%{cxx}', '%{compile_flags}', '%{link_flags}', '%{flags}', '%{exec}']:
         assert s in substitutions, "Required substitution {} was not provided".format(s)
 
+def _executeScriptInternal(test, litConfig, commands):
+  """
+  Returns (stdout, stderr, exitCode, timeoutInfo, parsedCommands)
+
+  TODO: This really should be easier to access from Lit itself
+  """
+  parsedCommands = parseScript(test, preamble=commands)
+
+  _, tmpBase = _getTempPaths(test)
+  execDir = os.path.dirname(test.getExecPath())
+  res = lit.TestRunner.executeScriptInternal(test, litConfig, tmpBase, parsedCommands, execDir)
+  if isinstance(res, lit.Test.Result): # Handle failure to parse the Lit test
+    res = ('', res.output, 127, None)
+  (out, err, exitCode, timeoutInfo) = res
+
+  # TODO: As a temporary workaround until https://reviews.llvm.org/D81892 lands, manually
+  #       split any stderr output that is included in stdout. It shouldn't be there, but
+  #       the Lit internal shell conflates stderr and stdout.
+  conflatedErrorOutput = re.search("(# command stderr:.+$)", out, flags=re.DOTALL)
+  if conflatedErrorOutput:
+    conflatedErrorOutput = conflatedErrorOutput.group(0)
+    out = out[:-len(conflatedErrorOutput)]
+    err += conflatedErrorOutput
+
+  return (out, err, exitCode, timeoutInfo, parsedCommands)
+
 def parseScript(test, preamble):
     """
     Extract the script from a test, with substitutions applied.
 
     Returns a list of commands ready to be executed.
 
     - test
         The lit.Test to parse.
class CxxStandardLibraryTest(lit.formats.TestFormat):
     FOO.compile.fail.cpp    - Does not compile successfully
 
     FOO.link.pass.cpp       - Compiles and links successfully, run not attempted
     FOO.link.pass.mm        - Same as .link.pass.cpp, but for Objective-C++
     FOO.link.fail.cpp       - Compiles successfully, but fails to link
 
     FOO.sh.<anything>       - A builtin Lit Shell test
 
+    FOO.gen.<anything>      - A .sh test that generates one or more Lit tests on the
+                              fly. Executing this test must generate one or more files
+                              as expected by LLVM split-file, and each generated file
+                              leads to a separate Lit test that runs that file as
+                              defined by the test format. This can be used to generate
+                              multiple Lit tests from a single source file, which is
+                              useful for testing repetitive properties in the library.
+                              Be careful not to abuse this since this is not a replacement
+                              for usual code reuse techniques.
+
     FOO.verify.cpp          - Compiles with clang-verify. This type of test is
                               automatically marked as UNSUPPORTED if the compiler
                               does not support Clang-verify.
 
     FOO.fail.cpp            - Compiled with clang-verify if clang-verify is
                               supported, and equivalent to a .compile.fail.cpp
                               test otherwise. This is supported only for backwards
                               compatibility with the test suite.
The test format will define the following substitutions for use inside tests:
             in conjunction with the %{build} substitution.
     """
     def getTestsInDirectory(self, testSuite, pathInSuite, litConfig, localConfig):
         SUPPORTED_SUFFIXES = ['[.]pass[.]cpp$', '[.]pass[.]mm$',
                               '[.]compile[.]pass[.]cpp$', '[.]compile[.]pass[.]mm$',
                               '[.]compile[.]fail[.]cpp$',
                               '[.]link[.]pass[.]cpp$', '[.]link[.]pass[.]mm$',
                               '[.]link[.]fail[.]cpp$',
-                              '[.]sh[.][^.]+$',
+                              '[.]sh[.][^.]+$', '[.]gen[.][^.]+$',
                               '[.]verify[.]cpp$',
                               '[.]fail[.]cpp$']
         sourcePath = testSuite.getSourcePath(pathInSuite)
         for filename in os.listdir(sourcePath):
             # Ignore dot files and excluded tests.
             if filename.startswith('.') or filename in localConfig.excludes:
                 continue
 
             filepath = os.path.join(sourcePath, filename)
             if not os.path.isdir(filepath):
                 if any([re.search(ext, filename) for ext in SUPPORTED_SUFFIXES]):
+                    # If this is a generated test, run the generation step and add
+                    # as many Lit tests as necessary.
+                    if re.search('[.]gen[.][^.]+$', filename):
+                        for test in self._generateGenTest(filepath, testSuite, pathInSuite, litConfig, localConfig):
+                            yield test
+                    else:
                         yield lit.Test.Test(testSuite, pathInSuite + (filename,), localConfig)
 
     def execute(self, test, litConfig):
         VERIFY_FLAGS = '-Xclang -verify -Xclang -verify-ignore-unexpected=note -ferror-limit=0'
         supportsVerify = 'verify-support' in test.config.available_features
         filename = test.path_in_suite[-1]
 
         if re.search('[.]sh[.][^.]+$', filename):
             steps = [ ] # The steps are already in the script
def _executeShTest(self, test, litConfig, steps):
             return script
 
         if litConfig.noExecute:
             return lit.Test.Result(lit.Test.XFAIL if test.isExpectedToFail() else lit.Test.PASS)
         else:
             _, tmpBase = _getTempPaths(test)
             useExternalSh = False
             return lit.TestRunner._runShTest(test, litConfig, useExternalSh, script, tmpBase)
+
+    def _generateGenTest(self, filepath, testSuite, pathInSuite, litConfig, localConfig):
+        generator = lit.Test.Test(testSuite, pathInSuite, localConfig, file_path=filepath)
+        steps = [] # Steps must already be in the script
+        (out, err, exitCode, _, _) = _executeScriptInternal(generator, litConfig, steps)
+        if exitCode != 0:
+            raise RuntimeError(f"Error while trying to generate gen test: {out}")
+
+        # Process the 'out' to get rid of Lit-injected crap
+        actualOut = re.search("# command output:\n(.+)\n$", out, flags=re.DOTALL)
+        actualOut = actualOut.group(1) if actualOut else ""
+
+        # Split the generated output into multiple files and generate one test for each file
+        for (subfile, content) in self._splitFile(actualOut):
+            generatedPathInSuite = pathInSuite + (os.path.basename(filepath), subfile)
+            generatedRealPath = testSuite.getExecPath(generatedPathInSuite)
+            os.makedirs(os.path.dirname(generatedRealPath), exist_ok=True)
+            with open(generatedRealPath, 'w') as f:
+                f.write(content)
+            yield lit.Test.Test(testSuite, generatedPathInSuite, localConfig, file_path=generatedRealPath)
+
+    def _splitFile(self, input):
+        DELIM = r'^.+---(.+)'
+        lines = input.splitlines()
+        currentFile = None
+        thisFileContent = []
+        for line in lines:
+            match = re.match(DELIM, line)
+            if match:
+                if currentFile is not None:
+                    yield (currentFile, '\n'.join(thisFileContent))
+                currentFile = match.group(1).strip()
+                thisFileContent = []
+            assert currentFile is not None, f"Some input to split-file doesn't belong to any file, input was:\n{input}"
+            thisFileContent.append(line)
+        yield (currentFile, '\n'.join(thisFileContent))

llvm/utils/lit/lit/Test.py

def getUsedFeatures(self):
         getUsedFeatures() -> list of strings
 
         Returns a list of all features appearing in XFAIL, UNSUPPORTED and
         REQUIRES annotations for this test.
         """
         import lit.TestRunner
 
         parsed = lit.TestRunner._parseKeywords(
-            self.getSourcePath(), require_script=False
+            self.getFilePath(), require_script=False
         )
         feature_keywords = ("UNSUPPORTED:", "REQUIRES:", "XFAIL:")
         boolean_expressions = itertools.chain.from_iterable(
             parsed[k] or [] for k in feature_keywords
         )
         tokens = itertools.chain.from_iterable(
             BooleanExpression.tokenize(expr)
             for expr in boolean_expressions
             if expr != "*"
         )
         matchExpressions = set(filter(BooleanExpression.isMatchExpression, tokens))
         return matchExpressions

llvm/utils/lit/lit/TestRunner.py

def colonNormalizePath(path):
     if kIsWindows:
         return re.sub(r"^(.):", r"\1", path.replace("\\", "/"))
     else:
         assert path[0] == "/"
         return path[1:]
 
 
 def getDefaultSubstitutions(test, tmpDir, tmpBase, normalize_slashes=False):
-    sourcepath = test.getSourcePath()
+    sourcepath = test.getFilePath()
     sourcedir = os.path.dirname(sourcepath)
 
     # Normalize slashes, if requested.
     if normalize_slashes:
         sourcepath = sourcepath.replace("\\", "/")
         sourcedir = sourcedir.replace("\\", "/")
         tmpDir = tmpDir.replace("\\", "/")
         tmpBase = tmpBase.replace("\\", "/")
def parseIntegratedTestScript(test, additional_parsers=[], require_script=True):
 
     If 'require_script' is False an empty script
     may be returned. This can be used for test formats where the actual script
     is optional or ignored.
     """
     # Parse the test sources and extract test properties
     try:
         parsed = _parseKeywords(
-            test.getSourcePath(), additional_parsers, require_script
+            test.getFilePath(), additional_parsers, require_script
         )
     except ValueError as e:
         return lit.Test.Result(Test.UNRESOLVED, str(e))
     script = parsed["RUN:"] or []
     assert parsed["DEFINE:"] == script
     assert parsed["REDEFINE:"] == script
     test.xfails += parsed["XFAIL:"] or []
     test.requires += parsed["REQUIRES:"] or []

llvm/utils/lit/lit/formats/base.py

 ###
 
 # Check exit code of a simple executable with no input
 class ExecutableTest(FileBasedTest):
     def execute(self, test, litConfig):
         if test.config.unsupported:
             return lit.Test.UNSUPPORTED
 
-        out, err, exitCode = lit.util.executeCommand(test.getSourcePath())
+        out, err, exitCode = lit.util.executeCommand(test.getFilePath())
 
         if not exitCode:
             return lit.Test.PASS, ""
 
         return lit.Test.FAIL, out + err