Generalize "check-all" umbrella targets, use for check-clang-tools

[sammccall]

Summary

The mechanism behind "check-all" is recording params of add_lit_testsuite()
calls in global variables LLVM_LIT_*, and then creating an extra suite with
their union at the end.
This avoids composing the check-* targets directly, which doesn't work well.

We generalize this by allowing multiple families of variables LLVM_{name}_LIT_*:

umbrella_lit_testsuite_begin(check-foo)
... test suites here will be added to LLVM_FOO_LIT_* variables ...
umbrella_lit_testsuite_end(check-foo)

(This also moves some implementation muck out of {llvm,clang}/CMakeLists.txt

This patch also changes check-clang-tools to use be an umbrella test target,
which means the clangd and clang-pseudo tests are included in it, along with the
the other testsuites that already are (like check-clang-extra-clang-tidy).

[sammccall]

This isn't ready for submission, there's some bits that touch the global variables in runtimes/, llvm/runtimes, and compiler-rt/test. Mostly to see the idea.

The main point is to allow check-clang-tools to cover the tests in clang-tools-extra without requiring them to be merged into a common test/ tree. (Thus enabling unmerging...)
It should also make it easy to add check-all targets to standalone builds of projects other than clang.

It probably makes sense to separate the behavior change for clang-tools-extra from the refactoring change to clang and llvm, combining them is mostly to make review clearer.

[sammccall]

Fix LLVM_ADDITIONAL_TEST_{TARGETS,DEPENDS}

[kbobyrev]

I think this is no longer [WIP] but rather review-ready, right?

[sammccall]

Ran into problems caused by check-clang-tools not running clangd tests again, probably worth

In D121838#3433022, @kbobyrev wrote:

I think this is no longer [WIP] but rather review-ready, right?

It's both WIP and review-ready :-)
There are some mechanical cleanups left to do as mentioned in the comment above, but there's no point doing them if the idea is wrong.

[aaron.ballman]

In D121838#3433022, @kbobyrev wrote:

I think this is no longer [WIP] but rather review-ready, right?

It's both WIP and review-ready :-)
There are some mechanical cleanups left to do as mentioned in the comment above, but there's no point doing them if the idea is wrong.

I edited the title to remove the WIP bit so reviewers know. :-)

Can you rebase the patch so that precommit CI can go through the paces?

[tstellar]

Can you elaborate more on the problem this is solving? Also, what are the user visible changes? Will check-all and check-clang run the same set of tests as before?

[sammccall]

In D121838#3433472, @tstellar wrote:

Can you elaborate more on the problem this is solving? Also, what are the user visible changes?

Most directly, check-clang-tools does not currently run clangd or clang-pseudo tests, but does after this patch.

More generally, it removes the requirement to intermingle the layout of projects (e.g. those under clang-tools-extra) that we want a single check-* target to cover.

Lit is a monolithic runner that expects to see all tests, manage parallelism and failures, and display interactive output.
This means if you want to run all tests in a subdirectory via e.g. ninja, you need to define a check-mydir target that knows about each lit test suite involved.

Examples of this pattern are:

• check-clang-tools for llvm-project/clang-tools-extra/, today we force the tests of each project in this directory into a single test suite directory. (clangd/ and pseudo/ do not follow this, and so their tests are not run by check-clang-tools).
• check-all for the llvm-project/ root, today we use global variables to keep track of all test suites. This is the mechanism this patch attempts to generalize.

Will check-all and check-clang run the same set of tests as before?

Yes, only check-clang-tools is intended to change.

[sammccall]

rebased, come and get me robots

[aaron.ballman]

I'm not an expert in CMake, but the changes look reasonable from what I do know. The current bot failures all look to be unrelated to the changes in this patch as best I can tell. Accepting, but you should probably wait for a second confirmation before landing.

[hokein]

+1, I think this is an improvement, it looks good overall, some nits.

clang-tools-extra/CMakeLists.txt

50

nit: remove the extra blank line.

clang/CMakeLists.txt

194–196

nit: add a # LLVM_INCLUDE_TESTS, this is a deeply-nested if statement, it would be easier to spot which if does the umbrella_lit_testsuite_begin stay.

llvm/cmake/modules/AddLLVM.cmake

1827

if I read the code correctly, input check-clangd the output is CLANGD, not CLANG.

1876

I don't know where is the name variable? it seems at this point we don't have a name var (unlike the code in the foreach(name ${gather_names}) below), I guess we probably should move it to the foreach?

[kbobyrev]

LG to me with the comments that Haojian made!

llvm/cmake/modules/AddLLVM.cmake

1827

Also, maybe the other way around (-> check-clang) since it's a generic LLVM CMake module, not everyone here knows what clangd is.

[sammccall]

Update references to global test-tracking variables is compiler-rt, runtimes,
and llvm/runtimes.

Tested compiler-rt and runtimes standalone build.

[sammccall]

llvm/cmake/modules/AddLLVM.cmake

1876

Wow, this was accidentally copied from line 1842. I suppose it was "mostly harmless". Removed.

[sammccall]

The changes to runtimes/ are mostly mechanical, but Herald says I'm supposed to get a review :-)
@ldionne, @phosek, looks like this touches the same bits from D121276, WDYT?

[ldionne]

I think @phosek is the right person to look at this. I looked at it and it seems fine, but I don't know how the runtimes tests are setup well enough to spot an issue if there were one.

Nit: I'd suggest rebasing and re-uploading the patch so that the runtimes CI can run again (and hopefully pass -- there was a flake last time).

[sammccall]

re-trigger CI

[sammccall]

In D121838#3470478, @ldionne wrote:

I think @phosek is the right person to look at this. I looked at it and it seems fine, but I don't know how the runtimes tests are setup well enough to spot an issue if there were one.

Nit: I'd suggest rebasing and re-uploading the patch so that the runtimes CI can run again (and hopefully pass -- there was a flake last time).

OK. I was requesting extra reviews because of the libc++ herald policy.
But since the changes are mechanical/cross-cutting, phosek isn't a libc++ approver, and there hasn't been more activity here I'm going to try to get this landed.

[JamesNagurne]

Hi Sam, I've got a downstream build system that is failing here and I'm trying to figure out why.

After your commit, I get an error during bootstrapped runtime builds:

05-27 14:41 __main__     INFO     [202/796] cd /path/to/builddir/runtimes/runtimes-target-bins && cmake --build /path/to/builddir/runtimes/runtimes-target-bins/ --target runtimes-test-depends --config Release
05-27 14:41 __main__     INFO     ninja: error: unknown target 'runtimes-test-depends'

We have a mono-project CMakeLists.txt that builds our entire cross-compilation toolchain, libraries and all.
The build system uses a bootstrapping strategy, but must separate the builds of llvm/clang tools and the runtime builds, so we have LLVM_ENABLE_RUNTIMES set but LLVM_BUILD_RUNTIMES=OFF

Why do we do this instead of using the "default build strategy", you may ask? Well, the default strategy didn't exist when we set everything up. Wouldn't be against trying that if it seems to be the only way to fix the issue I'm seeing.

Specifically, in-order, the actions we perform:

• Build clang/llvm tools (target llvm-build)
• Build in-house tools
• Build in-house C Library (uses just-built tools from above)
• Build libc++, libc++abi, compiler-rt using llvm/runtimes (target 'runtimes'. Needs C includes and just-built compiler from previous step)
• Build llvm-test-depends in preparation for check-all, etc. (arbitrarily ordered after runtime builds)

We do this by adding llvm as an ExternalProject and then adding step targets (llvm-runtimes, llvm-test-depends) that just map to the LLVM equivalent target.

I'll look more deeply into your commit and what's changed in our downstream build system, but I wanted to fire this comment off in case you've seen it or have a quick way to figure out what might be wrong.
We do have some downstream changes, including an abandoned commit (https://reviews.llvm.org/D72947) that could be causing issues. That's also a main target of scrutiny.

[JamesNagurne]

After some investigation, I found that we did not set LLVM_INCLUDE_TESTS from the top-level project. This seems like an oversight because we build the test-depends (and check-all) regularly.
After seeing LLVM_INCLUDE_TESTS to ON, the builds worked.

This commit cleaned things up and exposed that mistake.

[JamesNagurne]

In D121838#3543421, @JamesNagurne wrote:

After some investigation, I found that we did not set LLVM_INCLUDE_TESTS from the top-level project. This seems like an oversight because we build the test-depends (and check-all) regularly.
After seeing LLVM_INCLUDE_TESTS to ON, the builds worked.

This commit cleaned things up and exposed that mistake.

Sorry, this was a false positive. Building the 'test-depends' target still shows the LLVM build system telling the runtimes to build 'runtimes-test-depends', which is not defined in those ExternalProjects.

clang-tools-extra/CMakeLists.txt

 include(CMakeDependentOption)
 include(GNUInstallDirs)
 
 option(CLANG_TIDY_ENABLE_STATIC_ANALYZER
   "Include static analyzer checks in clang-tidy" ON)
 
+if(CLANG_INCLUDE_TESTS)
+  umbrella_lit_testsuite_begin(check-clang-tools)
+
+  option(CLANG_TOOLS_TEST_USE_VG "Run Clang tools' tests under Valgrind" OFF)
+  if(CLANG_TOOLS_TEST_USE_VG)
+    set_property(GLOBAL APPEND PROPERTY LLVM_CLANG_TOOLS_LIT_EXTRA_ARGS "--vg")
+  endif()
+endif()
+
 add_subdirectory(clang-apply-replacements)
 add_subdirectory(clang-reorder-fields)
 add_subdirectory(modularize)
 add_subdirectory(clang-tidy)
 
 add_subdirectory(clang-change-namespace)
 add_subdirectory(clang-doc)
 add_subdirectory(clang-include-fixer)
 add_subdirectory(clang-move)
 add_subdirectory(clang-query)
 add_subdirectory(include-cleaner)
 add_subdirectory(pp-trace)
 add_subdirectory(pseudo)
 add_subdirectory(tool-template)
 
-# Add the common testsuite after all the tools.
-if(CLANG_INCLUDE_TESTS)
-add_subdirectory(test)
-add_subdirectory(unittests)
-endif()
-
 option(CLANG_TOOLS_EXTRA_INCLUDE_DOCS "Generate build targets for the Clang Extra Tools docs."
   ${LLVM_INCLUDE_DOCS})
 if( CLANG_TOOLS_EXTRA_INCLUDE_DOCS )
   add_subdirectory(docs)
 endif()
 
 # clangd has its own CMake tree. It requires threads.
 CMAKE_DEPENDENT_OPTION(CLANG_ENABLE_CLANGD "Build clangd language server" ON
                        "LLVM_ENABLE_THREADS" OFF)
 if (CLANG_ENABLE_CLANGD)
   add_subdirectory(clangd)
 endif()
+
+# Add the common testsuite after all the tools.
+if(CLANG_INCLUDE_TESTS)
+  add_subdirectory(test)
+  add_subdirectory(unittests)
+  umbrella_lit_testsuite_end(check-clang-tools)
+endif()

clang-tools-extra/test/CMakeLists.txt

 
 configure_lit_site_cfg(
   ${CMAKE_CURRENT_SOURCE_DIR}/Unit/lit.site.cfg.py.in
   ${CMAKE_CURRENT_BINARY_DIR}/Unit/lit.site.cfg.py
   MAIN_CONFIG
   ${CMAKE_CURRENT_SOURCE_DIR}/Unit/lit.cfg.py
   )
 
-option(CLANG_TOOLS_TEST_USE_VG "Run Clang tools' tests under Valgrind" OFF)
-if(CLANG_TOOLS_TEST_USE_VG)
-  set(CLANG_TOOLS_TEST_EXTRA_ARGS ${CLANG_TEST_EXTRA_ARGS} "--vg")
-endif()
-
 set(CLANG_TOOLS_TEST_DEPS
   # For the clang-apply-replacements test that uses clang-rename.
   clang-rename
 
   # For the clang-doc tests that emit bitcode files.
   llvm-bcanalyzer
 
   # Individual tools we test.
if(TARGET CTTestTidyModule)
       if(CLANG_PLUGIN_SUPPORT AND (WIN32 OR CYGWIN))
         set(LLVM_LINK_COMPONENTS
           Support
         )
       endif()
   endif()
 endif()
 
-add_lit_testsuite(check-clang-tools "Running the Clang extra tools' regression tests"
-  ${CMAKE_CURRENT_BINARY_DIR}
-  DEPENDS ${CLANG_TOOLS_TEST_DEPS}
-  ARGS ${CLANG_TOOLS_TEST_EXTRA_ARGS}
-  )
-
-set_target_properties(check-clang-tools PROPERTIES FOLDER "Clang extra tools' tests")
-
 add_lit_testsuites(CLANG-EXTRA ${CMAKE_CURRENT_SOURCE_DIR}
   DEPENDS ${CLANG_TOOLS_TEST_DEPS}
   )

clang/CMakeLists.txt

if(LLVM_LIT)
 
       # On Win32 hosts, provide an option to specify the path to the GnuWin32 tools.
       if( WIN32 AND NOT CYGWIN )
         set(LLVM_LIT_TOOLS_DIR "" CACHE PATH "Path to GnuWin32 tools")
       endif()
     else()
       set(LLVM_INCLUDE_TESTS OFF)
     endif()
-  endif()
+
+    umbrella_lit_testsuite_begin(check-all)
+  endif() # LLVM_INCLUDE_TESTS
 
   set(BACKEND_PACKAGE_STRING "LLVM ${LLVM_PACKAGE_VERSION}")
 else()
   set(BACKEND_PACKAGE_STRING "${PACKAGE_STRING}")
 endif() # standalone
 
 if(NOT DEFINED LLVM_COMMON_CMAKE_UTILS)
   set(LLVM_COMMON_CMAKE_UTILS ${CMAKE_CURRENT_SOURCE_DIR}/../cmake)
if(EXISTS ${LLVM_MAIN_SRC_DIR}/utils/unittest/googletest/include/gtest/gtest.h)
     list(APPEND CLANG_TEST_PARAMS
       clang_unit_site_config=${CMAKE_CURRENT_BINARY_DIR}/test/Unit/lit.site.cfg
       )
   endif()
   add_subdirectory(test)
   add_subdirectory(bindings/python/tests)
 
   if(CLANG_BUILT_STANDALONE)
-    # Add a global check rule now that all subdirectories have been traversed
-    # and we know the total set of lit testsuites.
-    get_property(LLVM_LIT_TESTSUITES GLOBAL PROPERTY LLVM_LIT_TESTSUITES)
-    get_property(LLVM_LIT_PARAMS GLOBAL PROPERTY LLVM_LIT_PARAMS)
-    get_property(LLVM_LIT_DEPENDS GLOBAL PROPERTY LLVM_LIT_DEPENDS)
-    get_property(LLVM_LIT_EXTRA_ARGS GLOBAL PROPERTY LLVM_LIT_EXTRA_ARGS)
-    get_property(LLVM_ADDITIONAL_TEST_TARGETS
-                 GLOBAL PROPERTY LLVM_ADDITIONAL_TEST_TARGETS)
-    add_lit_target(check-all
-      "Running all regression tests"
-      ${LLVM_LIT_TESTSUITES}
-      PARAMS ${LLVM_LIT_PARAMS}
-      DEPENDS ${LLVM_LIT_DEPENDS} ${LLVM_ADDITIONAL_TEST_TARGETS}
-      ARGS ${LLVM_LIT_EXTRA_ARGS}
-      )
+    umbrella_lit_testsuite_end(check-all)
   endif()
   add_subdirectory(utils/perf-training)
 endif()
 
 option(CLANG_INCLUDE_DOCS "Generate build targets for the Clang docs."
   ${LLVM_INCLUDE_DOCS})
 if( CLANG_INCLUDE_DOCS )
   add_subdirectory(docs)

clang/bindings/python/tests/CMakeLists.txt

 # SystemZ has broken Python/FFI interface:
 # https://reviews.llvm.org/D52840#1265716
 if(${LLVM_NATIVE_ARCH} MATCHES "^(AArch64|Hexagon|Sparc|SystemZ)$")
   set(RUN_PYTHON_TESTS FALSE)
 endif()
 
 if(RUN_PYTHON_TESTS)
     set_property(GLOBAL APPEND PROPERTY
-                 LLVM_ADDITIONAL_TEST_TARGETS check-clang-python)
+                 LLVM_ALL_ADDITIONAL_TEST_TARGETS check-clang-python)
 endif()

clang/runtime/CMakeLists.txt

if(LLVM_INCLUDE_TESTS)
 
     # Add special target to run all compiler-rt test suites.
     get_ext_project_build_command(run_check_compiler_rt compiler-rt-test-depends)
     add_custom_target(compiler-rt-test-depends
       COMMAND ${run_check_compiler_rt}
       DEPENDS compiler-rt-build ${COMPILER_RT_TEST_DEPENDENCIES}
       WORKING_DIRECTORY ${BINARY_DIR}
       VERBATIM USES_TERMINAL)
-    set_property(GLOBAL APPEND PROPERTY LLVM_ADDITIONAL_TEST_DEPENDS compiler-rt-test-depends)
-    set_property(GLOBAL APPEND PROPERTY LLVM_ADDITIONAL_TEST_TARGETS check-compiler-rt)
+    set_property(GLOBAL APPEND PROPERTY LLVM_ALL_ADDITIONAL_TEST_DEPENDS compiler-rt-test-depends)
+    set_property(GLOBAL APPEND PROPERTY LLVM_ALL_ADDITIONAL_TEST_TARGETS check-compiler-rt)
   endif()
 endif()

compiler-rt/test/CMakeLists.txt

if(NOT COMPILER_RT_STANDALONE_BUILD AND NOT LLVM_RUNTIMES_BUILD)
     endif()
   endif()
   # Tests use C++ standard library headers.
   if (TARGET cxx-headers OR HAVE_LIBCXX)
     list(APPEND SANITIZER_COMMON_LIT_TEST_DEPS cxx-headers)
   endif()
 endif()
 
+if(COMPILER_RT_STANDALONE_BUILD)
+  umbrella_lit_testsuite_begin(check-compiler-rt)
+endif()
+
 function(compiler_rt_test_runtime runtime)
   string(TOUPPER ${runtime} runtime_uppercase)
   if(COMPILER_RT_HAS_${runtime_uppercase})
     add_subdirectory(${runtime})
     foreach(directory ${ARGN})
       add_subdirectory(${directory})
     endforeach()
   endif()
if(COMPILER_RT_CAN_EXECUTE_TESTS)
   # ShadowCallStack does not yet provide a runtime with compiler-rt, the tests
   # include their own minimal runtime
   add_subdirectory(shadowcallstack)
 endif()
 
 if(COMPILER_RT_STANDALONE_BUILD)
   # Now that we've traversed all the directories and know all the lit testsuites,
   # introduce a rule to run to run all of them.
-  get_property(LLVM_LIT_TESTSUITES GLOBAL PROPERTY LLVM_LIT_TESTSUITES)
-  get_property(LLVM_LIT_DEPENDS GLOBAL PROPERTY LLVM_LIT_DEPENDS)
-  add_lit_target(check-compiler-rt
-    "Running all regression tests"
-    ${LLVM_LIT_TESTSUITES}
-    DEPENDS ${LLVM_LIT_DEPENDS})
+  add_custom_target(compiler-rt-test-depends DEPENDS ${LLVM_COMPILER_RT_LIT_DEPENDS})
+  umbrella_lit_testsuite_end(check-compiler-rt)
+
   if(NOT TARGET check-all)
     add_custom_target(check-all)
   endif()
-  add_custom_target(compiler-rt-test-depends DEPENDS ${LLVM_LIT_DEPENDS})
   add_dependencies(check-all check-compiler-rt)
 endif()

llvm/CMakeLists.txt

 include(LLVMDistributionSupport)
 
 if( MINGW AND NOT "${CMAKE_CXX_COMPILER_ID}" MATCHES "Clang" )
   # People report that -O3 is unreliable on MinGW. The traditional
   # build also uses -O2 for that reason:
   llvm_replace_compiler_option(CMAKE_CXX_FLAGS_RELEASE "-O3" "-O2")
 endif()
 
+if(LLVM_INCLUDE_TESTS)
+  umbrella_lit_testsuite_begin(check-all)
+endif()
+
 # Put this before tblgen. Else we have a circular dependence.
 add_subdirectory(lib/Demangle)
 add_subdirectory(lib/Support)
 add_subdirectory(lib/TableGen)
 
 add_subdirectory(utils/TableGen)
 
 add_subdirectory(include/llvm)
if( LLVM_INCLUDE_TESTS )
   endif()
 
   if (WIN32)
     # This utility is used to prevent crashing tests from calling Dr. Watson on
     # Windows.
     add_subdirectory(utils/KillTheDoctor)
   endif()
 
-  # Add a global check rule now that all subdirectories have been traversed
-  # and we know the total set of lit testsuites.
-  get_property(LLVM_LIT_TESTSUITES GLOBAL PROPERTY LLVM_LIT_TESTSUITES)
-  get_property(LLVM_LIT_PARAMS GLOBAL PROPERTY LLVM_LIT_PARAMS)
-  get_property(LLVM_LIT_DEPENDS GLOBAL PROPERTY LLVM_LIT_DEPENDS)
-  get_property(LLVM_LIT_EXTRA_ARGS GLOBAL PROPERTY LLVM_LIT_EXTRA_ARGS)
-  get_property(LLVM_ADDITIONAL_TEST_TARGETS
-               GLOBAL PROPERTY LLVM_ADDITIONAL_TEST_TARGETS)
-  get_property(LLVM_ADDITIONAL_TEST_DEPENDS
-               GLOBAL PROPERTY LLVM_ADDITIONAL_TEST_DEPENDS)
-  add_lit_target(check-all
-    "Running all regression tests"
-    ${LLVM_LIT_TESTSUITES}
-    PARAMS ${LLVM_LIT_PARAMS}
-    DEPENDS ${LLVM_LIT_DEPENDS} ${LLVM_ADDITIONAL_TEST_TARGETS}
-    ARGS ${LLVM_LIT_EXTRA_ARGS}
-    )
+  umbrella_lit_testsuite_end(check-all)
+  get_property(LLVM_ALL_LIT_DEPENDS GLOBAL PROPERTY LLVM_ALL_LIT_DEPENDS)
+  get_property(LLVM_ALL_ADDITIONAL_TEST_DEPENDS
+      GLOBAL PROPERTY LLVM_ALL_ADDITIONAL_TEST_DEPENDS)
   add_custom_target(test-depends
-                    DEPENDS ${LLVM_LIT_DEPENDS} ${LLVM_ADDITIONAL_TEST_DEPENDS})
+      DEPENDS ${LLVM_ALL_LIT_DEPENDS} ${LLVM_ALL_ADDITIONAL_TEST_DEPENDS})
   set_target_properties(test-depends PROPERTIES FOLDER "Tests")
 endif()
 
 if (LLVM_INCLUDE_DOCS)
   add_subdirectory(docs)
 endif()
 
 add_subdirectory(cmake/modules)

llvm/cmake/modules/AddLLVM.cmake

function(add_lit_target target comment)
   if (ARG_DEPENDS)
     add_dependencies(${target} ${ARG_DEPENDS})
   endif()
 
   # Tests should be excluded from "Build Solution".
   set_target_properties(${target} PROPERTIES EXCLUDE_FROM_DEFAULT_BUILD ON)
 endfunction()
 
+# Convert a target name like check-clang to a variable name like CLANG.
+function(umbrella_lit_testsuite_var target outvar)
+  if (NOT target MATCHES "^check-")
+    message(FATAL_ERROR "umbrella lit suites must be check-*, not '${target}'")
+  endif()
+  string(SUBSTRING "${target}" 6 -1 var)
+  string(REPLACE "-" "_" var ${var})
+  string(TOUPPER "${var}" var)
+  set(${outvar} "${var}" PARENT_SCOPE)
+endfunction()
+
+# Start recording all lit test suites for a combined 'check-foo' target.
+# The recording continues until umbrella_lit_testsuite_end() creates the target.
+function(umbrella_lit_testsuite_begin target)
+  umbrella_lit_testsuite_var(${target} name)
+  set_property(GLOBAL APPEND PROPERTY LLVM_LIT_UMBRELLAS ${name})
+endfunction()
+
+# Create a combined 'check-foo' target for a set of related test suites.
+# It runs all suites added since the matching umbrella_lit_testsuite_end() call.
+# Tests marked EXCLUDE_FROM_CHECK_ALL are not gathered.
+function(umbrella_lit_testsuite_end target)
+  umbrella_lit_testsuite_var(${target} name)
+
+  get_property(testsuites GLOBAL PROPERTY LLVM_${name}_LIT_TESTSUITES)
+  get_property(params GLOBAL PROPERTY LLVM_${name}_LIT_PARAMS)
+  get_property(depends GLOBAL PROPERTY LLVM_${name}_LIT_DEPENDS)
+  get_property(extra_args GLOBAL PROPERTY LLVM_${name}_LIT_EXTRA_ARGS)
+  # Additional test targets are not gathered, but may be set externally.
+  get_property(additional_test_targets
+               GLOBAL PROPERTY LLVM_${name}_ADDITIONAL_TEST_TARGETS)
+
+  string(TOLOWER ${name} name)
+  add_lit_target(${target}
+    "Running ${name} regression tests"
+    ${testsuites}
+    PARAMS ${params}
+    DEPENDS ${depends} ${additional_test_targets}
+    ARGS ${extra_args}
+    )
+endfunction()
+
 # A function to add a set of lit test suites to be driven through 'check-*' targets.
 function(add_lit_testsuite target comment)
   cmake_parse_arguments(ARG "EXCLUDE_FROM_CHECK_ALL" "" "PARAMS;DEPENDS;ARGS" ${ARGN})
 
   # EXCLUDE_FROM_ALL excludes the test ${target} out of check-all.
   if(NOT ARG_EXCLUDE_FROM_CHECK_ALL)
-    # Register the testsuites, params and depends for the global check rule.
-    set_property(GLOBAL APPEND PROPERTY LLVM_LIT_TESTSUITES ${ARG_UNPARSED_ARGUMENTS})
-    set_property(GLOBAL APPEND PROPERTY LLVM_LIT_PARAMS ${ARG_PARAMS})
-    set_property(GLOBAL APPEND PROPERTY LLVM_LIT_DEPENDS ${ARG_DEPENDS})
-    set_property(GLOBAL APPEND PROPERTY LLVM_LIT_EXTRA_ARGS ${ARG_ARGS})
+    get_property(gather_names GLOBAL PROPERTY LLVM_LIT_UMBRELLAS)
+    foreach(name ${gather_names})
+    # Register the testsuites, params and depends for the umbrella check rule.
+      set_property(GLOBAL APPEND PROPERTY LLVM_${name}_LIT_TESTSUITES ${ARG_UNPARSED_ARGUMENTS})
+      set_property(GLOBAL APPEND PROPERTY LLVM_${name}_LIT_PARAMS ${ARG_PARAMS})
+      set_property(GLOBAL APPEND PROPERTY LLVM_${name}_LIT_DEPENDS ${ARG_DEPENDS})
+      set_property(GLOBAL APPEND PROPERTY LLVM_${name}_LIT_EXTRA_ARGS ${ARG_ARGS})
+    endforeach()
   endif()
 
   # Produce a specific suffixed check rule.
   add_lit_target(${target} ${comment}
     ${ARG_UNPARSED_ARGUMENTS}
     PARAMS ${ARG_PARAMS}
     DEPENDS ${ARG_DEPENDS}
     ARGS ${ARG_ARGS}

llvm/runtimes/CMakeLists.txt

foreach(component ${LLVM_RUNTIME_DISTRIBUTION_COMPONENTS})
     if(NOT ${component} IN_LIST SUB_COMPONENTS)
       list(APPEND extra_targets ${component} install-${component} install-${component}-stripped)
     endif()
   endforeach()
 
   if(LLVM_INCLUDE_TESTS)
     include(${LLVM_BINARY_DIR}/runtimes/Tests.cmake OPTIONAL)
     set_property(DIRECTORY APPEND PROPERTY CMAKE_CONFIGURE_DEPENDS ${LLVM_BINARY_DIR}/runtimes/Tests.cmake)
-    set_property(GLOBAL APPEND PROPERTY LLVM_LIT_TESTSUITES ${SUB_LIT_TESTSUITES})
-    set_property(GLOBAL APPEND PROPERTY LLVM_LIT_PARAMS ${SUB_LIT_PARAMS})
-    set_property(GLOBAL APPEND PROPERTY LLVM_LIT_EXTRA_ARGS ${SUB_LIT_EXTRA_ARGS})
+    set_property(GLOBAL APPEND PROPERTY LLVM_ALL_LIT_TESTSUITES ${SUB_LIT_TESTSUITES})
+    set_property(GLOBAL APPEND PROPERTY LLVM_ALL_LIT_PARAMS ${SUB_LIT_PARAMS})
+    set_property(GLOBAL APPEND PROPERTY LLVM_ALL_LIT_EXTRA_ARGS ${SUB_LIT_EXTRA_ARGS})
     list(APPEND test_targets runtimes-test-depends check-runtimes)
   endif()
 
   set(LLVM_ENABLE_PER_TARGET_RUNTIME_DIR_default ON)
   # AIX should fold 32-bit & 64-bit arch libraries into a single archive.
   if (LLVM_TARGET_TRIPLE MATCHES "aix")
     set(LLVM_ENABLE_PER_TARGET_RUNTIME_DIR_default OFF)
   endif()
foreach(component ${LLVM_RUNTIME_DISTRIBUTION_COMPONENTS})
     set(install-${component}-${name} install-${component})
     set(install-${component}-${name}-stripped install-${component}-stripped)
     list(APPEND ${name}_extra_targets ${component}-${name} install-${component}-${name} install-${component}-${name}-stripped)
   endforeach()
 
   if(LLVM_INCLUDE_TESTS)
     include(${LLVM_BINARY_DIR}/runtimes/${name}/Tests.cmake OPTIONAL)
     set_property(DIRECTORY APPEND PROPERTY CMAKE_CONFIGURE_DEPENDS ${LLVM_BINARY_DIR}/runtimes/${name}/Tests.cmake)
-    set_property(GLOBAL APPEND PROPERTY LLVM_LIT_TESTSUITES ${SUB_LIT_TESTSUITES})
-    set_property(GLOBAL APPEND PROPERTY LLVM_LIT_PARAMS ${SUB_LIT_PARAMS})
-    set_property(GLOBAL APPEND PROPERTY LLVM_LIT_EXTRA_ARGS ${SUB_LIT_EXTRA_ARGS})
+    set_property(GLOBAL APPEND PROPERTY LLVM_ALL_LIT_TESTSUITES ${SUB_LIT_TESTSUITES})
+    set_property(GLOBAL APPEND PROPERTY LLVM_ALL_LIT_PARAMS ${SUB_LIT_PARAMS})
+    set_property(GLOBAL APPEND PROPERTY LLVM_ALL_LIT_EXTRA_ARGS ${SUB_LIT_EXTRA_ARGS})
     set(runtimes-test-depends-${name} runtimes-test-depends)
     set(check-runtimes-${name} check-runtimes)
     list(APPEND ${name}_test_targets runtimes-test-depends-${name} check-runtimes-${name})
     list(APPEND test_targets ${${name}_test_targets})
     foreach(target_name IN LISTS SUB_CHECK_TARGETS)
       set(${target_name}-${name} ${target_name})
       list(APPEND ${name}_test_targets ${target_name}-${name})
       list(APPEND test_targets ${target_name}-${name})
if(NOT LLVM_BUILD_INSTRUMENTED AND CLANG_ENABLE_BOOTSTRAP)
     # that executes at LLVM configuration time to handle this case.
     add_dependencies(clang-bootstrap-deps runtimes-configure)
     # We need to add the runtimes as a dependency because compiler-rt can be
     # built as part of runtimes and we need the profile runtime for PGO
     add_dependencies(clang-bootstrap-deps runtimes)
   endif()
 
   if(LLVM_INCLUDE_TESTS)
-    set_property(GLOBAL APPEND PROPERTY LLVM_ADDITIONAL_TEST_DEPENDS runtimes-test-depends)
+    set_property(GLOBAL APPEND PROPERTY LLVM_ALL_ADDITIONAL_TEST_DEPENDS runtimes-test-depends)
 
     set(RUNTIMES_TEST_DEPENDS
         FileCheck
         count
         llvm-nm
         llvm-objdump
         llvm-xray
         not

runtimes/CMakeLists.txt

 endforeach()
 
 if(LLVM_INCLUDE_TESTS)
   set(LIT_ARGS_DEFAULT "-sv --show-xfail --show-unsupported")
   if (MSVC OR XCODE)
     set(LIT_ARGS_DEFAULT "${LIT_ARGS_DEFAULT} --no-progress-bar")
   endif()
   set(LLVM_LIT_ARGS "${LIT_ARGS_DEFAULT}" CACHE STRING "Default options for lit")
+
+  umbrella_lit_testsuite_begin(check-runtimes)
 endif()
 
 # llvm-libgcc incorporates both compiler-rt and libunwind as subprojects with very
 # specific flags, which causes clashes when they're independently built too.
 if("llvm-libgcc" IN_LIST runtimes)
   if("compiler-rt" IN_LIST runtimes OR "compiler-rt" IN_LIST LLVM_ENABLE_PROJECTS)
     message(FATAL_ERROR
       "Attempting to build both compiler-rt and llvm-libgcc will cause irreconcilable "
       "target clashes. Please choose one or the other, but not both.")
   endif()
 
   if("libunwind" IN_LIST runtimes)
     message(
       FATAL_ERROR
       "Attempting to build both libunwind and llvm-libgcc will cause irreconcilable "
       "target clashes. Please choose one or the other, but not both.")
   endif()
 endif()
 
 # We do this in two loops so that HAVE_* is set for each runtime before the
 # other runtimes are added.
 foreach(entry ${runtimes})
   get_filename_component(projName ${entry} NAME)
 
-  # Between each sub-project we want to cache and clear the LIT properties
-  set_property(GLOBAL PROPERTY LLVM_LIT_TESTSUITES)
-  set_property(GLOBAL PROPERTY LLVM_LIT_PARAMS)
-  set_property(GLOBAL PROPERTY LLVM_LIT_DEPENDS)
-  set_property(GLOBAL PROPERTY LLVM_LIT_EXTRA_ARGS)
-
   add_subdirectory(${entry} ${projName})
-
-  get_property(LLVM_LIT_TESTSUITES GLOBAL PROPERTY LLVM_LIT_TESTSUITES)
-  get_property(LLVM_LIT_PARAMS GLOBAL PROPERTY LLVM_LIT_PARAMS)
-  get_property(LLVM_LIT_DEPENDS GLOBAL PROPERTY LLVM_LIT_DEPENDS)
-  get_property(LLVM_LIT_EXTRA_ARGS GLOBAL PROPERTY LLVM_LIT_EXTRA_ARGS)
-
-  list(APPEND RUNTIMES_LIT_TESTSUITES ${LLVM_LIT_TESTSUITES})
-  list(APPEND RUNTIMES_LIT_PARAMS ${LLVM_LIT_PARAMS})
-  list(APPEND RUNTIMES_LIT_DEPENDS ${LLVM_LIT_DEPENDS})
-  list(APPEND RUNTIMES_LIT_EXTRA_ARGS ${LLVM_LIT_EXTRA_ARGS})
 endforeach()
 
 if(LLVM_INCLUDE_TESTS)
   # Add a global check rule now that all subdirectories have been traversed
   # and we know the total set of lit testsuites.
-  add_lit_target(check-runtimes
-    "Running all regression tests"
-    ${RUNTIMES_LIT_TESTSUITES}
-    PARAMS ${RUNTIMES_LIT_PARAMS}
-    DEPENDS ${RUNTIMES_LIT_DEPENDS}
-    ARGS ${RUNTIMES_LIT_EXTRA_ARGS}
-    )
-  add_custom_target(runtimes-test-depends DEPENDS ${RUNTIMES_LIT_DEPENDS})
+  umbrella_lit_testsuite_end(check-runtimes)
 
   if (NOT HAVE_LLVM_LIT)
     # If built by manually invoking cmake on this directory, we don't have
     # llvm-lit. If invoked via llvm/runtimes, the toplevel llvm cmake
     # invocation already generated the llvm-lit script.
     add_subdirectory(${LLVM_MAIN_SRC_DIR}/utils/llvm-lit
                      ${CMAKE_CURRENT_BINARY_DIR}/llvm-lit)
   endif()

runtimes/Tests.cmake.in

-set(SUB_LIT_TESTSUITES @RUNTIMES_LIT_TESTSUITES@)
-set(SUB_LIT_PARAMS @RUNTIMES_LIT_PARAMS@)
-set(SUB_LIT_EXTRA_ARGS @RUNTIMES_LIT_EXTRA_ARGS@)
+set(SUB_LIT_TESTSUITES @LLVM_RUNTIMES_LIT_TESTSUITES@)
+set(SUB_LIT_PARAMS @LLVM_RUNTIMES_LIT_PARAMS@)
+set(SUB_LIT_EXTRA_ARGS @LLVM_RUNTIMES_LIT_EXTRA_ARGS@)