[RFC][test] Adapt debug-info lit framework for more general purposes - part 1

[jhenderson]

Summary

Discussion thread: https://lists.llvm.org/pipermail/llvm-dev/2021-January/148048.html

Move debuginfo-test into a subdirectory of a new top-level directory, called cross-project-tests. The new name replaces "debuginfo-test" as an LLVM project enabled via LLVM_ENABLE_PROJECTS.

Depends on D101982.

[jhenderson]

Replace new testsuite with adaptation of the existing debuginfo-test testsuite.

[jhenderson]

I've iterated on this a few times locally, and feel like the easiest thing to do is to make the debuginfo-test directory a subdirectory of some new top-level directory. Further changes are to follow to make the whole thing a little bit more self-consistent.

I consider this now to be in a good state to review.

[jhenderson]

Ping? (Also all the patches that depend on this)

[jhenderson]

Rebase and ping!

[jhenderson]

Could someone please take a look at this patch series? It's been open for 6 weeks with no comment on this or 3 of the 4 other patches in the series.

We have a need for this mechanism downstream, and will shortly fallback to doing something like this locally, but it would be far preferable to do this upstream.

[mehdi_amini]

How does LLVM_ENABLE_PROJECTS=cross-project-tests will work if only configure a subset of the projects? Or will it requires all the projects to be configured?

[jhenderson]

In D95339#2626418, @mehdi_amini wrote:

How does LLVM_ENABLE_PROJECTS=cross-project-tests will work if only configure a subset of the projects? Or will it requires all the projects to be configured?

By the end of the patch series, the aim is to have tests marked as UNSUPPORTED if certain tools cannot be found, possibly due to them not being in LLVM_ENABLE_PROJECTS. If they are in LLVM_ENABLE_PROJECTS and cannot be found, the tests will fail.

The existing rules already mark some tests as UNSUPPORTED if lldb is not found in the environment somewhere (possibly because it is built due to being in LLVM_ENABLE_PROJECTS, or possibly because it is installed on the machine).

D96511 (the fourth patch in this series) will mark tests as UNSUPPORTED if clang can't be found and isn't in LLVM_ENABLED_PROJECTS. The configuration matrix in this case would be:

• clang in "LLVM_ENABLED_PROJECTS" and clang not found - error for missing tool.
• clang in "LLVM_ENABLED_PROJECTS" and clang found using existing rules (either in build output or via environment variables) - tests run.
• clang not in "LLVM_ENABLED_PROJECTS" and clang not found - debuginfo_tests (and any other tests added in the future with REQUIRES: clang) will be marked as UNSUPPORTED.

In D96510 (the third patch), I add LLD to the list of tools these tests look for, if it is in LLVM_ENABLE_PROJECTS, so that tests can be added that use REQUIRES: lld. This is subtly different to how the other tools work (the executable won't be searched for in the build paths if it is not in the LLVM_ENABLE_PROJECTS), but probably should be discussed in D96510.

[dblaikie]

In D95339#2628285, @jhenderson wrote:

In D95339#2626418, @mehdi_amini wrote:

How does LLVM_ENABLE_PROJECTS=cross-project-tests will work if only configure a subset of the projects? Or will it requires all the projects to be configured?

By the end of the patch series, the aim is to have tests marked as UNSUPPORTED if certain tools cannot be found, possibly due to them not being in LLVM_ENABLE_PROJECTS. If they are in LLVM_ENABLE_PROJECTS and cannot be found, the tests will fail.

The existing rules already mark some tests as UNSUPPORTED if lldb is not found in the environment somewhere (possibly because it is built due to being in LLVM_ENABLE_PROJECTS, or possibly because it is installed on the machine).

D96511 (the fourth patch in this series) will mark tests as UNSUPPORTED if clang can't be found and isn't in LLVM_ENABLED_PROJECTS. The configuration matrix in this case would be:

• clang in "LLVM_ENABLED_PROJECTS" and clang not found - error for missing tool.

Out of curiosity, any particular scenarios where you expect this ^ to come up. (does this configuration require the user to manually ensure the clang subproject was built before running the tests? So this error would be telling the user to go and run a separate build target before running the test target? If that's the case, it'd be good to have the dependencies added automatically if possible - in which case when would this error turn up? Only if there's some internal build system failure (akin to someone deleting a file out from underneath the build system - as unlikely as a .o file (under the purview of the build system) being missing from a linker invocation))

[jhenderson]

In D95339#2629533, @dblaikie wrote:

In D95339#2628285, @jhenderson wrote:

In D95339#2626418, @mehdi_amini wrote:

How does LLVM_ENABLE_PROJECTS=cross-project-tests will work if only configure a subset of the projects? Or will it requires all the projects to be configured?

By the end of the patch series, the aim is to have tests marked as UNSUPPORTED if certain tools cannot be found, possibly due to them not being in LLVM_ENABLE_PROJECTS. If they are in LLVM_ENABLE_PROJECTS and cannot be found, the tests will fail.

The existing rules already mark some tests as UNSUPPORTED if lldb is not found in the environment somewhere (possibly because it is built due to being in LLVM_ENABLE_PROJECTS, or possibly because it is installed on the machine).

D96511 (the fourth patch in this series) will mark tests as UNSUPPORTED if clang can't be found and isn't in LLVM_ENABLED_PROJECTS. The configuration matrix in this case would be:

• clang in "LLVM_ENABLED_PROJECTS" and clang not found - error for missing tool.

Out of curiosity, any particular scenarios where you expect this ^ to come up. (does this configuration require the user to manually ensure the clang subproject was built before running the tests? So this error would be telling the user to go and run a separate build target before running the test target? If that's the case, it'd be good to have the dependencies added automatically if possible - in which case when would this error turn up? Only if there's some internal build system failure (akin to someone deleting a file out from underneath the build system - as unlikely as a .o file (under the purview of the build system) being missing from a linker invocation))

Just to be clear, clang (and later lld in a future change) is already in the list of dependencies for check-debuginfo, so the dependency is there, if you run check-debuginfo to run the tests. The use case I've got in mind is where someone didn't use check-* to build the test dependencies. I routinely run subsets of the lit tests without running a check-* target, because I often am working on individual tools or tests, and there's no need for me to build the rest of LLVM (I build the one executable, say llvm-readelf, and that's about it). If a user happens to have cleared out their build directory beforehand and then try to run the tests in this way, lit will emit the error about a missing tool rather than the test itself failing, so you get quicker and clearer feedback.

[mehdi_amini]

either in build output or via environment variables

This "magic" seems somehow fragile to me: if I have an old lldb installed and I run the cross-project tests with clang+lld+flang configured, it may run the lldb test and incorrectly fails ninja check-all because I only intended to test clang+lld+flang that I'm building/testing in the repo. Can this be controlled by a flag instead so there is an explicit opt-in to use a prebuilt tool? (-DCROSS_PROJECT_LLDB_PATH=<path> straw man).

[jhenderson]

In D95339#2632968, @mehdi_amini wrote:

either in build output or via environment variables

This "magic" seems somehow fragile to me: if I have an old lldb installed and I run the cross-project tests with clang+lld+flang configured, it may run the lldb test and incorrectly fails ninja check-all because I only intended to test clang+lld+flang that I'm building/testing in the repo. Can this be controlled by a flag instead so there is an explicit opt-in to use a prebuilt tool? (-DCROSS_PROJECT_LLDB_PATH=<path> straw man).

Yeah, I was in two minds about it. I followed the approach I did because discussing with others, it's sometimes desired to run some of the tests (I think the ones to do with dexter) with any old compiler, because how it gets compiled is somewhat unimportant.

That being said, I don't think I'm changing anything with regards to LLDB at least. I'd be happy to change the behaviour so that you have to opt-in deliberately to using tools that aren't part of the explicit dependencies, if they can be found, but the obvious solution might impact other projects outside this area, so probably deserves a wider discussion on llvm-dev, if you think it's worth proceeding with that. I'll take a look and see what I can come up with.

[jhenderson]

In D95339#2633839, @jhenderson wrote:

In D95339#2632968, @mehdi_amini wrote:

either in build output or via environment variables

This "magic" seems somehow fragile to me: if I have an old lldb installed and I run the cross-project tests with clang+lld+flang configured, it may run the lldb test and incorrectly fails ninja check-all because I only intended to test clang+lld+flang that I'm building/testing in the repo. Can this be controlled by a flag instead so there is an explicit opt-in to use a prebuilt tool? (-DCROSS_PROJECT_LLDB_PATH=<path> straw man).

Yeah, I was in two minds about it. I followed the approach I did because discussing with others, it's sometimes desired to run some of the tests (I think the ones to do with dexter) with any old compiler, because how it gets compiled is somewhat unimportant.

That being said, I don't think I'm changing anything with regards to LLDB at least. I'd be happy to change the behaviour so that you have to opt-in deliberately to using tools that aren't part of the explicit dependencies, if they can be found, but the obvious solution might impact other projects outside this area, so probably deserves a wider discussion on llvm-dev, if you think it's worth proceeding with that. I'll take a look and see what I can come up with.

I went back over the lit code. The behaviour of use_clang, use_lld and use_llvm_tool is as I thought: all three already look for the tool in the PATH. The build directory is first, so if the tool has just been built, it will already use that. Otherwise, it will find an installed tool. That appears to be by design. I guess people could opt-out of using preinstalled tools by omitting the installation directories from the PATH when running lit. Do we need something more than that?

[mehdi_amini]

Do we need something more than that?

Yes: that may be how the debuginfo-tests was setup, but that seems like a no-go to me for generalizing this into a "cross-projects" that intends to serve the monorepo, for the reasons I mentioned above. I may be interested in some cross-project test in the future but unlikely to want to test lldb. Yet lldb is in /usr/bin/ on my Mac and this isn't a good "default" to use this for running the tests here IMO, especially when I don't intend this (and I obviously can't remove /usr/bin/ from my path without unintended consequences)

[dblaikie]

In D95339#2635435, @mehdi_amini wrote:

Do we need something more than that?

Yes: that may be how the debuginfo-tests was setup, but that seems like a no-go to me for generalizing this into a "cross-projects" that intends to serve the monorepo, for the reasons I mentioned above. I may be interested in some cross-project test in the future but unlikely to want to test lldb. Yet lldb is in /usr/bin/ on my Mac and this isn't a good "default" to use this for running the tests here IMO, especially when I don't intend this (and I obviously can't remove /usr/bin/ from my path without unintended consequences)

Perhaps Apple folks who implemented the original (@aprantl and @JDevlieghere I believe) can weigh in on their intended (& realized/actual) use cases for this functionality - though I agree that probably an explicit opt in to "I want to test with whatever binaries are in my PATH rather than just-built tools in this checkout" would be desirable here.

[aprantl]

Perhaps Apple folks who implemented the original (@aprantl and @JDevlieghere I believe) can weigh in on their intended (& realized/actual) use cases for this functionality - though I agree that probably an explicit opt in to "I want to test with whatever binaries are in my PATH rather than just-built tools in this checkout" would be desirable here.

This is more incidental than intentional behavior. The debuginfo-tests originally tested that the "system" /usr/bin/gdb would could debug clang-generated code. At some point I generalized the testsuite to also support LLDB, and kept the use-the-system-debugger behavior. In a monorepo world it makes much more sense to make this behavior configurable, but I would appreciate an option to test against an LLDB that is installed somewhere in the system and not just the latest version from the monorepo, so we can test some basic level backwards compatibility.

[jhenderson]

In D95339#2638619, @aprantl wrote:

Perhaps Apple folks who implemented the original (@aprantl and @JDevlieghere I believe) can weigh in on their intended (& realized/actual) use cases for this functionality - though I agree that probably an explicit opt in to "I want to test with whatever binaries are in my PATH rather than just-built tools in this checkout" would be desirable here.

This is more incidental than intentional behavior. The debuginfo-tests originally tested that the "system" /usr/bin/gdb would could debug clang-generated code. At some point I generalized the testsuite to also support LLDB, and kept the use-the-system-debugger behavior. In a monorepo world it makes much more sense to make this behavior configurable, but I would appreciate an option to test against an LLDB that is installed somewhere in the system and not just the latest version from the monorepo, so we can test some basic level backwards compatibility.

Okay, all sounds reasonable to me. I think there's a more general problem that the lit tests can use tools in the PATH, outside the build directory, if the tool hasn't been built (this is based on reading the lit source code - I need to confirm this in other test directories). I think we can resolve all of this with a new CMake variable, something like LLVM_USE_INSTALLED_TOOLS_FOR_TESTS (other name suggestions welcome). I'll take a look at this at some point in the coming days and post something on the mailing list to get more people's attention.

[jhenderson]

TL;DR: I think we can turn off looking for tools on the PATH when use_llvm_tool is used. This covers the use-cases in the debuginfo-tests/cross-project-tests area. Support already exists (but needs improving) for overriding using the just-built tool for these cases.

Full details:

I've dug into how tools are picked up. The exact behaviour depends on the tool in question. The first thing that's important to note is that for most tools, when you see an invocation like RUN: llvm-objcopy ..., the "llvm-objcopy" is substituted for the full path to the tool found in build directory. This means the tool won't be found on the PATH. The list of tools is defined in config files, e.g. https://github.com/llvm-mirror/llvm/blob/2c4ca6832fa6b306ee6a7010bfb80a3f2596f824/test/lit.cfg.py#L146.

A limited number of places use use_llvm_tool to find the executable. This is used by the debuginfo-tests to find LLDB, clang++, clang-cl and indirectly clang itself. There are various similar uses in other places throughout the project testsuites. use_llvm_tool allows you to specify an environment variable which points to the tool to use. In the debuginfo-tests, you can use the environment variable "CLANG" to specify the location of the clang executable. Unfortunately, it also specifies the location of LLDB (I think this is a bug)! If the environment variable is not specified, the tool is found on the PATH. The build directory is at the front of the PATH. This means that if the tool isn't in the build directory (i.e. a user hasn't built it), it will use the installed tool from the PATH.

Given the environment variable option, I think we could just disable using tools from the PATH where use_llvm_tool has been used. Note that we can't completely disable the use of PATH for all cases, because system tools like rm/cp etc need to be available.

[jhenderson]

Rebased on D101982. That patch now allows us to override the tool used with an environment variable. I'm still looking at preventing the tools being picked up from the PATH.

[jhenderson]

In D95339#2741545, @jhenderson wrote:

I'm still looking at preventing the tools being picked up from the PATH.

I dug into this. The behaviour is not specific to the debuginfo-tests/cross-project-test testsuite. Rather, any tool that is found via use_llvm_tool will use this process. Primarily, this means any tool that calls use_clang/use_lld will fallback to the PATH version if none is found in the build directory. Looking back at the history, for clang this has been the behaviour for at least 12 years (see https://github.com/llvm/llvm-project/commit/be4253a0caf6f011b2af12803db9cf980f48084e), quite possibly longer. Changing the behaviour to no longer rely on the PATH could impact various tests beyond this testsuite, in clangd and lldb as well as in the clang and lld testsuites themselves.

I'm not convinced we need the behaviour to use the PATH version now. With the CLANG and LLDB environment variables allowing to override which tool is used, and a similar patch for LLD planned for later today, I think we can probably remove the functionality. I've posted https://lists.llvm.org/pipermail/llvm-dev/2021-May/150421.html to discuss this further. Either way, I think the decision that the email chain will hopefully lead to shouldn't hold up proceeding with this patch series. Could I therefore ask people to resume reivewing this series, please?

[jhenderson]

In D95339#2741619, @jhenderson wrote:

and a similar patch for LLD planned for later today

This patch is up for review as D101995.

[jhenderson]

Ping!

[jhenderson]

In D95339#2741619, @jhenderson wrote:

I'm not convinced we need the behaviour to use the PATH version now. With the CLANG and LLDB environment variables allowing to override which tool is used, and a similar patch for LLD planned for later today, I think we can probably remove the functionality. I've posted https://lists.llvm.org/pipermail/llvm-dev/2021-May/150421.html to discuss this further. Either way, I think the decision that the email chain will hopefully lead to shouldn't hold up proceeding with this patch series. Could I therefore ask people to resume reivewing this series, please?

Quick update on this: D101982 fixes the environment variable for LLDB to be 'LLDB' not 'CLANG', so that it is now possible to specify that executable as requested by @aprantl. I have dropped the other new environment variables, as there has been no specific request for them, but can easily readd them, should the need arise. I will upload a new review, likely later today, to try disabling the wider PATH usage for these tools, but it is a somewhat independent item now.

[jhenderson]

Ping?

[jhenderson]

Ping!!

[jhenderson]

Ping #4!

[jhenderson]

Ping #5!!

The rest of the patches in this series have been approved. It's just this patch that needs looking at. There's no particular need to look at most of the files that were in the debuginfo-test folder, as they've simply been moved. I'd recommend reviewing the changes outside that folder, plus the CMake files.

[mehdi_amini]

Seems like there are only a few line changed other than the renaming? What kind of review is needed here? Just an agreement on the principle of moving these? I think it should be someone having a stake in the debug-info tests to give some sort of approval here.

[jhenderson]

In D95339#2831897, @mehdi_amini wrote:

Seems like there are only a few line changed other than the renaming? What kind of review is needed here? Just an agreement on the principle of moving these? I think it should be someone having a stake in the debug-info tests to give some sort of approval here.

Yeah, exactly. I've not had any positive confirmation that people are happy with the overall idea.

[mehdi_amini]

In D95339#2832498, @jhenderson wrote:

In D95339#2831897, @mehdi_amini wrote:

Seems like there are only a few line changed other than the renaming? What kind of review is needed here? Just an agreement on the principle of moving these? I think it should be someone having a stake in the debug-info tests to give some sort of approval here.

Yeah, exactly. I've not had any positive confirmation that people are happy with the overall idea.

@aprantl @dblaikie maybe?

[dblaikie]

In D95339#2833613, @mehdi_amini wrote:

In D95339#2832498, @jhenderson wrote:

In D95339#2831897, @mehdi_amini wrote:

Seems like there are only a few line changed other than the renaming? What kind of review is needed here? Just an agreement on the principle of moving these? I think it should be someone having a stake in the debug-info tests to give some sort of approval here.

Yeah, exactly. I've not had any positive confirmation that people are happy with the overall idea.

@aprantl @dblaikie maybe?

Yeah, generally I was leaving this up to @aprantl or whoever else has some more conext/history/ownership over the existing debug-info-tests repository.

[aprantl]

I guess we've discussed that on the mailing list and there was a consensus to move ahead this so I'm happy to makr the patch as accepted. Please take a close look at the output of https://green.lab.llvm.org/green/view/LLDB/job/lldb-cmake/ to make sure that the debuginfo-tests are still being run after landing it. If it doesn't work and the bot scripts need to be adapted feel free to reach out to me to make that happen. Sorry for the delay.

[jhenderson]

In D95339#2837491, @aprantl wrote:

I guess we've discussed that on the mailing list and there was a consensus to move ahead this so I'm happy to makr the patch as accepted. Please take a close look at the output of https://green.lab.llvm.org/green/view/LLDB/job/lldb-cmake/ to make sure that the debuginfo-tests are still being run after landing it. If it doesn't work and the bot scripts need to be adapted feel free to reach out to me to make that happen. Sorry for the delay.

Okay, thanks. I'll try to land the patch series today. I suspect this patch may need some rebasing first.

[sylvestre.ledru]

I pushed this small change to adjust the doc https://github.com/llvm/llvm-project/commit/2790498178b4bb7c8594c936d512410d55b91d98

This change should probably be mentioned in the release notes too :)

[jhenderson]

In D95339#3007765, @sylvestre.ledru wrote:

I pushed this small change to adjust the doc https://github.com/llvm/llvm-project/commit/2790498178b4bb7c8594c936d512410d55b91d98

This change should probably be mentioned in the release notes too :)

Thanks, I missed that one! I'm not familiar with the release note process for where the release is at such a late stage. @tstellar (or anybody else), are you able to help with this?

[tstellar]

In D95339#3008643, @jhenderson wrote:

In D95339#3007765, @sylvestre.ledru wrote:

I pushed this small change to adjust the doc https://github.com/llvm/llvm-project/commit/2790498178b4bb7c8594c936d512410d55b91d98

This change should probably be mentioned in the release notes too :)

Thanks, I missed that one! I'm not familiar with the release note process for where the release is at such a late stage. @tstellar (or anybody else), are you able to help with this?

Is this change in the release/13.x branch? If so, then you can push changes to update the release notes at any time.

[jhenderson]

In D95339#3009163, @tstellar wrote:

In D95339#3008643, @jhenderson wrote:

In D95339#3007765, @sylvestre.ledru wrote:

I pushed this small change to adjust the doc https://github.com/llvm/llvm-project/commit/2790498178b4bb7c8594c936d512410d55b91d98

This change should probably be mentioned in the release notes too :)

Thanks, I missed that one! I'm not familiar with the release note process for where the release is at such a late stage. @tstellar (or anybody else), are you able to help with this?

Is this change in the release/13.x branch? If so, then you can push changes to update the release notes at any time.

Thanks, I've created D110134 for it, as I wasn't 100% certain on the best point within the release notes for this note, and would also appreciate a second opinion on the wording.

README.md

* ``cmake -S llvm -B build -G <generator> [options]``
           solutions.
         * ``Xcode`` --- for generating Xcode projects.
 
         Some Common options:
 
         * ``-DLLVM_ENABLE_PROJECTS='...'`` --- semicolon-separated list of the LLVM
           sub-projects you'd like to additionally build. Can include any of: clang,
           clang-tools-extra, libcxx, libcxxabi, libunwind, lldb, compiler-rt, lld,
-          polly, or debuginfo-tests.
+          polly, or cross-project-tests.
 
           For example, to build LLVM, Clang, libcxx, and libcxxabi, use
           ``-DLLVM_ENABLE_PROJECTS="clang;libcxx;libcxxabi"``.
 
         * ``-DCMAKE_INSTALL_PREFIX=directory`` --- Specify for *directory* the full
           path name of where you want the LLVM tools and libraries to be installed
           (default ``/usr/local``).
 

cross-project-tests/CMakeLists.txt

 # Debug Info tests.  These tests invoke clang to generate programs with
 # various types of debug info, and then run those programs under a debugger
 # such as GDB or LLDB to verify the results.
 
 find_package(Python3 COMPONENTS Interpreter)
 
 add_llvm_executable(check-gdb-llvm-support
-        llvm-prettyprinters/gdb/llvm-support.cpp
+        debuginfo-tests/llvm-prettyprinters/gdb/llvm-support.cpp
 )
 target_link_libraries(check-gdb-llvm-support PRIVATE LLVMSupport)
 
 set(DEBUGINFO_TESTS_SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR})
 set(DEBUGINFO_TESTS_BINARY_DIR ${CMAKE_CURRENT_BINARY_DIR})
 
 set(DEBUGINFO_TEST_DEPS
   clang
   FileCheck
   count
   llvm-config
   llvm-objdump
   check-gdb-llvm-support
   not
   )
 
 if ("mlir" IN_LIST LLVM_ENABLE_PROJECTS)
   add_llvm_executable(check-gdb-mlir-support
-        llvm-prettyprinters/gdb/mlir-support.cpp
+        debuginfo-tests/llvm-prettyprinters/gdb/mlir-support.cpp
   )
   target_include_directories(check-gdb-mlir-support PRIVATE
                                ${LLVM_EXTERNAL_MLIR_SOURCE_DIR}/include
                                ${LLVM_BINARY_DIR}/tools/mlir/include)
   target_link_libraries(check-gdb-mlir-support PRIVATE MLIRIR)
   list(APPEND DEBUGINFO_TEST_DEPS check-gdb-mlir-support)
   set(MLIR_SOURCE_DIR  ${LLVM_EXTERNAL_MLIR_SOURCE_DIR})
 endif()

 
 configure_lit_site_cfg(
   ${CMAKE_CURRENT_SOURCE_DIR}/lit.site.cfg.py.in
   ${CMAKE_CURRENT_BINARY_DIR}/lit.site.cfg.py
   MAIN_CONFIG
   ${CMAKE_CURRENT_SOURCE_DIR}/lit.cfg.py
   )
 
-add_lit_testsuite(check-debuginfo "Running debug info integration tests"
+add_lit_testsuite(check-debuginfo "Running cross-project tests"
   ${CMAKE_CURRENT_BINARY_DIR}
   DEPENDS ${DEBUGINFO_TEST_DEPS}
   )
 
 # Add check-debuginfo-* targets.
 add_lit_testsuites(DEBUGINFO ${CMAKE_CURRENT_SOURCE_DIR}
   DEPENDS ${DEBUGINFO_TEST_DEPS}
   )
 
-set_target_properties(check-debuginfo PROPERTIES FOLDER "Debug info tests")
+set_target_properties(check-debuginfo PROPERTIES FOLDER "Tests")

cross-project-tests/lit.cfg.py

 config.suffixes = ['.c', '.cpp', '.m']
 
 # excludes: A list of directories to exclude from the testsuite. The 'Inputs'
 # subdirectories contain auxiliary inputs for various tests in their parent
 # directories.
 config.excludes = ['Inputs']
 
 # test_source_root: The root path where tests are located.
-config.test_source_root = os.path.join(config.debuginfo_tests_src_root)
+config.test_source_root = config.debuginfo_tests_src_root
 
 # test_exec_root: The root path where tests should be run.
 config.test_exec_root = config.debuginfo_tests_obj_root
 
 llvm_config.use_default_substitutions()
 
 tools = [
     ToolSubst('%test_debuginfo', command=os.path.join(
-        config.debuginfo_tests_src_root, 'llgdb-tests', 'test_debuginfo.pl')),
+        config.debuginfo_tests_src_root, 'debuginfo-tests',
+        'llgdb-tests', 'test_debuginfo.pl')),
     ToolSubst("%llvm_src_root", config.llvm_src_root),
     ToolSubst("%llvm_tools_dir", config.llvm_tools_dir),
 ]
 
 def get_required_attr(config, attr_name):
   attr_value = getattr(config, attr_name, None)
   if attr_value == None:
     lit_config.fatal(

 lldb_path = llvm_config.use_llvm_tool('lldb', search_env='LLDB')
 
 if lldb_path is not None:
     config.available_features.add('lldb')
 
 # Produce dexter path, lldb path, and combine into the %dexter substitution
 # for running a test.
 dexter_path = os.path.join(config.debuginfo_tests_src_root,
-                           'dexter', 'dexter.py')
+                           'debuginfo-tests', 'dexter', 'dexter.py')
 dexter_test_cmd = '"{}" "{}" test'.format(sys.executable, dexter_path)
 if lldb_path is not None:
   dexter_test_cmd += ' --lldb-executable "{}"'.format(lldb_path)
 tools.append(ToolSubst('%dexter', dexter_test_cmd))
 
 # For testing other bits of dexter that aren't under the "test" subcommand,
 # have a %dexter_base substitution.
 dexter_base_cmd = '"{}" "{}"'.format(sys.executable, dexter_path)

llvm/CMakeLists.txt

if (NOT CMAKE_BUILD_TYPE AND NOT CMAKE_CONFIGURATION_TYPES)
   message(STATUS "No build type selected, default to Debug")
   set(CMAKE_BUILD_TYPE "Debug" CACHE STRING "Build type (default Debug)" FORCE)
 endif()
 
 # Side-by-side subprojects layout: automatically set the
 # LLVM_EXTERNAL_${project}_SOURCE_DIR using LLVM_ALL_PROJECTS
 # This allows an easy way of setting up a build directory for llvm and another
 # one for llvm+clang+... using the same sources.
-set(LLVM_ALL_PROJECTS "clang;clang-tools-extra;compiler-rt;debuginfo-tests;libc;libclc;libcxx;libcxxabi;libunwind;lld;lldb;mlir;openmp;parallel-libs;polly;pstl")
+set(LLVM_ALL_PROJECTS "clang;clang-tools-extra;compiler-rt;cross-project-tests;libc;libclc;libcxx;libcxxabi;libunwind;lld;lldb;mlir;openmp;parallel-libs;polly;pstl")
 # The flang project is not yet part of "all" projects (see C++ requirements)
 set(LLVM_EXTRA_PROJECTS "flang")
 # List of all known projects in the mono repo
 set(LLVM_KNOWN_PROJECTS "${LLVM_ALL_PROJECTS};${LLVM_EXTRA_PROJECTS}")
 set(LLVM_ENABLE_PROJECTS "" CACHE STRING
   "Semicolon-separated list of projects to build (${LLVM_KNOWN_PROJECTS}), or \"all\".")
 if( LLVM_ENABLE_PROJECTS STREQUAL "all" )
   set( LLVM_ENABLE_PROJECTS ${LLVM_ALL_PROJECTS})

llvm/docs/CMake.rst

 **LLVM_ENABLE_PROJECTS**:STRING
   Semicolon-separated list of projects to build, or *all* for building all
   (clang, libcxx, libcxxabi, lldb, compiler-rt, lld, polly, etc) projects.
   This flag assumes that projects are checked out side-by-side and not nested,
   i.e. clang needs to be in parallel of llvm instead of nested in `llvm/tools`.
   This feature allows to have one build for only LLVM and another for clang+llvm
   using the same source checkout.
   The full list is:
-  ``clang;clang-tools-extra;compiler-rt;debuginfo-tests;libc;libclc;libcxx;libcxxabi;libunwind;lld;lldb;openmp;parallel-libs;polly;pstl``
+  ``clang;clang-tools-extra;compiler-rt;cross-project-tests;libc;libclc;libcxx;libcxxabi;libunwind;lld;lldb;openmp;parallel-libs;polly;pstl``
 
 **LLVM_ENABLE_RTTI**:BOOL
   Build LLVM with run-time type information. Defaults to OFF.
 
 **LLVM_ENABLE_SPHINX**:BOOL
   If specified, CMake will search for the ``sphinx-build`` executable and will make
   the ``SPHINX_OUTPUT_HTML`` and ``SPHINX_OUTPUT_MAN`` CMake options available.
   Defaults to OFF.

llvm/docs/GettingStarted.rst

* ``Visual Studio`` --- for generating Visual Studio projects and
        solutions.
      * ``Xcode`` --- for generating Xcode projects.
 
      Some Common options:
 
      * ``-DLLVM_ENABLE_PROJECTS='...'`` --- semicolon-separated list of the LLVM
        subprojects you'd like to additionally build. Can include any of: clang,
        clang-tools-extra, libcxx, libcxxabi, libunwind, lldb, compiler-rt, lld,
-       polly, or debuginfo-tests.
+       polly, or cross-project-tests.
 
        For example, to build LLVM, Clang, libcxx, and libcxxabi, use
        ``-DLLVM_ENABLE_PROJECTS="clang;libcxx;libcxxabi"``.
 
      * ``-DCMAKE_INSTALL_PREFIX=directory`` --- Specify for *directory* the full
        pathname of where you want the LLVM tools and libraries to be installed
        (default ``/usr/local``).
 

llvm/docs/TestingGuide.rst

 ---------------------------
 
 The test suite contains tests to check quality of debugging information.
 The test are written in C based languages or in LLVM assembly language.
 
 These tests are compiled and run under a debugger. The debugger output
 is checked to validate of debugging information. See README.txt in the
 test suite for more information. This test suite is located in the
-``debuginfo-tests`` Subversion module.
+``cross-project-tests/debuginfo-tests`` directory.
 
 Quick start
 ===========
 
 The tests are located in two separate Subversion modules. The unit and
 regression tests are in the main "llvm" module under the directories
 ``llvm/unittests`` and ``llvm/test`` (so you get these tests for free with the
 main LLVM tree). Use ``make check-all`` to run the unit and regression tests

llvm/projects/CMakeLists.txt

 # Discover the projects that use CMake in the subdirectories.
 # Note that explicit cmake invocation is required every time a new project is
 # added or removed.
 file(GLOB entries *)
 foreach(entry ${entries})
   if(IS_DIRECTORY ${entry} AND EXISTS ${entry}/CMakeLists.txt)
     if((NOT ${entry} STREQUAL ${CMAKE_CURRENT_SOURCE_DIR}/compiler-rt) AND
        (NOT ${entry} STREQUAL ${CMAKE_CURRENT_SOURCE_DIR}/dragonegg) AND
        (NOT ${entry} STREQUAL ${CMAKE_CURRENT_SOURCE_DIR}/libcxx) AND
        (NOT ${entry} STREQUAL ${CMAKE_CURRENT_SOURCE_DIR}/libcxxabi) AND
        (NOT ${entry} STREQUAL ${CMAKE_CURRENT_SOURCE_DIR}/libunwind) AND
        (NOT ${entry} STREQUAL ${CMAKE_CURRENT_SOURCE_DIR}/test-suite) AND
        (NOT ${entry} STREQUAL ${CMAKE_CURRENT_SOURCE_DIR}/parallel-libs) AND
        (NOT ${entry} STREQUAL ${CMAKE_CURRENT_SOURCE_DIR}/openmp) AND
-       (NOT ${entry} STREQUAL ${CMAKE_CURRENT_SOURCE_DIR}/debuginfo-tests))
+       (NOT ${entry} STREQUAL ${CMAKE_CURRENT_SOURCE_DIR}/cross-project-tests))
       get_filename_component(entry_name "${entry}" NAME)
       add_llvm_external_project(${entry_name})
     endif()
   endif()
 endforeach(entry)
 
 # Also add in libc++ and compiler-rt trees if present (and we have
 # a sufficiently recent version of CMake where required).
if(${LLVM_BUILD_RUNTIME})
   endif()
 endif()
 
 add_llvm_external_project(dragonegg)
 add_llvm_external_project(parallel-libs)
 add_llvm_external_project(openmp)
 
 if(LLVM_INCLUDE_TESTS)
-  add_llvm_external_project(debuginfo-tests)
+  add_llvm_external_project(cross-project-tests)
 endif()