[libc++] Allow specifying custom Lit config files

[ldionne]

Summary

Before this patch, the libc++ test suite first loads lit.site.cfg
(generated by CMake), and then lit.cfg. It's also possible to load
lit.cfg before lit.site.cfg and to point to a custom lit.site.cfg
file using '--param=libcxx_site_config'. However, in that case, lit.cfg
still relies on the site configuration filling up the 'config' object
like the default lit.site.cfg file does, which isn't flexible enough.

This commit simplifies the setup by having just a single Lit site config
file per CMake configuration, and always loading exactly that config file.
However, the config file to use can be selected when setting up CMake via
the LIBCXX_TEST_CONFIG setting. Furthermore, the site configs are entirely
standalone, which means that a new site config can be added that doesn't
need to conform what's expected by config.py.

[ldionne]

Adding reviewers whom I think might be impacted by this change.

[jwakely]

I don't think this impacts anything I do with lit (which isn't very much anyway).

[gargaroff]

Thanks for this patch. This definitely gets rid of one of the most annoying shortcomings we have at the moment, which is the need to run lit directly when we want to use a custom config.
So with this patch, if I want to use a custom config, I would simply copy-paste all the settings I care about from lit.site.cfg.in, add my own configuration logic and I'd be ready to go?

I would be okay with this as this solves my exact use-case, even though in a slightly different manner. I'm not a huge fan of having to copy-paste the config I'm interested in, because then I'd be concerned that the default config might change without me noticing. Would it be possible to load the default config from a custom config or is copy-pasting the stuff we're interested in the recommended way of setting up tests in this case?

By the way, I know this isn't directly related to this patch, but is there a minimum list of required configuration parameters for the new format?

[ldionne]

In D81846#2094828, @gargaroff wrote:

Thanks for this patch. This definitely gets rid of one of the most annoying shortcomings we have at the moment, which is the need to run lit directly when we want to use a custom config.
So with this patch, if I want to use a custom config, I would simply copy-paste all the settings I care about from lit.site.cfg.in, add my own configuration logic and I'd be ready to go?

Basically. However, with the new format, you don't really need that much boilerplate. Look at e.g. libcxx/test/configs/libcxx-trunk-shared.cfg.in in this patch: https://reviews.llvm.org/D81866.

I would be okay with this as this solves my exact use-case, even though in a slightly different manner. I'm not a huge fan of having to copy-paste the config I'm interested in, because then I'd be concerned that the default config might change without me noticing. Would it be possible to load the default config from a custom config or is copy-pasting the stuff we're interested in the recommended way of setting up tests in this case?

Ideally we wouldn't need to copy-paste a lot of stuff. It would help me to better understand your specific setup. How do you configure the test suite? Is there some code I can see somewhere?

By the way, I know this isn't directly related to this patch, but is there a minimum list of required configuration parameters for the new format?

It's documented in the new test format here: https://github.com/llvm/llvm-project/blob/master/libcxx/utils/libcxx/test/newformat.py#L139. Basically, you need to provide the following Lit substitutions and you're all done:

%{cxx}           - A command that can be used to invoke the compiler
%{compile_flags} - Flags to use when compiling a test case
%{link_flags}    - Flags to use when linking a test case
%{flags}         - Flags to use either when compiling or linking a test case
%{exec}          - A command to prefix the execution of executables

[STL_MSFT]

libcxx/test/lit.cfg.py

7–10

I think we started running lit directly as of https://github.com/microsoft/STL/pull/710 so this will affect us - but I am not yet deeply familiar with this area. (Not an objection to your change, thanks for the heads up!)

[gargaroff]

In D81846#2095851, @ldionne wrote:

Ideally we wouldn't need to copy-paste a lot of stuff. It would help me to better understand your specific setup. How do you configure the test suite? Is there some code I can see somewhere?

Unfortunately I can't show code (still need to convince management to open-source our backend 😄 ), but I should be able to go into just-enough details about our concrete use-case and challenges we are facing. Basically we have the following use-case: we have a downstream bare-metal target and we want to use the existing libcxx regression test suite to check whether our target-specific code (and obviously our compiler too) is working as intended. For that we need to cross-compile (managed to do that by creating a TargetInfo object) and remotely execute the tests on hardware. For executing a cross-compiled binary on hardware we have a script which does all of the heavy lifting for us.

So basically I simply need a minimum configuration which cross-compiles for my target and then uses our custom script to execute the cross-compiled binary on our hardware. We also have multiple architectures, but I can use the given target info object to determine which one and set compile and executions flags accordingly.

Since I didn't know which configuration parameters are actually required, I solved my use-case by implementing the ability to load an additional configuration, where I set up the substitutions and then it finally started working.

At this moment, our implementation doesn't support exception handling yet, so ideally I would like to disable all exception handling tests. Furthermore, I saw that there were some tests which rely on GNU extensions, which are not available in our C library. And also our C library has some quirks which makes it that we can't support some of the tests (e.g. include_as_c.sh.cpp or extern_c.pass.cpp) so we need to exclude all of those as well. I think that should be doable with a lit.local.cfg file which sets config.unsupported = True when running the tests for our target I guess? I'd rather not go through every single test and add a // UNSUPPORTED: ... to them.

Another challenge I'm facing at the moment is that we haven't implemented LLD support, so we are relying on an in-house linker, which we simply invoke from the driver. This linker is requires the libraries to be set in a certain order and with certain flags or otherwise we end up with undefined references. Is it problematic for the tests if I hard-code the linker flags, including libc++ and libc++abi in our configuration? I noticed that the test suite adds them to linker flags, but obviously not in a fashion that works for us.

Then, the final problem is that I have to disable warnings (config.enable_warnings = False) or otherwise the test suite adds a flag, which the compiler doesn't understand. I find it peculiar this happens as this flag is supposedly only added when it is supported by the compiler. So it would seem to me that either the wrong compiler is tested for this flag or that the check is not working. The flag in question is -Wno-aligned-allocation-unavailable and is added through this piece of code: self.cxx.addWarningFlagIfSupported('-Wno-aligned-allocation-unavailable').

By the way, I know this isn't directly related to this patch, but is there a minimum list of required configuration parameters for the new format?

It's documented in the new test format here: https://github.com/llvm/llvm-project/blob/master/libcxx/utils/libcxx/test/newformat.py#L139. Basically, you need to provide the following Lit substitutions and you're all done:

%{cxx}           - A command that can be used to invoke the compiler
%{compile_flags} - Flags to use when compiling a test case
%{link_flags}    - Flags to use when linking a test case
%{flags}         - Flags to use either when compiling or linking a test case
%{exec}          - A command to prefix the execution of executables

Oh, okay then! That is a lot less than I would have expected, given how much settings the old format required (or at least creates). In that case I really don't need the ability to load additional configurations along with the default one. When using a custom configuration, can the test suite still figure out to use the compiler that was used to build libc++ even if I don't add a substitution for %{cxx}? Or would I simply set it to "@LIBCXX_COMPILER@" and the CMake magic in this patch would fill in the right one?

[STL_MSFT]

libcxx/test/lit.cfg.py

7–10

We generate an stl-lit.py script so this shouldn't break microsoft/STL. Yay!

[ldionne]

In D81846#2097652, @gargaroff wrote:

In D81846#2095851, @ldionne wrote:

Ideally we wouldn't need to copy-paste a lot of stuff. It would help me to better understand your specific setup. How do you configure the test suite? Is there some code I can see somewhere?

Unfortunately I can't show code (still need to convince management to open-source our backend 😄 ), but I should be able to go into just-enough details about our concrete use-case and challenges we are facing. Basically we have the following use-case: we have a downstream bare-metal target and we want to use the existing libcxx regression test suite to check whether our target-specific code (and obviously our compiler too) is working as intended. For that we need to cross-compile (managed to do that by creating a TargetInfo object) and remotely execute the tests on hardware. For executing a cross-compiled binary on hardware we have a script which does all of the heavy lifting for us.

So basically I simply need a minimum configuration which cross-compiles for my target and then uses our custom script to execute the cross-compiled binary on our hardware. We also have multiple architectures, but I can use the given target info object to determine which one and set compile and executions flags accordingly.

Since I didn't know which configuration parameters are actually required, I solved my use-case by implementing the ability to load an additional configuration, where I set up the substitutions and then it finally started working.

Understood. The changes I made to the testing format were meant exactly to make this sort of use case easier, via the small number of configurations that are needed for the tests to run.

At this moment, our implementation doesn't support exception handling yet, so ideally I would like to disable all exception handling tests.

Does your compiler support -fno-exceptions? If so, just use --param enable_exceptions=False when running Lit and you're golden.

Furthermore, I saw that there were some tests which rely on GNU extensions, which are not available in our C library.

Which ones?

And also our C library has some quirks which makes it that we can't support some of the tests (e.g. include_as_c.sh.cpp or extern_c.pass.cpp) so we need to exclude all of those as well. I think that should be doable with a lit.local.cfg file which sets config.unsupported = True when running the tests for our target I guess? I'd rather not go through every single test and add a // UNSUPPORTED: ... to them.

I believe it might become useful to support specifying a set of UNSUPPORTED and/or XFAIL annotations externally to the test suite itself. This way, you could add additional annotations out of tree without risking merge conflicts or adding a lot of complexity. You'd have to maintain that out-of-tree list when we add new tests, but that's the case whichever way we decide to do it.

Another challenge I'm facing at the moment is that we haven't implemented LLD support, so we are relying on an in-house linker, which we simply invoke from the driver. This linker is requires the libraries to be set in a certain order and with certain flags or otherwise we end up with undefined references. Is it problematic for the tests if I hard-code the linker flags, including libc++ and libc++abi in our configuration? I noticed that the test suite adds them to linker flags, but obviously not in a fashion that works for us.

No, it shouldn't be an issue if you hardcode stuff. What's the problem you're seeing with adding your flags to %{link_flags}?

Then, the final problem is that I have to disable warnings (config.enable_warnings = False) or otherwise the test suite adds a flag, which the compiler doesn't understand. I find it peculiar this happens as this flag is supposedly only added when it is supported by the compiler. So it would seem to me that either the wrong compiler is tested for this flag or that the check is not working. The flag in question is -Wno-aligned-allocation-unavailable and is added through this piece of code: self.cxx.addWarningFlagIfSupported('-Wno-aligned-allocation-unavailable').

Your compiler doesn't support -Wno-aligned-allocation-unavailable? Does it fail when one tries to use that option?

By the way, I know this isn't directly related to this patch, but is there a minimum list of required configuration parameters for the new format?

It's documented in the new test format here: https://github.com/llvm/llvm-project/blob/master/libcxx/utils/libcxx/test/newformat.py#L139. Basically, you need to provide the following Lit substitutions and you're all done:

%{cxx}           - A command that can be used to invoke the compiler
%{compile_flags} - Flags to use when compiling a test case
%{link_flags}    - Flags to use when linking a test case
%{flags}         - Flags to use either when compiling or linking a test case
%{exec}          - A command to prefix the execution of executables

Oh, okay then! That is a lot less than I would have expected, given how much settings the old format required (or at least creates). In that case I really don't need the ability to load additional configurations along with the default one.

Nice.

When using a custom configuration, can the test suite still figure out to use the compiler that was used to build libc++ even if I don't add a substitution for %{cxx}? Or would I simply set it to "@LIBCXX_COMPILER@" and the CMake magic in this patch would fill in the right one?

A %{cxx} substitution needs to be provided to the test format. There's no magic. If you set %{cxx} to @CMAKE_CXX_COMPILER@, however, it'll get substituted at CMake generation time, like you would expect.

Since it looks like this is sufficient for your use case, I'll go forward with this patch.

[gargaroff]

In D81846#2100877, @ldionne wrote:

Does your compiler support -fno-exceptions? If so, just use --param enable_exceptions=False when running Lit and you're golden.

We have disabled exceptions through CMake (LIBCXX_ENABLE_EXCEPTIONS=OFF) which also sets config.enable_exceptions to False but we still see many many unresolved references errors (more than 500) for _Unwind_* functions (we don't provide libunwind yet). Also, at first we also disabled RTTI but still saw that all? most? just some select few? (didn't do a full investigation) of the RTTI tests were still running, so I was unsure whether the same was true for exception handling tests.

Furthermore, I saw that there were some tests which rely on GNU extensions, which are not available in our C library.

Which ones?

Sorry, I mixed up POSIX and GNU :)
Anyway, I see that some tests are using the debug_mode_helper.h which includes sys/wait.h. As far as I understand that's a POSIX header and is not available for our target (as well as all other POSIX-only headers).

Speaking of available features. There's another thing that just popped into my head: our C library uses the ISO-8859-X charmap and does not support UTF-8. Will we need to disable locale tests? When implementation our TargetInfo object I had to implement a function which tests whether some locales are supported and all of them where UTF-8 locales.

I believe it might become useful to support specifying a set of UNSUPPORTED and/or XFAIL annotations externally to the test suite itself. This way, you could add additional annotations out of tree without risking merge conflicts or adding a lot of complexity. You'd have to maintain that out-of-tree list when we add new tests, but that's the case whichever way we decide to do it.

Yes, that would indeed be a nice-to-have feature for us! I think it would make sense to have this as part of the config. I'm thinking something like config.unsupported_tests = [] and config.xfail_tests = [] which would take a list of paths (maybe even globs? or regex?). And since it would be part of the config, you could even make this dynamic.

No, it shouldn't be an issue if you hardcode stuff. What's the problem you're seeing with adding your flags to %{link_flags}?

No no, we didn't have any problem with adding it to %{link_flags}. We just wondered if it *would* be problematic for some tests if we did.

Your compiler doesn't support -Wno-aligned-allocation-unavailable? Does it fail when one tries to use that option?

Yes, clang complains about it being an unrecognized flag and exits right away.
By the way, setting config.enable_warnings = False has the nice side-effect that libcxx/selftest/newformat/pass.cpp/werror.pass.cpp and libcxx/selftest/newformat/sh.cpp/werror.sh.cpp are unexpectedly passing because -Werror is not passed as a flag anymore.

Since it looks like this is sufficient for your use case, I'll go forward with this patch.

Looking forward to using it! Thanks for the effort.

P.S. Since this is now quickly going to off-topic from this patch, we should probably move the discussion somewhere else again.

[mstorsjo]

Is there any docs on the canonical way of excuting the libcxx testsuite against an installed compiler and its C++ standard library (without compiling libc++ at all)? Before this change, I had a custom lit.site.cfg for that, and just ran lit.py with LIBCXX_SITE_CONFIG set pointing at the config. My current workaround is a small custom script similar to the generated bin/llvm-lit, that bootstraps things - which also seems to be what @STL_MSFT pointed out that they do in their repo. It'd be great to have that usecase somewhat documented though, as I'm afraid the setup will break soon again as long as it's a non-documented setup.

libcxx/CMakeLists.txt

 if (WIN32)
   set(ENABLE_FILESYSTEM_DEFAULT OFF)
 endif()
 option(LIBCXX_ENABLE_FILESYSTEM "Build filesystem as part of the main libc++ library"
     ${ENABLE_FILESYSTEM_DEFAULT})
 option(LIBCXX_INCLUDE_TESTS "Build the libc++ tests." ${LLVM_INCLUDE_TESTS})
 option(LIBCXX_ENABLE_PARALLEL_ALGORITHMS "Enable the parallel algorithms library. This requires the PSTL to be available." OFF)
 option(LIBCXX_TEST_GDB_PRETTY_PRINTERS "Test gdb pretty printers." OFF)
+set(LIBCXX_TEST_CONFIG "${CMAKE_CURRENT_SOURCE_DIR}/test/lit.site.cfg.in" CACHE STRING
+    "The Lit testing configuration to use when running the tests")
 
 # Benchmark options -----------------------------------------------------------
 option(LIBCXX_INCLUDE_BENCHMARKS "Build the libc++ benchmarks and their dependencies" ON)
 
 set(LIBCXX_BENCHMARK_TEST_ARGS_DEFAULT --benchmark_min_time=0.01)
 set(LIBCXX_BENCHMARK_TEST_ARGS "${LIBCXX_BENCHMARK_TEST_ARGS_DEFAULT}" CACHE STRING
     "Arguments to pass when running the benchmarks using check-cxx-benchmarks")
 

libcxx/docs/TestingLibcxx.rst

 Using a custom site configuration
 ---------------------------------
 
 By default, the libc++ test suite will use a site configuration that matches
 the current CMake configuration. It does so by generating a ``lit.site.cfg``
 file in the build directory from the ``libcxx/test/lit.site.cfg.in`` template,
 and pointing ``llvm-lit`` (which is a wrapper around ``llvm/utils/lit/lit.py``)
 to that file. So when you're running ``<build>/bin/llvm-lit``, the generated
-``lit.site.cfg`` file is always loaded first, followed by the actual config in
-``libcxx/test/lit.cfg``. However, it is sometimes desirable to use a custom
-site configuration. To do that, you can use ``--param=libcxx_site_config`` or
-the ``LIBCXX_SITE_CONFIG`` environment variable to point to the right site
-configuration file. However, you must stop using ``llvm-lit``, or else the
-generated ``lit.site.cfg`` will still be preferred:
+``lit.site.cfg`` file is always loaded instead of ``libcxx/test/lit.cfg.py``.
+If you want to use a custom site configuration, simply point the CMake build
+to it using ``-DLIBCXX_TEST_CONFIG=<path-to-site-config>``, and that site
+configuration will be used instead. That file can use CMake variables inside
+itself to make configuration easier.
 
    .. code-block:: bash
 
-     $ LIBCXX_SITE_CONFIG=path/to/your/site/configuration llvm/utils/lit/lit.py -sv ...
+     $ cmake <options> -DLIBCXX_TEST_CONFIG=<path-to-site-config>
+     $ make -C <build> check-cxx-deps
+     $ <build>/bin/llvm-lit -sv libcxx/test # will use your custom config file
 
-     $ llvm/utils/lit/lit.py -sv ... --param=libcxx_site_config=path/to/your/site/configuration
-
-In both of these cases, your custom site configuration should set up the
-``config`` object in a way that is compatible with what libc++'s ``config.py``
-module expects.
 
 LIT Options
 ===========
 
 :program:`lit` [*options*...] [*filenames*...]
 
 Command Line Options
 --------------------
.. option:: cxx_stdlib_under_test=<stdlib name>
   suite against other standard library implementations.
 
 .. option:: std=<standard version>
 
   **Values**: c++03, c++11, c++14, c++17, c++2a
 
   Change the standard version used when building the tests.
 
-.. option:: libcxx_site_config=<path/to/lit.site.cfg>
-
-  Specify the site configuration to use when running the tests.  This option
-  overrides the environment variable LIBCXX_SITE_CONFIG.
-
 .. option:: cxx_headers=<path/to/headers>
 
   Specify the c++ standard library headers that are tested. By default the
   headers in the source tree are used.
 
 .. option:: cxx_library_root=<path/to/lib/>
 
   Specify the directory of the libc++ library to be tested. By default the
.. option:: llvm_unwinder
 
   Enable the use of LLVM unwinder instead of libgcc.
 
 .. option:: builtins_library
 
   Path to the builtins library to use instead of libgcc.
 
 
-Environment Variables
----------------------
-
-.. envvar:: LIBCXX_SITE_CONFIG=<path/to/lit.site.cfg>
-
-  Specify the site configuration to use when running the tests.
-  Also see `libcxx_site_config`.
-
 Writing Tests
 -------------
 
 When writing tests for the libc++ test suite, you should follow a few guidelines.
 This will ensure that your tests can run on a wide variety of hardware and under
 a wide variety of configurations. We have several unusual configurations such as
 building the tests on one host but running them on a different host, which add a
 few requirements to the test suite. Here's some stuff you should know:

libcxx/test/CMakeLists.txt

if (LIBCXX_GDB)
     set(LIBCXX_GDB "${LIBCXX_GDB}")
     message(STATUS "gdb found: ${LIBCXX_GDB}")
   else()
     message(STATUS "gdb not found. Disabling dependent tests.")
   endif()
 endif()
 
 if (LIBCXX_INCLUDE_TESTS)
-  include(AddLLVM) # for configure_lit_site_cfg and add_lit_testsuit
+  include(AddLLVM) # for configure_lit_site_cfg and add_lit_testsuite
 
   configure_lit_site_cfg(
-    ${CMAKE_CURRENT_SOURCE_DIR}/lit.site.cfg.in
-    ${CMAKE_CURRENT_BINARY_DIR}/lit.site.cfg)
+    "${LIBCXX_TEST_CONFIG}"
+    ${CMAKE_CURRENT_BINARY_DIR}/lit.site.cfg
+    MAIN_CONFIG "${CMAKE_CURRENT_SOURCE_DIR}/lit.cfg.py")
 
   add_custom_target(check-cxx-deps
     DEPENDS cxx ${LIBCXX_TEST_DEPS}
     COMMENT "Builds dependencies required to run the test suite.")
 
   add_lit_testsuite(check-cxx
     "Running libcxx tests"
     ${CMAKE_CURRENT_BINARY_DIR}

libcxx/test/lit.cfg

-# -*- Python -*- vim: set ft=python ts=4 sw=4 expandtab tw=79:
-# Configuration file for the 'lit' test runner.
-import os
-import site
-
-site.addsitedir(os.path.join(os.path.dirname(os.path.dirname(__file__)), 'utils'))
-
-
-# Tell pylint that we know config and lit_config exist somewhere.
-if 'PYLINT_IMPORT' in os.environ:
-    config = object()
-    lit_config = object()
-
-# name: The name of this test suite.
-config.name = 'libc++'
-
-# suffixes: A list of file extensions to treat as test files.
-config.suffixes = ['.pass.cpp', '.fail.cpp', '.sh.cpp', '.pass.mm']
-
-# test_source_root: The root path where tests are located.
-config.test_source_root = os.path.dirname(__file__)
-
-# Allow expanding substitutions that are based on other substitutions
-config.recursiveExpansionLimit = 10
-
-loaded_site_cfg = getattr(config, 'loaded_site_config', False)
-if not loaded_site_cfg:
-    import libcxx.test.config
-    libcxx.test.config.loadSiteConfig(lit_config, config, 'libcxx_site_config',
-                                      'LIBCXX_SITE_CONFIG')
-
-# Infer the test_exec_root from the libcxx_object root.
-obj_root = getattr(config, 'libcxx_obj_root', None)
-
-# Check that the test exec root is known.
-if obj_root is None:
-    obj_root = getattr(config, 'libcxx_obj_root', None)
-    if obj_root is None:
-        import tempfile
-        obj_root = tempfile.mkdtemp(prefix='libcxx-testsuite-')
-        lit_config.warning('Creating temporary directory for object root: %s' %
-                           obj_root)
-
-if not config.test_exec_root:
-    config.test_exec_root = os.path.join(obj_root, 'test')
-
-cfg_variant = getattr(config, 'configuration_variant', 'libcxx')
-if cfg_variant:
-    lit_config.note('Using configuration variant: %s' % cfg_variant)
-
-# Load the Configuration class from the module name <cfg_variant>.test.config.
-config_module_name = '.'.join([cfg_variant, 'test', 'config'])
-config_module = __import__(config_module_name, fromlist=['Configuration'])
-
-configuration = config_module.Configuration(lit_config, config)
-configuration.configure()
-configuration.print_config_info()
-if lit_config.params.get('use_old_format', False):
-    lit_config.note("Using the old libc++ testing format")
-    config.test_format = configuration.get_test_format()
-else:
-    lit_config.note("Using the new libc++ testing format")
-    import libcxx.test.newformat
-    config.test_format = libcxx.test.newformat.CxxStandardLibraryTest()

libcxx/test/lit.cfg.py

+# All the Lit configuration is handled in the site configs -- this file is only
+# left as a canary to catch invocations of Lit that do not go through llvm-lit.
+#
+# Invocations that go through llvm-lit will automatically use the right Lit
+# site configuration inside the build directory.
+
+lit_config.fatal(
+    "You seem to be running Lit directly -- you should be running Lit through "
+    "<build>/bin/llvm-lit, which will ensure that the right Lit configuration "
+    "file is used.")

libcxx/test/lit.site.cfg.in

 @AUTO_GEN_COMMENT@
+
+import os
+import site
+
 config.cxx_under_test           = "@LIBCXX_COMPILER@"
 config.project_obj_root         = "@CMAKE_BINARY_DIR@"
 config.libcxx_src_root          = "@LIBCXX_SOURCE_DIR@"
 config.libcxx_obj_root          = "@LIBCXX_BINARY_DIR@"
 config.cxx_library_root         = "@LIBCXX_LIBRARY_DIR@"
 config.enable_exceptions        = @LIBCXX_ENABLE_EXCEPTIONS@
 config.enable_experimental      = @LIBCXX_ENABLE_EXPERIMENTAL_LIBRARY@
 config.enable_filesystem        = @LIBCXX_ENABLE_FILESYSTEM@

 config.cxx_ext_threads          = @LIBCXX_BUILD_EXTERNAL_THREAD_LIBRARY@
 config.pstl_src_root            = "@ParallelSTL_SOURCE_DIR@" if @LIBCXX_ENABLE_PARALLEL_ALGORITHMS@ else None
 config.pstl_obj_root            = "@ParallelSTL_BINARY_DIR@" if @LIBCXX_ENABLE_PARALLEL_ALGORITHMS@ else None
 config.libcxx_gdb               = "@LIBCXX_GDB@"
 
 # Code signing
 config.llvm_codesign_identity   = "@LLVM_CODESIGNING_IDENTITY@"
 
-# Let the main config do the real work.
-config.loaded_site_config = True
-lit_config.load_config(config, "@LIBCXX_SOURCE_DIR@/test/lit.cfg")
+site.addsitedir(os.path.join(config.libcxx_src_root, 'utils'))
+
+# name: The name of this test suite.
+config.name = 'libc++'
+
+# suffixes: A list of file extensions to treat as test files.
+config.suffixes = ['.pass.cpp', '.fail.cpp', '.sh.cpp', '.pass.mm']
+
+# test_source_root: The root path where tests are located.
+config.test_source_root = os.path.join(config.libcxx_src_root, 'test')
+
+# Allow expanding substitutions that are based on other substitutions
+config.recursiveExpansionLimit = 10
+
+# Infer the test_exec_root from the libcxx_object root.
+config.test_exec_root = os.path.join(config.libcxx_obj_root, 'test')
+
+lit_config.note('Using configuration variant: {}'.format(config.configuration_variant))
+import libcxx.test.config
+configuration = libcxx.test.config.Configuration(lit_config, config)
+configuration.configure()
+configuration.print_config_info()
+if lit_config.params.get('use_old_format', False):
+    lit_config.note("Using the old libc++ testing format")
+    config.test_format = configuration.get_test_format()
+else:
+    lit_config.note("Using the new libc++ testing format")
+    import libcxx.test.newformat
+    config.test_format = libcxx.test.newformat.CxxStandardLibraryTest()