[libc++] Add helper script libcxx-lit for running tests

[ldionne]

libcxx/utils/libcxx-lit

37

Does realpath work on Linux?

[jloser]

Thanks for adding this for others. This is the same script I use locally as well that I got from you ages ago.

libcxx/utils/libcxx-lit

37

It should be just fine per https://man7.org/linux/man-pages/man3/realpath.3.html

[arichardson]

libcxx/utils/libcxx-lit

37

I believe realpath should be available on most systems but may not always be installed by default. Also I seem to recall it not being there on macos? But is this actually needed? Shouldn't both cmake and lit be fine with a relative path?

[ldionne]

Drop realpath

libcxx/docs/TestingLibcxx.rst

.. note::
   ``cxx-test-depends`` target is instead named ``runtimes-test-depends``, and
   you will need to prefix ``<build>/runtimes/runtimes-<target>-bins/`` to the
   paths of all tests.
 
 In the default configuration, the tests are built against headers that form a
 fake installation root of libc++. This installation root has to be updated when
 changes are made to the headers, so you should re-run the ``cxx-test-depends``
 target before running the tests manually with ``lit`` when you make any sort of
-change, including to the headers.
+change, including to the headers. We recommend using the provided ``libcxx/utils/libcxx-lit``
+script to automate this so you don't have to think about building test dependencies
+every time:
+
+.. code-block:: bash
+
+  $ cd <monorepo-root>
+  $ libcxx/utils/libcxx-lit <build> -sv libcxx/test/std/re # Build testing dependencies and run all of the std::regex tests
 
 Sometimes you'll want to change the way LIT is running the tests. Custom options
 can be specified using the ``--param <name>=<val>`` flag. The most common option
 you'll want to change is the standard dialect (ie ``-std=c++XX``). By default the
 test suite will select the newest C++ dialect supported by the compiler and use
 that. However, you can manually specify the option like so if you want:
 
 .. code-block:: bash
 
-  $ <build>/bin/llvm-lit -sv libcxx/test/std/containers # Run the tests with the newest -std
-  $ <build>/bin/llvm-lit -sv libcxx/test/std/containers --param std=c++03 # Run the tests in C++03
+  $ libcxx/utils/libcxx-lit <build> -sv libcxx/test/std/containers # Run the tests with the newest -std
+  $ libcxx/utils/libcxx-lit <build> -sv libcxx/test/std/containers --param std=c++03 # Run the tests in C++03
 
 Other parameters are supported by the test suite. Those are defined in ``libcxx/utils/libcxx/test/params.py``.
 If you want to customize how to run the libc++ test suite beyond what is available
 in ``params.py``, you most likely want to use a custom site configuration instead.
 
 The libc++ test suite works by loading a site configuration that defines various
 "base" parameters (via Lit substitutions). These base parameters represent things
 like the compiler to use for running the tests, which default compiler and linker
 flags to use, and how to run an executable. This system is meant to be easily
 extended for custom needs, in particular when porting the libc++ test suite to
 new platforms.
 
 Using a Custom Site Configuration
 ---------------------------------
 
 By default, the libc++ test suite will use a site configuration that matches
 the current CMake configuration. It does so by generating a ``lit.site.cfg``
 file in the build directory from one of the configuration file templates in
 ``libcxx/test/configs/``, and pointing ``llvm-lit`` (which is a wrapper around
 ``llvm/utils/lit/lit.py``) to that file. So when you're running
-``<build>/bin/llvm-lit``, the generated ``lit.site.cfg`` file is always loaded
-instead of ``libcxx/test/lit.cfg.py``. If you want to use a custom site
-configuration, simply point the CMake build to it using
+``<build>/bin/llvm-lit`` either directly or indirectly, the generated ``lit.site.cfg``
+file is always loaded instead of ``libcxx/test/lit.cfg.py``. If you want to use a
+custom site configuration, simply point the CMake build to it using
 ``-DLIBCXX_TEST_CONFIG=<path-to-site-config>``, and that site configuration
 will be used instead. That file can use CMake variables inside it to make
 configuration easier.
 
    .. code-block:: bash
 
      $ cmake <options> -DLIBCXX_TEST_CONFIG=<path-to-site-config>
-     $ make -C <build> cxx-test-depends
-     $ <build>/bin/llvm-lit -sv libcxx/test # will use your custom config file
+     $ libcxx/utils/libcxx-lit <build> -sv libcxx/test # will use your custom config file
 
 Additional tools
 ----------------
 
 The libc++ test suite uses a few optional tools to improve the code quality.
 
 These tools are:
 - clang-tidy (you might need additional dev packages to compile libc++-specific clang-tidy checks)

libcxx/utils/libcxx-lit

+#!/usr/bin/env bash
+
+set -e
+
+PROGNAME="$(basename "${0}")"
+function usage() {
+cat <<EOF
+Usage:
+${PROGNAME} [-h|--help] <build-directory> [lit options...] tests...
+
+Shortcut to build the libc++ testing dependencies and run the libc++ tests with Lit.
+
+<build-directory>  The path to the build directory to use for building the library.
+[lit options...]   Optional options to pass to 'llvm-lit'.
+tests...           Paths of the tests to run. Those are paths relative to '<monorepo-root>/libcxx/test'.
+
+Example
+=======
+$ cmake -S runtimes -B build/ -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi"
+$ libcxx-lit build/ -sv libcxx/test/std/utilities/
+EOF
+}
+
+for arg in $@; do
+    if [[ "${arg}" == "-h" || "${arg}" == "--help" ]]; then
+        usage
+        exit 0
+    fi
+done
+
+if [[ $# -lt 1 ]]; then
+    usage
+    exit 1
+fi
+
+build_dir="${1}"
+shift
+
+cmake --build "${build_dir}" --target cxx-test-depends
+"${build_dir}/bin/llvm-lit" ${@}