[libc++] Add documentation for reproducing CI issues locally

[Mordante]

Thanks for working on this! LGTM modulo one request.

libcxx/docs/TestingLibcxx.rst

115

run-buildbot in our CI often sets CC and CXX. Can you add some documentation explaining that these environment variable might need to be exported before calling run-buildbot?

[ldionne]

Address comments.

[jloser]

Thanks for writing this up for others' benefit! I remember when you first showed me this a long time ago, I was really thankful for helping track down a CI failure.

libcxx/docs/TestingLibcxx.rst

 Additional tools
 ----------------
 
 The libc++ test suite uses a few optional tools to improve the code quality.
 
 These tools are:
 - clang-tidy (you might need additional dev packages to compile libc++-specific clang-tidy checks)
 
+Reproducing CI issues locally
+-----------------------------
+
+Libc++ has extensive CI that tests various configurations of the library. The testing for
+all these configurations is located in ``libcxx/utils/ci/run-buildbot``. Most of our
+CI jobs are being run on a Docker image for reproducibility. The definition of this Docker
+image is located in ``libcxx/utils/ci/Dockerfile``. If you are looking to reproduce the
+failure of a specific CI job locally, you should first drop into a Docker container that
+matches our CI images by running ``libcxx/utils/ci/run-buildbot-container``, and then run
+the specific CI job that you're interested in (from within the container) using the ``run-buildbot``
+script above. If you want to control which compiler is used, you can set the ``CC`` and the
+``CXX`` environment variables before calling ``run-buildbot`` to select the right compiler.
+Take note that some CI jobs are testing the library on specific platforms and are *not* run
+in our Docker image. In the general case, it is not possible to reproduce these failures
+locally, unless they aren't specific to the platform.
+
+Also note that the Docker container shares the same filesystem as your local machine, so
+modifying files on your local machine will also modify what the Docker container sees.
+This is useful for editing source files as you're testing your code in the Docker container.
+
 Writing Tests
 -------------
 
 When writing tests for the libc++ test suite, you should follow a few guidelines.
 This will ensure that your tests can run on a wide variety of hardware and under
 a wide variety of configurations. We have several unusual configurations such as
 building the tests on one host but running them on a different host, which add a
 few requirements to the test suite. Here's some stuff you should know:

libcxx/utils/ci/run-buildbot

CXX                 The C++ compiler to use, this value is used by CMake. This
                     variable is optional.
 
 GIT_CLANG_FORMAT    The git-clang-format binary to use in the 'format' builder.
                     When the value is omitted, it defaults to
                     'git-clang-format'. This variable or its fallback must work
                     when running the 'format' builder.
 
 ENABLE_CLANG_TIDY   Whether to compile and run clang-tidy checks. This variable
-                    is optional
+                    is optional.
 EOF
 }
 
 if [[ $# == 0 ]]; then
    usage
    exit 0
 fi