[libc++] Add a CMake target to re-generate files and revamp CONTRIBUTING.rst

[ldionne]

Summary

As we automate more and more things in the library, it becomes useful for
contributors to have a single target for running all the automation as
part of their workflow. This commit adds a new libcxx-generate-files
target that should re-generate all the auto-generated files in the library.

As a fly-by, I also revamped the documentation on Contributing to account
for this new target and present it as a bullet list of things to check
before committing. I also added a few things that are often overlooked
to that list, such as updating the synopsis and the status files.

[ldionne]

@cjdb I saw the script you added in D105932, saw it was missing from the documentation, and told myself: we need a single source of truth for running all the scripts now. This is it. Once this ships, can you rebase D105932 on top?

[Mordante]

Nice improvement. I noticed you already committed it after I started reviewing. But LGTM, just a few nits.

libcxx/docs/Contributing.rst

14

I think it would be nice to refer new contributors to the libcxx Discord channel.

41

Maybe a linebreak after the first ?

[• Quuxplusone]

libcxx/utils/CMakeLists.txt

18

I'd strongly prefer "one name for one thing": name the targets consistently with what scripts they run.
So:
libcxx-generate-header-inclusion-tests
libcxx-generate-header-tests
libcxx-generate-feature-test-macro-components

Otherwise LGTM.

libcxx/CMakeLists.txt

function(cxx_add_common_build_flags target)
   cxx_link_system_libraries(${target})
 endfunction()
 
 #===============================================================================
 # Setup Source Code And Tests
 #===============================================================================
 add_subdirectory(include)
 add_subdirectory(src)
+add_subdirectory(utils)
 
 set(LIBCXX_TEST_DEPS "")
 
 if (LIBCXX_ENABLE_EXPERIMENTAL_LIBRARY)
   list(APPEND LIBCXX_TEST_DEPS cxx_experimental)
 endif()
 
 if (LIBCXX_BUILD_EXTERNAL_THREAD_LIBRARY)

libcxx/docs/Contributing.rst

 .. _ContributingToLibcxx:
 
 ======================
 Contributing to libc++
 ======================
 
-.. contents::
-  :local:
-
-Please read `this document <https://www.llvm.org/docs/Contributing.html>`__ on general rules to contribute to LLVM projects.
-
-Tasks and processes
-===================
-
-This file contains notes about various tasks and processes specific to libc++.
+This file contains notes about various tasks and processes specific to contributing
+to libc++. If this is your first time contributing, please also read `this document
+<https://www.llvm.org/docs/Contributing.html>`__ on general rules for contributing to LLVM.
+
+For libc++, please make sure you follow `these instructions <https://www.llvm.org/docs/Phabricator.html#requesting-a-review-via-the-command-line>`_
+for submitting a code review from the command-line using ``arc``, since we have some
+automation (e.g. CI) that depends on the review being submitted that way.
 
 Looking for pre-existing reviews
 ================================
 
 Before you start working on any feature, please take a look at the open reviews
 to avoid duplicating someone else's work. You can do that by going to the website
 where code reviews are held, `Differential <https://reviews.llvm.org/differential>`__,
 and clicking on ``Libc++ Open Reviews`` in the sidebar to the left. If you see
 that your feature is already being worked on, please consider chiming in instead
 of duplicating work!
 
-Post-Release TODO
-=================
+Pre-commit check list
+=====================
+
+Before committing or creating a review, please go through this check-list to make
+sure you don't forget anything:
+
+- Do you have tests for every public class and/or function you're adding or modifying?
+- Did you update the synopsis of the relevant headers?
+- Did you update the relevant files to track implementation status (in ``docs/Status/``)?
+- If you added a header:
+
+  - Did you add it to ``include/module.modulemap``?
+  - Did you add it to ``include/CMakeLists.txt``?
+  - If it's a public header, did you add a test under ``test/libcxx`` that the new header defines ``_LIBCPP_VERSION``? See ``test/libcxx/algorithms/version.pass.cpp`` for an example. NOTE: This should be automated.
+  - If it's a public header, did you update ``utils/generate_header_inclusion_tests.py``?
+
+- Did you add the relevant feature test macro(s) for your feature? Did you update the ``generate_feature_test_macro_components.py`` script with it?
+- Did you run the ``libcxx-generate-files`` target and verify its output?
+
+Post-release check list
+=======================
 
 After branching for an LLVM release:
 
 1. Update ``_LIBCPP_VERSION`` in ``include/__config``
 2. Update the ``include/__libcpp_version`` file
 3. Update the version number in ``docs/conf.py``
 
-Modifying feature-test macros
-=============================
-
-When adding or updating feature-test macros, you should update the corresponding tests.
-To do that, modify ``feature_test_macros`` table in the script
-``utils/generate_feature_test_macro_components.py``, run it, and commit updated
-files. Running ``utils/generate_feature_test_macro_components.py`` should never
-generate diffs in a clean checkout; feel free to run it in your local checkout
-any time you want.
-
-
-Adding a new header TODO
-========================
-
-When adding a new header to libc++:
-
-1. Add a test under ``test/libcxx`` that the new header defines ``_LIBCPP_VERSION``. See ``test/libcxx/algorithms/version.pass.cpp`` for an example.
-2. Run ``python utils/generate_header_tests.py``; verify and commit the changes.
-3. Modify ``python utils/generate_header_inclusion_tests.py``; run it; verify and commit the changes.
-4. Create a submodule in ``include/module.modulemap`` for the new header.
-5. Update the ``include/CMakeLists.txt`` file to include the new header.
-
 Exporting new symbols from the library
 ======================================
 
 When exporting new symbols from libc++, you must update the ABI lists located in ``lib/abi``.
 To test whether the lists are up-to-date, please run the target ``check-cxx-abilist``.
 To regenerate the lists, use the target ``generate-cxx-abilist``.
 The ABI lists must be updated for all supported platforms; currently Linux and
 Apple.  If you don't have access to one of these platforms, you can download an
 updated list from the failed build at
 `Buildkite <https://buildkite.com/llvm-project/libcxx-ci>`__.
 Look for the failed build and select the ``artifacts`` tab. There, download the
 abilist for the platform, e.g.:
 
 * C++20 for the Linux platform.
 * MacOS C++20 for the Apple platform.

libcxx/utils/CMakeLists.txt

+
+add_custom_target(libcxx-generate-public-header-transitive-inclusion-tests
+    COMMAND "${Python3_EXECUTABLE}" "${LIBCXX_SOURCE_DIR}/utils/generate_header_inclusion_tests.py"
+    COMMENT "Generate tests checking for mandated transitive includes in public headers.")
+
+add_custom_target(libcxx-generate-public-header-tests
+    COMMAND "${Python3_EXECUTABLE}" "${LIBCXX_SOURCE_DIR}/utils/generate_header_tests.py"
+    COMMENT "Generate tests for including public headers.")
+
+add_custom_target(libcxx-generate-feature-test-macros
+    COMMAND "${Python3_EXECUTABLE}" "${LIBCXX_SOURCE_DIR}/utils/generate_feature_test_macro_components.py"
+    COMMENT "Generate the <version> header and tests for feature test macros.")
+
+add_custom_target(libcxx-generate-files
+    DEPENDS libcxx-generate-public-header-transitive-inclusion-tests
+            libcxx-generate-public-header-tests
+            libcxx-generate-feature-test-macros
+    COMMENT "Create all the auto-generated files in libc++ and its tests.")

libcxx/utils/ci/run-buildbot

git-clang-format \
         | tee ${BUILD_DIR}/clang-format.patch
     # Check if the diff is empty, fail otherwise.
     ! grep -q '^--- a' ${BUILD_DIR}/clang-format.patch
 ;;
 check-generated-output)
     # `! foo` doesn't work properly with `set -e`, use `! foo || false` instead.
     # https://stackoverflow.com/questions/57681955/set-e-does-not-respect-logical-not
     clean
-    echo "+++ Checking the output of the generator scripts"
-    mkdir -p ${BUILD_DIR}
-    # Reject patches that don't update the generated output correctly.
-    python3 libcxx/utils/generate_feature_test_macro_components.py
-    python3 libcxx/utils/generate_header_inclusion_tests.py
-    python3 libcxx/utils/generate_header_tests.py
+    generate-cmake
+
+    # Reject patches that forgot to re-run the generator scripts.
+    echo "+++ Making sure the generator scripts were run"
+    ${NINJA} -vC "${BUILD_DIR}" libcxx-generate-files
     git diff | tee ${BUILD_DIR}/generated_output.patch
-    # Check if the diffs are empty, fail otherwise.
     ! grep -q '^--- a' ${BUILD_DIR}/generated_output.patch || false
+
     # Reject patches that introduce non-ASCII characters or hard tabs.
     # Depends on LC_COLLATE set at the top of this script.
     ! grep -rn '[^ -~]' libcxx/include/ || false
-    # Check that no dependency cycles have been introduced.
+
+    # Reject patches that introduce dependency cycles in the headers.
     python3 libcxx/utils/graph_header_deps.py >/dev/null
 ;;
 generic-cxx03)
     # TODO: Switch to compiling with Clang ToT once issues with https://reviews.llvm.org/D104500 are resolved.
     export CC=clang-12
     export CXX=clang++-12
     clean
     generate-cmake -C "${MONOREPO_ROOT}/libcxx/cmake/caches/Generic-cxx03.cmake" \