[libc++][Modules] Generate the __std_clang_module header

[iana]

Summary

Use header_information to generate the __std_clang_module header. Instead of using lit_header_restrictions like the manually written header did, make a new header_include_requirements to codify what can be included rather than what can be fully tested.

[Mordante]

Thanks for working on this! I see several not related changes, for now let's keep them but land them separately.

I would like to see a slightly different design before reviewing in-depth.

libcxx/docs/Contributing.rst

55 ↗

(On Diff #548075)

Typically our generated files are stored in git and manually regenerated. This makes it easier to track changes.

The 'generated output` CI job regenerated the file and complains when a difference happens.

[iana]

libcxx/docs/Contributing.rst

55 ↗

(On Diff #548075)

Are you saying this should move to the libcxx-generate-files target? Or it should be a static file that we have to remember to manually regenerate when we add/remove public headers?

[iana]

In D157364#4570331, @Mordante wrote:

I see several not related changes, for now let's keep them but land them separately.

Sorry, which changes should be split?

[Mordante]

libcxx/docs/Contributing.rst

55 ↗

(On Diff #548075)

It should be generated by libcxx-generate-files and the output should be committed in the CI. Then when you forget to regenerate the file the CI will warn you. That way the file is available in the release tarball and we are sure it's kept up-to-date.

[iana]

libcxx/docs/Contributing.rst

55 ↗

(On Diff #548075)

Alright, I'll get that done, thanks.

[iana]

Move the generator script over to libcxx-generate-files

[iana]

Revert lint_headers.sh.py

[iana]

Run black over the Python script

[iana]

Rebase

[Mordante]

Thanks for working on this! In general happy with the changes, but I like to see the CI green before approving.

libcxx/test/libcxx/assertions/headers_declare_verbose_abort.gen.py

17 ↗

(On Diff #548798)

I would like to see this change and its related changes in a separate review.

libcxx/utils/CMakeLists.txt

7

libcxx/utils/generate_std_clang_module_header.py

2

Please add a copyright block.

29

Similar warnings are used in other generated headers.

48

The format script complains about the sorting, mainly hunks like

-#include <limits>
 #include <limits.h>
+#include <limits>

I suspect this is something what happens on Apple where the ordering of . and '>` differs. I know that @ldionne has seen this too. Would it help to force the C locale? This is something we use in other places where we sort data in the CI.

49–51

I feel this is a bit more readable, can you look at using f-strings at other places too.

libcxx/utils/libcxx/header_information.py

17–20

I would like to see these two changes committed separately. When the CI is green feel free to commit these two.

[iana]

libcxx/utils/CMakeLists.txt

7

But the one above says Generate the <version> header

[Mordante]

libcxx/utils/CMakeLists.txt

7

I missed that. Then let's keep it as is.

[iana]

libcxx/test/libcxx/assertions/headers_declare_verbose_abort.gen.py

17 ↗

(On Diff #548798)

https://reviews.llvm.org/D157639

libcxx/utils/libcxx/header_information.py

17–20

I put them in https://reviews.llvm.org/D157639, is that ok or should I make one more review?

[iana]

libcxx/utils/generate_std_clang_module_header.py

48

It's because it's only sorting the header names, i.e. it has "limits.h" and "limits", it doesn't have "<limits.h>" and "<limits>".

49–51

Sure

[iana]

Addressed review comments

[iana]

Ran black over the Python sources again

[iana]

Fix the extra newline at the end of the header

[iana]

libcxx/utils/libcxx/header_information.py

61

The module map requires and lit_header_restrictions looked like there were going to be headers that had multiple requirements, which is why the key is requirement tuples. But it turned out that all the headers only have one requirement. Should I keep the tuple keys and support for multiples in the script, or just make the keys strings?

[Mordante]

Thanks for the improvement, this really helps to reduce the maintenance when adding new headers. We don't do it often so more automation is really nice to have.

LGTM!

libcxx/utils/generate_std_clang_module_header.py

60

Great comment, thanks!

[ldionne]

libcxx/utils/libcxx/header_information.py

61

My preference would be to reinstate something like what we had before https://reviews.llvm.org/D151893.

[iana]

libcxx/utils/libcxx/header_information.py

61

Sorry for landing so quickly, I'll redo this one and make another PR.

libcxx/include/__std_clang_module

 // -*- C++ -*-
 //===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
+// WARNING, this entire header is generated by
+// utils/generate_std_clang_module_header.py
+// DO NOT MODIFY!
+
 // This header should not be directly included, it's exclusively to import all
 // of the libc++ public clang modules for the `std` clang module to export. In
 // other words, it's to facilitate `@import std;` in Objective-C++ and `import std`
 // in Swift to expose all of the libc++ interfaces. This is generally not
 // recommended, however there are some clients that need to import all of libc++
 // without knowing what "all" is.
 #if !__building_module(std)
 #  error "Do not include this header directly, include individual headers instead"
 #endif
 
-#include <__availability>
 #include <__config>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #  pragma GCC system_header
 #endif
 
 #include <algorithm>
 #include <any>
 #include <array>
-#include <atomic>
 #include <bit>
 #include <bitset>
+#include <cassert>
+#include <ccomplex>
+#include <cctype>
+#include <cerrno>
+#include <cfenv>
+#include <cfloat>
 #include <charconv>
 #include <chrono>
+#include <cinttypes>
+#include <ciso646>
+#include <climits>
+#include <cmath>
 #include <compare>
+#include <complex.h>
 #include <complex>
 #include <concepts>
 #include <condition_variable>
+#include <coroutine>
+#include <csetjmp>
+#include <csignal>
+#include <cstdarg>
+#include <cstdbool>
+#include <cstddef>
+#include <cstdint>
+#include <cstdio>
+#include <cstdlib>
+#include <cstring>
+#include <ctgmath>
+#include <ctime>
+#include <ctype.h>
+#include <cuchar>
 #include <deque>
+#include <errno.h>
 #include <exception>
 #include <execution>
 #include <expected>
+#include <experimental/deque>
+#include <experimental/forward_list>
+#include <experimental/iterator>
+#include <experimental/list>
+#include <experimental/map>
+#include <experimental/memory_resource>
+#include <experimental/propagate_const>
+#include <experimental/set>
+#include <experimental/simd>
+#include <experimental/string>
+#include <experimental/type_traits>
+#include <experimental/unordered_map>
+#include <experimental/unordered_set>
+#include <experimental/utility>
+#include <experimental/vector>
+#include <fenv.h>
+#include <filesystem>
+#include <float.h>
 #include <format>
 #include <forward_list>
 #include <functional>
 #include <initializer_list>
+#include <inttypes.h>
 #include <iosfwd>
 #include <iterator>
+#include <limits.h>
 #include <limits>
 #include <list>
 #include <map>
+#include <math.h>
 #include <mdspan>
 #include <memory>
 #include <memory_resource>
+#include <mutex>
 #include <new>
 #include <numbers>
 #include <numeric>
 #include <optional>
+#include <print>
 #include <queue>
 #include <random>
 #include <ranges>
 #include <ratio>
 #include <scoped_allocator>
 #include <set>
+#include <setjmp.h>
 #include <source_location>
 #include <span>
 #include <stack>
+#include <stdbool.h>
+#include <stddef.h>
 #include <stdexcept>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
 #include <string>
 #include <string_view>
 #include <system_error>
+#include <tgmath.h>
 #include <tuple>
 #include <type_traits>
 #include <typeindex>
 #include <typeinfo>
+#include <uchar.h>
 #include <unordered_map>
 #include <unordered_set>
 #include <utility>
 #include <valarray>
 #include <variant>
 #include <vector>
 #include <version>
 
-#include <cassert>
-#include <ccomplex>
-#include <cctype>
-#include <cerrno>
-#include <cfenv>
-#include <cfloat>
-#include <cinttypes>
-#include <ciso646>
-#include <climits>
-#include <cmath>
-#include <csetjmp>
-#include <csignal>
-#include <cstdarg>
-#include <cstdbool>
-#include <cstddef>
-#include <cstdint>
-#include <cstdio>
-#include <cstdlib>
-#include <cstring>
-#include <ctgmath>
-#include <ctime>
-#include <cuchar>
-
-#include <complex.h>
-#include <ctype.h>
-#include <errno.h>
-#include <fenv.h>
-#include <float.h>
-#include <inttypes.h>
-#include <limits.h>
-#include <math.h>
-#include <setjmp.h>
-#include <stdbool.h>
-#include <stddef.h>
-#include <stdint.h>
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-#include <tgmath.h>
-#include <uchar.h>
+#ifndef _LIBCPP_HAS_NO_ATOMIC_HEADER
+#  include <atomic>
+#  include <stdatomic.h>
+#endif
 
 #ifndef _LIBCPP_HAS_NO_LOCALIZATION
+#  include <clocale>
 #  include <codecvt>
-#  ifndef _LIBCPP_HAS_NO_FILESYSTEM
-#    include <fstream>
-#  endif
+#  include <experimental/regex>
+#  include <fstream>
 #  include <iomanip>
 #  include <ios>
 #  include <iostream>
 #  include <istream>
+#  include <locale.h>
 #  include <locale>
 #  include <ostream>
 #  include <regex>
 #  include <sstream>
 #  include <streambuf>
 #  include <strstream>
-
-#  include <clocale>
-
-#  include <locale.h>
-#endif
-
-#ifndef _LIBCPP_HAS_NO_WIDE_CHARACTERS
-#  include <cwchar>
-#  include <cwctype>
-
-#  include <wchar.h>
-#  include <wctype.h>
 #endif
 
-#ifdef _LIBCPP_AVAILABILITY_TO_CHARS_FLOATING_POINT
-#  include <print>
-#endif
-
-#ifndef _LIBCPP_CXX03_LANG
-#  ifndef _LIBCPP_HAS_NO_THREADS
-#    include <future>
-#    include <mutex>
-#    include <thread>
-#  endif
-
-#  include <experimental/deque>
-#  include <experimental/forward_list>
-#  include <experimental/iterator>
-#  include <experimental/list>
-#  include <experimental/map>
-#  include <experimental/memory_resource>
-#  include <experimental/propagate_const>
-#  ifndef _LIBCPP_HAS_NO_LOCALIZATION
-#    include <experimental/regex>
-#  endif
-#  include <experimental/set>
-#  include <experimental/simd>
-#  include <experimental/string>
-#  include <experimental/type_traits>
-#  include <experimental/unordered_map>
-#  include <experimental/unordered_set>
-#  include <experimental/utility>
-#  include <experimental/vector>
-#endif
-
-#if _LIBCPP_STD_VER >= 14
-#  ifndef _LIBCPP_HAS_NO_THREADS
-#    include <shared_mutex>
-#  endif
-#endif
-
-#if _LIBCPP_STD_VER >= 17
-#  ifndef _LIBCPP_HAS_NO_FILESYSTEM
-#    include <filesystem>
-#  endif
-#endif
-
-#if _LIBCPP_STD_VER >= 20
-#  include <coroutine>
-
-#  ifndef _LIBCPP_HAS_NO_THREADS
-#    include <barrier>
-#    include <latch>
-#    include <semaphore>
-#    include <stop_token>
-#  endif
+#ifndef _LIBCPP_HAS_NO_THREADS
+#  include <barrier>
+#  include <future>
+#  include <latch>
+#  include <semaphore>
+#  include <shared_mutex>
+#  include <stop_token>
+#  include <thread>
 #endif
 
-#if _LIBCPP_STD_VER >= 23
-#  ifndef _LIBCPP_HAS_NO_THREADS
-#    include <stdatomic.h>
-#  endif
+#ifndef _LIBCPP_HAS_NO_WIDE_CHARACTERS
+#  include <cwchar>
+#  include <cwctype>
+#  include <wchar.h>
+#  include <wctype.h>
 #endif

libcxx/test/libcxx/modules_include.gen.py

 # libc++ headers with modules enabled.
 
 # RUN: %{python} %s %{libcxx}/utils
 
 import sys
 sys.path.append(sys.argv[1])
 from libcxx.header_information import lit_header_restrictions, public_headers
 
-for header in public_headers:
-  BLOCKLIT = '' # block Lit from interpreting a RUN/XFAIL/etc inside the generation script
+
+for header in public_headers:
   print(f"""\
 //--- {header}.compile.pass.cpp
 // RUN{BLOCKLIT}: %{{cxx}} %s %{{flags}} %{{compile_flags}} -fmodules -fcxx-modules -fmodules-cache-path=%t -fsyntax-only
 
 // UNSUPPORTED{BLOCKLIT}: use_module_std
 
 // GCC doesn't support -fcxx-modules
 // UNSUPPORTED{BLOCKLIT}: gcc

 // TODO: Investigate this failure
 // UNSUPPORTED{BLOCKLIT}: LIBCXX-FREEBSD-FIXME
 
 {lit_header_restrictions.get(header, '')}
 
 #include <{header}>
 """)
 
-print(f"""
+print(f"""\
 //--- __std_clang_module.compile.pass.mm
 // RUN{BLOCKLIT}: %{{cxx}} %s %{{flags}} %{{compile_flags}} -fmodules -fcxx-modules -fmodules-cache-path=%t -fsyntax-only
 
 // REQUIRES{BLOCKLIT}: modules-build
 // UNSUPPORTED{BLOCKLIT}: use_module_std
 
 // GCC doesn't support -fcxx-modules
 // UNSUPPORTED{BLOCKLIT}: gcc
 
 // The Windows headers don't appear to be compatible with modules
 // UNSUPPORTED{BLOCKLIT}: windows
 // UNSUPPORTED{BLOCKLIT}: buildhost=windows
 
 // The AIX headers don't appear to be compatible with modules
 // UNSUPPORTED{BLOCKLIT}: LIBCXX-AIX-FIXME
 
 // The Android headers don't appear to be compatible with modules yet
 // XFAIL{BLOCKLIT}: LIBCXX-ANDROID-FIXME
 
 // TODO: Investigate this failure
 // UNSUPPORTED{BLOCKLIT}: LIBCXX-FREEBSD-FIXME
 
-// Lit seems to compile this twice: once with the default flags and once with with
-// the flags specified in the RUN directive. Guard the first compile from failing.
-#if __has_feature(modules)
 @import std;
-#endif
 
 """)

libcxx/utils/CMakeLists.txt

 add_custom_target(libcxx-generate-feature-test-macros
     COMMAND "${Python3_EXECUTABLE}" "${LIBCXX_SOURCE_DIR}/utils/generate_feature_test_macro_components.py"
     COMMENT "Generate the <version> header and tests for feature test macros.")
 
+add_custom_target(libcxx-generate-std-clang-module-header
+  COMMAND "${Python3_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/generate_std_clang_module_header.py"
+  COMMENT "Generate the <__std_clang_module> header")
+
 add_custom_target(libcxx-generate-extended-grapheme-cluster-tables
     COMMAND
         "${Python3_EXECUTABLE}"
         "${LIBCXX_SOURCE_DIR}/utils/generate_extended_grapheme_cluster_table.py"
         "${LIBCXX_SOURCE_DIR}/include/__format/extended_grapheme_cluster_table.h"
     COMMENT "Generate the extended grapheme cluster header.")
 
 add_custom_target(libcxx-generate-extended-grapheme-cluster-tests

 add_custom_target(libcxx-generate-iwyu-mapping
     COMMAND
         "${Python3_EXECUTABLE}"
         "${LIBCXX_SOURCE_DIR}/utils/generate_iwyu_mapping.py"
     COMMENT "Generate the mapping file for include-what-you-use")
 
 add_custom_target(libcxx-generate-files
     DEPENDS libcxx-generate-feature-test-macros
+            libcxx-generate-std-clang-module-header
             libcxx-generate-extended-grapheme-cluster-tables
             libcxx-generate-extended-grapheme-cluster-tests
             libcxx-generate-escaped-output-table
             libcxx-generate-width-estimation-table
             libcxx-generate-iwyu-mapping
     COMMENT "Create all the auto-generated files in libc++ and its tests.")

libcxx/utils/generate_std_clang_module_header.py

+# ===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+# ===----------------------------------------------------------------------===##
+
+import operator
+import os.path
+
+import libcxx.header_information
+
+public_headers = libcxx.header_information.public_headers
+header_include_requirements = libcxx.header_information.header_include_requirements
+always_available_headers = frozenset(public_headers).difference(
+    *header_include_requirements.values()
+)
+
+libcxx_include_directory = os.path.join(
+    os.path.dirname(os.path.dirname(os.path.realpath(__file__))), "include"
+)
+with open(
+    os.path.join(libcxx_include_directory, "__std_clang_module"), "w"
+) as std_clang_module_header:
+    std_clang_module_header.write(
+        """\
+// -*- C++ -*-
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// WARNING, this entire header is generated by
+// utils/generate_std_clang_module_header.py
+// DO NOT MODIFY!
+
+// This header should not be directly included, it's exclusively to import all
+// of the libc++ public clang modules for the `std` clang module to export. In
+// other words, it's to facilitate `@import std;` in Objective-C++ and `import std`
+// in Swift to expose all of the libc++ interfaces. This is generally not
+// recommended, however there are some clients that need to import all of libc++
+// without knowing what "all" is.
+#if !__building_module(std)
+#  error "Do not include this header directly, include individual headers instead"
+#endif
+
+#include <__config>
+
+#if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
+#  pragma GCC system_header
+#endif
+
+"""
+    )
+    # Include the angle brackets in sorting so that <a.h> sorts before <a>
+    # like check-format wants.
+    for include in sorted([f"<{header}>" for header in always_available_headers]):
+        std_clang_module_header.write(f"#include {include}\n")
+
+    for requirements, headers in sorted(
+        header_include_requirements.items(), key=operator.itemgetter(0)
+    ):
+        std_clang_module_header.write("\n")
+        if len(requirements) == 1:
+            std_clang_module_header.write("#ifndef ")
+            std_clang_module_header.write(requirements[0])
+        else:
+            std_clang_module_header.write("#if")
+            for index, requirement in enumerate(requirements):
+                if index > 0:
+                    std_clang_module_header.write(" &&")
+                std_clang_module_header.write(f" !defined({requirement})")
+        std_clang_module_header.write("\n")
+
+        for include in sorted([f"<{header}>" for header in headers]):
+            std_clang_module_header.write(f"#  include {include}\n")
+
+        std_clang_module_header.write("#endif\n")

libcxx/utils/libcxx/header_information.py

 # ===----------------------------------------------------------------------===##
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
 # ===----------------------------------------------------------------------===##
 
 import os, pathlib
 
 lit_header_restrictions = {
     "barrier": "// UNSUPPORTED: no-threads, c++03, c++11, c++14, c++17",
     "clocale": "// UNSUPPORTED: no-localization",
     "codecvt": "// UNSUPPORTED: no-localization",
     "coroutine": "// UNSUPPORTED: c++03, c++11, c++14, c++17",
     "cwchar": "// UNSUPPORTED: no-wide-characters",
     "cwctype": "// UNSUPPORTED: no-wide-characters",
     "experimental/deque": "// UNSUPPORTED: c++03",
     "experimental/forward_list": "// UNSUPPORTED: c++03",
     "experimental/iterator": "// UNSUPPORTED: c++03",
     "experimental/list": "// UNSUPPORTED: c++03",
     "experimental/map": "// UNSUPPORTED: c++03",
     "experimental/memory_resource": "// UNSUPPORTED: c++03",
     "experimental/propagate_const": "// UNSUPPORTED: c++03",
     "experimental/regex": "// UNSUPPORTED: no-localization, c++03",
     "experimental/set": "// UNSUPPORTED: c++03",
     "experimental/simd": "// UNSUPPORTED: c++03",
     "experimental/string": "// UNSUPPORTED: c++03",
lit_header_restrictions = {
     "stop_token": "// UNSUPPORTED: no-threads, c++03, c++11, c++14, c++17",
     "streambuf": "// UNSUPPORTED: no-localization",
     "strstream": "// UNSUPPORTED: no-localization",
     "thread": "// UNSUPPORTED: no-threads, c++03",
     "wchar.h": "// UNSUPPORTED: no-wide-characters",
     "wctype.h": "// UNSUPPORTED: no-wide-characters",
 }
 
+header_include_requirements = {
+    ("_LIBCPP_HAS_NO_ATOMIC_HEADER",): (
+        # headers with #error directives
+        "atomic",
+        # transitive includers of the above headers
+        "stdatomic.h",
+    ),
+    ("_LIBCPP_HAS_NO_LOCALIZATION",): (
+        # headers with #error directives
+        "ios",
+        "locale.h",
+        # transitive includers of the above headers
+        "clocale",
+        "codecvt",
+        "experimental/regex",
+        "fstream",
+        "iomanip",
+        "iostream",
+        "istream",
+        "locale",
+        "ostream",
+        "regex",
+        "sstream",
+        "streambuf",
+        "strstream",
+    ),
+    ("_LIBCPP_HAS_NO_THREADS",): (
+        # headers with #error directives
+        "barrier",
+        "future",
+        "latch",
+        "semaphore",
+        "shared_mutex",
+        "stop_token",
+        "thread",
+    ),
+    ("_LIBCPP_HAS_NO_WIDE_CHARACTERS",): (
+        # headers with #error directives
+        "wchar.h",
+        "wctype.h",
+        # transitive includers of the above headers
+        "cwchar",
+        "cwctype",
+    ),
+}
+
 # This table was produced manually, by grepping the TeX source of the Standard's
 # library clauses for the string "#include". Each header's synopsis contains
 # explicit "#include" directives for its mandatory inclusions.
 # For example, [algorithm.syn] contains "#include <initializer_list>".
 mandatory_inclusions = {
     "algorithm": ["initializer_list"],
     "array": ["compare", "initializer_list"],
     "bitset": ["iosfwd", "string"],