[Draft][libc++][modules] Adds std module.

[Mordante]

Summary

Implements parts of

• P2465R3 Standard Library Modules std and std.compat

This patch is based on work by @ChuanqiXu.

This patch not intended to be landed is this form, it's intended to get
a feeling of modules in libc++ and improve upon it. This patch will be
used to discuss the further approach regarding modules in libc++.

[ChuanqiXu]

libcxx/docs/Modules.rst

43

Yeah, the <build-dir> is tied to specific machine. But why is it a problem? From my mind, we will distribute the *.cppm files like the header files. Then the users' build system will generate the P1689 output according to the *.cppm files in their machine. I think the key point here is that the P1689 output is a by-product of the building process and we (users) don't need to care about it.

[aaronmondal]

libcxx/docs/Modules.rst

43

Hmm P1689 states that these files assume

uniformity of the environment between creation and usage; only used within one build of a project

AFAIU the P1689 output is not supposed to be distributed to users. It's supposed to be generated and consumed during the same build.

[ChuanqiXu]

About distributing the std modules, this is what we do in the downstream:

# std-module.spec

...

Requires: clang_xxx # for example clang17

%prep
%build
%install

mkdir -p <path-to-the-install-destination-of-std-module>
cp `.cppm` files to  <path-to-the-install-destination-of-std-module>
cp ../../../build.sh <path-to-the-install-destination-of-std-module>

...

%post
<path-to-the-install-destination-of-std-module>/build.sh

The commands line after %post will execute the corresponding instructions when the user input yum install std-modules and the build.sh contains the instructions to compile the std modules.

Then in our downstream, we will hardcode the <path-to-the-install-destination-of-std-module> to the compiler to lookup the .pcm files and linking libraries. Hope the sharing helps.

---

From my understanding, we can't do the above things easily in the open-source community like LLVM. There should be some more discussing about this. So I suggest that the current patch shouldn't care about installation and distribution. It will be good enough if we have an available std module which pass the tests. Then we can discuss about the installation and distribution step by step.

[Mordante]

In D144994#4170655, @ChuanqiXu wrote:

About distributing the std modules, this is what we do in the downstream:

# std-module.spec

...

Requires: clang_xxx # for example clang17

%prep
%build
%install

mkdir -p <path-to-the-install-destination-of-std-module>
cp `.cppm` files to  <path-to-the-install-destination-of-std-module>
cp ../../../build.sh <path-to-the-install-destination-of-std-module>

...

%post
<path-to-the-install-destination-of-std-module>/build.sh

The commands line after %post will execute the corresponding instructions when the user input yum install std-modules and the build.sh contains the instructions to compile the std modules.

Then in our downstream, we will hardcode the <path-to-the-install-destination-of-std-module> to the compiler to lookup the .pcm files and linking libraries. Hope the sharing helps.

---

This is somewhat along the lines what I am thinking about too. However I use a CMake file for users. One of the issues in libc++ is that vendors can decide to disable filesystem header. Since parts of that library are in the dylib the std module can't provide what's in filesystem. Does your method also have that feature?

From my understanding, we can't do the above things easily in the open-source community like LLVM. There should be some more discussing about this. So I suggest that the current patch shouldn't care about installation and distribution. It will be good enough if we have an available std module which pass the tests. Then we can discuss about the installation and distribution step by step.

I think we can discuss that and see whether it's a blocker for landing the patch or not. For now I still have issues with ranges (which I have not started to investigate). I have been experimenting with running the test-suite for the modular build. So I will have some more information on that later today, probably on Discord.

libcxx/docs/Modules.rst

43

Let's discuss this further in an upcoming meeting. I don't object against not shipping it.

[ChuanqiXu]

One of the issues in libc++ is that vendors can decide to disable filesystem header. Since parts of that library are in the dylib the std module can't provide what's in filesystem. Does your method also have that feature?

I disabled the filesystem header directly in the downstream... so I don't have a lot experience in this.

I think we can discuss that and see whether it's a blocker for landing the patch or not. For now I still have issues with ranges (which I have not started to investigate). I have been experimenting with running the test-suite for the modular build. So I will have some more information on that later today, probably on Discord.

Yeah, it is always good to have more experience. And I didn't expect that we can convert all headers and pass all the tests in one shot. It is good enough for me to have a std module which contains a lot of headers and can pass a lot of tests.

[EricWF]

Cool deal. Thanks for working on this.

[ldionne]

libcxx/stdmodules/std-format.cppm

12 ↗

(On Diff #502156)

Well, if nobody like synopsis in headers, we can also remove them :-). Let's chat about that.

To be clear, you are not opposed to keeping the synopsis comment in test files that describes what we're testing, right? I think those are definitely useful and we should not stop those.

Pinging @var-const @huixie90 for thoughts. If folks agree, I don't want to impose it on people.

FWIW I do agree that storing those in the .cppm files results in something less readable than I thought it would.

[ldionne]

Pinging @iana @Bigcheese for awareness since they are working on Modules.

[philnik]

libcxx/stdmodules/std-format.cppm

12 ↗

(On Diff #502156)

Well, if nobody like synopsis in headers, we can also remove them :-). Let's chat about that.

To be clear, you are not opposed to keeping the synopsis comment in test files that describes what we're testing, right? I think those are definitely useful and we should not stop those.

I'm not sure they are always super useful, but they are definitely in enough cases that it would probably be a bad idea to remove them. They are also much easier to maintain and less error-prone.

[Mordante]

In D144994#4176269, @EricWF wrote:

Cool deal. Thanks for working on this.

Thanks!

In D144994#4173764, @ChuanqiXu wrote:

One of the issues in libc++ is that vendors can decide to disable filesystem header. Since parts of that library are in the dylib the std module can't provide what's in filesystem. Does your method also have that feature?

I disabled the filesystem header directly in the downstream... so I don't have a lot experience in this.

Thanks for the information!

I think we can discuss that and see whether it's a blocker for landing the patch or not. For now I still have issues with ranges (which I have not started to investigate). I have been experimenting with running the test-suite for the modular build. So I will have some more information on that later today, probably on Discord.

Yeah, it is always good to have more experience. And I didn't expect that we can convert all headers and pass all the tests in one shot. It is good enough for me to have a std module which contains a lot of headers and can pass a lot of tests.

I also didn't expect that, but we need to look at when we're comfortable with shipping something. Since libc++ is a system library on some platforms we need to consider that carefully. Obviously there are other non-official ways to ship things too.

It seems quite some tests fail due to using size_t instead of std::size_t and similar C-types. I'm working on patches to use the qualified name instead.

libcxx/stdmodules/std-format.cppm

12 ↗

(On Diff #502156)

Well, if nobody like synopsis in headers, we can also remove them :-). Let's chat about that.

To be clear, you are not opposed to keeping the synopsis comment in test files that describes what we're testing, right? I think those are definitely useful and we should not stop those.

Pinging @var-const @huixie90 for thoughts. If folks agree, I don't want to impose it on people.

FWIW I do agree that storing those in the .cppm files results in something less readable than I thought it would.

+1 for discussing the synopsis. But let's take it to Discord or a libc++ meeting.

[Mordante]

Various improvements.

• Incorporates D145596 CMake 3.26 fix.
• Removes new tests libc++ certainly doesn't want to duplicate all their tests for modules. (The final solution still is unknown.)
• Adds the std.compat module.
• Propagate some libc++ configuration flags to the modules
• Prepared a CI job, this is not used yet since the CI does not have CMake 3.26 and Ninja 1.11 yet.
• Update the test transformation script and did initial tried it locally.
• Small fixes to the module file.
• Added a few now module files.

[aaronmondal]

libcxx/docs/Modules.rst

84

Ah yes this fails due to template (mis)deduction with lambdas if I remember correctly. This works around it but it was decided some time ago that this shouldn't be changed in libc++ due to "work around compiler bug" not being a good reason for a change.

diff --git a/libcxx/include/__compare/synth_three_way.h b/libcxx/include/__compare/synth_three_way.h
index fa8cbda79ba2..a6df1b936c72 100644
--- a/libcxx/include/__compare/synth_three_way.h
+++ b/libcxx/include/__compare/synth_three_way.h
@@ -25,8 +25,8 @@ _LIBCPP_BEGIN_NAMESPACE_STD
 
 // [expos.only.func]
 
-_LIBCPP_HIDE_FROM_ABI inline constexpr auto __synth_three_way =
-  []<class _Tp, class _Up>(const _Tp& __t, const _Up& __u)
+template<class _Tp, class _Up>
+_LIBCPP_HIDE_FROM_ABI inline constexpr auto __synth_three_way(const _Tp& __t, const _Up& __u)
     requires requires {
       { __t < __u } -> __boolean_testable;
       { __u < __t } -> __boolean_testable;

libcxx/stdmodules/std-string_view.cppm

47 ↗

(On Diff #504447)

[h-vetinari]

libcxx/docs/Modules.rst

66

But we need to find the "proper" location to store them. Since this is something new this probably needs some involvement of our packagers. I don't expect the Linux Standard Base has an entry for C++20 modules.

As mentioned already, I think there are strong parallels with distributing headers. The bare minimum should be that they are automatically discoverable by the toolchain that produced them (should be somewhere under INSTALL_PREFIX obviously), but ideally it's in a standard location that's uniform across toolchains (again, relative to a PREFIX), like the headers and libs are too. I'm not sure if SG21 has had discussions about this, or if CMake has already thought about this.

Personally, I think the best might be a separate folder like $PREFIX/something_module_related/foo.cppm, but I wouldn't care much or at all if it's under $PREFIX/include/something_module_related/foo.cppm or $PREFIX/lib/<libname>/foo.cppm, etc., as long as it's discoverable and works.

CC @mgorny as another packager & @ben.boeckel for the CMake side.

[ChuanqiXu]

libcxx/docs/Modules.rst

66

I'm not sure if SG21 has had discussions about this, or if CMake has already thought about this.

As far as I know, sg15 has discussed this for a long time. (If I recall this correctly,) sg15 don't get the final consensus yet. But @ruoso has already made a lot of proposals. We can look at them later.

Personally, I think the best might be a separate folder like $PREFIX/something_module_related/foo.cppm, but I wouldn't care much or at all if it's under $PREFIX/include/something_module_related/foo.cppm or $PREFIX/lib/<libname>/foo.cppm, etc., as long as it's discoverable and works.

I think we need to discuss this in the discourse. Because the patch is going to be a large patch and the topic about distribution is another big topic. It looks not good to discuss two big things in the same page. The direction may be lost.

libcxx/stdmodules/std-compat.cppm

1 ↗

(On Diff #504447)

std.compat.cppm may be a better name. Since std-compat.cppm looks like a partition of the std module to me.

[ChuanqiXu]

libcxx/docs/Modules.rst

66

I created https://discourse.llvm.org/t/rfc-about-the-default-location-for-std-modules/69191 and let's discuss this there.

libcxx/stdmodules/CMakeLists.txt.in

71–75 ↗

(On Diff #504447)

This is what I did in the downstream. I imaged that we will merge this one with libc++.so in the upstream. But maybe I am wrong.
I feel we need to give it another name like libc++-modules. Otherwise it may be conflict with the std modules provided by libstdc++ later.

[ben.boeckel]

libcxx/CMakeLists.txt

108–111

This help string has embedded newlines and indentation (though I see other options do this…). I don't think the UI editors show the help well in this case (testing with ccmake seems to bear this out).

426

Somewhere under libdir would be good unless we know these files are actually arch-independent.

libcxx/docs/Modules.rst

26–27

Note that buildsystems should handle this. I don't know how viable the "just run the compiler by hand" is in the long-term…

43

You cannot ship the P1689 *output*, but it needs to be generated by anything *consuming* it so that the collation step can know to add the right flags (-fmodule-file=…) to the consuming TU compilation.

52–55

Nor should there be IMO. Note that a single build tree may need *multiple* compilations of the module, so I don't think a single "scanning output" is suitable to ship.

66

Personally, I think the best might be a separate folder like $PREFIX/something_module_related/foo.cppm, but I wouldn't care much or at all if it's under $PREFIX/include/something_module_related/foo.cppm or $PREFIX/lib/<libname>/foo.cppm, etc., as long as it's discoverable and works.

Note that libstdc++ is also going to ship std.cppm (possibly with a different extension), so the libname being part of the path sounds best to me. I'll jump in over on Discourse.

72

"not used as a compile flag" perhaps?

117

IMO, no.

163–165

Is this intended to be a stable interface? IMO, support for std modules needs to come from CMake because this is going to be different for every compiler otherwise.

libcxx/stdmodules/CMakeLists.txt

43 ↗

(On Diff #501265)

Instead we need to focus on distributing .cppm files and offer the tools for the user to create the .pcm files for their project. This is something I will investigate further.

I think this is more the job of CMake since asking users to do

cmake
if (<is_clang>)
  setup_clang_libcpp()
elseif (<is_gcc>)
  setup_gcc_libstdcpp()
elseif (<is_msvc>)
  setup_msvc_stl()
endif ()

is awfully narrow IMO. Highly related question: what do you expect to happen in the case of clang with libstdc++ (as many distros end up doing)?

libcxx/stdmodules/CMakeLists.txt.in

7 ↗

(On Diff #504447)

3.26 should have this.

9 ↗

(On Diff #504447)

This UUID changes…regularly. There will probably need to be a table to do this "properly".

10 ↗

(On Diff #504447)

This is specific to CMake's test suite and shouldn't be relevant.

[ldionne]

libcxx/stdmodules/std-cstddef.cppm

13 ↗

(On Diff #504447)

Would this make sense to avoid users having to define all the config site macros? #include <__config_site>

[tahonermann]

libcxx/stdmodules/std-cstddef.cppm

13 ↗

(On Diff #504447)

Named modules don't export/leak macros when imported, so the addition of the #include would have no effect on importers of the module.

[Mordante]

libcxx/CMakeLists.txt

108–111

Yes I indeed copied this from the other scripts. Are there better ways in CMake to have multi-line options?

426

This is used for the .cppm source files so they are arch independent.

libcxx/docs/Modules.rst

66

I'm not sure if SG21 has had discussions about this, or if CMake has already thought about this.

As far as I know, sg15 has discussed this for a long time. (If I recall this correctly,) sg15 don't get the final consensus yet. But @ruoso has already made a lot of proposals. We can look at them later.

Do you have a paper number?

Personally, I think the best might be a separate folder like $PREFIX/something_module_related/foo.cppm, but I wouldn't care much or at all if it's under $PREFIX/include/something_module_related/foo.cppm or $PREFIX/lib/<libname>/foo.cppm, etc., as long as it's discoverable and works.

I think we need to discuss this in the discourse. Because the patch is going to be a large patch and the topic about distribution is another big topic. It looks not good to discuss two big things in the same page. The direction may be lost.

I agree, this list is mainly added for documentation so we know what the status is. It was not intended to be used for discussions during the review.

163–165

I agree this is not the way users should use it. But without build system support this is how it works now.
So I feel this is "good enough" for early adapters. However when we (libc++) know how we want to distribute it
we should reach out to build system vendors and ask them to add proper support in their tools.

I'm not even sure what direction we want to take. For the experimental library we have a Clang flag which
automatically link that library. Do we want to do something similar here or leave it entirely in the hands
of the build system?

libcxx/stdmodules/CMakeLists.txt

43 ↗

(On Diff #501265)

I agree this is narrow and not great. However this patch's main focus it to be able to get modules working in libc++ in the first place.
We definitely need to think about the distribution in a user-friendly way.

I'm not sure whether libc++ needs to consider the configuration of Clang + libstdc++. Do you have something specific in mind?
I expect there need to be things done on the Clang side.

libcxx/stdmodules/CMakeLists.txt.in

7 ↗

(On Diff #504447)

Thanks for the information! I see it landed recently in CMake.

9 ↗

(On Diff #504447)

Is there a public available table for these UUIDs? The last time I needed the one for CMake 3.25 I looked in the CMake git history.

libcxx/stdmodules/std-cstddef.cppm

13 ↗

(On Diff #504447)

The goal is not to leak the macros, but to use the macros to conditionally export symbols. For example in the next file in the patch I use #ifndef _LIBCPP_HAS_NO_WIDE_CHARACTERS to avoid exporting wide character functions when the user configured libc++ without wide character support.

When building the BMI we need to know the configuration of libc++. In a similar fashion we need to disable symbols based on the language version used. For now there are some examples for C++20 vs C++23. Not that we have decided that we want to back port this feature, but we will need this for C++23 vs C++26.

[ben.boeckel]

libcxx/docs/Modules.rst

163–165

I think that some build-system agnostic data (similar to the MSVC STL JSON file linked to on Discourse is best). I don't think every compiler trying to chase umpteen build tool integrations (as compilers tend to be *far* harder to upgrade in the Real World than build system tools IME).

ISO C++'s SG15 is the best place for communication IMO.

libcxx/stdmodules/CMakeLists.txt.in

9 ↗

(On Diff #504447)

The UUID is only documented in the CMake repository's Help/dev/experimental.rst file.

[Mordante]

Adds more modules.

This mostely completes the modules. These modules have been compile
tested but nothing more. They also do not take disabled libc++ parts
into account.

Updates the documentation.

[ldionne]

libcxx/test/libcxx/language.support/support.types/cstddef.compile.pass.cpp

1

IMO it would make sense to split this test like this:

/////////////////////////////////////////////////////////////////////////////
// cstddef.compile.pass.cpp
#include <cstddef>
#include "test_macros.h"

using PtrdiffT = ::ptrdiff_t;
using SizeT = ::size_t;
#if TEST_STD_VER >= 11
using MaxAlignT = ::max_align_t;
#endif

// Supported in C++03 mode too for backwards compatibility with previous versions of libc++
using NullptrT = ::nullptr_t;

// Also ensure that we provide std::nullptr_t in C++03 mode, which is an extension too.
using StdNullptrT = std::nullptr_t;
/////////////////////////////////////////////////////////////////////////////

/////////////////////////////////////////////////////////////////////////////
// cstddef.std.module.verify.cpp
// REQUIRES: modules-build (????)
import std;

#include "test_macros.h"

using PtrdiffT = ::ptrdiff_t; // expected-error {{no ptrdiff_t in global namespace}}
using SizeT = ::size_t; // expected-error {{no size_t in global namespace}}

[ etc... ]
/////////////////////////////////////////////////////////////////////////////

/////////////////////////////////////////////////////////////////////////////
// cstddef.stdcompat.module.compile.pass.cpp
import std.compat;

#include "test_macros.h"

using PtrdiffT = ::ptrdiff_t;
using SizeT = ::size_t;

[ etc... ]
/////////////////////////////////////////////////////////////////////////////

This one would be split up manually since it is extremely relevant to test this with C++23 modules.

However, many other tests are not as useful to test with/without modules. As we discussed, we could perhaps run use_modules_in_test.py on the fly to create a modularized version of the test that we wouldn't check into source control.

[Mordante]

Minor improvements

Ran the tests using the compat module with the tests converted using
the test conversion script. After some minor fixes 222 tests fail and
need triaging. (This are more tests than before, but now the tests use
less headers since most headers are now available as module.)

• Improved the conversion script
  • Imports are moved the the end.
  • Modularized header #ifndefs are grouped.
  • Disables tests using deprecated features. These use a compiler switch that doesn't work with modules.
• Updated the documentation.
• Minor module fixes.

[Mordante]

Updated due to Discourse discussion and minor improvements.

• Use the installation location as proposed on Discourse. This adds a proof-of-concept installer.
• Add CI jobs to test modules with parts disabled.
• Fix modules to work with no localization or no wide characters. Other configurations are errors.
• Adds a few empty modules, their headers export nothing.
• Improve the algorithm module.

[tschuett]

libcxx/docs/Modules.rst

104

C++20

136

I believe this is confusing. I thought that you want to store everything underneath c++? Additionally you use the same path for includes and modules.

[aaronmondal]

libcxx/docs/Modules.rst

136

Typo? Should probably be "Includes are stored in <prefix>/include/c++/v1, modules in <prefix>/modules/c++/v1".

I think this makes sense. Modules can't be #included anyways so there is a need for them to live in an include/ directory. And for those using a different ABIs this leaves the possibility to have

.../include/c++/v1
.../include/c++/v2
.../modules/c++/v1
.../modules/c++/v2

or similar to clearly differentiate the headers/module interfaces.

[tschuett]

There was objection against using /usr/modules on discourse. I assumed that libc++ owns /usr/include/c++ and hides everything under there.

[ruoso]

libcxx/docs/Modules.rst

136

$PREFIX/modules is a new path that doesn't exist in the Filesystem Hierarchy Standard.

The source files for the module units is indeed a arch-independent resource, therefore the correct directory would be something under $PREFIX/share

If we ever intend to ship BMI files, they would belong in $libdir.

Here's how it would look like if we go in that direction:

$PREFIX/
   $libdir/
       libc++.so
       libc++.so.module-info
       c++/
           modules/
                libc++/
                      std.gcm.deadbeef1234
   share/
       c++/
           modules/
                 libc++/
                       std.cppm

the libc++.so.module-info file would have the metadata necessary for someone to understand how to produce their own BMI as well as potentially reuse the shipped BMI if it just so happen that they can.

[ruoso]

libcxx/docs/Modules.rst

136

There is one important bit that worries about splitting off the source files, tho.

The module metadata shipped alongside the library itself needs to reference those source files, and I am concerned that requiring the use of ../../../../share/ in order to address the source location can lead to fragility in the deployment.

So, even though it's technically arch-independent, I would also consider the following option:

$PREFIX/
   $libdir/
       libc++.so
       libc++.so.module-info
       c++/
           modules/
                libc++/
                      std.gcm.deadbeef1234
                      std.cppm

Because at that point, the metadata could reference a relative path from the library location without the awkwardness that using ../ can cause when directories are symlinks.

[ChuanqiXu]

Add CI jobs to test modules with parts disabled.

So, is the CI going to be ready?

libcxx/modules/std.compat/cassert.cppm

11–17

What's the meaning for the file? It looks useless.

[Mordante]

In D144994#4208434, @ChuanqiXu wrote:

Add CI jobs to test modules with parts disabled.

So, is the CI going to be ready?

Not at the moment. There are some CI issues not related to modules which should be resolved first. However I am testing with the CI configuration locally. I'm still fixing bugs in the modules and investigating issue which might be Clang bugs.

I'm also still busy to remove symbols from the global namespace so I can start to test the std module, for now I only test with the std.compat module.

libcxx/docs/Modules.rst

26–27

I fully agree. For now we will also ship a CMake file to aid users. But once build systems catch up that CMake file will be removed. At the moment we're not even sure where our .cppm files will be installed, so asking buildsystems to support libc++ is a bit premature. Having this information makes it possible for early adaptors to get started.

43

I've removed the section.

136

This is indeed a typo.

163–165

I will consider attending in SG15 to discuss this.

libcxx/modules/std.compat/cassert.cppm

11–17

It kind of is, but it's easier to have tooling that makes sure every header is a module. This way there need to be no exceptions. Some of the "empty" modules may get symbols in the future.

[ChuanqiXu]

Not at the moment. There are some CI issues not related to modules which should be resolved first. However I am testing with the CI configuration locally. I'm still fixing bugs in the modules and investigating issue which might be Clang bugs.

Oh, got it. Never be hurry : )

libcxx/modules/std.compat/cassert.cppm

11–17

Got your point. I remember the paper said clearly that assert and errno is not exported in the std modules. But it doesn't matter really.

[Mordante]

libcxx/modules/std.compat/cassert.cppm

11–17

I can't find that in the paper nor in the WP. There is http://eel.is/c++draft/std.modules#note-2 which states

[Note 2: Like all named modules, the C++ library modules do not make macros visible ([module.import]), such as assert ([cassert.syn]), errno ([cerrno.syn]), offsetof ([cstddef.syn]), and va_arg ([cstdarg.syn]).
— end note]

[ChuanqiXu]

libcxx/modules/std.compat/cassert.cppm

11–17

Oh, my bad for confusing : )