This is an archive of the discontinued LLVM Phabricator instance.

[Draft][libc++][modules] Adds std module.
AbandonedPublic

Authored by Mordante on Feb 28 2023, 11:37 AM.

Details

Reviewers
ldionne
ChuanqiXu
aaronmondal
EricWF
jdoerfert
Group Reviewers
Restricted Project
Restricted Project
Restricted Project
Summary

Implements parts of

  • P2465R3 Standard Library Modules std and std.compat

This patch is based on work by @ChuanqiXu.

This patch not intended to be landed is this form, it's intended to get
a feeling of modules in libc++ and improve upon it. This patch will be
used to discuss the further approach regarding modules in libc++.

Diff Detail

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
EricWF added a comment.Mar 7 2023, 1:58 PM

Cool deal. Thanks for working on this.

ldionne added inline comments.
libcxx/stdmodules/std-format.cppm
13

Well, if nobody like synopsis in headers, we can also remove them :-). Let's chat about that.

To be clear, you are not opposed to keeping the synopsis comment in test files that describes what we're testing, right? I think those are definitely useful and we should not stop those.

Pinging @var-const @huixie90 for thoughts. If folks agree, I don't want to impose it on people.

FWIW I do agree that storing those in the .cppm files results in something less readable than I thought it would.

Pinging @iana @Bigcheese for awareness since they are working on Modules.

philnik added inline comments.Mar 8 2023, 3:58 PM
libcxx/stdmodules/std-format.cppm
13

Well, if nobody like synopsis in headers, we can also remove them :-). Let's chat about that.

To be clear, you are not opposed to keeping the synopsis comment in test files that describes what we're testing, right? I think those are definitely useful and we should not stop those.

I'm not sure they are always super useful, but they are definitely in enough cases that it would probably be a bad idea to remove them. They are also much easier to maintain and less error-prone.

Cool deal. Thanks for working on this.

Thanks!

One of the issues in libc++ is that vendors can decide to disable filesystem header. Since parts of that library are in the dylib the std module can't provide what's in filesystem. Does your method also have that feature?

I disabled the filesystem header directly in the downstream... so I don't have a lot experience in this.

Thanks for the information!

I think we can discuss that and see whether it's a blocker for landing the patch or not. For now I still have issues with ranges (which I have not started to investigate). I have been experimenting with running the test-suite for the modular build. So I will have some more information on that later today, probably on Discord.

Yeah, it is always good to have more experience. And I didn't expect that we can convert all headers and pass all the tests in one shot. It is good enough for me to have a std module which contains a lot of headers and can pass a lot of tests.

I also didn't expect that, but we need to look at when we're comfortable with shipping something. Since libc++ is a system library on some platforms we need to consider that carefully. Obviously there are other non-official ways to ship things too.

It seems quite some tests fail due to using size_t instead of std::size_t and similar C-types. I'm working on patches to use the qualified name instead.

libcxx/stdmodules/std-format.cppm
13

Well, if nobody like synopsis in headers, we can also remove them :-). Let's chat about that.

To be clear, you are not opposed to keeping the synopsis comment in test files that describes what we're testing, right? I think those are definitely useful and we should not stop those.

Pinging @var-const @huixie90 for thoughts. If folks agree, I don't want to impose it on people.

FWIW I do agree that storing those in the .cppm files results in something less readable than I thought it would.

+1 for discussing the synopsis. But let's take it to Discord or a libc++ meeting.

Mordante updated this revision to Diff 504447.Mar 12 2023, 8:53 AM

Various improvements.

  • Incorporates D145596 CMake 3.26 fix.
  • Removes new tests libc++ certainly doesn't want to duplicate all their tests for modules. (The final solution still is unknown.)
  • Adds the std.compat module.
  • Propagate some libc++ configuration flags to the modules
  • Prepared a CI job, this is not used yet since the CI does not have CMake 3.26 and Ninja 1.11 yet.
  • Update the test transformation script and did initial tried it locally.
  • Small fixes to the module file.
  • Added a few now module files.
Herald added a project: Restricted Project. · View Herald TranscriptMar 12 2023, 8:53 AM
Herald added a reviewer: Restricted Project. · View Herald Transcript
aaronmondal added inline comments.Mar 12 2023, 9:36 AM
libcxx/docs/Modules.rst
84

Ah yes this fails due to template (mis)deduction with lambdas if I remember correctly. This works around it but it was decided some time ago that this shouldn't be changed in libc++ due to "work around compiler bug" not being a good reason for a change.

diff --git a/libcxx/include/__compare/synth_three_way.h b/libcxx/include/__compare/synth_three_way.h
index fa8cbda79ba2..a6df1b936c72 100644
--- a/libcxx/include/__compare/synth_three_way.h
+++ b/libcxx/include/__compare/synth_three_way.h
@@ -25,8 +25,8 @@ _LIBCPP_BEGIN_NAMESPACE_STD
 
 // [expos.only.func]
 
-_LIBCPP_HIDE_FROM_ABI inline constexpr auto __synth_three_way =
-  []<class _Tp, class _Up>(const _Tp& __t, const _Up& __u)
+template<class _Tp, class _Up>
+_LIBCPP_HIDE_FROM_ABI inline constexpr auto __synth_three_way(const _Tp& __t, const _Up& __u)
     requires requires {
       { __t < __u } -> __boolean_testable;
       { __u < __t } -> __boolean_testable;
libcxx/stdmodules/std-string_view.cppm
47 ↗(On Diff #504447)
h-vetinari added inline comments.
libcxx/docs/Modules.rst
65

But we need to find the "proper" location to store them. Since this is something new this probably needs some involvement of our packagers. I don't expect the Linux Standard Base has an entry for C++20 modules.

As mentioned already, I think there are strong parallels with distributing headers. The bare minimum should be that they are automatically discoverable by the toolchain that produced them (should be somewhere under INSTALL_PREFIX obviously), but ideally it's in a standard location that's uniform across toolchains (again, relative to a PREFIX), like the headers and libs are too. I'm not sure if SG21 has had discussions about this, or if CMake has already thought about this.

Personally, I think the best might be a separate folder like $PREFIX/something_module_related/foo.cppm, but I wouldn't care much or at all if it's under $PREFIX/include/something_module_related/foo.cppm or $PREFIX/lib/<libname>/foo.cppm, etc., as long as it's discoverable and works.

CC @mgorny as another packager & @ben.boeckel for the CMake side.

ChuanqiXu added inline comments.
libcxx/docs/Modules.rst
65

I'm not sure if SG21 has had discussions about this, or if CMake has already thought about this.

As far as I know, sg15 has discussed this for a long time. (If I recall this correctly,) sg15 don't get the final consensus yet. But @ruoso has already made a lot of proposals. We can look at them later.

Personally, I think the best might be a separate folder like $PREFIX/something_module_related/foo.cppm, but I wouldn't care much or at all if it's under $PREFIX/include/something_module_related/foo.cppm or $PREFIX/lib/<libname>/foo.cppm, etc., as long as it's discoverable and works.

I think we need to discuss this in the discourse. Because the patch is going to be a large patch and the topic about distribution is another big topic. It looks not good to discuss two big things in the same page. The direction may be lost.

libcxx/stdmodules/std-compat.cppm
1 ↗(On Diff #504447)

std.compat.cppm may be a better name. Since std-compat.cppm looks like a partition of the std module to me.

ChuanqiXu added inline comments.Mar 13 2023, 1:30 AM
libcxx/docs/Modules.rst
65
libcxx/stdmodules/CMakeLists.txt.in
72–76

This is what I did in the downstream. I imaged that we will merge this one with libc++.so in the upstream. But maybe I am wrong.
I feel we need to give it another name like libc++-modules. Otherwise it may be conflict with the std modules provided by libstdc++ later.

ben.boeckel added inline comments.Mar 13 2023, 9:24 AM
libcxx/CMakeLists.txt
108–111

This help string has embedded newlines and indentation (though I see other options do this…). I don't think the UI editors show the help well in this case (testing with ccmake seems to bear this out).

422

Somewhere under libdir would be good unless we know these files are actually arch-independent.

libcxx/docs/Modules.rst
26–27

Note that buildsystems should handle this. I don't know how viable the "just run the compiler by hand" is in the long-term…

42

You cannot ship the P1689 *output*, but it needs to be generated by anything *consuming* it so that the collation step can know to add the right flags (-fmodule-file=…) to the consuming TU compilation.

52–55

Nor should there be IMO. Note that a single build tree may need *multiple* compilations of the module, so I don't think a single "scanning output" is suitable to ship.

65

Personally, I think the best might be a separate folder like $PREFIX/something_module_related/foo.cppm, but I wouldn't care much or at all if it's under $PREFIX/include/something_module_related/foo.cppm or $PREFIX/lib/<libname>/foo.cppm, etc., as long as it's discoverable and works.

Note that libstdc++ is also going to ship std.cppm (possibly with a different extension), so the libname being part of the path sounds best to me. I'll jump in over on Discourse.

72

"not used as a compile flag" perhaps?

117

IMO, no.

163–165

Is this intended to be a stable interface? IMO, support for std modules needs to come from CMake because this is going to be different for every compiler otherwise.

libcxx/stdmodules/CMakeLists.txt
44

Instead we need to focus on distributing .cppm files and offer the tools for the user to create the .pcm files for their project. This is something I will investigate further.

I think this is more the job of CMake since asking users to do

cmake
if (<is_clang>)
  setup_clang_libcpp()
elseif (<is_gcc>)
  setup_gcc_libstdcpp()
elseif (<is_msvc>)
  setup_msvc_stl()
endif ()

is awfully narrow IMO. Highly related question: what do you expect to happen in the case of clang with libstdc++ (as many distros end up doing)?

libcxx/stdmodules/CMakeLists.txt.in
8

3.26 should have this.

10

This UUID changes…regularly. There will probably need to be a table to do this "properly".

11

This is specific to CMake's test suite and shouldn't be relevant.

ldionne added inline comments.Mar 14 2023, 10:04 AM
libcxx/stdmodules/std-cstddef.cppm
13 ↗(On Diff #504447)

Would this make sense to avoid users having to define all the config site macros? #include <__config_site>

tahonermann added inline comments.
libcxx/stdmodules/std-cstddef.cppm
13 ↗(On Diff #504447)

Named modules don't export/leak macros when imported, so the addition of the #include would have no effect on importers of the module.

Mordante added inline comments.Mar 14 2023, 11:41 AM
libcxx/CMakeLists.txt
108–111

Yes I indeed copied this from the other scripts. Are there better ways in CMake to have multi-line options?

422

This is used for the .cppm source files so they are arch independent.

libcxx/docs/Modules.rst
65

I'm not sure if SG21 has had discussions about this, or if CMake has already thought about this.

As far as I know, sg15 has discussed this for a long time. (If I recall this correctly,) sg15 don't get the final consensus yet. But @ruoso has already made a lot of proposals. We can look at them later.

Do you have a paper number?

Personally, I think the best might be a separate folder like $PREFIX/something_module_related/foo.cppm, but I wouldn't care much or at all if it's under $PREFIX/include/something_module_related/foo.cppm or $PREFIX/lib/<libname>/foo.cppm, etc., as long as it's discoverable and works.

I think we need to discuss this in the discourse. Because the patch is going to be a large patch and the topic about distribution is another big topic. It looks not good to discuss two big things in the same page. The direction may be lost.

I agree, this list is mainly added for documentation so we know what the status is. It was not intended to be used for discussions during the review.

163–165

I agree this is not the way users should use it. But without build system support this is how it works now.
So I feel this is "good enough" for early adapters. However when we (libc++) know how we want to distribute it
we should reach out to build system vendors and ask them to add proper support in their tools.

I'm not even sure what direction we want to take. For the experimental library we have a Clang flag which
automatically link that library. Do we want to do something similar here or leave it entirely in the hands
of the build system?

libcxx/stdmodules/CMakeLists.txt
44

I agree this is narrow and not great. However this patch's main focus it to be able to get modules working in libc++ in the first place.
We definitely need to think about the distribution in a user-friendly way.

I'm not sure whether libc++ needs to consider the configuration of Clang + libstdc++. Do you have something specific in mind?
I expect there need to be things done on the Clang side.

libcxx/stdmodules/CMakeLists.txt.in
8

Thanks for the information! I see it landed recently in CMake.

10

Is there a public available table for these UUIDs? The last time I needed the one for CMake 3.25 I looked in the CMake git history.

libcxx/stdmodules/std-cstddef.cppm
13 ↗(On Diff #504447)

The goal is not to leak the macros, but to use the macros to conditionally export symbols. For example in the next file in the patch I use #ifndef _LIBCPP_HAS_NO_WIDE_CHARACTERS to avoid exporting wide character functions when the user configured libc++ without wide character support.

When building the BMI we need to know the configuration of libc++. In a similar fashion we need to disable symbols based on the language version used. For now there are some examples for C++20 vs C++23. Not that we have decided that we want to back port this feature, but we will need this for C++23 vs C++26.

ben.boeckel added inline comments.Mar 14 2023, 12:19 PM
libcxx/docs/Modules.rst
163–165

I think that some build-system agnostic data (similar to the MSVC STL JSON file linked to on Discourse is best). I don't think every compiler trying to chase umpteen build tool integrations (as compilers tend to be *far* harder to upgrade in the Real World than build system tools IME).

ISO C++'s SG15 is the best place for communication IMO.

libcxx/stdmodules/CMakeLists.txt.in
10

The UUID is only documented in the CMake repository's Help/dev/experimental.rst file.

Mordante updated this revision to Diff 505553.Mar 15 2023, 10:24 AM

Adds more modules.

This mostely completes the modules. These modules have been compile
tested but nothing more. They also do not take disabled libc++ parts
into account.

Updates the documentation.

ldionne added inline comments.Mar 17 2023, 11:28 AM
libcxx/test/libcxx/language.support/support.types/cstddef.compile.pass.cpp
1 ↗(On Diff #505553)

IMO it would make sense to split this test like this:

/////////////////////////////////////////////////////////////////////////////
// cstddef.compile.pass.cpp
#include <cstddef>
#include "test_macros.h"

using PtrdiffT = ::ptrdiff_t;
using SizeT = ::size_t;
#if TEST_STD_VER >= 11
using MaxAlignT = ::max_align_t;
#endif

// Supported in C++03 mode too for backwards compatibility with previous versions of libc++
using NullptrT = ::nullptr_t;

// Also ensure that we provide std::nullptr_t in C++03 mode, which is an extension too.
using StdNullptrT = std::nullptr_t;
/////////////////////////////////////////////////////////////////////////////

/////////////////////////////////////////////////////////////////////////////
// cstddef.std.module.verify.cpp
// REQUIRES: modules-build (????)
import std;

#include "test_macros.h"

using PtrdiffT = ::ptrdiff_t; // expected-error {{no ptrdiff_t in global namespace}}
using SizeT = ::size_t; // expected-error {{no size_t in global namespace}}

[ etc... ]
/////////////////////////////////////////////////////////////////////////////

/////////////////////////////////////////////////////////////////////////////
// cstddef.stdcompat.module.compile.pass.cpp
import std.compat;

#include "test_macros.h"

using PtrdiffT = ::ptrdiff_t;
using SizeT = ::size_t;

[ etc... ]
/////////////////////////////////////////////////////////////////////////////

This one would be split up manually since it is extremely relevant to test this with C++23 modules.

However, many other tests are not as useful to test with/without modules. As we discussed, we could perhaps run use_modules_in_test.py on the fly to create a modularized version of the test that we wouldn't check into source control.

Mordante updated this revision to Diff 506302.Mar 18 2023, 9:59 AM

Minor improvements

Ran the tests using the compat module with the tests converted using
the test conversion script. After some minor fixes 222 tests fail and
need triaging. (This are more tests than before, but now the tests use
less headers since most headers are now available as module.)

  • Improved the conversion script
    • Imports are moved the the end.
    • Modularized header #ifndefs are grouped.
    • Disables tests using deprecated features. These use a compiler switch that doesn't work with modules.
  • Updated the documentation.
  • Minor module fixes.
Mordante updated this revision to Diff 506637.Mar 20 2023, 10:18 AM

Updated due to Discourse discussion and minor improvements.

  • Use the installation location as proposed on Discourse. This adds a proof-of-concept installer.
  • Add CI jobs to test modules with parts disabled.
  • Fix modules to work with no localization or no wide characters. Other configurations are errors.
  • Adds a few empty modules, their headers export nothing.
  • Improve the algorithm module.
tschuett added inline comments.Mar 20 2023, 10:26 AM
libcxx/docs/Modules.rst
105

C++20

137

I believe this is confusing. I thought that you want to store everything underneath c++? Additionally you use the same path for includes and modules.

aaronmondal added inline comments.Mar 20 2023, 11:20 AM
libcxx/docs/Modules.rst
137

Typo? Should probably be "Includes are stored in <prefix>/include/c++/v1, modules in <prefix>/modules/c++/v1".

I think this makes sense. Modules can't be #included anyways so there is a need for them to live in an include/ directory. And for those using a different ABIs this leaves the possibility to have

.../include/c++/v1
.../include/c++/v2
.../modules/c++/v1
.../modules/c++/v2

or similar to clearly differentiate the headers/module interfaces.

There was objection against using /usr/modules on discourse. I assumed that libc++ owns /usr/include/c++ and hides everything under there.

ruoso added inline comments.Mar 20 2023, 11:44 AM
libcxx/docs/Modules.rst
137

$PREFIX/modules is a new path that doesn't exist in the Filesystem Hierarchy Standard.

The source files for the module units is indeed a arch-independent resource, therefore the correct directory would be something under $PREFIX/share

If we ever intend to ship BMI files, they would belong in $libdir.

Here's how it would look like if we go in that direction:

$PREFIX/
   $libdir/
       libc++.so
       libc++.so.module-info
       c++/
           modules/
                libc++/
                      std.gcm.deadbeef1234
   share/
       c++/
           modules/
                 libc++/
                       std.cppm

the libc++.so.module-info file would have the metadata necessary for someone to understand how to produce their own BMI as well as potentially reuse the shipped BMI if it just so happen that they can.

ruoso added inline comments.Mar 20 2023, 12:05 PM
libcxx/docs/Modules.rst
137

There is one important bit that worries about splitting off the source files, tho.

The module metadata shipped alongside the library itself needs to reference those source files, and I am concerned that requiring the use of ../../../../share/ in order to address the source location can lead to fragility in the deployment.

So, even though it's technically arch-independent, I would also consider the following option:

$PREFIX/
   $libdir/
       libc++.so
       libc++.so.module-info
       c++/
           modules/
                libc++/
                      std.gcm.deadbeef1234
                      std.cppm

Because at that point, the metadata could reference a relative path from the library location without the awkwardness that using ../ can cause when directories are symlinks.

Add CI jobs to test modules with parts disabled.

So, is the CI going to be ready?

libcxx/modules/std.compat/cassert.cppm
11–17 ↗(On Diff #506637)

What's the meaning for the file? It looks useless.

Mordante marked 5 inline comments as done.Mar 21 2023, 9:38 AM

Add CI jobs to test modules with parts disabled.

So, is the CI going to be ready?

Not at the moment. There are some CI issues not related to modules which should be resolved first. However I am testing with the CI configuration locally. I'm still fixing bugs in the modules and investigating issue which might be Clang bugs.

I'm also still busy to remove symbols from the global namespace so I can start to test the std module, for now I only test with the std.compat module.

libcxx/docs/Modules.rst
26–27

I fully agree. For now we will also ship a CMake file to aid users. But once build systems catch up that CMake file will be removed. At the moment we're not even sure where our .cppm files will be installed, so asking buildsystems to support libc++ is a bit premature. Having this information makes it possible for early adaptors to get started.

42

I've removed the section.

137

This is indeed a typo.

163–165

I will consider attending in SG15 to discuss this.

libcxx/modules/std.compat/cassert.cppm
11–17 ↗(On Diff #506637)

It kind of is, but it's easier to have tooling that makes sure every header is a module. This way there need to be no exceptions. Some of the "empty" modules may get symbols in the future.

Not at the moment. There are some CI issues not related to modules which should be resolved first. However I am testing with the CI configuration locally. I'm still fixing bugs in the modules and investigating issue which might be Clang bugs.

Oh, got it. Never be hurry : )

libcxx/modules/std.compat/cassert.cppm
11–17 ↗(On Diff #506637)

Got your point. I remember the paper said clearly that assert and errno is not exported in the std modules. But it doesn't matter really.

Mordante marked 2 inline comments as done.Mar 21 2023, 10:01 AM
Mordante added inline comments.
libcxx/modules/std.compat/cassert.cppm
11–17 ↗(On Diff #506637)

I can't find that in the paper nor in the WP. There is http://eel.is/c++draft/std.modules#note-2 which states

[Note 2: Like all named modules, the C++ library modules do not make macros visible ([module.import]), such as assert ([cassert.syn]), errno ([cerrno.syn]), offsetof ([cstddef.syn]), and va_arg ([cstdarg.syn]).
— end note]
ChuanqiXu added inline comments.Mar 21 2023, 7:00 PM
libcxx/modules/std.compat/cassert.cppm
11–17 ↗(On Diff #506637)

Oh, my bad for confusing : )

Mordante updated this revision to Diff 512849.Apr 12 2023, 8:32 AM
Mordante marked an inline comment as done.

Several minor improvements.

  • updates comments
  • removes C++20 modules
  • adds new module specific tests using AST matchers
  • fixes several module bugs, mainly found by the new tests
Mordante updated this revision to Diff 512886.Apr 12 2023, 10:03 AM

Rebased on main and resolved merge conflicts.

ChuanqiXu added inline comments.Apr 12 2023, 7:00 PM
libcxx/test/libcxx/module_std.sh.cpp
227 ↗(On Diff #512886)

I can't search libcpp-header-exportable-declarations. Is there a patch that haven't been uploaded?

Mordante marked an inline comment as done.Apr 14 2023, 9:05 AM
Mordante added inline comments.Apr 14 2023, 9:05 AM
libcxx/test/libcxx/module_std.sh.cpp
227 ↗(On Diff #512886)

It's in the file libcxx/test/tools/clang_tidy_checks/header_exportable_declarations.cpp which is part of this patch.
(It still has some issues, but I wanted to post some progress again.)

philnik added inline comments.Apr 14 2023, 9:39 AM
libcxx/test/tools/clang_tidy_checks/header_exportable_declarations.cpp
92 ↗(On Diff #512886)
102–106 ↗(On Diff #512886)

I think that's the idiomatic way to do this.

141–156 ↗(On Diff #512886)

We have something very similar in ugilfy_attributes.cpp. Maybe we should add a header for a few utilities?

165 ↗(On Diff #512886)

Is this equivalent to decl->getName()?

172 ↗(On Diff #512886)

What is this doing?

libcxx/test/tools/clang_tidy_checks/header_exportable_declarations.hpp
21 ↗(On Diff #512886)

Would an unordered_set be better?

Mordante marked 5 inline comments as done.Apr 14 2023, 10:32 AM

Thanks for the review!

libcxx/test/tools/clang_tidy_checks/header_exportable_declarations.cpp
92 ↗(On Diff #512886)

Thanks I already spotted this locally.

102–106 ↗(On Diff #512886)

Interesting I wasn't aware llvm::isa multiple arguments. For now I keep it as is, but leave this comment open,

Once the code is working as intended I will see whether this change improves the code.

141–156 ↗(On Diff #512886)

That might be useful. BTW if (str[0] == '_' && str[1] >= 'A' && str[1] <= 'Z' does not work properly in EBCDIC.

165 ↗(On Diff #512886)

The documentation in not fully clear, but since getName() returns a StringRef I'm sure it's not the same.
getName() would return fill where get_qualified_name() returns std::fill or std::ranges::fill.

172 ↗(On Diff #512886)

Dumping a line like using std::vector. This checker is not a real checker, but extracts information.
The output of a toplevel header is compared with the output of a module.
This is done in libcxx/test/libcxx/module_std.sh.cpp (which very much under review and I'm still looking at the failures.)

libcxx/test/tools/clang_tidy_checks/header_exportable_declarations.hpp
21 ↗(On Diff #512886)

I haven't tested. Most headers don't have a lot of declarations. The module type_traits which is one of the larger modules has less than 300 entries. (In total there are less than 4000 entries in the Standard library.)

ChuanqiXu added inline comments.Apr 17 2023, 11:37 PM
libcxx/modules/CMakeLists.txt.in
48 ↗(On Diff #512886)

I guess we need a better library name before landing this.

Mordante marked 4 inline comments as done.Apr 18 2023, 9:40 AM
Mordante added inline comments.
libcxx/modules/CMakeLists.txt.in
48 ↗(On Diff #512886)

What would you propose?
I'm not even sure we really need to, but this is something to determine during a review.

ChuanqiXu added inline comments.Apr 18 2023, 7:02 PM
libcxx/docs/Modules.rst
84–99

Will these problems be blockers actually? Specially for the "header-module" order problem, we can't fix it fundamentally. And I feel 'std module' is still workable in some degree with these defects. If these problems are not blockers actually, could we move the "limitations" section?

libcxx/modules/CMakeLists.txt.in
48 ↗(On Diff #512886)

What would you propose?

I use libstdmodule.a in the downstream. But it looks not good enough for the upstream.

I'm not even sure we really need to, but this is something to determine during a review.

Ideally this should be handled by cmake (or other build systems). But we need it now to make progress. How about something like 'libc++_std_module.a'?

Mordante added inline comments.Apr 19 2023, 8:28 AM
libcxx/docs/Modules.rst
84–99

I think this one isn't I think we should downgrade this one to limitation.

Any idea how to get this diff from phabricator in a way that doesn't include the changes to the modules directory? That part somehow messes up Bazel's patching.

libcxx/modules/CMakeLists.txt.in
48 ↗(On Diff #512886)

Don't have a strong opinion on this, though this will probably be painful to change once users adopt any name.

libc++-std.a, lib++_std.a? Though I personally think just std would be fine.

The std.compat archive should follow a similar scheme though.

ChuanqiXu added inline comments.Apr 19 2023, 7:01 PM
libcxx/docs/Modules.rst
84–99

What do you mean for this one?

aaronmondal added a comment.EditedApr 19 2023, 11:29 PM

Started testing the Bazel build. Seems to almost build. The __new is a slight inconvenience, but easily worked around and I don't think we can do too much about it.

Only std:memory doesn't build yet on my end.

I'm also noticing that this will place quite the significant restriction on the compiler building the modules. It's a bit unintuitive to build libc++ with C++20 and the interfaces/partitions with C++23. It's more intuitive to build the entire libc++ (sources) with C++23 (which works), but I'd assume that the only compiler capable of doing that at the moment upstream Clang itself 🤣

libcxx/modules/CMakeLists.txt.in
48 ↗(On Diff #512886)

I think I'm misunderstanding this target. What are the contents of this archive?

libcxx/modules/std/memory.cppm
183–184 ↗(On Diff #512886)

This seems to break the build for me with "no member atomic in namespace std". Where is this symbol coming from?

The <atomic> include is disabled for C++23. Maybe I'm just missing some config flag?

Started testing the Bazel build. Seems to almost build. The __new is a slight inconvenience, but easily worked around and I don't think we can do too much about it.

Yeah we can't use new since it's a keyword.

I'm also noticing that this will place quite the significant restriction on the compiler building the modules. It's a bit unintuitive to build libc++ with C++20 and the interfaces/partitions with C++23. It's more intuitive to build the entire libc++ (sources) with C++23 (which works), but I'd assume that the only compiler capable of doing that at the moment upstream Clang itself 🤣

We can't nicely build libc++ with C++23, this requires a newer CMake. (I have a patch for it, but I'm waiting for the last buildbots to be updated.)
It could be done with older CMake versions, but that is quite inconvenient.

Since we have a CI it will be trivial to test whether our supported compilers have sufficient C++23 support to build libc++

libcxx/docs/Modules.rst
84–99

This bullet point.

libcxx/modules/CMakeLists.txt.in
48 ↗(On Diff #512886)

This is the library for the module std. So next to the .pcm files there is a library component.

libcxx/modules/std/memory.cppm
183–184 ↗(On Diff #512886)

I'm not sure, but there are several errors in the modules. I copy-paste the synopis and manually cleaned them, so errors were always bound to occur.

There is a test module_std.sh.cpp which validates the declarations in the include files and in the modules.
This found quite some issues, some parts are not implemented in libc++, the synopsis only sometimes doesn't have all declarations (bitmask types for example), etc/
I'm not sure whether the test flags this one, but I have some patches to fix some of the issue found in libc++ itself.

Yeah we can't use new since it's a keyword.

Actually, what about renaming the new.cppm to __new.cppm? The issue with new.cppm is that you can't grep/ls all files in the libcxx/modules directory and extract the partition names from the filenames that way. Since the partition is named std:__new, naming the file same seems more natural.

The downside of this is that __new doesn't 1:1 correspond to the <new> header name. Since there is no include/__new in the same way that there is e.g. include/__algorithm there is little room for confusion though, and users can't directly import the partitions anyways.

ldionne added inline comments.Apr 20 2023, 10:08 AM
libcxx/test/std/time/time.syn/formatter_tests.h
22 ↗(On Diff #512886)

This doesn't seem necessary at all, it's included above.

I've just opened https://github.com/llvm/llvm-project/issues/62269 to track progress on the irreproducibility of BMIs. After thinking about this for a while, believe that this could be more of an issue than I initially thought. IMO we should add a warning that Module precompilation currently leaks absolute paths into BMIs.

ChuanqiXu added inline comments.Apr 20 2023, 6:59 PM
libcxx/docs/Modules.rst
84–99

Oh, this bullet has 3 issues. I thought you referred to one of them.

As discussed, let's split the cppm files into a separate review so we can get the bulk of the mechanical changes in.

libcxx/modules/std/format.cppm
10 ↗(On Diff #512886)

As discussed, I think this makes it too hard to distinguish what is a comment and what is an actual declaration. Thanks so much for doing the experiment -- I think it was a good idea on paper but it turns out to be a bit too hard to read.

libcxx/modules/std/memory.cppm
31 ↗(On Diff #512886)

This has been fixed now. Also some stuff in <mutex> and <utility>, see D145589.

Mordante updated this revision to Diff 517937.Apr 28 2023, 8:30 AM

Various cleanups
Documentation updates
Removed magic header from modules, this requires Clang 17

Mordante updated this revision to Diff 518456.May 1 2023, 8:39 AM

Removes some no longer needed work-arounds.
Adds a work-around for as_rvalue, which will be put in a separate review too.

Regarding compilation of the BMIs themselves the only issue left seems to be that <type_traits> doesn't export some symbols and <functional> needs to be included in its cppm at the moment. Other than that everything builds and std is importable.

Some std:: symbols still breaks in a few unsurprising places, e.g. format/ranges.

Herald added a project: Restricted Project. · View Herald TranscriptMay 7 2023, 11:53 AM
Herald added a reviewer: Restricted Project. · View Herald Transcript
Mordante marked 2 inline comments as done.May 7 2023, 12:24 PM

Regarding compilation of the BMIs themselves the only issue left seems to be that <type_traits> doesn't export some symbols and <functional> needs to be included in its cppm at the moment. Other than that everything builds and std is importable.

Some std:: symbols still breaks in a few unsurprising places, e.g. format/ranges.

The first part for functional is D149351 which moves these declarations in the proper primary header.
On my list there are some issues with invocable too.

Once these are done I can start to finalize the cppm files.

Mordante updated this revision to Diff 524029.May 20 2023, 5:31 AM

Polish the standard modules.

Mordante updated this revision to Diff 524030.May 20 2023, 5:34 AM

Rebased on main.

Mordante updated this revision to Diff 524034.May 20 2023, 7:39 AM

Final review of the std module

  • fixes some minor issues
  • enables the declarations that have been recently implemented by various contributors

I split of the .cppm files for the std module in D151030 once that lands I'll rebase this patch on top of that.

ldionne added inline comments.
libcxx/utils/use_modules_in_test.py
1 ↗(On Diff #524034)

Let's extract this to its own patch.

159 ↗(On Diff #524034)

Could we instead use this annotation in the test itself, with a relevant comment?

// UNSUPPORTED: use_module_std && !(c++11 || c++14 || c++17)

Or actually, we can use just

// UNSUPPORTED: use_module_std

since we never run with use_module_std in C++11/14/17.

169–176 ↗(On Diff #524034)

The fact that we need to disable these tests is quite interesting. I think it surfaces a pretty big problem with modules (as a language feature). Today, we use macros and compiler flags to customize the behavior of the library on a per translation-unit basis. That is an extremely important property of some aspects of the library (*). If we want everything to transition to modules (and we do), we need a way to satisfy those use cases. I don't mind whether that's done by defining macros, via some compiler flags or some other way, but those use cases are valid and we need to figure out how we're going to support them. Otherwise, we'll be left with large parts of our user base that can't switch over to C++ modules because they don't support some required use cases.

(*) For example, we use _LIBCPP_ENABLE_ASSERTIONS to control whether assertions are enabled or not on a per-TU basis. This is extremely important: when converting code bases to assertions, users might want to prevent assertions in some performance critical parts of the code, while still enabling assertions everywhere else. This approach allows for an incremental adoption of the feature and that's an important aspect of it being deployable in the real world.

I would like us to discuss avenues we could use to satisfy these use cases. For example, could we ship multiple "versions" of the library, each configured differently, and then the compiler would use a different version based on the compiler flags it is passed? I think pretty much all vendors are going to have similar questions, and that's going to severely limit how useful C++ modules can be in the real world. @jwakely @CaseyCarter I assume that's also something that you folks have thought about?

CC @ChuanqiXu

ChuanqiXu added inline comments.Jun 27 2023, 7:19 PM
libcxx/utils/use_modules_in_test.py
169–176 ↗(On Diff #524034)

(I am not 100% sure that I understand your question)

A background (I guess you already know this. But I'd like to repeat it again to avoid misunderstandings): we're going to ship (*.cppm) files. And the compilation of the std modules would happen in the users' machine. Also the compilation of the std modules may occur multiple times due to different configuration of the build. For example, the Release build and the Debug build will compile std modules twice with different options. And ideally the process would be controlled by the build systems (cmake).

The consensus in SG15 now is that we need to ship source files (*.cppm files) and a metadata file to describe the requirement/information of the library/module. Note that SG15 doesn't have a consensus (or even a feeling as far as I know) how the metadata should be.

So a temporary conclusion here is that possibly there is not an off-the-shelf practice that we can take directly. We would have to explore the method during the practice.

Also, I strongly suggest to send this question to SG15 and there are multiple experts who studied the problem for a relative long time. Maybe it will be more helpful than the small group discussion here.

CC: @ruoso @ben.boeckel

ldionne added inline comments.Jun 29 2023, 7:37 AM
libcxx/utils/use_modules_in_test.py
169–176 ↗(On Diff #524034)

we're going to ship (*.cppm) files. And the compilation of the std modules would happen in the users' machine.

Yes, we're on the same page. I'm not sure why I mentioned shipping various versions of the library, I was thinking about the need to pre-build multiple versions of the library for testing purposes (which is not related). Let me try to clarify my concern, and you can let me know whether I'm getting agitated for no good reason :-). Let's take for example _LIBCPP_ENABLE_ASSERTIONS. Today, that's a macro you define as -D_LIBCPP_ENABLE_ASSERTIONS=1 on the command-line or #define _LIBCPP_ENABLE_ASSERTIONS 1 as the first thing in your file, and the result is that libc++ functions like operator[] will be compiled with some assertions inside. This can be controlled on a per-TU basis today using the #include model we all know.

How do we do that in modules world if you're using import std instead? You can still pass -D_LIBCPP_ENABLE_ASSERTIONS=1 on the command-line, however that won't do anything because the compiler is using the BMI files that have already been compiled and _LIBCPP_ENABLE_ASSERTIONS was set to 0 when those were compiled. Right?

You can also still use #define _LIBCPP_ENABLE_ASSERTIONS=1 as the first thing in your file, but that also won't do anything because import std; doesn't have any way (or desire) to somehow resolve to a different BMI depending on the current macro environment. Right?

Basically, my question is: "What's the WG21-blessed alternative for this use case"? Are we expected to lift this mechanism from -D_LIBCPP_ENABLE_ASSERTIONS=1 to some other compiler flag like -flibcxx-enable-assertions (with a better name)? Then users would tell their build-system to build specific TUs using that flag, and that would result in different BMIs being compiled and used for those TUs? I think that would work, but it would create some challenges for sure.

If that's the expectation, then it means that we either need to build multiple configurations of the library BMIs during CMake invocation for the test suite to reuse, or lit needs to be able to somehow rebuild BMIs (or ask CMake to do it) based on the requirement of various tests in the test suite. For example, you'd tell lit that you're using -fsized-deallocation in your test, so it would go and rebuild the BMIs for libc++ with that set of compiler flags, and then set the appropriate substitution for that test to use the right BMIs for -fsized-deallocation. Possible, but fun to implement :-)

davidstone added inline comments.Jun 29 2023, 11:11 AM
libcxx/utils/use_modules_in_test.py
169–176 ↗(On Diff #524034)

The point that you would define _LIBCPP_ENABLE_ASSERTIONS=1 is when you compile the module in which you want to enable assertions. In other words, you can control it on a per-TU basis for the code that has the assertions, but not for the code using the code that has assertions. In general, you get ODR violations if the user's code is using multiple versions of library code.

So the user would not define the macro in their source file, nor would they (in CMake terms) associate the macro with their own target, they would either define the macro globally in their build system or associate it (somehow) with the standard library build target.

ChuanqiXu added inline comments.Jun 30 2023, 1:50 AM
libcxx/utils/use_modules_in_test.py
169–176 ↗(On Diff #524034)

@ldionne Oh, I got your point. You're mainly talking about testing instead of distributing. (IIUC).

How do we do that in modules world if you're using import std instead? You can still pass -D_LIBCPP_ENABLE_ASSERTIONS=1 on the command-line, however that won't do anything because the compiler is using the BMI files that have already been compiled and _LIBCPP_ENABLE_ASSERTIONS was set to 0 when those were compiled. Right?

You can also still use #define _LIBCPP_ENABLE_ASSERTIONS=1 as the first thing in your file, but that also won't do anything because import std; doesn't have any way (or desire) to somehow resolve to a different BMI depending on the current macro environment. Right?

Yes. This is by design. We don't want the build of modules to be affected by its users.

Basically, my question is: "What's the WG21-blessed alternative for this use case"?

The expected solution may be add a metadata file (which haven't been defined yet) to the library and the metadata file will contain informations about the interested flags of the library.

And if there are two (multiple) users use the same library and all the flags they used in the interesting flag set are the same, they can reuse the same BMI file. But if there is any different flag in the interesting flag set, they have to use different BMI file.

Let's come back to the example of testing of libcxx. Here every lit test with a configuration is a user of the std module. But the same lit test with different configuration are considered to be different users of the std module. And in your example, both -D_LIBCPP_ENABLE_ASSERTIONS and -fsized-deallocation are the interesting flags.

BTW, the intention of the design is that, (I guess you already know this too. Just repeat again to be informative. Sorry if this is wordy), the build system will rebuild the BMIs if the input compilation flag contains -DINTERESTING_MACRO=NEW_VALUE but the build system are capable to reuse the BMIs if the input compilation flag contains -DABCDEFG. This is important because almost every library has local macro definitions.

Given the lit tests are internal users, maybe we can assume all the flags to be interesting flags.

If that's the expectation, then it means that we either need to build multiple configurations of the library BMIs during CMake invocation for the test suite to reuse, or lit needs to be able to somehow rebuild BMIs (or ask CMake to do it) based on the requirement of various tests in the test suite.

If I read correctly, yes.

My mental model for the general case would be:

  1. the user (a lit test with a configuration in this example) would ask the build system about the location of the BMI file with a certain module name and a list of flags.
  2. The build systems will lookup if the appropriate BMI exists. If yes, return the location. If not, build it and return the location.

And in the testing case, maybe we need to implement a mechanism for lit and cmake to communicate with each other.

ChuanqiXu added inline comments.Jun 30 2023, 1:53 AM
libcxx/utils/use_modules_in_test.py
169–176 ↗(On Diff #524034)

We don't want the build of modules to be affected by its users.

We don't want the build of modules to be affected by its users surprisingly. That said, the build of the modules can be affected by its users if and only if in the way designed by the modules.

ben.boeckel added inline comments.Jul 1 2023, 3:56 PM
libcxx/utils/use_modules_in_test.py
169–176 ↗(On Diff #524034)

We don't want the build of modules to be affected by its users surprisingly.

Alas, consumers do need to agree on a lot of flags. We'll need at least a BMI per consuming standard. Probably per flag like -D_LIBCPP_* and the like, whatever -f and -m flags affect ABI, etc. I'd love for "I build module X one way and everyone agrees", but implementors have told me that it doesn't work that way. The ODR probably pops in at the other extreme as well.

Mordante added inline comments.Jul 2 2023, 8:15 AM
libcxx/utils/use_modules_in_test.py
169–176 ↗(On Diff #524034)

Is there a way to indicate BMIs are incompatible due to ODR violations? For example when the size of a struct changes depending on compiler flags used?

ben.boeckel added inline comments.Jul 2 2023, 9:09 AM
libcxx/utils/use_modules_in_test.py
169–176 ↗(On Diff #524034)

My understanding is that Clang stores the flags used to make the BMI in the BMI and if they mismatch "enough", it is incompatible. Given that…just about every flag does this (e.g., anything that affects __has_attribute, __has_builtin, or other preprocessor state), the BMI compatibility contract is very narrow. Named modules *might* be able to be a bit looser given how preprocessor state is far more isolated, but header units are going to leak these state differences far more easily.

ChuanqiXu added inline comments.Jul 2 2023, 7:12 PM
libcxx/utils/use_modules_in_test.py
169–176 ↗(On Diff #524034)

Currently clang will try to perform a strict check when loading BMIs. For example if a TU imports module A and B, and the same declaration in module A and B differ, clang will emit an error.

For example when the size of a struct changes depending on compiler flags used?

But this may not be possible IIUC.

@Mordante As your direction is 'lifting' exported names from the GMF into the module purview I've initiated core issue CWG2783 https://cplusplus.github.io/CWG/issues/2783.html to strengthen the foundations.

@Mordante As your direction is 'lifting' exported names from the GMF into the module purview I've initiated core issue CWG2783 https://cplusplus.github.io/CWG/issues/2783.html to strengthen the foundations.

Thanks @DanielaE! After our last discussion I had looking at this at my todo list.

Mordante abandoned this revision.Sep 11 2023, 10:21 AM

Since we move to GitHub PRs we're cleaning up our review queue. I abandon this patch and provide GitHub PRs for the not yet landed parts.