This makes sure that the PSTL integration doesn't regress. Eventually,
when we're ready to ship PSTL, this configuration will be enabled by
default in all builds and we can drop this additional CI configuration.
|3032–3037 ↗||(On Diff #348057)|
This change is necessary for the PSTL tests to succeed.
I am fairly certain that we can reduce that failure sufficiently to show that we actually have a bug in std::not_fn. @zoecarver would you be able to take a look since you implemented __perfect_forward_impl?
I am trying to update this review to continue enabling of PSTL CI with necessary modifications of the pipeline and fixes in PSTL source/tests, but I cannot do that. Seems like I don't have enough permissions to update the review I am not the author of. I am using web interface and when I am pressing "Update diff" I can only create the new revision and don't have an option to update the current one. Probably it's because I don't have direct Phabricator account (I have been using the GitHub one from the very beginning) or probably the reason is different. Together with @MikeDvorskiy we tried to do the same from his account (using web interface as well) and everything works just fine.
Could you please help me with the issue?
P.S. That problem was observed even earlier so, it seems I've never had such permission
How realistic is it that this still makes it into LLVM 15? I'd really like to start testing the pstl in our ecosystem a bit, happy to file bug reports too if so, but I'd need a way to be able to build it...
Unfortunately, I don't think we'll cherry-pick this to LLVM 15. Proper integration requires additional work, and that would be too much for LLVM 15 since we've already branched. We focused on Ranges and Format for LLVM 15. This should make it for LLVM 16 though.
I think the PSTL should be more tightly integrated with libc++, and we should probably be including libc++'s internal headers directly from it to break these kinds of circular dependencies.
This is going to be non-trivial though because there was an intent to share the PSTL between libc++ and libstdc++ -- I don't know whether that is still desired, and we may need to revisit this if we want to ease the integration between libc++ and PSTL.
@rodgert What is the current state of PSTL within libstdc++? How do you integrate it nowadays? Back in the days, I think you had a script that basically copied the sources to libstdc++ -- if that's still the case, do you have plans to keep it up to date, and can you link to that script so we can take a look at how the integration works?
We should be installing the PSTL headers to include/c++/v1 so that we find them alongside other libc++ headers out-of-the-box, and users don't need to add an additional include.
I suspect that means we'll have to start running the tests against a fake-installation of libc++/libc++abi/PSTL/whatever, which is what we should do anyways because it's closer to what users get. I've been meaning to do that for a while, I'll see if I can do this in the next few weeks.
This bit (and a few others) can land separately.
I pulled a new version in 2020 and have not rebased since. I had been waiting on the tag dispatch rewrite before considering the process again. I also not seen any progress on that since January, so maybe I will rethink matters if that continues.
I think any changes to include libc++'s internal headers would be detrimental to the continued to sharing of this codebase. I deliberately have not done that so far with libstdc++ (or I maintain those changes downstream only) because that would almost certainly introduce massive and likely unresolvable breakage for libc++.
I just had a discussion with @philnik about this patch. I have come to think that attempting to use the shared version of pstl inside libc++ was not a good idea to begin with.
Standard libraries have subtly different requirements and ways of doing things. For example, we use an ABI tag on our implementation-detail functions so that we have the freedom to change them however we please from release to release. Last I checked, I think libstdc++ handles this differently by not making pre/post condition changes even in implementation-detail functions. Those are both valid ways of doing things and I am not criticizing, however trying to share *header* code between the two standard libraries will surface these kinds of fundamental differences, some of which can't be reconciled with any amount of macros. Another example is circular header dependencies -- without being able to include implementation-detail headers of libc++ from the pstl, we may not be able to get rid of some circular dependencies, and we certainly won't be able to achieve the same level of granularity that we have in libc++. Yet another example is how we deal with removing incidental transitive includes -- we have a process for doing it, but it's not clear to me how we'd establish this in a shared pstl library.
In other words, I think that sharing header-level APIs between standard libraries is not a good idea. Sharing implementation that lives in a dylib or static archive is 100% fine since we can treat it as a third-party dependency. Sharing a well-defined backend implementation is also probably fine, but sharing headers is a different can of worms.
My opinion about this is even stronger after seeing what the decision of using an external code-base in our headers has given us so far -- not much. Because of the complexity of integrating pstl properly into libc++, we've basically been sitting on this and a few other difficult-ish questions ever since it was contributed, leading to us not shipping anything for years. Instead, if we had this code directly in libc++, I am confident we would have shipped *something* at this point and started making improvements to it.
@philnik I would suggest we abandon this revision and instead integrate the pstl directly into libc++, marking it as an experimental feature per our usual conventions. We should strive not to touch the backends too much so as to make it easy (but not necessarily automatic) to keep that in sync with the actual pstl code. That's where 90% of changes happen anyway. Rough suggested path:
- Copy over as-is to libc++
- Add a CI job that enables _LIBCPP_HAS_PARALLEL_ALGORITHMS -- no big CMake changes needed
- Clang-format the public API, but don't touch the back-ends since we're going to try to share that
- Apply libc++ macros as needed (e.g. _LIBCPP_HIDE_FROM_ABI)
- Use the right namespaces
- Get rid of pstl_config.h in favour of <__config>
- Granularize public API headers
- Now we should be able to simplify the glue like <__pstl_algorithm> inclusion to instead include our own granularized headers from <algorithm>
- We should be able to drop the custom _LIBCPP_HAS_PARALLEL_ALGORITHMS and use our existing machinery for experimental features
- We should also import the tests, since right now we're not testing anything
Doing that, we should strive to maintain the clear boundary between the backends and the public API. The public API is ours and we can change it -- the backend is where the real "work" is, and we want to share that with the pstl as much as possible since folks are making updates and improvements to it.
@rodgert This repo is not going to go anywhere, so this will not have an impact on your ability to keep taking updates made to pstl. We (libc++) will just start porting changes from pstl into libc++ manually when they happen (which isn't *that* often so far).
Sorry for disappearing. That was because of Intel business suspension in Russia. I was not allowed to work on that kind of stuff.
I am glad that things are moving forward with making Parallel algorithms available as a part of libc++.
Now I am available and ready to help.
Circling back on this -
I don't disagree with any of Louis' points. Regarding circular header
dependencies, I would also like to switch the PSTL to use libstdc++'s
internal implementation headers. I am not really sure how to do that and
still incorporate upstream changes going forward.
I expect to start rebasing libstdc++'s implementation on the current
version of the upstream next month.