[libc++] Include <__config_site> from <__config>

[phosek]

Summary

Prior to this patch, we would generate a fancy <config> header by
concatenating <config_site> and <__config>. This complexifies the
build system and also increases the difference between what's tested
and what's actually installed.

This patch removes that complexity and instead simply installs <config_site>
alongside the libc++ headers. <config_site> is then included by <config>,
which is much simpler. Doing this also opens the door to having different
<config_site> headers depending on the target, which was impossible before.

It does change the workflow for testing header-only changes to libc++.
Previously, we would run lit against the headers in libcxx/include.
After this patch, we run it against a fake installation root of the
headers (containing a proper <__config_site> header). This makes use
closer to testing what we actually install, which is good, however it
does mean that we have to update that root before testing header changes.
Thus, we now need to run ninja check-cxx-deps before running lit by
hand.

This commit was originally applied in 5d796645d and reverted in 48e4b0fd3
because it broke several bots. The issues should have been addressed now.

[phosek]

@ldionne this is an attempt at reviving D89041, I tested this locally and it seems to be working, at least for the Fuchsia runtimes build. It might break out some parts of this patch to make it more manageable, like the s/-isystem/-cxx-isystem/ in compiler-rt.

[ldionne]

I think it would be great to try it out again. To be honest, I've been meaning to try it out again (I had a similar patch locally), but I've been too scared to do so. Last time the world just exploded and I spent like a week trying to make heads or tail of the situation.

I'm entirely favorable to doing this, but can we try splitting out changes as much as possible? Ideally, we'd land everything before, and then at the end we'd land the libc++ changes, and only those.

Also, it would be awesome to set up a true runtimes build on our pre-commit CI infrastructure. The GCE instances we use could probably handle the load, they're very beefy. Would you be willing to help set that up?

[curdeius]

libcxx/cmake/Modules/HandleLibCXXABI.cmake

54–55

If it's always true, then the condition should be removed, no?

[ldionne]

libcxx/cmake/Modules/HandleLibCXXABI.cmake

54–55

I think at the time it was an attempt to make the diff as small as possible, since that change was already very risky. If we manage to split this out into smaller patches, then yeah, I think we could probably just remove this if and unconditionally execute its body.

[• Quuxplusone]

Minor drive-by comments. I'm extremely unlikely to voice either approval or disapproval in an official capacity. :)

libcxx/CMakeLists.txt

407

This comment is scary but uninformative. What "other projects" do you mean? By "these dependencies" do you mean the dependency from those other projects to the specific string "c++/v1" (in which case surely you should say They not We)? or by "these dependencies" do you mean one or more of lines 408-424?
Do you mean They should use LIBCXX_GENERATED_INCLUDE_DIR instead of hard-coding include/c++/v1?

libcxx/docs/TestingLibcxx.rst

44

Is running [ninja|make] cxx specifically not enough? (I'm guessing the answer is "yes, not enough.") That answer should be called out explicitly in the text.

libcxxabi/CMakeLists.txt

173 ↗

(On Diff #329712)

Lines 170 and 172 are identical except for /I versus -I. I know Microsoft's command-line compiler driver supports -I as a synonym for /I. So could we combine these five lines into one line using -I in both cases? Or is the slash character required because it's magic to CMake, or something like that?

[phosek]

libcxx/docs/TestingLibcxx.rst

44

This section specifically is about using llvm-lit directly, if you're using ninja or make you can just run check-cxx.

[phosek]

@ldionne I've already landed all changes that could be meaningfully carved out, would you be OK if we tried to land this again?

[phosek]

Looks like presubmit checks are passing now.

[ldionne]

In D97572#2648233, @phosek wrote:

Looks like presubmit checks are passing now.

I'm pretty sure we'll need the cxx-headers change in libcxxabi (D98367) to make this change, no? Otherwise, we might break some people building and testing libc++abi in funky ways. I'd also wait until https://reviews.llvm.org/D97888 has landed so that the pre-commit tests include the Runtimes build. But except for that, I think this is good and I'm excited/scared to try this again.

[phosek]

In D97572#2651312, @ldionne wrote:

In D97572#2648233, @phosek wrote:

Looks like presubmit checks are passing now.

I'm pretty sure we'll need the cxx-headers change in libcxxabi (D98367) to make this change, no? Otherwise, we might break some people building and testing libc++abi in funky ways. I'd also wait until https://reviews.llvm.org/D97888 has landed so that the pre-commit tests include the Runtimes build. But except for that, I think this is good and I'm excited/scared to try this again.

This should be ready to land now.

[ldionne]

LGTM with nitpick (and assuming CI comes back green).

I still think we should watch out for any unintended consequence of this change. Basically, anyone including libcxx/include instead of the properly installed headers will now run into issues after this change (since __config_site will not have been included and thus won't be found). That's why this change has so much possibility for breakage. However, once that's fixed, we'll be in a much better place knowing that all users are using the correctly installed (and hence configured) libc++ headers.

libcxx/docs/TestingLibcxx.rst

42

This target name has been renamed since the original patch. It's now cxx-test-depends.

[thakis]

We're seeing two test failures on macOS in bootstrap builds after this: https://bugs.chromium.org/p/chromium/issues/detail?id=1194745

[phosek]

In D97572#2662599, @thakis wrote:

We're seeing two test failures on macOS in bootstrap builds after this: https://bugs.chromium.org/p/chromium/issues/detail?id=1194745

D99706 should hopefully address this.

[muiez]

libcxx/cmake/Modules/HandleLibCXXABI.cmake

57

Wouldn't this copy the libc++abi headers to include/c++/v1/include/c++/v1/ (on z/OS for example)? I ask because LIBCXX_GENERATED_INCLUDE_DIR is already set to ${LLVM_BINARY_DIR}/include/c++/v1 above. Nonetheless, the old change copies them to include/c++/v1 instead. In other words, was this intended?

[phosek]

libcxx/cmake/Modules/HandleLibCXXABI.cmake

57

This was unintended, thanks for spotting it, I fixed it in rGeea5cbc8583d0857e0a9e429c61f7e87122b4dd6.

[MarcoFalke]

I suspect this broke the OSS-Fuzz integration

Step #4: + ./libcxx/utils/ci/oss-fuzz.sh
Step #4: In file included from libcxx/test/libcxx/fuzzing/make_heap.pass.cpp:11:
Step #4: In file included from libcxx/include/algorithm:648:
Step #4: libcxx/include/__config:13:10: fatal error: '__config_site' file not found
Step #4: #include <__config_site>
Step #4:          ^~~~~~~~~~~~~~~
Step #4: 1 error generated.
Step #4: ********************************************************************************
Step #4: Failed to build.
Step #4: To reproduce, run:
Step #4: python infra/helper.py build_image llvm_libcxx
Step #4: python infra/helper.py build_fuzzers --sanitizer address --engine libfuzzer --architecture x86_64 llvm_libcxx
Step #4: ********************************************************************************
Finished Step #4
ERROR
ERROR: build step 4 "gcr.io/oss-fuzz/llvm_libcxx" failed: step exited with non-zero status: 1

See https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=32788

[bjope]

I added some questions related to this in D103386 (as I started to see failures with not finding __config_siteafter that patch, but I guess it is related to this patch).

We been using -DCLANG_DEFAULT_RTLIB=compiler-rt -DCLANG_DEFAULT_CXX_STDLIB=libc++ when building clang. And we also use -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON.
We also build clang/llc for all targets, but we only build libcxx for a subset of targets (e.g. x86_64-unknown-linux-gnu).

When I only build libcxx for x86_64-unknown-linux-gnu, I only get a config_site file for that specific triple inside bin/../include/x86_64-unknown-linux-gnu/c++/v1/config_site in the build result.
But then when compiling for a target such as powerpc64le-unknown-linux-gnu (the test case added in D103386), when including bin/../include/c++/v1/stddef.h, that wants wants to include <__config_site>, I run into trouble because there is no such header for the powerpc64le-unknown-linux-gnu triple.

Is this kind of setup totally stupid and unsupported?
(I kind of understand that using libcxx for powerpc64le-unknown-linux-gnu when not building libcxx for that target kind of is like begging for problems, but I doubt the test case should depend on that, or if I override CLANG_DEFAULT_CXX_STDLIB)

Another problem is that a user was using my clang build together with "--target=x86_64-wrs-linux". This has worked fine in the past, but now when we only get "include/x86_64-unknown-linux-gnu/c++/v1/__config_site" in the compiler-clang install the user get failures about not finding __config_site. I could mention that this user is not even interested in linking the program, just using clang for static analysis. But the conclusion is that even for doing static analysis one need to match the target with one of the targets for which we have built libcxx.
Maybe this is as expected? Or maybe it has been forseen?

Should there perhaps always be some kind of generic __config_site that is used if there is not target override?

[bjope]

In D97572#2795853, @bjope wrote:

I added some questions related to this in D103386 (as I started to see failures with not finding __config_siteafter that patch, but I guess it is related to this patch).

We been using -DCLANG_DEFAULT_RTLIB=compiler-rt -DCLANG_DEFAULT_CXX_STDLIB=libc++ when building clang. And we also use -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON.
We also build clang/llc for all targets, but we only build libcxx for a subset of targets (e.g. x86_64-unknown-linux-gnu).

When I only build libcxx for x86_64-unknown-linux-gnu, I only get a config_site file for that specific triple inside bin/../include/x86_64-unknown-linux-gnu/c++/v1/config_site in the build result.
But then when compiling for a target such as powerpc64le-unknown-linux-gnu (the test case added in D103386), when including bin/../include/c++/v1/stddef.h, that wants wants to include <__config_site>, I run into trouble because there is no such header for the powerpc64le-unknown-linux-gnu triple.

Is this kind of setup totally stupid and unsupported?
(I kind of understand that using libcxx for powerpc64le-unknown-linux-gnu when not building libcxx for that target kind of is like begging for problems, but I doubt the test case should depend on that, or if I override CLANG_DEFAULT_CXX_STDLIB)

Another problem is that a user was using my clang build together with "--target=x86_64-wrs-linux". This has worked fine in the past, but now when we only get "include/x86_64-unknown-linux-gnu/c++/v1/__config_site" in the compiler-clang install the user get failures about not finding __config_site. I could mention that this user is not even interested in linking the program, just using clang for static analysis. But the conclusion is that even for doing static analysis one need to match the target with one of the targets for which we have built libcxx.
Maybe this is as expected? Or maybe it has been forseen?

Should there perhaps always be some kind of generic __config_site that is used if there is not target override?

I still have these questions. Is there anyone that could help clarifying this a bit, how is it supposed to work?

[phosek]

In D97572#2818674, @bjope wrote:

In D97572#2795853, @bjope wrote:

I added some questions related to this in D103386 (as I started to see failures with not finding __config_siteafter that patch, but I guess it is related to this patch).

We been using -DCLANG_DEFAULT_RTLIB=compiler-rt -DCLANG_DEFAULT_CXX_STDLIB=libc++ when building clang. And we also use -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON.
We also build clang/llc for all targets, but we only build libcxx for a subset of targets (e.g. x86_64-unknown-linux-gnu).

When I only build libcxx for x86_64-unknown-linux-gnu, I only get a config_site file for that specific triple inside bin/../include/x86_64-unknown-linux-gnu/c++/v1/config_site in the build result.
But then when compiling for a target such as powerpc64le-unknown-linux-gnu (the test case added in D103386), when including bin/../include/c++/v1/stddef.h, that wants wants to include <__config_site>, I run into trouble because there is no such header for the powerpc64le-unknown-linux-gnu triple.

Is this kind of setup totally stupid and unsupported?
(I kind of understand that using libcxx for powerpc64le-unknown-linux-gnu when not building libcxx for that target kind of is like begging for problems, but I doubt the test case should depend on that, or if I override CLANG_DEFAULT_CXX_STDLIB)

Another problem is that a user was using my clang build together with "--target=x86_64-wrs-linux". This has worked fine in the past, but now when we only get "include/x86_64-unknown-linux-gnu/c++/v1/__config_site" in the compiler-clang install the user get failures about not finding __config_site. I could mention that this user is not even interested in linking the program, just using clang for static analysis. But the conclusion is that even for doing static analysis one need to match the target with one of the targets for which we have built libcxx.
Maybe this is as expected? Or maybe it has been forseen?

Should there perhaps always be some kind of generic __config_site that is used if there is not target override?

I still have these questions. Is there anyone that could help clarifying this a bit, how is it supposed to work?

I missed your question earlier so I apologize for the belated response.

This is not the scenario that I considered when implementing this change but it makes sense. We could generate a generic fallback __config_site, my only concern is that it could potentially mask a case where the target __config_site is missing where today you'd get an include error whereas with the fallback __config_site you might see more subtle issues.

We could put it behind a CMake option that's going to be opt-in, for example LIBCXX_GENERIC_CONFIG_SITE, would that work?

@ldionne do you have any other ideas? Would this be OK with you?

[bjope]

In D97572#2835202, @phosek wrote:

In D97572#2818674, @bjope wrote:

In D97572#2795853, @bjope wrote:

I added some questions related to this in D103386 (as I started to see failures with not finding __config_siteafter that patch, but I guess it is related to this patch).

We been using -DCLANG_DEFAULT_RTLIB=compiler-rt -DCLANG_DEFAULT_CXX_STDLIB=libc++ when building clang. And we also use -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON.
We also build clang/llc for all targets, but we only build libcxx for a subset of targets (e.g. x86_64-unknown-linux-gnu).

When I only build libcxx for x86_64-unknown-linux-gnu, I only get a config_site file for that specific triple inside bin/../include/x86_64-unknown-linux-gnu/c++/v1/config_site in the build result.
But then when compiling for a target such as powerpc64le-unknown-linux-gnu (the test case added in D103386), when including bin/../include/c++/v1/stddef.h, that wants wants to include <__config_site>, I run into trouble because there is no such header for the powerpc64le-unknown-linux-gnu triple.

Is this kind of setup totally stupid and unsupported?
(I kind of understand that using libcxx for powerpc64le-unknown-linux-gnu when not building libcxx for that target kind of is like begging for problems, but I doubt the test case should depend on that, or if I override CLANG_DEFAULT_CXX_STDLIB)

Another problem is that a user was using my clang build together with "--target=x86_64-wrs-linux". This has worked fine in the past, but now when we only get "include/x86_64-unknown-linux-gnu/c++/v1/__config_site" in the compiler-clang install the user get failures about not finding __config_site. I could mention that this user is not even interested in linking the program, just using clang for static analysis. But the conclusion is that even for doing static analysis one need to match the target with one of the targets for which we have built libcxx.
Maybe this is as expected? Or maybe it has been forseen?

Should there perhaps always be some kind of generic __config_site that is used if there is not target override?

I still have these questions. Is there anyone that could help clarifying this a bit, how is it supposed to work?

I missed your question earlier so I apologize for the belated response.

This is not the scenario that I considered when implementing this change but it makes sense. We could generate a generic fallback __config_site, my only concern is that it could potentially mask a case where the target __config_site is missing where today you'd get an include error whereas with the fallback __config_site you might see more subtle issues.

We could put it behind a CMake option that's going to be opt-in, for example LIBCXX_GENERIC_CONFIG_SITE, would that work?

Just to understand, the idea is to put the generic file in include/c++/v1/ and rely on that if there is a target specific override that will be included instead (based on order of include paths)?

I do not have a problem with hiding such a thing behind a cmake option.

[ldionne]

In D97572#2835202, @phosek wrote:

We could put it behind a CMake option that's going to be opt-in, for example LIBCXX_GENERIC_CONFIG_SITE, would that work?

@ldionne do you have any other ideas? Would this be OK with you?

Stumbled upon this months later while clearing my email queue, I hope the discussion is still relevant.

@bjope What is the intent when compiling for --target=powerpc64le-unknown-linux-gnu with a library that has only been built for x86_64-unknown-linux-gnu? Is the intent that the __config_site used for powerpc64le-unknown-linux-gnu is the same as the one for x86_64-unknown-linux-gnu? To me, that sounds like something we can't really guess. I think what I'm wondering here is why you're using LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON at all.

Having a fallback is indeed tricky because we really want to avoid at all costs using the wrong __config_site by mistake (this could lead to incredibly tricky errors including ABI breaks). I would also really like to avoid adding an opt-in for this - there has to be a better solution.

[bjope]

In D97572#3015406, @ldionne wrote:

In D97572#2835202, @phosek wrote:

We could put it behind a CMake option that's going to be opt-in, for example LIBCXX_GENERIC_CONFIG_SITE, would that work?

@ldionne do you have any other ideas? Would this be OK with you?

Stumbled upon this months later while clearing my email queue, I hope the discussion is still relevant.

@bjope What is the intent when compiling for --target=powerpc64le-unknown-linux-gnu with a library that has only been built for x86_64-unknown-linux-gnu? Is the intent that the __config_site used for powerpc64le-unknown-linux-gnu is the same as the one for x86_64-unknown-linux-gnu? To me, that sounds like something we can't really guess. I think what I'm wondering here is why you're using LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON at all.

Having a fallback is indeed tricky because we really want to avoid at all costs using the wrong __config_site by mistake (this could lead to incredibly tricky errors including ABI breaks). I would also really like to avoid adding an opt-in for this - there has to be a better solution.

Not sure exactly the history related to using LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON, but out downstream compiler has support for a couple of different DSP targets and then we also support x86_64-unknown-linux-gnu (some users are basically using that for unittests running on the build server). So our release includes builtins/runtimes for those different targets.
However, we do not limit LLVM_TARGETS_TO_BUILD to only those target when we build/test/release the compiler, so clang will support targets such as powerpc (but we do not build/release any runtimes for powerpc).

If I recall correctly one of the problems that we started to see with this commit was that the test case clang/test/CodeGen/ppc-xmmintrin.c started to fail. That is the test case that is using powerpc64le-unknown-linux-gnu, and that would fail with fatal error: '__config_site' file not found.
So we could not use ninja check-all with that build setup.

Our current "workaround" is to avoid using -DCLANG_DEFAULT_CXX_STDLIB=libc++. So we kind of postponed the idea of building the compiler with libc++.

[mstorsjo]

In D97572#3015517, @bjope wrote:

If I recall correctly one of the problems that we started to see with this commit was that the test case clang/test/CodeGen/ppc-xmmintrin.c started to fail. That is the test case that is using powerpc64le-unknown-linux-gnu, and that would fail with fatal error: '__config_site' file not found.
So we could not use ninja check-all with that build setup.

Our current "workaround" is to avoid using -DCLANG_DEFAULT_CXX_STDLIB=libc++. So we kind of postponed the idea of building the compiler with libc++.

That sounds to me like that testcase needs to be adjusted so that it's not reliant on the surrounding setup - the clang tests shouldn't rely on clang itself having a working C++ standard library available at all, in general.

[phosek]

In D97572#3015537, @mstorsjo wrote:

In D97572#3015517, @bjope wrote:

If I recall correctly one of the problems that we started to see with this commit was that the test case clang/test/CodeGen/ppc-xmmintrin.c started to fail. That is the test case that is using powerpc64le-unknown-linux-gnu, and that would fail with fatal error: '__config_site' file not found.
So we could not use ninja check-all with that build setup.

Our current "workaround" is to avoid using -DCLANG_DEFAULT_CXX_STDLIB=libc++. So we kind of postponed the idea of building the compiler with libc++.

That sounds to me like that testcase needs to be adjusted so that it's not reliant on the surrounding setup - the clang tests shouldn't rely on clang itself having a working C++ standard library available at all, in general.

I agree, that looks like an issue with the test that should be addressed.

[ldionne]

Agreed with @phosek and @mstorsjo. Also, I looked at ppc-xmmintrin.c and traced down the includes, and I don't see what would cause this issue. @bjope if you'd like to try to -DCLANG_DEFAULT_CXX_STDLIB=libc++ again, I'll do my best to help you troubleshoot the issue. If you do, it's easiest to reach out to me on Discord in the #libcxx channel.

[bjope]

In D97572#3015742, @ldionne wrote:

Agreed with @phosek and @mstorsjo. Also, I looked at ppc-xmmintrin.c and traced down the includes, and I don't see what would cause this issue. @bjope if you'd like to try to -DCLANG_DEFAULT_CXX_STDLIB=libc++ again, I'll do my best to help you troubleshoot the issue. If you do, it's easiest to reach out to me on Discord in the #libcxx channel.

I actually started out by complaining about the test case here https://reviews.llvm.org/D103386#2790821 but I don't think it was corrected.

So that was one of the reasons for starting to ask questions in this review.

But I also wanted to know how this is supposed to work (or what shouldn't work) in a broader perspective. And there was also this other scenario that isn't helped out by adjusting a lit test in the llvm-project repo:

Another problem is that a user was using my clang build together with "--target=x86_64-wrs-linux". This has worked fine in the past, but now when we only get "include/x86_64-unknown-linux-gnu/c++/v1/__config_site" in the compiler-clang install the user get failures about not finding __config_site. I could mention that this user is not even interested in linking the program, just using clang for static analysis. But the conclusion is that even for doing static analysis one need to match the target with one of the targets for which we have built libcxx. Maybe this is as expected? Or maybe it has been forseen?

[nickdesaulniers]

Hi, I'm hacking on trying to bootstrap a build of clang from sources linked against musl. I'm hitting a curious set of failures trying to bootstrap libcxxabi:

llvm-project/libcxx/include/__config:13:10: fatal error: '__config_site' file not found
#include <__config_site>

@compnerd mentioned this commit. It no longer can be reverted cleanly, but if I check out the commit before this, I no longer have this error bootstrapping.

(This isn't necessarily a regression; my scripts never worked on the first place. I only started writing them recently; much later than when this landed. That said, bootstrapping seems a bit trick at the moment and it's not clear how to fix).

[MaskRay]

In D97572#3015920, @nickdesaulniers wrote:

Hi, I'm hacking on trying to bootstrap a build of clang from sources linked against musl. I'm hitting a curious set of failures trying to bootstrap libcxxabi:

llvm-project/libcxx/include/__config:13:10: fatal error: '__config_site' file not found
#include <__config_site>

@compnerd mentioned this commit. It no longer can be reverted cleanly, but if I check out the commit before this, I no longer have this error bootstrapping.

(This isn't necessarily a regression; my scripts never worked on the first place. I only started writing them recently; much later than when this landed. That said, bootstrapping seems a bit trick at the moment and it's not clear how to fix).

include/__config includes the generated include/x86_64-unknown-linux-gnu/c++/v1/__config_site.
__config_site has target specific customization.
clang --target=xxx using include/yyy/c++/v1/__config_site does not guarantee to work.

In practice, we can think that xxx and yyy may be similar enough and it may work if you just copy include/yyy/c++/v1/__config_site to include/xxx/c++/v1/__config_site

[ldionne]

@nickdesaulniers More generally, you must be setting -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON when you build? Or is that the default on Linux now? If you're setting it explicitly, then you probably don't want to be doing that. LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON is explicitly meant to have one include/ and lib/ dir for each target. If that's not your intent, then just use the good old way of doing things, i.e. just a single include/c++/v1 directory with all the headers, including the __config_site header. If LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON is the default on Linux, then perhaps you should explicitly specify LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF.

libcxx/CMakeLists.txt

 if (LIBCXX_ABI_FORCE_ITANIUM AND LIBCXX_ABI_FORCE_MICROSOFT)
   message(FATAL_ERROR "Only one of LIBCXX_ABI_FORCE_ITANIUM and LIBCXX_ABI_FORCE_MICROSOFT can be specified.")
 endif ()
 
 #===============================================================================
 # Configure System
 #===============================================================================
 
+# TODO: Projects that depend on libc++ should use LIBCXX_GENERATED_INCLUDE_DIR
+# instead of hard-coding include/c++/v1.
 if(LLVM_ENABLE_PER_TARGET_RUNTIME_DIR AND NOT APPLE)
   set(LIBCXX_LIBRARY_DIR ${LLVM_LIBRARY_OUTPUT_INTDIR}/${LLVM_DEFAULT_TARGET_TRIPLE}/c++)
-  set(LIBCXX_HEADER_DIR ${LLVM_BINARY_DIR})
+  set(LIBCXX_GENERATED_INCLUDE_DIR "${LLVM_BINARY_DIR}/include/c++/v1")
   set(LIBCXX_INSTALL_LIBRARY_DIR lib${LLVM_LIBDIR_SUFFIX}/${LLVM_DEFAULT_TARGET_TRIPLE}/c++)
   if(LIBCXX_LIBDIR_SUBDIR)
     string(APPEND LIBCXX_LIBRARY_DIR /${LIBCXX_LIBDIR_SUBDIR})
     string(APPEND LIBCXX_INSTALL_LIBRARY_DIR /${LIBCXX_LIBDIR_SUBDIR})
   endif()
 elseif(LLVM_LIBRARY_OUTPUT_INTDIR)
   set(LIBCXX_LIBRARY_DIR ${LLVM_LIBRARY_OUTPUT_INTDIR})
-  set(LIBCXX_HEADER_DIR  ${LLVM_BINARY_DIR})
+  set(LIBCXX_GENERATED_INCLUDE_DIR "${LLVM_BINARY_DIR}/include/c++/v1")
   set(LIBCXX_INSTALL_LIBRARY_DIR lib${LIBCXX_LIBDIR_SUFFIX})
 else()
   set(LIBCXX_LIBRARY_DIR ${CMAKE_BINARY_DIR}/lib${LIBCXX_LIBDIR_SUFFIX})
-  set(LIBCXX_HEADER_DIR  ${CMAKE_BINARY_DIR})
+  set(LIBCXX_GENERATED_INCLUDE_DIR "${CMAKE_BINARY_DIR}/include/c++/v1")
   set(LIBCXX_INSTALL_LIBRARY_DIR lib${LIBCXX_LIBDIR_SUFFIX})
 endif()
 
 file(MAKE_DIRECTORY "${LIBCXX_BINARY_INCLUDE_DIR}")
 
 set(LIBCXX_INSTALL_PREFIX "" CACHE STRING "Define libc++ destination prefix.")
 set(LIBCXX_INSTALL_HEADER_PREFIX "" CACHE STRING "Define libc++ header destination prefix.")
 

libcxx/benchmarks/CMakeLists.txt

 include(ExternalProject)
 include(CheckCXXCompilerFlag)
 
 #==============================================================================
 # Build Google Benchmark for libc++
 #==============================================================================
 
 set(CMAKE_FOLDER "${CMAKE_FOLDER}/Benchmarks")
 
 set(BENCHMARK_LIBCXX_COMPILE_FLAGS
     -Wno-unused-command-line-argument
     -nostdinc++
-    -isystem ${LIBCXX_SOURCE_DIR}/include
+    -isystem "${LIBCXX_GENERATED_INCLUDE_DIR}"
     -L${LIBCXX_LIBRARY_DIR}
     -Wl,-rpath,${LIBCXX_LIBRARY_DIR}
     ${SANITIZER_FLAGS}
     )
 if (DEFINED LIBCXX_CXX_ABI_LIBRARY_PATH)
   list(APPEND BENCHMARK_LIBCXX_COMPILE_FLAGS
           -L${LIBCXX_CXX_ABI_LIBRARY_PATH}
           -Wl,-rpath,${LIBCXX_CXX_ABI_LIBRARY_PATH})
 endif()
-list(APPEND BENCHMARK_LIBCXX_COMPILE_FLAGS -include "${LIBCXX_BINARY_DIR}/__config_site")
 split_list(BENCHMARK_LIBCXX_COMPILE_FLAGS)
 
 ExternalProject_Add(google-benchmark-libcxx
         EXCLUDE_FROM_ALL ON
         DEPENDS cxx cxx-headers
         PREFIX benchmark-libcxx
         SOURCE_DIR ${LIBCXX_SOURCE_DIR}/utils/google-benchmark
         INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/benchmark-libcxx

libcxx/cmake/Modules/HandleLibCXXABI.cmake

foreach(incpath ${LIBCXX_CXX_ABI_INCLUDE_PATHS})
         set(src ${incpath}/${fpath})
 
         set(dst ${LIBCXX_BINARY_INCLUDE_DIR}/${dstdir}/${ifile})
         add_custom_command(OUTPUT ${dst}
             DEPENDS ${src}
             COMMAND ${CMAKE_COMMAND} -E copy_if_different ${src} ${dst}
             COMMENT "Copying C++ ABI header ${fpath}...")
         list(APPEND abilib_headers "${dst}")
 
-        if (LIBCXX_HEADER_DIR)
-          set(dst "${LIBCXX_HEADER_DIR}/include/c++/v1/${dstdir}/${fpath}")
-          add_custom_command(OUTPUT ${dst}
-              DEPENDS ${src}
-              COMMAND ${CMAKE_COMMAND} -E copy_if_different ${src} ${dst}
-              COMMENT "Copying C++ ABI header ${fpath}...")
-          list(APPEND abilib_headers "${dst}")
-        endif()
+        # TODO: libc++ shouldn't be responsible for copying the libc++abi
+        # headers into the right location.
+        set(dst "${LIBCXX_GENERATED_INCLUDE_DIR}/include/c++/v1/${dstdir}/${fpath}")
 
         if (LIBCXX_INSTALL_HEADERS)
           install(FILES "${LIBCXX_BINARY_INCLUDE_DIR}/${fpath}"
             DESTINATION ${LIBCXX_INSTALL_HEADER_PREFIX}include/c++/v1/${dstdir}
             COMPONENT cxx-headers
             PERMISSIONS OWNER_READ OWNER_WRITE GROUP_READ WORLD_READ
             )
         endif()

libcxx/docs/TestingLibcxx.rst

 .. code-block:: bash
 
   $ cd <monorepo-root>
   $ make -C <build> cxx-test-depends # If you want to make sure the targets get rebuilt
   $ <build>/bin/llvm-lit -sv libcxx/test/std/re # Run all of the std::regex tests
   $ <build>/bin/llvm-lit -sv libcxx/test/std/depr/depr.c.headers/stdlib_h.pass.cpp # Run a single test
   $ <build>/bin/llvm-lit -sv libcxx/test/std/atomics libcxx/test/std/threads # Test std::thread and std::atomic
 
+In the default configuration, the tests are built against headers that form a
+fake installation root of libc++. This installation root has to be updated when
+changes are made to the headers, so you should re-run the `check-cxx-deps` target
+before running the tests manually with `lit` when you make any sort of change,
+including to the headers.
+
 Sometimes you'll want to change the way LIT is running the tests. Custom options
 can be specified using the `--param=<name>=<val>` flag. The most common option
 you'll want to change is the standard dialect (ie -std=c++XX). By default the
 test suite will select the newest C++ dialect supported by the compiler and use
 that. However if you want to manually specify the option like so:
 
 .. code-block:: bash
 

libcxx/include/CMakeLists.txt

 set(files
   __availability
   __bit_reference
   __bits
   __bsd_locale_defaults.h
   __bsd_locale_fallbacks.h
+  __config
   __debug
   __errc
   __functional_03
   __functional_base
   __functional_base_03
   __hash_table
   __libcpp_version
   __locale
set(files
   valarray
   variant
   vector
   version
   wchar.h
   wctype.h
   )
 
-configure_file("__config_site.in"
-               "${LIBCXX_BINARY_DIR}/__config_site"
-               @ONLY)
+configure_file("__config_site.in" "${LIBCXX_GENERATED_INCLUDE_DIR}/__config_site" @ONLY)
 
-# Generate a custom __config header. The new header is created
-# by prepending __config_site to the current __config header.
-add_custom_command(OUTPUT ${LIBCXX_BINARY_DIR}/__generated_config
-  COMMAND ${Python3_EXECUTABLE} ${LIBCXX_SOURCE_DIR}/utils/cat_files.py
-    ${LIBCXX_BINARY_DIR}/__config_site
-    ${LIBCXX_SOURCE_DIR}/include/__config
-    -o ${LIBCXX_BINARY_DIR}/__generated_config
-  DEPENDS ${LIBCXX_SOURCE_DIR}/include/__config
-          ${LIBCXX_BINARY_DIR}/__config_site
-)
-# Add a target that executes the generation commands.
-add_custom_target(cxx-generated-config ALL
-  DEPENDS ${LIBCXX_BINARY_DIR}/__generated_config)
-
-if(LIBCXX_HEADER_DIR)
-  set(output_dir ${LIBCXX_HEADER_DIR}/include/c++/v1)
-
-  set(out_files)
-  foreach(f ${files})
-    set(src ${CMAKE_CURRENT_SOURCE_DIR}/${f})
-    set(dst ${output_dir}/${f})
-    add_custom_command(OUTPUT ${dst}
-      DEPENDS ${src}
-      COMMAND ${CMAKE_COMMAND} -E copy_if_different ${src} ${dst}
-      COMMENT "Copying CXX header ${f}")
-    list(APPEND out_files ${dst})
-  endforeach()
+set(_all_includes "${LIBCXX_GENERATED_INCLUDE_DIR}/__config_site")
+  set(src "${CMAKE_CURRENT_SOURCE_DIR}/${f}")
+  set(dst "${LIBCXX_GENERATED_INCLUDE_DIR}/${f}")
+  list(APPEND _all_includes "${dst}")
 
-  # Copy the generated header as __config into build directory.
-  set(src ${LIBCXX_BINARY_DIR}/__generated_config)
-  set(dst ${output_dir}/__config)
-  add_custom_command(OUTPUT ${dst}
-      DEPENDS ${src} cxx-generated-config
-      COMMAND ${CMAKE_COMMAND} -E copy_if_different ${src} ${dst}
-      COMMENT "Copying CXX __config")
-  list(APPEND out_files ${dst})
-  add_custom_target(generate-cxx-headers ALL DEPENDS ${out_files})
+add_custom_target(generate-cxx-headers DEPENDS ${_all_includes})
 
-  add_library(cxx-headers INTERFACE)
-  add_dependencies(cxx-headers generate-cxx-headers ${LIBCXX_CXX_ABI_HEADER_TARGET})
-  # TODO: Use target_include_directories once we figure out why that breaks the runtimes build
-  if("${CMAKE_CXX_COMPILER_ID}" STREQUAL "MSVC" OR "${CMAKE_CXX_SIMULATE_ID}" STREQUAL "MSVC")
-    target_compile_options(cxx-headers INTERFACE /I "${output_dir}")
-  else()
-    target_compile_options(cxx-headers INTERFACE -I "${output_dir}")
-  endif()
-
-  # Make sure the generated __config_site header is included when we build the library.
-  if("${CMAKE_CXX_COMPILER_ID}" STREQUAL "MSVC" OR "${CMAKE_CXX_SIMULATE_ID}" STREQUAL "MSVC")
-    target_compile_options(cxx-headers INTERFACE /FI "${LIBCXX_BINARY_DIR}/__config_site")
-  else()
-    target_compile_options(cxx-headers INTERFACE -include "${LIBCXX_BINARY_DIR}/__config_site")
-  endif()
-else()
-  add_library(cxx-headers INTERFACE)
+  target_compile_options(cxx-headers INTERFACE /I "${LIBCXX_GENERATED_INCLUDE_DIR}")
+  target_compile_options(cxx-headers INTERFACE -I "${LIBCXX_GENERATED_INCLUDE_DIR}")
 endif()
 
 if (LIBCXX_INSTALL_HEADERS)
   foreach(file ${files})
     get_filename_component(dir ${file} DIRECTORY)
     install(FILES ${file}
       DESTINATION ${LIBCXX_INSTALL_HEADER_PREFIX}include/c++/v1/${dir}
       COMPONENT cxx-headers
       PERMISSIONS OWNER_READ OWNER_WRITE GROUP_READ WORLD_READ
     )
   endforeach()
 
-  # Install the generated header as __config.
-  install(FILES ${LIBCXX_BINARY_DIR}/__generated_config
+  # Install the generated __config_site.
+  install(FILES ${LIBCXX_GENERATED_INCLUDE_DIR}/__config_site
     DESTINATION ${LIBCXX_INSTALL_HEADER_PREFIX}include/c++/v1
     PERMISSIONS OWNER_READ OWNER_WRITE GROUP_READ WORLD_READ
-    RENAME __config
     COMPONENT cxx-headers)
 
   if (NOT CMAKE_CONFIGURATION_TYPES)
     add_custom_target(install-cxx-headers
-                      DEPENDS cxx-headers cxx-generated-config
+                      DEPENDS cxx-headers
                       COMMAND "${CMAKE_COMMAND}"
                               -DCMAKE_INSTALL_COMPONENT=cxx-headers
                               -P "${CMAKE_BINARY_DIR}/cmake_install.cmake")
     # Stripping is a no-op for headers
     add_custom_target(install-cxx-headers-stripped DEPENDS install-cxx-headers)
   endif()
 endif()

libcxx/include/__config

 // -*- C++ -*-
 //===--------------------------- __config ---------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef _LIBCPP_CONFIG
 #define _LIBCPP_CONFIG
 
+#include <__config_site>
+
 #if defined(_MSC_VER) && !defined(__clang__)
 #  if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #    define _LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER
 #  endif
 #endif
 
 #ifndef _LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER
 #pragma GCC system_header

libcxx/test/configs/legacy.cfg.in

 @AUTO_GEN_COMMENT@
 
 import os
 import site
 
+config.cxx_headers              = "@LIBCXX_GENERATED_INCLUDE_DIR@"
 config.cxx_under_test           = "@CMAKE_CXX_COMPILER@"
 config.project_obj_root         = "@CMAKE_BINARY_DIR@"
 config.libcxx_src_root          = "@LIBCXX_SOURCE_DIR@"
 config.libcxx_obj_root          = "@LIBCXX_BINARY_DIR@"
 config.cxx_library_root         = "@LIBCXX_LIBRARY_DIR@"
 config.abi_library_root         = "@LIBCXX_CXX_ABI_LIBRARY_PATH@"
 config.enable_exceptions        = @LIBCXX_ENABLE_EXCEPTIONS@
 config.enable_debug_tests       = @LIBCXX_ENABLE_DEBUG_MODE_SUPPORT@

libcxx/utils/ci/run-buildbot

legacy-standalone)
 
     cmake -S "${MONOREPO_ROOT}/libcxxabi" \
           -B "${BUILD_DIR}/libcxxabi" \
           -GNinja -DCMAKE_MAKE_PROGRAM="${NINJA}" \
           -DCMAKE_BUILD_TYPE=RelWithDebInfo \
           -DCMAKE_INSTALL_PREFIX="${INSTALL_DIR}" \
           -DLLVM_PATH="${MONOREPO_ROOT}/llvm" \
           -DLIBCXXABI_LIBCXX_PATH="${MONOREPO_ROOT}/libcxx" \
-          -DLIBCXXABI_LIBCXX_INCLUDES="${MONOREPO_ROOT}/libcxx/include" \
+          -DLIBCXXABI_LIBCXX_INCLUDES="${BUILD_DIR}/libcxx/include/c++/v1" \
           -DLIBCXXABI_LIBCXX_LIBRARY_PATH="${BUILD_DIR}/libcxx/lib"
 
+    echo "+++ Generating libc++ headers"
+    ${NINJA} -vC "${BUILD_DIR}/libcxx" generate-cxx-headers
+
     echo "+++ Building libc++abi"
     ${NINJA} -vC "${BUILD_DIR}/libcxxabi" cxxabi
 
     echo "+++ Building libc++"
     ${NINJA} -vC "${BUILD_DIR}/libcxx" cxx
 
     echo "+++ Running the libc++ tests"
     ${NINJA} -vC "${BUILD_DIR}/libcxx" check-cxx

libcxx/utils/libcxx/test/config.py

def configure_default_compile_flags(self):
         if pstl_src_root is not None and pstl_obj_root is not None:
             self.cxx.compile_flags += ['-I' + os.path.join(pstl_src_root, 'include')]
             self.cxx.compile_flags += ['-I' + os.path.join(pstl_obj_root, 'generated_headers')]
             self.cxx.compile_flags += ['-I' + os.path.join(pstl_src_root, 'test')]
             self.config.available_features.add('parallel-algorithms')
 
     def configure_compile_flags_header_includes(self):
         support_path = os.path.join(self.libcxx_src_root, 'test', 'support')
-        self.configure_config_site_header()
         if self.cxx_stdlib_under_test != 'libstdc++' and \
            not self.target_info.is_windows() and \
            not self.target_info.is_zos():
             self.cxx.compile_flags += [
                 '-include', os.path.join(support_path, 'nasty_macros.h')]
         if self.cxx_stdlib_under_test == 'msvc':
             self.cxx.compile_flags += [
                 '-include', os.path.join(support_path,
                                          'msvc_stdlib_force_include.h')]
             pass
         if self.target_info.is_windows() and self.debug_build and \
                 self.cxx_stdlib_under_test != 'msvc':
             self.cxx.compile_flags += [
                 '-include', os.path.join(support_path,
                                          'set_windows_crt_report_mode.h')
             ]
         cxx_headers = self.get_lit_conf('cxx_headers')
-        if cxx_headers == '' or (cxx_headers is None
-                                 and self.cxx_stdlib_under_test != 'libc++'):
+        if cxx_headers is None and self.cxx_stdlib_under_test != 'libc++':
             self.lit_config.note('using the system cxx headers')
             return
         self.cxx.compile_flags += ['-nostdinc++']
-        if cxx_headers is None:
-            cxx_headers = os.path.join(self.libcxx_src_root, 'include')
         if not os.path.isdir(cxx_headers):
-            self.lit_config.fatal("cxx_headers='%s' is not a directory."
-                                  % cxx_headers)
+            self.lit_config.fatal("cxx_headers='{}' is not a directory.".format(cxx_headers))
         self.cxx.compile_flags += ['-I' + cxx_headers]
         if self.libcxx_obj_root is not None:
             cxxabi_headers = os.path.join(self.libcxx_obj_root, 'include',
                                           'c++build')
             if os.path.isdir(cxxabi_headers):
                 self.cxx.compile_flags += ['-I' + cxxabi_headers]
 
-    def configure_config_site_header(self):
-        # Check for a possible __config_site in the build directory. We
-        # use this if it exists.
-        if self.libcxx_obj_root is None:
-            return
-        config_site_header = os.path.join(self.libcxx_obj_root, '__config_site')
-        if not os.path.isfile(config_site_header):
-            return
-        self.cxx.compile_flags += ['-include', config_site_header]
-
     def configure_link_flags(self):
         # Configure library path
         self.configure_link_flags_cxx_library_path()
         self.configure_link_flags_abi_library_path()
 
         # Configure libraries
         if self.cxx_stdlib_under_test == 'libc++':
             if self.target_info.is_mingw():

libcxxabi/test/libcxxabi/test/config.py

def configure_compile_flags(self):
         if self.get_lit_bool('enable_exceptions', True):
             self.cxx.compile_flags += ['-funwind-tables']
         if not self.get_lit_bool('enable_threads', True):
             self.cxx.compile_flags += ['-D_LIBCXXABI_HAS_NO_THREADS']
             self.config.available_features.add('libcxxabi-no-threads')
         super(Configuration, self).configure_compile_flags()
 
     def configure_compile_flags_header_includes(self):
-        self.configure_config_site_header()
         cxx_headers = self.get_lit_conf('cxx_headers', None) or \
             os.path.join(self.libcxxabi_hdr_root, 'include', 'c++', 'v1')
         if cxx_headers == '':
             self.lit_config.note('using the systems c++ headers')
         else:
             self.cxx.compile_flags += ['-nostdinc++']
         if not os.path.isdir(cxx_headers):
             self.lit_config.fatal("cxx_headers='%s' is not a directory."

libunwind/test/libunwind/test/config.py

def configure_compile_flags(self):
         if 'linux' in self.config.target_triple:
             self.cxx.link_flags += ['-Wl,--export-dynamic']
         if not self.get_lit_bool('enable_threads', True):
             self.cxx.compile_flags += ['-D_LIBUNWIND_HAS_NO_THREADS']
             self.config.available_features.add('libunwind-no-threads')
         super(Configuration, self).configure_compile_flags()
 
     def configure_compile_flags_header_includes(self):
-        self.configure_config_site_header()
-
         libunwind_headers = self.get_lit_conf(
             'libunwind_headers',
             os.path.join(self.libunwind_src_root, 'include'))
         if not os.path.isdir(libunwind_headers):
             self.lit_config.fatal("libunwind_headers='%s' is not a directory."
                                   % libunwind_headers)
         self.cxx.compile_flags += ['-I' + libunwind_headers]
 
     def configure_link_flags_cxx_library(self):
         # libunwind tests should not link with libc++
         pass
 
     def configure_link_flags_abi_library(self):
         # libunwind tests should not link with libc++abi
         pass