[libcxx] Support per-target __config_site in per-target runtime build

[phosek]

Summary

When using the per-target runtime build, it may be desirable to have
different __config_site headers for each target where all targets cannot
share a single configuration.

The layout used for libc++ headers after this change is:

include/
  c++/
    v1/
      <libc++ headers except for __config_site>
  <target1>/
    c++/
      v1/
        __config_site
  <target2>/
    c++/
      v1/
        __config_site
  <other targets>

This is the most optimal layout since it avoids duplication, the only
headers that's per-target is __config_site, all other headers are
shared across targets. This also means that we no need two
-isystem flags: one for the target-agnostic headers and one for
the target specific headers.

[phosek]

This is the per-target __config (or more preferable __config_site) support I mentioned in D88843.

[phosek]

@ldionne this is ready for review now that D97572 landed.

[MaskRay]

Regarding

/usr/include/x86_64-unknown-linux-gnu/c++/v1
/usr/include/c++/v1

With GCC multiarch, some include search paths are preceded by machine-os-env specific suffix directories.
Note: 'vendor' is not there. So you may see .../include/x86_64-linux-gnu but there is no .../include/x86_64-{pc,unknown}-linux-gnu

/usr/local/include/x86_64-linux-gnu         # affected by sysroot, multiarch, usually nonexistent
/usr/local/include                          # affected by sysroot
/tmp/opt/gcc-debug/include
/tmp/opt/gcc-debug/lib/gcc/x86_64-pc-linux-gnu/11.0.1/include-fixed
/tmp/opt/gcc-debug/lib/gcc/x86_64-pc-linux-gnu/11.0.1/../../../../x86_64-pc-linux-gnu/include
/usr/include/x86_64-linux-gnu               # affected by sysroot, multiarch
/usr/include                                # affected by sysroot

On Debian x86_64, /usr/include/x86_64-linux-gnu/ exists. Adding x86_64-pc-linux-gnu (an possible value of LLVM_DEFAULT_TARGET_TRIPLE) may add more confusion.

[phosek]

In D89013#2660407, @MaskRay wrote:

Regarding

/usr/include/x86_64-unknown-linux-gnu/c++/v1
/usr/include/c++/v1

With GCC multiarch, some include search paths are preceded by machine-os-env specific suffix directories.
Note: 'vendor' is not there. So you may see .../include/x86_64-linux-gnu but there is no .../include/x86_64-{pc,unknown}-linux-gnu

/usr/local/include/x86_64-linux-gnu         # affected by sysroot, multiarch, usually nonexistent
/usr/local/include                          # affected by sysroot
/tmp/opt/gcc-debug/include
/tmp/opt/gcc-debug/lib/gcc/x86_64-pc-linux-gnu/11.0.1/include-fixed
/tmp/opt/gcc-debug/lib/gcc/x86_64-pc-linux-gnu/11.0.1/../../../../x86_64-pc-linux-gnu/include
/usr/include/x86_64-linux-gnu               # affected by sysroot, multiarch
/usr/include                                # affected by sysroot

On Debian x86_64, /usr/include/x86_64-linux-gnu/ exists. Adding x86_64-pc-linux-gnu (an possible value of LLVM_DEFAULT_TARGET_TRIPLE) may add more confusion.

This was already discussed pretty extensively in the past, but I'll try to give a summary.

In the case of GCC, compiler can only compile binaries for a single target that's set at configuration time. For example if you configured GCC with --target=x86_64-acme-linux-gnu, the build links the binary as x86_64-acme-linux-gnu-gcc and that binary would look for headers in /usr/include/x86_64-acme-linux-gnu and libraries in /usr/lib/x86_64-acme-linux-gnu (among other paths) so there's little ambiguity.

In the case of Clang, the situation is a bit more difficult because a single compiler supports multiple targets via --target= option and there are different ways to spell the same target triple. For example, x86_64-linux-gnu and x86_64-unknown-linux-gnu are considered equivalent, the latter is a normalized version of the former (since you can omit emit components). The question then is where to look for headers and libraries. Currently, we always look in two locations: (1) we use --target=<triple as specified by the user and (2) normalized version of that target. For example if the user passes --target=x86_64-linux-gnu, we would first check <prefix>/lib/x86_64-linux-gnu and then <prefix>/lib/x86_64-unknown-linux-gnu.

In Fuchsia, we always prefer using normalized triples because that's less error prone. For example, in the past we would install libraries in <prefix>/lib/x86_64-fuchsia which works fine if you invoke the compiler with --target=x86_64-fuchsia, but if you instead use --target=x86_64-unknown-fuchsia (which some build systems would do), compiler would fail to find libraries because it doesn't know how to denormalize the triple. Using normalized triples addresses the issue because we can always normalize the triple to get a canonical spelling.

[MaskRay]

In D89013#2662167, @phosek wrote:

In D89013#2660407, @MaskRay wrote:

Regarding

/usr/include/x86_64-unknown-linux-gnu/c++/v1
/usr/include/c++/v1

With GCC multiarch, some include search paths are preceded by machine-os-env specific suffix directories.
Note: 'vendor' is not there. So you may see .../include/x86_64-linux-gnu but there is no .../include/x86_64-{pc,unknown}-linux-gnu

/usr/local/include/x86_64-linux-gnu         # affected by sysroot, multiarch, usually nonexistent
/usr/local/include                          # affected by sysroot
/tmp/opt/gcc-debug/include
/tmp/opt/gcc-debug/lib/gcc/x86_64-pc-linux-gnu/11.0.1/include-fixed
/tmp/opt/gcc-debug/lib/gcc/x86_64-pc-linux-gnu/11.0.1/../../../../x86_64-pc-linux-gnu/include
/usr/include/x86_64-linux-gnu               # affected by sysroot, multiarch
/usr/include                                # affected by sysroot

On Debian x86_64, /usr/include/x86_64-linux-gnu/ exists. Adding x86_64-pc-linux-gnu (an possible value of LLVM_DEFAULT_TARGET_TRIPLE) may add more confusion.

This was already discussed pretty extensively in the past, but I'll try to give a summary.

In the case of GCC, compiler can only compile binaries for a single target that's set at configuration time. For example if you configured GCC with --target=x86_64-acme-linux-gnu, the build links the binary as x86_64-acme-linux-gnu-gcc and that binary would look for headers in /usr/include/x86_64-acme-linux-gnu and libraries in /usr/lib/x86_64-acme-linux-gnu (among other paths) so there's little ambiguity.

I'd love if vendor is in the multiarch string, i.e. /usr/include/x86_64-acme-linux-gnu and /usr/lib/x86_64-acme-linux-gnu, but unfortunately that is not the case. See my previous dump with a GCC configured with x86_64-pc-linux-gnu. Note that some paths are derived from the GCC installation, which has vendor part, while others are derived from $multiarch, which has no vendor part.

The inconsistency between the proposed /usr/include/x86_64-unknown-linux-gnu/c++/v1 and /usr/include/x86_64-linux-gnu is something I want to avoid.

In the case of Clang, the situation is a bit more difficult because a single compiler supports multiple targets via --target= option and there are different ways to spell the same target triple. For example, x86_64-linux-gnu and x86_64-unknown-linux-gnu are considered equivalent, the latter is a normalized version of the former (since you can omit emit components). The question then is where to look for headers and libraries. Currently, we always look in two locations: (1) we use --target=<triple as specified by the user and (2) normalized version of that target. For example if the user passes --target=x86_64-linux-gnu, we would first check <prefix>/lib/x86_64-linux-gnu and then <prefix>/lib/x86_64-unknown-linux-gnu.

x86_64-linux-gnu and x86_64-unknown-linux-gnu are mostly equivalent, but they matter for the driver constructed paths from GCC installation (see the ../../.. path components in /tmp/opt/gcc-debug/lib/gcc/x86_64-pc-linux-gnu/11.0.1/../../../../lib64).

I think it makes sense to use un-normalized (--target=) for both lib/gcc/$triple/11.0.1 derived paths and /usr/include/$triple and /usr/lib/$triple and such a change will likely make our rules more reasonable (and more useful if people do want to leverage vendor for different directories).
We can then drop clang/lib/Driver/ToolChains/Linux.cpp Linux::getMultiarchTriple.
But as it stands, Linux::getMultiarchTriple constructed triples have no vendor part, so this patch /usr/include/c++/$triple/c++/v1 will introduce some inconsistency.

The change will look like breaking but it actually works on Debian/Ubuntu and their derivatives (--enable-multi-arch GCC). Because their GCC installation paths have no vendor part, e.g. lib/gcc/x86_64-linux-gnu.

Summary:

• clang a.cc => auto detection still works and will find lib/gcc/x86_64-linux-gnu
• clang --target=x86_64-linux-gnu a.cc => find lib/gcc/x86_64-linux-gnu
• clang --target=x86_64-unknown-linux-gnu a.cc => found lib/gcc/x86_64-linux-gnu with the current behavior, but unsupported in the future. I don't think people do this today.

I believe this change will not need anything on @thakis' side but just wanted to give him a heads-up.

In Fuchsia, we always prefer using normalized triples because that's less error prone. For example, in the past we would install libraries in <prefix>/lib/x86_64-fuchsia which works fine if you invoke the compiler with --target=x86_64-fuchsia, but if you instead use --target=x86_64-unknown-fuchsia (which some build systems would do), compiler would fail to find libraries because it doesn't know how to denormalize the triple. Using normalized triples addresses the issue because we can always normalize the triple to get a canonical spelling.

[ldionne]

I'm personally not so concerned with the Clang driver side of things, but primarily with adding more complexity to the libc++ build. Shouldn't we be driving things from the runtimes build and setting a different CMAKE_INSTALL_PREFIX to install stuff to the right location instead? Then you can run N builds of libc++ for N targets, each setting the appropriate install path.

clang/lib/Driver/ToolChains/Fuchsia.cpp

387

Doesn't that break #include_next if both directories exist? Generally speaking, I think you only ever want to have a single directory containing the libc++ headers on your header search path.

[phosek]

In D89013#2662434, @ldionne wrote:

I'm personally not so concerned with the Clang driver side of things, but primarily with adding more complexity to the libc++ build. Shouldn't we be driving things from the runtimes build and setting a different CMAKE_INSTALL_PREFIX to install stuff to the right location instead? Then you can run N builds of libc++ for N targets, each setting the appropriate install path.

We could do that but problem with the approach is that we would end up duplicating all headers across targets. Since all headers except __config_site should be the same, we only want to have per-target version of that header which is what this change is doing. The final directory layout you end up with:

include/
  c++/
    v1/
      <headers except for __config_site>
  x86_64-linux-gnu/
    c++/
      v1/
        __config_site
  <other targets>

This is the most optimal layout but I don't think it's something we can achieve purely with CMAKE_INSTALL_PREFIX without additional build support.

[phosek]

clang/lib/Driver/ToolChains/Fuchsia.cpp

387

See my comment, the per-target include path will only contain __config_site whereas the generic one contains all the headers other than __config_site so this works fine. It would even work if you had another copy of __config_site in the generic one which may be useful on platforms that don't currently support the multiarch layout like Darwin.

[phosek]

@ldionne I have update the change description to explain the new layout.

[MaskRay]

This layout matches the layout use by other compilers like GCC.

Clarification: this is similar to but not match GCC's layout, where the multiarch path component (e.g. /x86_64-linux-gnu) is appended to .../include/c++/$V

% clang++ --target=aarch64-pc-linux-gnu a.cc '-###' |& sed -E 's/ "?-[LiI]/\n&/g'
clang version 13.0.0
Target: aarch64-pc-linux-gnu
Thread model: posix
InstalledDir: /tmp/RelA/bin
 "/tmp/RelA/bin/clang-13" "-cc1" "-triple" "aarch64-pc-linux-gnu" "-emit-obj" "-mrelax-all" "--mrelax-relocations" "-disable-free" "-main-file-name" "a.cc" "-mrelocation-model" "static" "-mframe-pointer=non-leaf" "-fmath-errno" "-fno-rounding-math" "-mconstructor-aliases" "-munwind-tables" "-target-cpu" "generic" "-target-feature" "+neon" "-target-abi" "aapcs" "-fallow-half-arguments-and-returns" "-debugger-tuning=gdb" "-fcoverage-compilation-dir=/tmp/c" "-resource-dir" "/tmp/RelA/lib/clang/13.0.0"
 "-internal-isystem" "/usr/lib/gcc-cross/aarch64-linux-gnu/10/../../../../aarch64-linux-gnu/include/c++/10"
 "-internal-isystem" "/usr/lib/gcc-cross/aarch64-linux-gnu/10/../../../../aarch64-linux-gnu/include/c++/10/aarch64-linux-gnu"
 "-internal-isystem" "/usr/lib/gcc-cross/aarch64-linux-gnu/10/../../../../aarch64-linux-gnu/include/c++/10/backward"
...

[phosek]

@ldionne any more thoughts on this?

[ldionne]

I agree the design is clean from the "what we ship" perspective. I really like the idea of shipping the whole library in a generic spot and only having one file embody all the target specific configuration of the library. I think it's a very elegant way to handle the configuration of libc++, and it will have positive effects beyond your intended use case like giving us a place and a way to push platform-specific hacks to.

Where this really stinks IMO is in the way we build the library. I don't want us to have a separate LLVM_ENABLE_PER_TARGET_RUNTIME_DIR setting and a different mode for building libc++. This is at the heart of many discussions we've been having lately. Basically, the way I do things:

1. Build libc++ for one target using the most straightforward possible CMake settings
2. Then, post-process that by moving headers around, creating universal binaries, etc, all of it from a separate script.

That's how I set up the libc++ build at Apple because I didn't want to add too many platform specific things in the CMake. Otherwise we'd have e.g. a mode for building for many architectures (x86_64 and arm64) at once on Apple platforms and creating a universal dylib from that. That's not the way CMake is intended to be used unless I fundamentally missed something. Also note that it would be much better if CMake had proper support for multi-arch, but I'm not aware of anything like that.

On the contrary, your approach (please correct me if I'm mis-speaking) is to make the CMake work out of the box exactly as you want it, even if that means adding complexity. This has the nice benefit that everything is done in a single place, however it forces us to go against the way CMake was designed in a few places, for example when it comes to building the library for several platforms.

My only push back on this patch, and on several patches in the same vein, is about this fundamental difference in approach that we have.

[phosek]

@ldionne friendly ping.

In D89013#2686799, @ldionne wrote:

I agree the design is clean from the "what we ship" perspective. I really like the idea of shipping the whole library in a generic spot and only having one file embody all the target specific configuration of the library. I think it's a very elegant way to handle the configuration of libc++, and it will have positive effects beyond your intended use case like giving us a place and a way to push platform-specific hacks to.

Where this really stinks IMO is in the way we build the library. I don't want us to have a separate LLVM_ENABLE_PER_TARGET_RUNTIME_DIR setting and a different mode for building libc++. This is at the heart of many discussions we've been having lately. Basically, the way I do things:

1. Build libc++ for one target using the most straightforward possible CMake settings
2. Then, post-process that by moving headers around, creating universal binaries, etc, all of it from a separate script.

That's how I set up the libc++ build at Apple because I didn't want to add too many platform specific things in the CMake. Otherwise we'd have e.g. a mode for building for many architectures (x86_64 and arm64) at once on Apple platforms and creating a universal dylib from that. That's not the way CMake is intended to be used unless I fundamentally missed something. Also note that it would be much better if CMake had proper support for multi-arch, but I'm not aware of anything like that.

On the contrary, your approach (please correct me if I'm mis-speaking) is to make the CMake work out of the box exactly as you want it, even if that means adding complexity. This has the nice benefit that everything is done in a single place, however it forces us to go against the way CMake was designed in a few places, for example when it comes to building the library for several platforms.

My only push back on this patch, and on several patches in the same vein, is about this fundamental difference in approach that we have.

I agree with you, I'm just not sure if what you're describing is possible, at least not yet. There is one additional requirement we have, which is that build layout matches the installation layout. This is what many of our tools and tests rely on. For example, when you build clang++ and then use it, it's going to look for C++ headers in ../include/c++/v1 and without needing to run ninja install. This is where a lot of the complexity comes from and changing this assumption would be non-trivial (and it's not even clear to me that the alternative would be better from usability perspective).

If you want to simplify the libc++ build, we could consider moving the path setup (that is the LLVM_ENABLE_PER_TARGET_RUNTIME_DIR branch) to the runtimes build, but we would still need expose all of the variables to parametrize where different build artifacts are placed. If we decide to go that way, I'd prefer to do it in a follow up change since it's technically orthogonal.

I'd also point out that we support a lot of different ways to do things which is where a lot of the complexity comes from, one example is having multiple different ways to build runtimes which you brought up in your RFC, but there are others. LLVM_ENABLE_PER_TARGET_RUNTIME_DIR is actually an example of that, I'd really prefer if that was the default and the only supported way, but we would need to ensure that all vendors have switched to it, just as I'd like for everyone to use the runtimes build.

[ldionne]

Well, one way to see things is that this patch only makes yet another step in a direction that we're already rather far in (in terms of CMake complexity). And as I said, I think the separation of __config_site in its own platform-specific directory makes a lot of sense. So don't consider yourself blocked on libc++.

I do have some questions regarding the driver side of things though. What platforms is this change going to impact on the Driver side? Fuchsia is self-explanatory, but what does the Gnu driver impact? Also, have you thought about using this scheme instead? Why did you opt for the scheme you selected?

include/
  c++/
    v1/
      <libc++ headers except for __config_site>
      <target1>/
          __config_site
      <target2>/
          __config_site

[ldionne]

As discussed offline, I think this is fine. This layout seems to be closest to what other compilers use.

@phosek said he would send out an RFC to make the multiarch layout become the default for all platforms. I'd be interested in using that on Darwin too (the only difference is that it would be apple-<platform> without the architecture, since we use universal binaries).

[ldionne]

Oh, and can you make sure the Runtimes build passes CI before shipping this?

[MaskRay]

The path component x86_64-unknown-linux-gnu (not multiarch x86_64-linux-gnu) looks good to me

clang/test/Driver/linux-header-search.cpp

16

You may want to use the -SAME: {{^}} pattern in linux-cross.cpp to improve the checks.

[sylvestre.ledru]

Building with this patch Debian packages is failing with:

"/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/build-llvm/tools/clang/stage2-bins/bin/clang++" -DHAVE___CXA_THREAD_ATEXIT_IMPL -DLIBCXXABI_USE_LLVM_UNWINDER -D_LIBCPP_BUILDING_LIBRARY -D_LIBCPP_DISABLE_EXTERN_TEMPLATE -D_LIBCXXABI_BUILDING_LIBRARY -D_LIBCXXABI_LINK_PTHREAD_LIB -I../include -I"/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/libunwind/include" -I"/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/libcxx/src" -g -O2 -fdebug-prefix-map=/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 --target=x86_64-pc-linux-gnu  -O2 -g -DNDEBUG -fPIC --target=x86_64-pc-linux-gnu -nostdinc++ -Werror=return-type -W -Wall -Wchar-subscripts -Wconversion -Wmismatched-tags -Wmissing-braces -Wnewline-eof -Wunused-function -Wshadow -Wshorten-64-to-32 -Wsign-compare -Wsign-conversion -Wstrict-aliasing=2 -Wstrict-overflow=4 -Wunused-parameter -Wunused-variable -Wwrite-strings -Wundef -Wno-suggest-override -Wno-error -pedantic -fstrict-aliasing -funwind-tables -D_DEBUG -I "/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/build-llvm/tools/clang/stage2-bins/include/c++/v1" -std=c++20 -MD -MT src/CMakeFiles/cxxabi_shared.dir/cxa_demangle.cpp.o -MF src/CMakeFiles/cxxabi_shared.dir/cxa_demangle.cpp.o.d -o src/CMakeFiles/cxxabi_shared.dir/cxa_demangle.cpp.o -c ../src/cxa_demangle.cpp
In file included from ../src/cxa_demangle.cpp:13:
In file included from ../src/demangle/ItaniumDemangle.h:21:
In file included from ../src/demangle/DemangleConfig.h:14:
In file included from /home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/build-llvm/tools/clang/stage2-bins/include/c++/v1/ciso646:18:
/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/build-llvm/tools/clang/stage2-bins/include/c++/v1/__config:13:10: fatal error: '__config_site' file not found
#include <__config_site>
         ^~~~~~~~~~~~~~~
1 error generated.
ninja: build stopped: subcommand failed.

cmake:

 cmake \
-S . -B build \
-G Ninja \
-DCMAKE_INSTALL_PREFIX=/usr/lib/llvm-14 -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DLLVM_CONFIG_PATH=/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/build-llvm/tools/clang/stage2-bins/bin/llvm-config -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_CXX_COMPILER=/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/build-llvm/tools/clang/stage2-bins/bin/clang++ -DCMAKE_C_COMPILER=/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/build-llvm/tools/clang/stage2-bins/bin/clang -DCMAKE_CXX_FLAGS="-g -O2 -fdebug-prefix-map=/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2" -DCMAKE_EXE_LINKER_FLAGS="" -DCMAKE_SHARED_LINKER_FLAGS="" -DCMAKE_MODULE_LINKER_FLAGS="" -DCMAKE_AR=/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/build-llvm/tools/clang/stage2-bins/bin/llvm-ar -DCMAKE_RANLIB=/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/build-llvm/tools/clang/stage2-bins/bin/llvm-ranlib -DLLVM_EXTERNAL_LIT=/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/llvm/utils/lit/lit.py -DLLVM_TEMPORARILY_ALLOW_OLD_TOOLCHAIN=ON -DPYTHON_EXECUTABLE=/usr/bin/python3 -DCLANG_SYSTEMZ_DEFAULT_ARCH=z196 -DLLVM_BINUTILS_INCDIR=/usr/include/ -DLLVM_USE_PERF=yes -DLLVM_ENABLE_LIBPFM=ON \
-DLIBCXXABI_LIBCXX_PATH=/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/libcxx \
-DLIBCXXABI_LIBCXX_LIBRARY_PATH=/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/libcxx/build/lib \
-DLIBCXXABI_LIBCXX_INCLUDES=/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/build-llvm/tools/clang/stage2-bins/include/c++/v1/ \
-DLIBCXXABI_ENABLE_EXCEPTIONS=ON \
-DLIBCXXABI_USE_COMPILER_RT=ON \
-DLLVM_ENABLE_RTTI=ON \
-DLIBCXXABI_USE_LLVM_UNWINDER=ON -DLIBUNWIND_USE_COMPILER_RT=ON

Do you want a bug report about this?

[phosek]

In D89013#3017317, @sylvestre.ledru wrote:

Building with this patch Debian packages is failing with:

"/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/build-llvm/tools/clang/stage2-bins/bin/clang++" -DHAVE___CXA_THREAD_ATEXIT_IMPL -DLIBCXXABI_USE_LLVM_UNWINDER -D_LIBCPP_BUILDING_LIBRARY -D_LIBCPP_DISABLE_EXTERN_TEMPLATE -D_LIBCXXABI_BUILDING_LIBRARY -D_LIBCXXABI_LINK_PTHREAD_LIB -I../include -I"/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/libunwind/include" -I"/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/libcxx/src" -g -O2 -fdebug-prefix-map=/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 --target=x86_64-pc-linux-gnu  -O2 -g -DNDEBUG -fPIC --target=x86_64-pc-linux-gnu -nostdinc++ -Werror=return-type -W -Wall -Wchar-subscripts -Wconversion -Wmismatched-tags -Wmissing-braces -Wnewline-eof -Wunused-function -Wshadow -Wshorten-64-to-32 -Wsign-compare -Wsign-conversion -Wstrict-aliasing=2 -Wstrict-overflow=4 -Wunused-parameter -Wunused-variable -Wwrite-strings -Wundef -Wno-suggest-override -Wno-error -pedantic -fstrict-aliasing -funwind-tables -D_DEBUG -I "/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/build-llvm/tools/clang/stage2-bins/include/c++/v1" -std=c++20 -MD -MT src/CMakeFiles/cxxabi_shared.dir/cxa_demangle.cpp.o -MF src/CMakeFiles/cxxabi_shared.dir/cxa_demangle.cpp.o.d -o src/CMakeFiles/cxxabi_shared.dir/cxa_demangle.cpp.o -c ../src/cxa_demangle.cpp
In file included from ../src/cxa_demangle.cpp:13:
In file included from ../src/demangle/ItaniumDemangle.h:21:
In file included from ../src/demangle/DemangleConfig.h:14:
In file included from /home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/build-llvm/tools/clang/stage2-bins/include/c++/v1/ciso646:18:
/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/build-llvm/tools/clang/stage2-bins/include/c++/v1/__config:13:10: fatal error: '__config_site' file not found
#include <__config_site>
         ^~~~~~~~~~~~~~~
1 error generated.
ninja: build stopped: subcommand failed.

cmake:

 cmake \
-S . -B build \
-G Ninja \
-DCMAKE_INSTALL_PREFIX=/usr/lib/llvm-14 -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DLLVM_CONFIG_PATH=/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/build-llvm/tools/clang/stage2-bins/bin/llvm-config -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_CXX_COMPILER=/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/build-llvm/tools/clang/stage2-bins/bin/clang++ -DCMAKE_C_COMPILER=/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/build-llvm/tools/clang/stage2-bins/bin/clang -DCMAKE_CXX_FLAGS="-g -O2 -fdebug-prefix-map=/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2" -DCMAKE_EXE_LINKER_FLAGS="" -DCMAKE_SHARED_LINKER_FLAGS="" -DCMAKE_MODULE_LINKER_FLAGS="" -DCMAKE_AR=/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/build-llvm/tools/clang/stage2-bins/bin/llvm-ar -DCMAKE_RANLIB=/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/build-llvm/tools/clang/stage2-bins/bin/llvm-ranlib -DLLVM_EXTERNAL_LIT=/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/llvm/utils/lit/lit.py -DLLVM_TEMPORARILY_ALLOW_OLD_TOOLCHAIN=ON -DPYTHON_EXECUTABLE=/usr/bin/python3 -DCLANG_SYSTEMZ_DEFAULT_ARCH=z196 -DLLVM_BINUTILS_INCDIR=/usr/include/ -DLLVM_USE_PERF=yes -DLLVM_ENABLE_LIBPFM=ON \
-DLIBCXXABI_LIBCXX_PATH=/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/libcxx \
-DLIBCXXABI_LIBCXX_LIBRARY_PATH=/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/libcxx/build/lib \
-DLIBCXXABI_LIBCXX_INCLUDES=/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/build-llvm/tools/clang/stage2-bins/include/c++/v1/ \
-DLIBCXXABI_ENABLE_EXCEPTIONS=ON \
-DLIBCXXABI_USE_COMPILER_RT=ON \
-DLLVM_ENABLE_RTTI=ON \
-DLIBCXXABI_USE_LLVM_UNWINDER=ON -DLIBUNWIND_USE_COMPILER_RT=ON

Do you want a bug report about this?

How was the toolchain in /home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/build-llvm built?

[sylvestre.ledru]

How was the toolchain in /home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-snapshot-14~++20210922093928+5da21338bcd0/build-llvm built?

Only using PROJECTS using stage2, not RUNTIMES (it is planned)

[sylvestre.ledru]

@phosek
It was probably my fault (mix of different in-tree / out of tree builds)
Quick question, why the path to __config_site contains this triple
x86_64-pc-linux-gnu
(in /usr/lib/llvm-14/include/x86_64-pc-linux-gnu/c++/v1/__config_site)
when it is usually:
x86_64-linux-gnu
(does not contain -pc)
thanks?

[phosek]

In D89013#3022332, @sylvestre.ledru wrote:

@phosek
It was probably my fault (mix of different in-tree / out of tree builds)
Quick question, why the path to __config_site contains this triple
x86_64-pc-linux-gnu
(in /usr/lib/llvm-14/include/x86_64-pc-linux-gnu/c++/v1/__config_site)
when it is usually:
x86_64-linux-gnu
(does not contain -pc)
thanks?

See the comments earlier in this review (such as https://reviews.llvm.org/D89013#2662244) where we discussed whether to use Debian multiarch triples or normalized LLVM triples. We've decided to use normalized LLVM triples to reduce the amount of custom logic needed and increase consistency across platforms.

The reason why __config_site uses x86_64-pc-linux-gnu is because that's the default target that was used by the build (you can also override it via -DLLVM_DEFAULT_TARGET_TRIPLE=). We would normalize x86_64-linux-gnu to x86_64-unknown-linux-gnu. The -pc vendor component was returned by the config.guess script, I believe that's going to be addressed by removal of that script in D109837.

[MaskRay]

Supplement:
Even if you omit the vendor part in -DLLVM_DEFAULT_TARGET_TRIPLE (if unspecified, use -DLLVM_HOST_TRIPLE), x86_64-linux-gnu, you will see
Target: x86_64-unknown-linux-gnu in clang -v output (and similarly in -dumpmachine/--print-target-triple).

The reason is that clang normalizes the target triple quite early in toolchain selection.
(You may need debugging to figure out where.)

It'd probably make sense to honor -DLLVM_DEFAULT_TARGET_TRIPLE and don't add the omitted vendor part (unknown or pc), but it requires some work ;-)
Then Debian/Ubuntu and their derivatives will see x86_64-linux-gnu everywhere.

As a bonus, we can drop many x86_64-linux-gnu like elements from *Triples in clang/lib/Driver/ToolChains/Gnu.cpp:CollectLibDirsAndTriples (see D109727 for the idea).

---

PS: I believe that only the Debian multiarch scheme omits vendor part from the target triple.
Vanilla GCC configured on Debian and non-Debian all use x86_64-unknown-linux-gnu by default.

clang/lib/Driver/ToolChains/Fuchsia.cpp

 }
 
 void Fuchsia::AddClangCXXStdlibIncludeArgs(const ArgList &DriverArgs,
                                            ArgStringList &CC1Args) const {
   if (DriverArgs.hasArg(options::OPT_nostdlibinc) ||
       DriverArgs.hasArg(options::OPT_nostdincxx))
     return;
 
+  const Driver &D = getDriver();
+  std::string MultiarchTriple = getMultiarchTriple(D, getTriple(), D.SysRoot);
+
+  auto AddIncludePath = [&](StringRef Path) {
+    std::string Version = detectLibcxxVersion(Path);
+    if (Version.empty())
+      return;
+
+    SmallString<128> P;
+
+    // First add the per-target include path.
+    P.assign(Path);
+    llvm::sys::path::append(P, MultiarchTriple, "c++", Version);
+    if (getVFS().exists(P))
+      addSystemInclude(DriverArgs, CC1Args, P);
+
+    // Second add the generic one.
+    P.assign(Path);
+    llvm::sys::path::append(P, "c++", Version);
+    addSystemInclude(DriverArgs, CC1Args, P);
+  };
+
   switch (GetCXXStdlibType(DriverArgs)) {
   case ToolChain::CST_Libcxx: {
-    SmallString<128> P(getDriver().Dir);
+    SmallString<128> P(D.Dir);
     llvm::sys::path::append(P, "..", "include");
-    std::string Version = detectLibcxxVersion(P);
-    llvm::sys::path::append(P, "c++", Version);
-    addSystemInclude(DriverArgs, CC1Args, P.str());
+    AddIncludePath(P);
     break;
   }
 
   default:
     llvm_unreachable("invalid stdlib name");
   }
 }
 

clang/lib/Driver/ToolChains/Gnu.cpp

case ToolChain::CST_Libstdcxx:
     addLibStdCxxIncludePaths(DriverArgs, CC1Args);
     break;
   }
 }
 
 void
 Generic_GCC::addLibCxxIncludePaths(const llvm::opt::ArgList &DriverArgs,
                                    llvm::opt::ArgStringList &CC1Args) const {
+  const Driver &D = getDriver();
+  std::string SysRoot = computeSysRoot();
+  std::string MultiarchTriple = getMultiarchTriple(D, getTriple(), SysRoot);
+
   auto AddIncludePath = [&](std::string Path) {
     std::string Version = detectLibcxxVersion(Path);
-    std::string IncludePath = Path + "/c++/" + Version;
-    if (Version.empty() || !getVFS().exists(IncludePath))
+    if (Version.empty())
       return false;
-    addSystemInclude(DriverArgs, CC1Args, IncludePath);
+
+    // First add the multiarch include path if it exists.
+    std::string MultiarchDir = Path + "/" + MultiarchTriple + "/c++/" + Version;
+    if (!MultiarchTriple.empty() && D.getVFS().exists(MultiarchDir))
+      addSystemInclude(DriverArgs, CC1Args, MultiarchDir);
+
+    // Second add the generic one.
+    addSystemInclude(DriverArgs, CC1Args, Path + "/c++/" + Version);
     return true;
   };
+
   // Android never uses the libc++ headers installed alongside the toolchain,
   // which are generally incompatible with the NDK libraries anyway.
   if (!getTriple().isAndroid())
     if (AddIncludePath(getDriver().Dir + "/../include"))
       return;
   // If this is a development, non-installed, clang, libcxx will
   // not be found at ../include/c++ but it likely to be found at
   // one of the following two locations:
-  std::string SysRoot = computeSysRoot();
   if (AddIncludePath(SysRoot + "/usr/local/include"))
     return;
   if (AddIncludePath(SysRoot + "/usr/include"))
     return;
 }
 
 bool Generic_GCC::addLibStdCXXIncludePaths(Twine IncludeDir, StringRef Triple,
                                            Twine IncludeSuffix,

clang/test/Driver/fuchsia.cpp

 // RUN:     | FileCheck -check-prefixes=CHECK,CHECK-RISCV64 %s
 // CHECK: {{.*}}clang{{.*}}" "-cc1"
 // CHECK-X86_64: "-triple" "x86_64-unknown-fuchsia"
 // CHECK-AARCH64: "-triple" "aarch64-unknown-fuchsia"
 // CHECK-RISCV64: "-triple" "riscv64-unknown-fuchsia"
 // CHECK-NOT: "-fno-use-init-array"
 // CHECK: "-resource-dir" "[[RESOURCE_DIR:[^"]+]]"
 // CHECK: "-isysroot" "[[SYSROOT:[^"]+]]"
+// CHECK-X86_64: "-internal-isystem" "{{.*[/\\]}}include{{/|\\\\}}x86_64-unknown-fuchsia{{/|\\\\}}c++{{/|\\\\}}v1"
+// CHECK-AARCH64: "-internal-isystem" "{{.*[/\\]}}include{{/|\\\\}}aarch64-unknown-fuchsia{{/|\\\\}}c++{{/|\\\\}}v1"
+// CHECK-RISCV64: "-internal-isystem" "{{.*[/\\]}}include{{/|\\\\}}riscv64-unknown-fuchsia{{/|\\\\}}c++{{/|\\\\}}v1"
 // CHECK: "-internal-isystem" "{{.*[/\\]}}include{{/|\\\\}}c++{{/|\\\\}}v1"
 // CHECK: "-internal-externc-isystem" "[[SYSROOT]]{{/|\\\\}}include"
 // CHECK: {{.*}}ld.lld{{.*}}" "-z" "now" "-z" "rodynamic" "-z" "separate-loadable-segments"
 // CHECK: "--sysroot=[[SYSROOT]]"
 // CHECK: "-pie"
 // CHECK: "--build-id"
 // CHECK: "-dynamic-linker" "ld.so.1"
 // CHECK: Scrt1.o

clang/test/Driver/linux-header-search.cpp

 // General tests that the header search paths detected by the driver and passed
 // to CC1 are sane.
 //
 // Test a simulated installation of libc++ on Linux, both through sysroot and
 // the installation path of Clang.
 // RUN: %clang -no-canonical-prefixes %s -### -fsyntax-only 2>&1 \
 // RUN:     -target x86_64-unknown-linux-gnu \
 // RUN:     -stdlib=libc++ \
 // RUN:     -ccc-install-dir %S/Inputs/basic_linux_tree/usr/bin \
 // RUN:     -resource-dir=%S/Inputs/resource_dir \
 // RUN:     --sysroot=%S/Inputs/basic_linux_libcxx_tree \
 // RUN:     --gcc-toolchain="" \
 // RUN:   | FileCheck --check-prefix=CHECK-BASIC-LIBCXX-SYSROOT %s
 // CHECK-BASIC-LIBCXX-SYSROOT: "{{[^"]*}}clang{{[^"]*}}" "-cc1"
 // CHECK-BASIC-LIBCXX-SYSROOT: "-isysroot" "[[SYSROOT:[^"]+]]"
+// CHECK-BASIC-LIBCXX-SYSROOT: "-internal-isystem" "[[SYSROOT]]/usr/include/x86_64-linux-gnu/c++/v1"
 // CHECK-BASIC-LIBCXX-SYSROOT: "-internal-isystem" "[[SYSROOT]]/usr/include/c++/v1"
 // CHECK-BASIC-LIBCXX-SYSROOT: "-internal-isystem" "[[SYSROOT]]/usr/local/include"
 // RUN: %clang -no-canonical-prefixes %s -### -fsyntax-only 2>&1 \
 // RUN:     -target x86_64-unknown-linux-gnu \
 // RUN:     -stdlib=libc++ \
 // RUN:     -ccc-install-dir %S/Inputs/basic_linux_libcxx_tree/usr/bin \
 // RUN:     -resource-dir=%S/Inputs/resource_dir \
 // RUN:     --sysroot=%S/Inputs/basic_linux_libcxx_tree \
 // RUN:     --gcc-toolchain="" \
 // RUN:   | FileCheck --check-prefix=CHECK-BASIC-LIBCXX-INSTALL %s
 // CHECK-BASIC-LIBCXX-INSTALL: "{{[^"]*}}clang{{[^"]*}}" "-cc1"
 // CHECK-BASIC-LIBCXX-INSTALL: "-isysroot" "[[SYSROOT:[^"]+]]"
+// CHECK-BASIC-LIBCXX-INSTALL: "-internal-isystem" "[[SYSROOT]]/usr/bin/../include/x86_64-linux-gnu/c++/v1"
 // CHECK-BASIC-LIBCXX-INSTALL: "-internal-isystem" "[[SYSROOT]]/usr/bin/../include/c++/v1"
 // CHECK-BASIC-LIBCXX-INSTALL: "-internal-isystem" "[[SYSROOT]]/usr/local/include"
 //
 // RUN: %clang -no-canonical-prefixes %s -### -fsyntax-only 2>&1 \
 // RUN:     -target x86_64-unknown-linux-gnu \
 // RUN:     -stdlib=libc++ \
 // RUN:     -ccc-install-dir %S/Inputs/basic_linux_tree/usr/bin \
 // RUN:     -resource-dir=%S/Inputs/resource_dir \
 // RUN:     --sysroot=%S/Inputs/basic_linux_libcxxv2_tree \
 // RUN:     --gcc-toolchain="" \
 // RUN:   | FileCheck --check-prefix=CHECK-BASIC-LIBCXXV2-SYSROOT %s
 // CHECK-BASIC-LIBCXXV2-SYSROOT: "{{[^"]*}}clang{{[^"]*}}" "-cc1"
 // CHECK-BASIC-LIBCXXV2-SYSROOT: "-isysroot" "[[SYSROOT:[^"]+]]"
+// CHECK-BASIC-LIBCXXV2-SYSROOT: "-internal-isystem" "[[SYSROOT]]/usr/include/x86_64-linux-gnu/c++/v2"
 // CHECK-BASIC-LIBCXXV2-SYSROOT: "-internal-isystem" "[[SYSROOT]]/usr/include/c++/v2"
 // CHECK-BASIC-LIBCXXV2-SYSROOT: "-internal-isystem" "[[SYSROOT]]/usr/local/include"
 // RUN: %clang -no-canonical-prefixes %s -### -fsyntax-only 2>&1 \
 // RUN:     -target x86_64-unknown-linux-gnu \
 // RUN:     -stdlib=libc++ \
 // RUN:     -ccc-install-dir %S/Inputs/basic_linux_libcxxv2_tree/usr/bin \
 // RUN:     -resource-dir=%S/Inputs/resource_dir \
 // RUN:     --sysroot=%S/Inputs/basic_linux_libcxxv2_tree \
 // RUN:     --gcc-toolchain="" \
 // RUN:   | FileCheck --check-prefix=CHECK-BASIC-LIBCXXV2-INSTALL %s
 // CHECK-BASIC-LIBCXXV2-INSTALL: "{{[^"]*}}clang{{[^"]*}}" "-cc1"
 // CHECK-BASIC-LIBCXXV2-INSTALL: "-isysroot" "[[SYSROOT:[^"]+]]"
+// CHECK-BASIC-LIBCXXV2-INSTALL: "-internal-isystem" "[[SYSROOT]]/usr/bin/../include/x86_64-linux-gnu/c++/v2"
 // CHECK-BASIC-LIBCXXV2-INSTALL: "-internal-isystem" "[[SYSROOT]]/usr/bin/../include/c++/v2"
 // CHECK-BASIC-LIBCXXV2-INSTALL: "-internal-isystem" "[[SYSROOT]]/usr/local/include"
 //
 // Test Linux with both libc++ and libstdc++ installed.
 // RUN: %clang -no-canonical-prefixes %s -### -fsyntax-only 2>&1 \
 // RUN:     -target x86_64-unknown-linux-gnu \
 // RUN:     -stdlib=libc++ \
 // RUN:     -ccc-install-dir %S/Inputs/basic_linux_tree/usr/bin \
 // RUN:     -resource-dir=%S/Inputs/resource_dir \
 // RUN:     --sysroot=%S/Inputs/basic_linux_libstdcxx_libcxxv2_tree \
 // RUN:     --gcc-toolchain="" \
 // RUN:   | FileCheck --check-prefix=CHECK-BASIC-LIBSTDCXX-LIBCXXV2-SYSROOT %s
 // CHECK-BASIC-LIBSTDCXX-LIBCXXV2-SYSROOT: "{{[^"]*}}clang{{[^"]*}}" "-cc1"
 // CHECK-BASIC-LIBSTDCXX-LIBCXXV2-SYSROOT: "-isysroot" "[[SYSROOT:[^"]+]]"
+// CHECK-BASIC-LIBSTDCXX-LIBCXXV2-SYSROOT: "-internal-isystem" "[[SYSROOT]]/usr/include/x86_64-linux-gnu/c++/v2"
 // CHECK-BASIC-LIBSTDCXX-LIBCXXV2-SYSROOT: "-internal-isystem" "[[SYSROOT]]/usr/include/c++/v2"
 // CHECK-BASIC-LIBSTDCXX-LIBCXXV2-SYSROOT: "-internal-isystem" "[[SYSROOT]]/usr/local/include"
 
 // Test Gentoo's weirdness both before and after they changed it in their GCC
 // 4.6.4 release.
 // RUN: %clang -no-canonical-prefixes %s -### -fsyntax-only 2>&1 \
 // RUN:     -target x86_64-unknown-linux-gnu -stdlib=libstdc++ \
 // RUN:     --sysroot=%S/Inputs/gentoo_linux_gcc_4.6.2_tree \

libcxx/CMakeLists.txt

 # Configure System
 #===============================================================================
 
 # TODO: Projects that depend on libc++ should use LIBCXX_GENERATED_INCLUDE_DIR
 # instead of hard-coding include/c++/v1.
 if(LLVM_ENABLE_PER_TARGET_RUNTIME_DIR AND NOT APPLE)
   set(LIBCXX_LIBRARY_DIR ${LLVM_LIBRARY_OUTPUT_INTDIR}/${LLVM_DEFAULT_TARGET_TRIPLE})
   set(LIBCXX_GENERATED_INCLUDE_DIR "${LLVM_BINARY_DIR}/include/c++/v1")
+  set(LIBCXX_GENERATED_INCLUDE_TARGET_DIR "${LLVM_BINARY_DIR}/include/${LLVM_DEFAULT_TARGET_TRIPLE}/c++/v1")
   set(LIBCXX_INSTALL_LIBRARY_DIR lib${LLVM_LIBDIR_SUFFIX}/${LLVM_DEFAULT_TARGET_TRIPLE})
+  set(LIBCXX_INSTALL_INCLUDE_DIR "include/c++/v1")
+  set(LIBCXX_INSTALL_INCLUDE_TARGET_DIR "include/${LLVM_DEFAULT_TARGET_TRIPLE}/c++/v1")
   if(LIBCXX_LIBDIR_SUBDIR)
     string(APPEND LIBCXX_LIBRARY_DIR /${LIBCXX_LIBDIR_SUBDIR})
     string(APPEND LIBCXX_INSTALL_LIBRARY_DIR /${LIBCXX_LIBDIR_SUBDIR})
   endif()
 elseif(LLVM_LIBRARY_OUTPUT_INTDIR)
   set(LIBCXX_LIBRARY_DIR ${LLVM_LIBRARY_OUTPUT_INTDIR})
   set(LIBCXX_GENERATED_INCLUDE_DIR "${LLVM_BINARY_DIR}/include/c++/v1")
+  set(LIBCXX_GENERATED_INCLUDE_TARGET_DIR "${LIBCXX_GENERATED_INCLUDE_DIR}")
   set(LIBCXX_INSTALL_LIBRARY_DIR lib${LIBCXX_LIBDIR_SUFFIX})
+  set(LIBCXX_INSTALL_INCLUDE_DIR "include/c++/v1")
+  set(LIBCXX_INSTALL_INCLUDE_TARGET_DIR "${LIBCXX_INSTALL_INCLUDE_DIR}")
 else()
   set(LIBCXX_LIBRARY_DIR ${CMAKE_BINARY_DIR}/lib${LIBCXX_LIBDIR_SUFFIX})
   set(LIBCXX_GENERATED_INCLUDE_DIR "${CMAKE_BINARY_DIR}/include/c++/v1")
+  set(LIBCXX_GENERATED_INCLUDE_TARGET_DIR "${LIBCXX_GENERATED_INCLUDE_DIR}")
   set(LIBCXX_INSTALL_LIBRARY_DIR lib${LIBCXX_LIBDIR_SUFFIX})
+  set(LIBCXX_INSTALL_INCLUDE_DIR "include/c++/v1")
+  set(LIBCXX_INSTALL_INCLUDE_TARGET_DIR "${LIBCXX_INSTALL_INCLUDE_DIR}")
 endif()
 
 file(MAKE_DIRECTORY "${LIBCXX_BINARY_INCLUDE_DIR}")
 
 set(CMAKE_ARCHIVE_OUTPUT_DIRECTORY ${LIBCXX_LIBRARY_DIR})
 set(CMAKE_LIBRARY_OUTPUT_DIRECTORY ${LIBCXX_LIBRARY_DIR})
 set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${LIBCXX_LIBRARY_DIR})
 

libcxx/benchmarks/CMakeLists.txt

 set(BENCHMARK_LIBCXX_COMPILE_FLAGS
     -Wno-unused-command-line-argument
     -nostdinc++
     -isystem "${LIBCXX_GENERATED_INCLUDE_DIR}"
     -L${LIBCXX_LIBRARY_DIR}
     -Wl,-rpath,${LIBCXX_LIBRARY_DIR}
     ${SANITIZER_FLAGS}
     )
+if(LLVM_ENABLE_PER_TARGET_RUNTIME_DIR AND NOT APPLE)
+  list(APPEND BENCHMARK_LIBCXX_COMPILE_FLAGS
+    -isystem "${LIBCXX_GENERATED_INCLUDE_TARGET_DIR}")
+endif()
 if (DEFINED LIBCXX_CXX_ABI_LIBRARY_PATH)
   list(APPEND BENCHMARK_LIBCXX_COMPILE_FLAGS
           -L${LIBCXX_CXX_ABI_LIBRARY_PATH}
           -Wl,-rpath,${LIBCXX_CXX_ABI_LIBRARY_PATH})
 endif()
 split_list(BENCHMARK_LIBCXX_COMPILE_FLAGS)
 
 ExternalProject_Add(google-benchmark-libcxx

libcxx/include/CMakeLists.txt

set(files
   valarray
   variant
   vector
   version
   wchar.h
   wctype.h
   )
 
-configure_file("__config_site.in" "${LIBCXX_GENERATED_INCLUDE_DIR}/__config_site" @ONLY)
+configure_file("__config_site.in" "${LIBCXX_GENERATED_INCLUDE_TARGET_DIR}/__config_site" @ONLY)
 
-set(_all_includes "${LIBCXX_GENERATED_INCLUDE_DIR}/__config_site")
+set(_all_includes "${LIBCXX_GENERATED_INCLUDE_TARGET_DIR}/__config_site")
 foreach(f ${files})
   set(src "${CMAKE_CURRENT_SOURCE_DIR}/${f}")
   set(dst "${LIBCXX_GENERATED_INCLUDE_DIR}/${f}")
   add_custom_command(OUTPUT ${dst}
     DEPENDS ${src}
     COMMAND ${CMAKE_COMMAND} -E copy_if_different ${src} ${dst}
     COMMENT "Copying CXX header ${f}")
   list(APPEND _all_includes "${dst}")
 endforeach()
 
 add_custom_target(generate-cxx-headers ALL DEPENDS ${_all_includes})
 
 add_library(cxx-headers INTERFACE)
 add_dependencies(cxx-headers generate-cxx-headers ${LIBCXX_CXX_ABI_HEADER_TARGET})
 # TODO: Use target_include_directories once we figure out why that breaks the runtimes build
 if("${CMAKE_CXX_COMPILER_ID}" STREQUAL "MSVC" OR "${CMAKE_CXX_SIMULATE_ID}" STREQUAL "MSVC")
-  target_compile_options(cxx-headers INTERFACE /I "${LIBCXX_GENERATED_INCLUDE_DIR}")
+  target_compile_options(cxx-headers INTERFACE /I "${LIBCXX_GENERATED_INCLUDE_DIR}"
+                                     INTERFACE /I "${LIBCXX_GENERATED_INCLUDE_TARGET_DIR}")
 else()
-  target_compile_options(cxx-headers INTERFACE -I "${LIBCXX_GENERATED_INCLUDE_DIR}")
+  target_compile_options(cxx-headers INTERFACE -I${LIBCXX_GENERATED_INCLUDE_DIR}
+                                     INTERFACE -I${LIBCXX_GENERATED_INCLUDE_TARGET_DIR})
 endif()
 
 if (LIBCXX_INSTALL_HEADERS)
   foreach(file ${files})
     get_filename_component(dir ${file} DIRECTORY)
     install(FILES ${file}
-      DESTINATION include/c++/v1/${dir}
+      DESTINATION ${LIBCXX_INSTALL_INCLUDE_DIR}/${dir}
       COMPONENT cxx-headers
       PERMISSIONS OWNER_READ OWNER_WRITE GROUP_READ WORLD_READ
     )
   endforeach()
 
   # Install the generated __config_site.
-  install(FILES ${LIBCXX_GENERATED_INCLUDE_DIR}/__config_site
-    DESTINATION include/c++/v1
+  install(FILES ${LIBCXX_GENERATED_INCLUDE_TARGET_DIR}/__config_site
+    DESTINATION ${LIBCXX_INSTALL_INCLUDE_TARGET_DIR}
     PERMISSIONS OWNER_READ OWNER_WRITE GROUP_READ WORLD_READ
     COMPONENT cxx-headers)
 
   if (NOT CMAKE_CONFIGURATION_TYPES)
     add_custom_target(install-cxx-headers
                       DEPENDS cxx-headers
                       COMMAND "${CMAKE_COMMAND}"
                               -DCMAKE_INSTALL_COMPONENT=cxx-headers
                               -P "${CMAKE_BINARY_DIR}/cmake_install.cmake")
     # Stripping is a no-op for headers
     add_custom_target(install-cxx-headers-stripped DEPENDS install-cxx-headers)
   endif()
 endif()