[CMake] Enable LLVM_ENABLE_PER_TARGET_RUNTIME_DIR by default on Linux

[MaskRay]

Summary

This makes the LLVM_ENABLE_PROJECTS mode (supported for compiler-rt, deprecated for libcxx/libcxxabi/libunwind)
closer to https://libcxx.llvm.org/BuildingLibcxx.html#bootstrapping-build .
The layout is arguably superior because different libraries of target triples
are in different directories, similar to GCC/Debian multiarch.

When LLVM_DEFAULT_TARGET_TRIPLE is x86_64-unknown-linux-gnu,
lib/clang/15.0.0/lib/libclang_rt.asan-x86_64.a
is moved to
lib/clang/15.0.0/lib/x86_64-unknown-linux-gnu/libclang_rt.asan.a.

In addition, if the host compiler supports -m32 (multilib),
lib/clang/15.0.0/lib/libclang_rt.asan-i386.a
is moved to
lib/clang/15.0.0/lib/i386-unknown-linux-gnu/libclang_rt.asan.a.

[manojgupta]

llvm/CMakeLists.txt

674–679

Hmm, trying to understand the reason to make it default since users can already opt in. On Chrome OS, we'll have to disable it explictly (LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF ?)

[MaskRay]

llvm/CMakeLists.txt

674–679

For most Linux distributions, they need to do nothing.

ninja asan uses the new path. clang -fsanitize=address picks the new path. ninja install installs the new file. Every step is transparent.
If a distributor copies the whole lib/clang/14.0.0, no care is needed.

If a build system does something special, e.g. a runtime file may be used in an ad-hoc way bypassing Clang, some adjustment may be needed.
If the distributor doesn't want to adjustment, -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=off is the simplest approach.

Does Chrome OS do some stuff with the runtime paths so that this needs to be disabled?

[manojgupta]

llvm/CMakeLists.txt

674–679

We have some complications because of various build systems, mix of packages using clang/gcc etc. Not all packages are consistent in passing "-target <abi>" (often inside packages that do mixed host/target compiles).
In addition, we have been using the old layout since 2016 so various internal tools etc. are relying on the existing locations. Since (so far), we do not support multiple ABIs per ISA in Chrome OS, this has not been important and the cleanup will be long time consuming. We do customization per device using the device's sysroot e.g. install device specfic c++ etc. at /build/<device>/usr/lib.

[sylvestre.ledru]

I love this change. many thanks :)

[sylvestre.ledru]

It should probably be mentioned in the release notes btw

[sylvestre.ledru]

Not related to your change but with llvm 12:
-DLLVM_ENABLE_RUNTIMES=ON
fails with:

-- Registering Polly as a pass plugin (static build: ON)
CMake Error at runtimes/CMakeLists.txt:21 (message):
  LLVM_ENABLE_RUNTIMES requests ON but directory not found:
  /home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-12-12.0.1/llvm/runtimes/../../ON

[phosek]

In D107799#2937699, @sylvestre.ledru wrote:

Not related to your change but with llvm 12:
-DLLVM_ENABLE_RUNTIMES=ON
fails with:

-- Registering Polly as a pass plugin (static build: ON)
CMake Error at runtimes/CMakeLists.txt:21 (message):
  LLVM_ENABLE_RUNTIMES requests ON but directory not found:
  /home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-12-12.0.1/llvm/runtimes/../../ON

LLVM_ENABLE_RUNTIMES takes a list of projects just like LLVM_ENABLE_PROJECTS, not a boolean. You can also use LLVM_ENABLE_RUNTIMES=all to enable all supported runtimes.

[sylvestre.ledru]

LLVM_ENABLE_RUNTIMES takes a list of projects just like LLVM_ENABLE_PROJECTS, not a boolean.

Oh, silly me. Maybe the error message should be updated

[phosek]

@MaskRay is there a reason why we're restricting this only to Linux? This layout is supported on all platforms except for Darwin which already has special handling so setting LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON everywhere is going to work.

llvm/CMakeLists.txt

674–679

@manojgupta Would having to use additional option be too much of an overhead? There are always going to be some exceptions that need additional customization like Chrome OS, but we trying to choose the right default that works for most projects.

[manojgupta]

llvm/CMakeLists.txt

674–679

This change is not a problem since we can disable it. I was mostly letting the author know that that on Chrome OS we'll be disabling this.

[MaskRay]

In D107799#2937768, @phosek wrote:

@MaskRay is there a reason why we're restricting this only to Linux? This layout is supported on all platforms except for Darwin which already has special handling so setting LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON everywhere is going to work.

I'd love to enable it for other OSes as well.

However, some *.cmake files have LLVM_ENABLE_PER_TARGET_RUNTIME_DIR AND NOT APPLE.
And I do not know what the Windows case is. @mstorsjo

Perhaps let all ELF OSes enable this by default?

if(CMAKE_SYSTEM_NAME MATCHES "Darwin|Windows")  # I can add more OSes here
  set(LLVM_ENABLE_PER_TARGET_RUNTIME_DIR_default OFF)
else()
  set(LLVM_ENABLE_PER_TARGET_RUNTIME_DIR_default ON)
endif()

?

(@joerg NetBSD; @ro SunOS; @daltenty AIX)

[mstorsjo]

I can give it a shot to enable it in mingw based builds and see how it works or if it requires some more changes there. I don't often build it in MSVC mode though, can someone else try that out, @rnk or @zero9178?

[phosek]

In D107799#2937886, @MaskRay wrote:

In D107799#2937768, @phosek wrote:

@MaskRay is there a reason why we're restricting this only to Linux? This layout is supported on all platforms except for Darwin which already has special handling so setting LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON everywhere is going to work.

I'd love to enable it for other OSes as well.

Regarding the LLVM_ENABLE_PER_TARGET_RUNTIME_DIR AND NOT APPLE, this is the Darwin exception I mentioned earlier. We never figured out what LLVM_ENABLE_PER_TARGET_RUNTIME_DIR should look like on Darwin hence the carve out.

Darwin uses fat libraries, so using target triples like we do on other platforms doesn't make sense since the same library can be used across multiple architectures. We could just keep the status quo for Darwin where all the runtime end up in lib/clang/14.0.0/lib/darwin.

Alternative would be to try and split the runtimes for different systems into different directories. Today, you'll end up with:

lib/clang/14.0.0/lib/darwin/libclang_rt.profile_ios.a
lib/clang/14.0.0/lib/darwin/libclang_rt.profile_iossim.a
lib/clang/14.0.0/lib/darwin/libclang_rt.profile_osx.a
...

Instead we could do:

lib/clang/14.0.0/lib/ios/libclang_rt.profile.a
lib/clang/14.0.0/lib/iossim/libclang_rt.profile.a
lib/clang/14.0.0/lib/osx/libclang_rt.profile.a
...

That would better match other platforms, but I'd like to hear from Apple folks like @beanz whether this is something that would make sense.

However, some *.cmake files have LLVM_ENABLE_PER_TARGET_RUNTIME_DIR AND NOT APPLE.
And I do not know what the Windows case is. @mstorsjo

We use LLVM_ENABLE_PER_TARGET_RUNTIME_DIR on Windows (in MSVC) with our toolchain without any issues. That toolchain is also used by several projects like Fluter, Dart and Pigweed and they haven't reported any issues.

[phosek]

This comment was removed by phosek.

[MaskRay]

Enable on non-Darwin

[mstorsjo]

In D107799#2938070, @phosek wrote:

However, some *.cmake files have LLVM_ENABLE_PER_TARGET_RUNTIME_DIR AND NOT APPLE.
And I do not know what the Windows case is. @mstorsjo

We use LLVM_ENABLE_PER_TARGET_RUNTIME_DIR on Windows (in MSVC) with our toolchain without any issues. That toolchain is also used by several projects like Fluter, Dart and Pigweed and they haven't reported any issues.

FWIW I build my toolchains by building compiler-rt/libcxx/libcxxabi/libunwind separately, so the default set here won't affect me directly (but others may be building it for mingw this way so it matters if it works). For compiler-rt builtins libraries I would presume it works fine (but I'll test), but I see that it does affect where the output from libunwind/libc++ end up as well, and I'm not quite as confident about whether those are picked up as they should in the mingw setups. (There, most things are already sorted by target architecture in the form of <compiler_binary>/../<target_triple>/{include,lib}, and e.g. the runtimes are built with <compiler_binary>/../<target_triple> as CMAKE_INSTALL_PREFIX. But I'll try it out for the different libraries and see how it behaves.

I'd prefer if you can hold off enabling it for Windows until I've confirmed that it works in mingw setups.

[tstellar]

Is there any reason why the prefix couldn't be lib/clang/14/ instead of lib/clang/14.0.0/ ?

[mstorsjo]

I tested using the per-target runtimes feature for my mingw builds, and enabling it requires a bunch of minor tweaks in the build scripts, but there's one fatal missing feature which IMO blocks enabling the feature by default anywhere outside of Linux and Fuchsia: Clang got support for finding the libc++ include files from the per-target include directory in ea12d779bc238c387511fe7462020f4ecf4a8246 / D89013, but it was only added to the Gnu and Fuchsia toolchain files, so __config_site won't be found even for other ELF platforms.

Secondly, using per-target runtimes makes the mingw use case much more picky about the form of triples used; the OS part of x86_64-w64-mingw32 is normalized into x86_64-w64-windows-gnu, but the vendor string, w64 is left intact. This causes issues where one transparently could use e.g. i686-pc-mingw32 just as well as i686-w64-mingw32 before. @phosek When normalizing the triple for this use case, do you think we should strip out the vendor part of the string altogether?

Other than that, I run into a number of minor inconveniences; I'll be sending patches for those. But the missing include directory is the main blocker IMO.

[beanz]

In D107799#2938302, @tstellar wrote:

Is there any reason why the prefix couldn't be lib/clang/14/ instead of lib/clang/14.0.0/ ?

I think keeping the full version number is good in case we rev any of the headers or libraries shipped with clang in a patch release. It is good to be able to ensure those directories don't overlap or have stale contents if clang gets installed into the same root directory.

In D107799#2938070, @phosek wrote:

That would better match other platforms, but I'd like to hear from Apple folks like @beanz whether this is something that would make sense.

I'm probably not the best person from Apple to weigh in here since I'm not actively involved with clang toolchains anymore. @steven_wu & @arphaman are probably better people to consult since they'd need to be involved.

I always thought it would be best to get Darwin to start building thin-archives for all the compiler-rt libraries, and to move to triple-based directory structure. There's no reason the compiler can't operate on thin files, and it would allow much of the driver and build system code to become unified. Which would be a huge win.

I suspect part of how we got to the place we're at now is because the driver doesn't share a lot of code between targets, and back in the old makefile days each target basically had its own makefiles for compiler-rt. CMake always tried to share logic, but became a mess trying to do _very_ different things.

It would be a big project to clean up and unify all of that, but totally worth doing.

[steven_wu]

In D107799#2939530, @beanz wrote:

In D107799#2938302, @tstellar wrote:

Is there any reason why the prefix couldn't be lib/clang/14/ instead of lib/clang/14.0.0/ ?

I think keeping the full version number is good in case we rev any of the headers or libraries shipped with clang in a patch release. It is good to be able to ensure those directories don't overlap or have stale contents if clang gets installed into the same root directory.

In D107799#2938070, @phosek wrote:

That would better match other platforms, but I'd like to hear from Apple folks like @beanz whether this is something that would make sense.

I'm probably not the best person from Apple to weigh in here since I'm not actively involved with clang toolchains anymore. @steven_wu & @arphaman are probably better people to consult since they'd need to be involved.

I always thought it would be best to get Darwin to start building thin-archives for all the compiler-rt libraries, and to move to triple-based directory structure. There's no reason the compiler can't operate on thin files, and it would allow much of the driver and build system code to become unified. Which would be a huge win.

I suspect part of how we got to the place we're at now is because the driver doesn't share a lot of code between targets, and back in the old makefile days each target basically had its own makefiles for compiler-rt. CMake always tried to share logic, but became a mess trying to do _very_ different things.

It would be a big project to clean up and unify all of that, but totally worth doing.

I think it is a good direction to unify all the platforms but it might not be an easy task. The problem is that I am pretty sure changing the path of libclang_rt will break some existing users since not all access to libclang_rt is going through clang driver. Constructing triple on darwin platform is probably not as trivial as other platforms since many legacy build system do not have native support for that.

[MaskRay]

Ping:)

I'll assume that @mstorsjo has fixed the Windows issues (or I can exclude Windows like Darwin).

Apologies that we want to do this for Linux and can make it a bit inconvenient for Chrome OS (@manojgupta), but we will keep in mind the Linux support for the LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=off configuration should not retire.

[mstorsjo]

In D107799#2963198, @MaskRay wrote:

Ping:)

I'll assume that @mstorsjo has fixed the Windows issues (or I can exclude Windows like Darwin).

Apologies that we want to do this for Linux and can make it a bit inconvenient for Chrome OS (@manojgupta), but we will keep in mind the Linux support for the LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=off configuration should not retire.

I've fixed some Windows issues (D107893) but others (D107892 and D107895) are still stalled.

The biggest blocker is not fixed in general though: Clang in general does not add the libc++ per-target include directory to its include path. In D89013 this was added specifically only to the Gnu and Fuchsia drivers, and D107893 added it for the mingw driver. Even for other ELF drivers, like e.g. FreeBSD, the per-target include directory isn't added. See e.g. https://github.com/llvm/llvm-project/blob/204038d52e03811cc39c18cc073c6020fbae4a4b/clang/lib/Driver/ToolChains/FreeBSD.cpp#L405-L409 vs https://github.com/llvm/llvm-project/blob/204038d52e03811cc39c18cc073c6020fbae4a4b/clang/lib/Driver/ToolChains/MinGW.cpp#L591-L602.

[phosek]

In D107799#2963311, @mstorsjo wrote:

In D107799#2963198, @MaskRay wrote:

Ping:)

I'll assume that @mstorsjo has fixed the Windows issues (or I can exclude Windows like Darwin).

Apologies that we want to do this for Linux and can make it a bit inconvenient for Chrome OS (@manojgupta), but we will keep in mind the Linux support for the LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=off configuration should not retire.

I've fixed some Windows issues (D107893) but others (D107892 and D107895) are still stalled.

The biggest blocker is not fixed in general though: Clang in general does not add the libc++ per-target include directory to its include path. In D89013 this was added specifically only to the Gnu and Fuchsia drivers, and D107893 added it for the mingw driver. Even for other ELF drivers, like e.g. FreeBSD, the per-target include directory isn't added. See e.g. https://github.com/llvm/llvm-project/blob/204038d52e03811cc39c18cc073c6020fbae4a4b/clang/lib/Driver/ToolChains/FreeBSD.cpp#L405-L409 vs https://github.com/llvm/llvm-project/blob/204038d52e03811cc39c18cc073c6020fbae4a4b/clang/lib/Driver/ToolChains/MinGW.cpp#L591-L602.

I'd prefer moving the header search logic for libc++ into ToolChain. There's a comment that says this logic should be handled in subclasses (https://github.com/llvm/llvm-project/blob/b97ca3aca12148bd6ddd0dd78d73003a92f8e553/clang/lib/Driver/ToolChain.cpp#L905) but that leads to a lot of duplication and I'm not sure whether that comment is still relevant. We already have the generic logic for finding the library in ToolChain so I don't see why we shouldn't have the header logic there as well. If some platforms need special handling, they can always override the method.

[mstorsjo]

In D107799#2963397, @phosek wrote:

I'd prefer moving the header search logic for libc++ into ToolChain. There's a comment that says this logic should be handled in subclasses (https://github.com/llvm/llvm-project/blob/b97ca3aca12148bd6ddd0dd78d73003a92f8e553/clang/lib/Driver/ToolChain.cpp#L905) but that leads to a lot of duplication and I'm not sure whether that comment is still relevant. We already have the generic logic for finding the library in ToolChain so I don't see why we shouldn't have the header logic there as well. If some platforms need special handling, they can always override the method.

That sounds good to me as long as it doesn't regress any of the drivers in the process.

[MaskRay]

exclude Windows which isn't ready.

[mstorsjo]

With the proper checks modified, I guess this could be fine, after the libc++ handling in clang has been fixed/generalized for other ELF drivers.

llvm/CMakeLists.txt

411

It looks to me like you modified the wrong condition

675

... as this one still only checks for darwin.

[MaskRay]

only change Linux

[mstorsjo]

Thanks, this looks like it should work.

[eugenis]

FYI this broke several sanitizer bots like this one: https://lab.llvm.org/buildbot/#/builders/168/builds/2015

[MaskRay]

In D107799#2968007, @eugenis wrote:

FYI this broke several sanitizer bots like this one: https://lab.llvm.org/buildbot/#/builders/168/builds/2015

Is the problem sanitizer bots' use of -L?
Add -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=off to their cmake configurations for now?

---

edit: thanks for D108786

[eugenis]

There is also something wrong with -shared-libsan, which finds the wrong runtime library in a multiarch configuration (because they have the same name now!):
https://lab.llvm.org/buildbot/#/builders/37/builds/6346/steps/9/logs/stdio

[mstorsjo]

This also broke the libcxx CI, see e.g. this scheduled build (regular builds of the main branch): https://buildkite.com/llvm-project/libcxx-ci/builds/5060#5ca44812-fb35-4984-9e08-af785d5b9c81

That ends up with errors like these:

/home/libcxx-builder/.buildkite-agent/builds/c06bce848068-1/llvm-project/libcxx-ci/build/generic-cxx17/include/c++/v1/__config:13:10: fatal error: '__config_site' file not found

You should be able to reproduce this by just invoking libcxx/utils/ci/run-buildbot generic-cxx17 in the llvm-project root directory.

I think we need to revert this for now, to unblock libcxx.

[ldionne]

In D107799#2968625, @mstorsjo wrote:

This also broke the libcxx CI, see e.g. this scheduled build (regular builds of the main branch): https://buildkite.com/llvm-project/libcxx-ci/builds/5060#5ca44812-fb35-4984-9e08-af785d5b9c81

That ends up with errors like these:

/home/libcxx-builder/.buildkite-agent/builds/c06bce848068-1/llvm-project/libcxx-ci/build/generic-cxx17/include/c++/v1/__config:13:10: fatal error: '__config_site' file not found

You should be able to reproduce this by just invoking libcxx/utils/ci/run-buildbot generic-cxx17 in the llvm-project root directory.

I think we need to revert this for now, to unblock libcxx.

Right, I got to this commit just now while trying to investigate the libc++ CI failures.

I'm going to revert this for now because we need to get the CI green ASAP, but I'm more than willing to help figure out what's the correct thing to do (we probably need to teach libc++ where to find the headers).

[ldionne]

Note that if you want to try it out with libc++, just make a dummy modification inside the libcxx/ subfolder and that'll trigger the CI for libc++ here. That will allow us to iterate on fixing the libc++ issues right here in this review.

[MaskRay]

Add a blank line in libcxx/TODO.txt to trigger buildbot.

libcxx/utils/ci/run-buildbot generic-cxx17 doesn't work on many Linux distro.
It uses /usr/bin/c++ (g++) to compile tests which fail with:

c++: error: unrecognized command-line option '-nostdlib++'; did you mean '-nostdlib'?

[MaskRay]

@ldionne

My build directory is /tmp/RelA.

I have:

• /tmp/RelA/include/x86_64-unknown-linux-gnu/c++/v1/__config_site
• /tmp/RelA/lib/x86_64-unknown-linux-gnu/libc++.so

check-cxx works fine because these options are available "-I/tmp/RelA/include/x86_64-unknown-linux-gnu/c++/v1" "-L/tmp/RelA/./lib/x86_64-unknown-linux-gnu" "-Wl,-rpath,/tmp/RelA/./lib/x86_64-unknown-linux-gnu"

---

libcxx/utils/ci/run-buildbot generic-cxx17 invokes clang with -nostdinc++ -nostdlib++ and manual -I / -L / -Wl,-rpath (via some scripts which I don't know yet).

The script passes:

• -isystem path/to/build/generic-cxx17/include/c++/v1. build/generic-cxx17/include/x86_64-unknown-linux-gnu/c++/v1 (__config_site) is missing.
• -L path/to/build/generic-cxx17/lib. Should be build/generic-cxx17/lib/x86_64-unknown-linux-gnu instead

The root cause is that libcxx/test/configs/libcxx-trunk-shared.cfg.in does not recognize the -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=on include and lib hierarchy.

Note that libcxx/utils/libcxx/test/config.py has some -I/-L/-Wl,-rpath, logic as well. libcxx/test/configs/libcxx-trunk-shared.cfg.in appears to duplicate some configuration there.

[eugenis]

This breaks libc++ on the sanitizer bot in a similar way:
https://lab.llvm.org/buildbot/#/builders/37/builds/6371

/b/sanitizer-x86_64-linux/build/symbolizer_build32/symbolizer/libcxx/include/c++/v1/__config:13:10: fatal error: '__config_site' file not found

This line needs to add a target dir to the header path:
https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/sanitizer_common/symbolizer/scripts/build_symbolizer.sh#L126

[MaskRay]

In D107799#2969787, @eugenis wrote:

This breaks libc++ on the sanitizer bot in a similar way:
https://lab.llvm.org/buildbot/#/builders/37/builds/6371

/b/sanitizer-x86_64-linux/build/symbolizer_build32/symbolizer/libcxx/include/c++/v1/__config:13:10: fatal error: '__config_site' file not found

This line needs to add a target dir to the header path:
https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/sanitizer_common/symbolizer/scripts/build_symbolizer.sh#L126

LLVM_FLAGS="${FLAGS} -nostdinc++ -I${ZLIB_BUILD} -I${LIBCXX_BUILD}/include/c++/v1 -Wno-error=global-constructors" should use a scheme similar to D108836.
@eugenis Can you make such a change to sanitizer bots?

Perhaps switch from -I to -isystem while touching the line.

[eugenis]

symbolizer build fixed in D108841.
There is still the asan test failure - you don't need any fancy build to reproduce it, just check-asan in a regular cmake build. It's in TestCases/replaceable_new_delete.cpp.
Sorry I can not chase it right now - it's a very busy week for me. Please do not reland until it is fixed though.

[ldionne]

In D107799#2969650, @MaskRay wrote:

@ldionne

[...]

The root cause is that libcxx/test/configs/libcxx-trunk-shared.cfg.in does not recognize the -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=on include and lib hierarchy.

Thanks for the analysis. I saw your other patch to fix libc++'s test suite harness (D108836). I think this patch should be OK once that has been resolved.

Note that libcxx/utils/libcxx/test/config.py has some -I/-L/-Wl,-rpath, logic as well. libcxx/test/configs/libcxx-trunk-shared.cfg.in appears to duplicate some configuration there.

Yeah, it's a bit confusing but basically we're migrating from config.py to the smaller libcxx/test/configs/libcxx-trunk-shared.cfg.in files. It's a gradual transition.

[ldionne]

You should re-upload this to ping the CI. LGTM if libc++ CI passes.

[MaskRay]

rebase to trigger libc++ build bots

[MaskRay]

I'll push D108859 (sanitizer -m32 fix) and reland this patch (with the libcxx touch removed) tomorrow.

[DavidSpickett]

Since this landed our armhf bots have been failing to find compiler RT libs. I've put a fix up for review https://reviews.llvm.org/D110142.

[lebedev.ri]

Hello. This broke the world for me.
Both @sylvestre.ledru's llvm-main package for debian sid (https://apt.llvm.org/)
and clean local builds are broken:

$ git bisect log
git bisect start
# bad: [20c02807333a47000879e0f673cdf2d6b07148dd] [LiveIntervals] Repair subreg ranges in processTiedPairs
git bisect bad 20c02807333a47000879e0f673cdf2d6b07148dd
# bad: [3a998c06a8e93989319238e12b56a731198cc1c2] Revert "Recommit "Revert "[CVP] processSwitch: Remove default case when switch cover all possible values."""
git bisect bad 3a998c06a8e93989319238e12b56a731198cc1c2
# bad: [6cd382bf2894f87a6a68e2d962bdbfc2f0fb3d85] Revert "[CMake] Add debuginfo-tests to LLVM_ALL_PROJECTS after D110016"
git bisect bad 6cd382bf2894f87a6a68e2d962bdbfc2f0fb3d85
# good: [c82dbc2924bd37fc497cdfe581a21810aace608d] [lldb] Skip TestGuiBasicDebug due to pr51833
git bisect good c82dbc2924bd37fc497cdfe581a21810aace608d
# bad: [a2c319fdc6b35ae6d8578459f07c89042775e5ec] [LLVM][CMake][NFC] Resolve FIXME: Rename LLVM_CMAKE_PATH to LLVM_CMAKE_DIR throughout the project
git bisect bad a2c319fdc6b35ae6d8578459f07c89042775e5ec
# good: [a32300a68f6c94b7b275e3560ed31e9174cec5ad] Make the --mlir-disable-threading command line option overrides the C++ API usage
git bisect good a32300a68f6c94b7b275e3560ed31e9174cec5ad
# bad: [87c00878d3ace60fe07cc2a9f8bd4566ab817d26] SplitKit: Remove decade old live interval hack
git bisect bad 87c00878d3ace60fe07cc2a9f8bd4566ab817d26
# good: [1a3b3301d7aa9ab25a8bdf045c77298b087e3930] [HWASan] Catch cases where libc populated jmp_buf.
git bisect good 1a3b3301d7aa9ab25a8bdf045c77298b087e3930
# bad: [1b79efdc72e640fb7a70e5974d0e7352d72ce997] [mlir] Fix printing of EmitC attrs/types with escape characters
git bisect bad 1b79efdc72e640fb7a70e5974d0e7352d72ce997
# bad: [f9dbca68d48e705f6d45df8f58d6b2ee88bce76c] [CMake] Enable LLVM_ENABLE_PER_TARGET_RUNTIME_DIR by default on Linux
git bisect bad f9dbca68d48e705f6d45df8f58d6b2ee88bce76c
# good: [0a07789fe93d8994163798ed6d14937412274dc2] [HWASan] Add missing newlines.
git bisect good 0a07789fe93d8994163798ed6d14937412274dc2
# good: [36ef65adc356699832f786f9fbd3244527e4b79d] [InstCombine] Update test checks through autogeneration, add more tests. NFC
git bisect good 36ef65adc356699832f786f9fbd3244527e4b79d

$ cat /tmp/bisect.sh 
#!/bin/bash

rm -rf /tmp/build-llvm-project
mkdir /tmp/build-llvm-project
cd /tmp/build-llvm-project
CC=clang-12 CXX=clang++-12 cmake -GNinja -DCMAKE_BUILD_TYPE:String=Release -DLLVM_TARGETS_TO_BUILD=all -DLLVM_ENABLE_PROJECTS=all -DLLVM_BUILD_TESTS=ON -DLLVM_INCLUDE_TESTS=ON -DLLVM_BUILD_BENCHMARKS=ON -DLLVM_INCLUDE_BENCHMARKS=ON -DLLVM_APPEND_VC_REV=ON -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_EH=ON -DLLVM_ENABLE_RTTI=ON -DLLVM_ENABLE_EXPENSIVE_CHECKS=OFF -DLLVM_ABI_BREAKING_CHECKS=FORCE_ON -DLLVM_USE_LINKER=lld -DLLVM_USE_SPLIT_DWARF=ON -DBUILD_SHARED_LIBS=ON -DCMAKE_C_FLAGS:String='-O3 -march=native -UNDEBUG -DDEBUG -gsplit-dwarf -gz' -DCMAKE_CXX_FLAGS:String='-O3 -march=native -UNDEBUG -DDEBUG -gsplit-dwarf -gz' -DCMAKE_C_FLAGS_RELEASE:String='-O3 -march=native -UNDEBUG -DDEBUG -g -gsplit-dwarf -gz' -DCMAKE_CXX_FLAGS_RELEASE:String='-O3 -march=native -UNDEBUG -DDEBUG -g -gsplit-dwarf -gz' /tmp/llvm-project/llvm && cp /tmp/llvm.ninja_log .ninja_log && ninja clang
rm -rf /tmp/test-project/build
mkdir /tmp/test-project/build
cd /tmp/test-project/build
CC=/tmp/build-llvm-project/bin/clang CXX=/tmp/build-llvm-project/bin/clang++ cmake .. &> log
grep "\-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version \"1.2.11\")" log || exit 1
exit 0

$ cat CMakeLists.txt 
cmake_minimum_required(VERSION 3.0)

project(test)

find_package(ZLIB)

Good output (before this patch / with this patch reverted):

$ CC=/tmp/build-llvm-project/bin/clang CXX=/tmp/build-llvm-project/bin/clang++ cmake ..
-- The C compiler identification is Clang 14.0.0
-- The CXX compiler identification is Clang 14.0.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /tmp/build-llvm-project/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /tmp/build-llvm-project/bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.11") 
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/test-project/build

Bad output (with this patch):

$ CC=clang CXX=clang++ cmake ..
-- The C compiler identification is Clang 14.0.0
-- The CXX compiler identification is Clang 14.0.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/local/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/local/bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Could NOT find ZLIB (missing: ZLIB_LIBRARY) (found version "1.2.11")
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/test-project/build

Instead of looking in /usr/lib/x86_64-linux-gnu/ it looks in /usr/lib/x86_64-pc-linux-gnu/, which does not exist.
I have no idea where the fix should lie, but it would be good to have a working compiler in the meanwhile.

$ CC=/tmp/build-llvm-project/bin/clang CXX=/tmp/build-llvm-project/bin/clang++ cmake .. --debug-find &> ../log.good
$ rm -rf *
$ CC=clang CXX=clang++ cmake .. --debug-find &> ../log.bad
$ diff -p -U0 ../log.good ../log.bad 
--- ../log.good 2021-09-28 18:16:56.232438873 +0300
+++ ../log.bad  2021-09-28 18:17:13.964680087 +0300
@@ -74 +74 @@ CMake Debug Log at /usr/share/cmake-3.21
-    NAMES: "/tmp/build-llvm-project/bin/clang"
+    NAMES: "/usr/local/bin/clang"
@@ -93 +93 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /tmp/build-llvm-project/bin/clang
+    /usr/local/bin/clang
@@ -153,4 +152,0 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /tmp/build-llvm-project/bin/llvm-ar
-    /home/lebedevri/bin/llvm-ar
-    /opt/cov-analysis-linux64-8.5.0.1/bin/llvm-ar
-
@@ -188,4 +183,0 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /tmp/build-llvm-project/bin/llvm-ranlib
-    /home/lebedevri/bin/llvm-ranlib
-    /opt/cov-analysis-linux64-8.5.0.1/bin/llvm-ranlib
-
@@ -223 +215 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /tmp/build-llvm-project/bin/llvm-strip
+    /usr/local/bin/llvm-strip
@@ -226 +217,0 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /usr/local/bin/llvm-strip
@@ -259,4 +249,0 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /tmp/build-llvm-project/bin/ld.lld
-    /home/lebedevri/bin/ld.lld
-    /opt/cov-analysis-linux64-8.5.0.1/bin/ld.lld
-
@@ -294,4 +280,0 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /tmp/build-llvm-project/bin/llvm-nm
-    /home/lebedevri/bin/llvm-nm
-    /opt/cov-analysis-linux64-8.5.0.1/bin/llvm-nm
-
@@ -329,4 +311,0 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /tmp/build-llvm-project/bin/llvm-objdump
-    /home/lebedevri/bin/llvm-objdump
-    /opt/cov-analysis-linux64-8.5.0.1/bin/llvm-objdump
-
@@ -364,4 +342,0 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /tmp/build-llvm-project/bin/llvm-objcopy
-    /home/lebedevri/bin/llvm-objcopy
-    /opt/cov-analysis-linux64-8.5.0.1/bin/llvm-objcopy
-
@@ -399 +374 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /tmp/build-llvm-project/bin/llvm-readelf
+    /usr/local/bin/llvm-readelf
@@ -402 +376,0 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /usr/local/bin/llvm-readelf
@@ -435 +409 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /tmp/build-llvm-project/bin/llvm-dlltool
+    /usr/local/bin/llvm-dlltool
@@ -438 +411,0 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /usr/local/bin/llvm-dlltool
@@ -471 +444 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /tmp/build-llvm-project/bin/llvm-addr2line
+    /usr/local/bin/llvm-addr2line
@@ -474 +446,0 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /usr/local/bin/llvm-addr2line
@@ -479 +451 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /tmp/build-llvm-project/bin/addr2line
+    /usr/local/bin/addr2line
@@ -482 +453,0 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /usr/local/bin/addr2line
@@ -516 +487,2 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /tmp/build-llvm-project/bin/llvm-ar-14.0
+    /usr/lib/llvm-14/bin/llvm-ar-14.0
+    /usr/local/bin/llvm-ar-14.0
@@ -519 +490,0 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /usr/local/bin/llvm-ar-14.0
@@ -524 +495,2 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /tmp/build-llvm-project/bin/llvm-ar-14
+    /usr/lib/llvm-14/bin/llvm-ar-14
+    /usr/local/bin/llvm-ar-14
@@ -527 +498,0 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /usr/local/bin/llvm-ar-14
@@ -561 +532,2 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /tmp/build-llvm-project/bin/llvm-ranlib-14.0
+    /usr/lib/llvm-14/bin/llvm-ranlib-14.0
+    /usr/local/bin/llvm-ranlib-14.0
@@ -564 +535,0 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /usr/local/bin/llvm-ranlib-14.0
@@ -569 +540,2 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /tmp/build-llvm-project/bin/llvm-ranlib-14
+    /usr/lib/llvm-14/bin/llvm-ranlib-14
+    /usr/local/bin/llvm-ranlib-14
@@ -572 +543,0 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /usr/local/bin/llvm-ranlib-14
@@ -587 +558 @@ CMake Debug Log at /usr/share/cmake-3.21
-    NAMES: "/tmp/build-llvm-project/bin/clang++"
+    NAMES: "/usr/local/bin/clang++"
@@ -603 +574 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /tmp/build-llvm-project/bin/clang++
+    /usr/local/bin/clang++
@@ -664 +635,2 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /tmp/build-llvm-project/bin/llvm-ar-14.0
+    /usr/lib/llvm-14/bin/llvm-ar-14.0
+    /usr/local/bin/llvm-ar-14.0
@@ -667 +638,0 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /usr/local/bin/llvm-ar-14.0
@@ -672 +643,2 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /tmp/build-llvm-project/bin/llvm-ar-14
+    /usr/lib/llvm-14/bin/llvm-ar-14
+    /usr/local/bin/llvm-ar-14
@@ -675 +646,0 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /usr/local/bin/llvm-ar-14
@@ -709 +680,2 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /tmp/build-llvm-project/bin/llvm-ranlib-14.0
+    /usr/lib/llvm-14/bin/llvm-ranlib-14.0
+    /usr/local/bin/llvm-ranlib-14.0
@@ -712 +683,0 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /usr/local/bin/llvm-ranlib-14.0
@@ -717 +688,2 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /tmp/build-llvm-project/bin/llvm-ranlib-14
+    /usr/lib/llvm-14/bin/llvm-ranlib-14
+    /usr/local/bin/llvm-ranlib-14
@@ -720 +691,0 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /usr/local/bin/llvm-ranlib-14
@@ -733 +704 @@ Call Stack (most recent call first):
--- Check for working C compiler: /tmp/build-llvm-project/bin/clang - skipped
+-- Check for working C compiler: /usr/local/bin/clang - skipped
@@ -738 +709 @@ Call Stack (most recent call first):
--- Check for working CXX compiler: /tmp/build-llvm-project/bin/clang++ - skipped
+-- Check for working CXX compiler: /usr/local/bin/clang++ - skipped
@@ -776,2 +747,2 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /usr/local/include/x86_64-linux-gnu/include/zlib.h
-    /usr/local/include/x86_64-linux-gnu/zlib.h
+    /usr/local/include/x86_64-pc-linux-gnu/include/zlib.h
+    /usr/local/include/x86_64-pc-linux-gnu/zlib.h
@@ -782,2 +753,2 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /usr/include/x86_64-linux-gnu/include/zlib.h
-    /usr/include/x86_64-linux-gnu/zlib.h
+    /usr/include/x86_64-pc-linux-gnu/include/zlib.h
+    /usr/include/x86_64-pc-linux-gnu/zlib.h
@@ -889,10 +860,10 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /usr/local/lib/x86_64-linux-gnu/lib/(lib)z(\.so|\.a)
-    /usr/local/lib/x86_64-linux-gnu/lib/(lib)zlib(\.so|\.a)
-    /usr/local/lib/x86_64-linux-gnu/lib/(lib)zdll(\.so|\.a)
-    /usr/local/lib/x86_64-linux-gnu/lib/(lib)zlib1(\.so|\.a)
-    /usr/local/lib/x86_64-linux-gnu/lib/(lib)zlibstatic(\.so|\.a)
-    /usr/local/lib/x86_64-linux-gnu/(lib)z(\.so|\.a)
-    /usr/local/lib/x86_64-linux-gnu/(lib)zlib(\.so|\.a)
-    /usr/local/lib/x86_64-linux-gnu/(lib)zdll(\.so|\.a)
-    /usr/local/lib/x86_64-linux-gnu/(lib)zlib1(\.so|\.a)
-    /usr/local/lib/x86_64-linux-gnu/(lib)zlibstatic(\.so|\.a)
+    /usr/local/lib/x86_64-pc-linux-gnu/lib/(lib)z(\.so|\.a)
+    /usr/local/lib/x86_64-pc-linux-gnu/lib/(lib)zlib(\.so|\.a)
+    /usr/local/lib/x86_64-pc-linux-gnu/lib/(lib)zdll(\.so|\.a)
+    /usr/local/lib/x86_64-pc-linux-gnu/lib/(lib)zlib1(\.so|\.a)
+    /usr/local/lib/x86_64-pc-linux-gnu/lib/(lib)zlibstatic(\.so|\.a)
+    /usr/local/lib/x86_64-pc-linux-gnu/(lib)z(\.so|\.a)
+    /usr/local/lib/x86_64-pc-linux-gnu/(lib)zlib(\.so|\.a)
+    /usr/local/lib/x86_64-pc-linux-gnu/(lib)zdll(\.so|\.a)
+    /usr/local/lib/x86_64-pc-linux-gnu/(lib)zlib1(\.so|\.a)
+    /usr/local/lib/x86_64-pc-linux-gnu/(lib)zlibstatic(\.so|\.a)
@@ -919,7 +890,170 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /usr/lib/x86_64-linux-gnu/lib/(lib)z(\.so|\.a)
-    /usr/lib/x86_64-linux-gnu/lib/(lib)zlib(\.so|\.a)
-    /usr/lib/x86_64-linux-gnu/lib/(lib)zdll(\.so|\.a)
-    /usr/lib/x86_64-linux-gnu/lib/(lib)zlib1(\.so|\.a)
-    /usr/lib/x86_64-linux-gnu/lib/(lib)zlibstatic(\.so|\.a)
-
-  The item was found at
+    /usr/lib/x86_64-pc-linux-gnu/lib/(lib)z(\.so|\.a)
+    /usr/lib/x86_64-pc-linux-gnu/lib/(lib)zlib(\.so|\.a)
+    /usr/lib/x86_64-pc-linux-gnu/lib/(lib)zdll(\.so|\.a)
+    /usr/lib/x86_64-pc-linux-gnu/lib/(lib)zlib1(\.so|\.a)
+    /usr/lib/x86_64-pc-linux-gnu/lib/(lib)zlibstatic(\.so|\.a)
+    /usr/lib/x86_64-pc-linux-gnu/(lib)z(\.so|\.a)
+    /usr/lib/x86_64-pc-linux-gnu/(lib)zlib(\.so|\.a)
+    /usr/lib/x86_64-pc-linux-gnu/(lib)zdll(\.so|\.a)
+    /usr/lib/x86_64-pc-linux-gnu/(lib)zlib1(\.so|\.a)
+    /usr/lib/x86_64-pc-linux-gnu/(lib)zlibstatic(\.so|\.a)
+    /usr/lib/lib/(lib)z(\.so|\.a)
+    /usr/lib/lib/(lib)zlib(\.so|\.a)
+    /usr/lib/lib/(lib)zdll(\.so|\.a)
+    /usr/lib/lib/(lib)zlib1(\.so|\.a)
+    /usr/lib/lib/(lib)zlibstatic(\.so|\.a)
+    /usr/lib/(lib)z(\.so|\.a)
+    /usr/lib/(lib)zlib(\.so|\.a)
+    /usr/lib/(lib)zdll(\.so|\.a)
+    /usr/lib/(lib)zlib1(\.so|\.a)
+    /usr/lib/(lib)zlibstatic(\.so|\.a)
+    /usr/lib/(lib)z(\.so|\.a)
+    /usr/lib/(lib)zlib(\.so|\.a)
+    /usr/lib/(lib)zdll(\.so|\.a)
+    /usr/lib/(lib)zlib1(\.so|\.a)
+    /usr/lib/(lib)zlibstatic(\.so|\.a)
+    /usr/(lib)z(\.so|\.a)
+    /usr/(lib)zlib(\.so|\.a)
+    /usr/(lib)zdll(\.so|\.a)
+    /usr/(lib)zlib1(\.so|\.a)
+    /usr/(lib)zlibstatic(\.so|\.a)
+    /lib/x86_64-pc-linux-gnu/lib/(lib)z(\.so|\.a)
+    /lib/x86_64-pc-linux-gnu/lib/(lib)zlib(\.so|\.a)
+    /lib/x86_64-pc-linux-gnu/lib/(lib)zdll(\.so|\.a)
+    /lib/x86_64-pc-linux-gnu/lib/(lib)zlib1(\.so|\.a)
+    /lib/x86_64-pc-linux-gnu/lib/(lib)zlibstatic(\.so|\.a)
+    /lib/x86_64-pc-linux-gnu/(lib)z(\.so|\.a)
+    /lib/x86_64-pc-linux-gnu/(lib)zlib(\.so|\.a)
+    /lib/x86_64-pc-linux-gnu/(lib)zdll(\.so|\.a)
+    /lib/x86_64-pc-linux-gnu/(lib)zlib1(\.so|\.a)
+    /lib/x86_64-pc-linux-gnu/(lib)zlibstatic(\.so|\.a)
+    /lib/lib/(lib)z(\.so|\.a)
+    /lib/lib/(lib)zlib(\.so|\.a)
+    /lib/lib/(lib)zdll(\.so|\.a)
+    /lib/lib/(lib)zlib1(\.so|\.a)
+    /lib/lib/(lib)zlibstatic(\.so|\.a)
+    /lib/(lib)z(\.so|\.a)
+    /lib/(lib)zlib(\.so|\.a)
+    /lib/(lib)zdll(\.so|\.a)
+    /lib/(lib)zlib1(\.so|\.a)
+    /lib/(lib)zlibstatic(\.so|\.a)
+    /usr/X11R6/lib/x86_64-pc-linux-gnu/lib/(lib)z(\.so|\.a)
+    /usr/X11R6/lib/x86_64-pc-linux-gnu/lib/(lib)zlib(\.so|\.a)
+    /usr/X11R6/lib/x86_64-pc-linux-gnu/lib/(lib)zdll(\.so|\.a)
+    /usr/X11R6/lib/x86_64-pc-linux-gnu/lib/(lib)zlib1(\.so|\.a)
+    /usr/X11R6/lib/x86_64-pc-linux-gnu/lib/(lib)zlibstatic(\.so|\.a)
+    /usr/X11R6/lib/x86_64-pc-linux-gnu/(lib)z(\.so|\.a)
+    /usr/X11R6/lib/x86_64-pc-linux-gnu/(lib)zlib(\.so|\.a)
+    /usr/X11R6/lib/x86_64-pc-linux-gnu/(lib)zdll(\.so|\.a)
+    /usr/X11R6/lib/x86_64-pc-linux-gnu/(lib)zlib1(\.so|\.a)
+    /usr/X11R6/lib/x86_64-pc-linux-gnu/(lib)zlibstatic(\.so|\.a)
+    /usr/X11R6/lib/lib/(lib)z(\.so|\.a)
+    /usr/X11R6/lib/lib/(lib)zlib(\.so|\.a)
+    /usr/X11R6/lib/lib/(lib)zdll(\.so|\.a)
+    /usr/X11R6/lib/lib/(lib)zlib1(\.so|\.a)
+    /usr/X11R6/lib/lib/(lib)zlibstatic(\.so|\.a)
+    /usr/X11R6/lib/(lib)z(\.so|\.a)
+    /usr/X11R6/lib/(lib)zlib(\.so|\.a)
+    /usr/X11R6/lib/(lib)zdll(\.so|\.a)
+    /usr/X11R6/lib/(lib)zlib1(\.so|\.a)
+    /usr/X11R6/lib/(lib)zlibstatic(\.so|\.a)
+    /usr/X11R6/lib/(lib)z(\.so|\.a)
+    /usr/X11R6/lib/(lib)zlib(\.so|\.a)
+    /usr/X11R6/lib/(lib)zdll(\.so|\.a)
+    /usr/X11R6/lib/(lib)zlib1(\.so|\.a)
+    /usr/X11R6/lib/(lib)zlibstatic(\.so|\.a)
+    /usr/X11R6/(lib)z(\.so|\.a)
+    /usr/X11R6/(lib)zlib(\.so|\.a)
+    /usr/X11R6/(lib)zdll(\.so|\.a)
+    /usr/X11R6/(lib)zlib1(\.so|\.a)
+    /usr/X11R6/(lib)zlibstatic(\.so|\.a)
+    /usr/pkg/lib/x86_64-pc-linux-gnu/lib/(lib)z(\.so|\.a)
+    /usr/pkg/lib/x86_64-pc-linux-gnu/lib/(lib)zlib(\.so|\.a)
+    /usr/pkg/lib/x86_64-pc-linux-gnu/lib/(lib)zdll(\.so|\.a)
+    /usr/pkg/lib/x86_64-pc-linux-gnu/lib/(lib)zlib1(\.so|\.a)
+    /usr/pkg/lib/x86_64-pc-linux-gnu/lib/(lib)zlibstatic(\.so|\.a)
+    /usr/pkg/lib/x86_64-pc-linux-gnu/(lib)z(\.so|\.a)
+    /usr/pkg/lib/x86_64-pc-linux-gnu/(lib)zlib(\.so|\.a)
+    /usr/pkg/lib/x86_64-pc-linux-gnu/(lib)zdll(\.so|\.a)
+    /usr/pkg/lib/x86_64-pc-linux-gnu/(lib)zlib1(\.so|\.a)
+    /usr/pkg/lib/x86_64-pc-linux-gnu/(lib)zlibstatic(\.so|\.a)
+    /usr/pkg/lib/lib/(lib)z(\.so|\.a)
+    /usr/pkg/lib/lib/(lib)zlib(\.so|\.a)
+    /usr/pkg/lib/lib/(lib)zdll(\.so|\.a)
+    /usr/pkg/lib/lib/(lib)zlib1(\.so|\.a)
+    /usr/pkg/lib/lib/(lib)zlibstatic(\.so|\.a)
+    /usr/pkg/lib/(lib)z(\.so|\.a)
+    /usr/pkg/lib/(lib)zlib(\.so|\.a)
+    /usr/pkg/lib/(lib)zdll(\.so|\.a)
+    /usr/pkg/lib/(lib)zlib1(\.so|\.a)
+    /usr/pkg/lib/(lib)zlibstatic(\.so|\.a)
+    /usr/pkg/lib/(lib)z(\.so|\.a)
+    /usr/pkg/lib/(lib)zlib(\.so|\.a)
+    /usr/pkg/lib/(lib)zdll(\.so|\.a)
+    /usr/pkg/lib/(lib)zlib1(\.so|\.a)
+    /usr/pkg/lib/(lib)zlibstatic(\.so|\.a)
+    /usr/pkg/(lib)z(\.so|\.a)
+    /usr/pkg/(lib)zlib(\.so|\.a)
+    /usr/pkg/(lib)zdll(\.so|\.a)
+    /usr/pkg/(lib)zlib1(\.so|\.a)
+    /usr/pkg/(lib)zlibstatic(\.so|\.a)
+    /opt/lib/x86_64-pc-linux-gnu/lib/(lib)z(\.so|\.a)
+    /opt/lib/x86_64-pc-linux-gnu/lib/(lib)zlib(\.so|\.a)
+    /opt/lib/x86_64-pc-linux-gnu/lib/(lib)zdll(\.so|\.a)
+    /opt/lib/x86_64-pc-linux-gnu/lib/(lib)zlib1(\.so|\.a)
+    /opt/lib/x86_64-pc-linux-gnu/lib/(lib)zlibstatic(\.so|\.a)
+    /opt/lib/x86_64-pc-linux-gnu/(lib)z(\.so|\.a)
+    /opt/lib/x86_64-pc-linux-gnu/(lib)zlib(\.so|\.a)
+    /opt/lib/x86_64-pc-linux-gnu/(lib)zdll(\.so|\.a)
+    /opt/lib/x86_64-pc-linux-gnu/(lib)zlib1(\.so|\.a)
+    /opt/lib/x86_64-pc-linux-gnu/(lib)zlibstatic(\.so|\.a)
+    /opt/lib/lib/(lib)z(\.so|\.a)
+    /opt/lib/lib/(lib)zlib(\.so|\.a)
+    /opt/lib/lib/(lib)zdll(\.so|\.a)
+    /opt/lib/lib/(lib)zlib1(\.so|\.a)
+    /opt/lib/lib/(lib)zlibstatic(\.so|\.a)
+    /opt/lib/(lib)z(\.so|\.a)
+    /opt/lib/(lib)zlib(\.so|\.a)
+    /opt/lib/(lib)zdll(\.so|\.a)
+    /opt/lib/(lib)zlib1(\.so|\.a)
+    /opt/lib/(lib)zlibstatic(\.so|\.a)
+    /opt/lib/(lib)z(\.so|\.a)
+    /opt/lib/(lib)zlib(\.so|\.a)
+    /opt/lib/(lib)zdll(\.so|\.a)
+    /opt/lib/(lib)zlib1(\.so|\.a)
+    /opt/lib/(lib)zlibstatic(\.so|\.a)
+    /opt/(lib)z(\.so|\.a)
+    /opt/(lib)zlib(\.so|\.a)
+    /opt/(lib)zdll(\.so|\.a)
+    /opt/(lib)zlib1(\.so|\.a)
+    /opt/(lib)zlibstatic(\.so|\.a)
+    /usr/lib/X11/lib/(lib)z(\.so|\.a)
+    /usr/lib/X11/lib/(lib)zlib(\.so|\.a)
+    /usr/lib/X11/lib/(lib)zdll(\.so|\.a)
+    /usr/lib/X11/lib/(lib)zlib1(\.so|\.a)
+    /usr/lib/X11/lib/(lib)zlibstatic(\.so|\.a)
+    /usr/lib/X11/(lib)z(\.so|\.a)
+    /usr/lib/X11/(lib)zlib(\.so|\.a)
+    /usr/lib/X11/(lib)zdll(\.so|\.a)
+    /usr/lib/X11/(lib)zlib1(\.so|\.a)
+    /usr/lib/X11/(lib)zlibstatic(\.so|\.a)
+    /registry/lib/(lib)z(\.so|\.a)
+    /registry/lib/(lib)zlib(\.so|\.a)
+    /registry/lib/(lib)zdll(\.so|\.a)
+    /registry/lib/(lib)zlib1(\.so|\.a)
+    /registry/lib/(lib)zlibstatic(\.so|\.a)
+    /registry/(lib)z(\.so|\.a)
+    /registry/(lib)zlib(\.so|\.a)
+    /registry/(lib)zdll(\.so|\.a)
+    /registry/(lib)zlib1(\.so|\.a)
+    /registry/(lib)zlibstatic(\.so|\.a)
+    /zlib/lib/(lib)z(\.so|\.a)
+    /zlib/lib/(lib)zlib(\.so|\.a)
+    /zlib/lib/(lib)zdll(\.so|\.a)
+    /zlib/lib/(lib)zlib1(\.so|\.a)
+    /zlib/lib/(lib)zlibstatic(\.so|\.a)
+    /zlib/(lib)z(\.so|\.a)
+    /zlib/(lib)zlib(\.so|\.a)
+    /zlib/(lib)zdll(\.so|\.a)
+    /zlib/(lib)zlib1(\.so|\.a)
+    /zlib/(lib)zlibstatic(\.so|\.a)
@@ -927 +1061 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /usr/lib/x86_64-linux-gnu/libz.so
+  The item was not found.
@@ -1043,12 +1177,12 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /usr/local/lib/x86_64-linux-gnu/lib/(lib)zd(\.so|\.a)
-    /usr/local/lib/x86_64-linux-gnu/lib/(lib)zlibd(\.so|\.a)
-    /usr/local/lib/x86_64-linux-gnu/lib/(lib)zdlld(\.so|\.a)
-    /usr/local/lib/x86_64-linux-gnu/lib/(lib)zlibd1(\.so|\.a)
-    /usr/local/lib/x86_64-linux-gnu/lib/(lib)zlib1d(\.so|\.a)
-    /usr/local/lib/x86_64-linux-gnu/lib/(lib)zlibstaticd(\.so|\.a)
-    /usr/local/lib/x86_64-linux-gnu/(lib)zd(\.so|\.a)
-    /usr/local/lib/x86_64-linux-gnu/(lib)zlibd(\.so|\.a)
-    /usr/local/lib/x86_64-linux-gnu/(lib)zdlld(\.so|\.a)
-    /usr/local/lib/x86_64-linux-gnu/(lib)zlibd1(\.so|\.a)
-    /usr/local/lib/x86_64-linux-gnu/(lib)zlib1d(\.so|\.a)
-    /usr/local/lib/x86_64-linux-gnu/(lib)zlibstaticd(\.so|\.a)
+    /usr/local/lib/x86_64-pc-linux-gnu/lib/(lib)zd(\.so|\.a)
+    /usr/local/lib/x86_64-pc-linux-gnu/lib/(lib)zlibd(\.so|\.a)
+    /usr/local/lib/x86_64-pc-linux-gnu/lib/(lib)zdlld(\.so|\.a)
+    /usr/local/lib/x86_64-pc-linux-gnu/lib/(lib)zlibd1(\.so|\.a)
+    /usr/local/lib/x86_64-pc-linux-gnu/lib/(lib)zlib1d(\.so|\.a)
+    /usr/local/lib/x86_64-pc-linux-gnu/lib/(lib)zlibstaticd(\.so|\.a)
+    /usr/local/lib/x86_64-pc-linux-gnu/(lib)zd(\.so|\.a)
+    /usr/local/lib/x86_64-pc-linux-gnu/(lib)zlibd(\.so|\.a)
+    /usr/local/lib/x86_64-pc-linux-gnu/(lib)zdlld(\.so|\.a)
+    /usr/local/lib/x86_64-pc-linux-gnu/(lib)zlibd1(\.so|\.a)
+    /usr/local/lib/x86_64-pc-linux-gnu/(lib)zlib1d(\.so|\.a)
+    /usr/local/lib/x86_64-pc-linux-gnu/(lib)zlibstaticd(\.so|\.a)
@@ -1079,12 +1213,12 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /usr/lib/x86_64-linux-gnu/lib/(lib)zd(\.so|\.a)
-    /usr/lib/x86_64-linux-gnu/lib/(lib)zlibd(\.so|\.a)
-    /usr/lib/x86_64-linux-gnu/lib/(lib)zdlld(\.so|\.a)
-    /usr/lib/x86_64-linux-gnu/lib/(lib)zlibd1(\.so|\.a)
-    /usr/lib/x86_64-linux-gnu/lib/(lib)zlib1d(\.so|\.a)
-    /usr/lib/x86_64-linux-gnu/lib/(lib)zlibstaticd(\.so|\.a)
-    /usr/lib/x86_64-linux-gnu/(lib)zd(\.so|\.a)
-    /usr/lib/x86_64-linux-gnu/(lib)zlibd(\.so|\.a)
-    /usr/lib/x86_64-linux-gnu/(lib)zdlld(\.so|\.a)
-    /usr/lib/x86_64-linux-gnu/(lib)zlibd1(\.so|\.a)
-    /usr/lib/x86_64-linux-gnu/(lib)zlib1d(\.so|\.a)
-    /usr/lib/x86_64-linux-gnu/(lib)zlibstaticd(\.so|\.a)
+    /usr/lib/x86_64-pc-linux-gnu/lib/(lib)zd(\.so|\.a)
+    /usr/lib/x86_64-pc-linux-gnu/lib/(lib)zlibd(\.so|\.a)
+    /usr/lib/x86_64-pc-linux-gnu/lib/(lib)zdlld(\.so|\.a)
+    /usr/lib/x86_64-pc-linux-gnu/lib/(lib)zlibd1(\.so|\.a)
+    /usr/lib/x86_64-pc-linux-gnu/lib/(lib)zlib1d(\.so|\.a)
+    /usr/lib/x86_64-pc-linux-gnu/lib/(lib)zlibstaticd(\.so|\.a)
+    /usr/lib/x86_64-pc-linux-gnu/(lib)zd(\.so|\.a)
+    /usr/lib/x86_64-pc-linux-gnu/(lib)zlibd(\.so|\.a)
+    /usr/lib/x86_64-pc-linux-gnu/(lib)zdlld(\.so|\.a)
+    /usr/lib/x86_64-pc-linux-gnu/(lib)zlibd1(\.so|\.a)
+    /usr/lib/x86_64-pc-linux-gnu/(lib)zlib1d(\.so|\.a)
+    /usr/lib/x86_64-pc-linux-gnu/(lib)zlibstaticd(\.so|\.a)
@@ -1115,12 +1249,12 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /lib/x86_64-linux-gnu/lib/(lib)zd(\.so|\.a)
-    /lib/x86_64-linux-gnu/lib/(lib)zlibd(\.so|\.a)
-    /lib/x86_64-linux-gnu/lib/(lib)zdlld(\.so|\.a)
-    /lib/x86_64-linux-gnu/lib/(lib)zlibd1(\.so|\.a)
-    /lib/x86_64-linux-gnu/lib/(lib)zlib1d(\.so|\.a)
-    /lib/x86_64-linux-gnu/lib/(lib)zlibstaticd(\.so|\.a)
-    /lib/x86_64-linux-gnu/(lib)zd(\.so|\.a)
-    /lib/x86_64-linux-gnu/(lib)zlibd(\.so|\.a)
-    /lib/x86_64-linux-gnu/(lib)zdlld(\.so|\.a)
-    /lib/x86_64-linux-gnu/(lib)zlibd1(\.so|\.a)
-    /lib/x86_64-linux-gnu/(lib)zlib1d(\.so|\.a)
-    /lib/x86_64-linux-gnu/(lib)zlibstaticd(\.so|\.a)
+    /lib/x86_64-pc-linux-gnu/lib/(lib)zd(\.so|\.a)
+    /lib/x86_64-pc-linux-gnu/lib/(lib)zlibd(\.so|\.a)
+    /lib/x86_64-pc-linux-gnu/lib/(lib)zdlld(\.so|\.a)
+    /lib/x86_64-pc-linux-gnu/lib/(lib)zlibd1(\.so|\.a)
+    /lib/x86_64-pc-linux-gnu/lib/(lib)zlib1d(\.so|\.a)
+    /lib/x86_64-pc-linux-gnu/lib/(lib)zlibstaticd(\.so|\.a)
+    /lib/x86_64-pc-linux-gnu/(lib)zd(\.so|\.a)
+    /lib/x86_64-pc-linux-gnu/(lib)zlibd(\.so|\.a)
+    /lib/x86_64-pc-linux-gnu/(lib)zdlld(\.so|\.a)
+    /lib/x86_64-pc-linux-gnu/(lib)zlibd1(\.so|\.a)
+    /lib/x86_64-pc-linux-gnu/(lib)zlib1d(\.so|\.a)
+    /lib/x86_64-pc-linux-gnu/(lib)zlibstaticd(\.so|\.a)
@@ -1139,12 +1273,12 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /usr/X11R6/lib/x86_64-linux-gnu/lib/(lib)zd(\.so|\.a)
-    /usr/X11R6/lib/x86_64-linux-gnu/lib/(lib)zlibd(\.so|\.a)
-    /usr/X11R6/lib/x86_64-linux-gnu/lib/(lib)zdlld(\.so|\.a)
-    /usr/X11R6/lib/x86_64-linux-gnu/lib/(lib)zlibd1(\.so|\.a)
-    /usr/X11R6/lib/x86_64-linux-gnu/lib/(lib)zlib1d(\.so|\.a)
-    /usr/X11R6/lib/x86_64-linux-gnu/lib/(lib)zlibstaticd(\.so|\.a)
-    /usr/X11R6/lib/x86_64-linux-gnu/(lib)zd(\.so|\.a)
-    /usr/X11R6/lib/x86_64-linux-gnu/(lib)zlibd(\.so|\.a)
-    /usr/X11R6/lib/x86_64-linux-gnu/(lib)zdlld(\.so|\.a)
-    /usr/X11R6/lib/x86_64-linux-gnu/(lib)zlibd1(\.so|\.a)
-    /usr/X11R6/lib/x86_64-linux-gnu/(lib)zlib1d(\.so|\.a)
-    /usr/X11R6/lib/x86_64-linux-gnu/(lib)zlibstaticd(\.so|\.a)
+    /usr/X11R6/lib/x86_64-pc-linux-gnu/lib/(lib)zd(\.so|\.a)
+    /usr/X11R6/lib/x86_64-pc-linux-gnu/lib/(lib)zlibd(\.so|\.a)
+    /usr/X11R6/lib/x86_64-pc-linux-gnu/lib/(lib)zdlld(\.so|\.a)
+    /usr/X11R6/lib/x86_64-pc-linux-gnu/lib/(lib)zlibd1(\.so|\.a)
+    /usr/X11R6/lib/x86_64-pc-linux-gnu/lib/(lib)zlib1d(\.so|\.a)
+    /usr/X11R6/lib/x86_64-pc-linux-gnu/lib/(lib)zlibstaticd(\.so|\.a)
+    /usr/X11R6/lib/x86_64-pc-linux-gnu/(lib)zd(\.so|\.a)
+    /usr/X11R6/lib/x86_64-pc-linux-gnu/(lib)zlibd(\.so|\.a)
+    /usr/X11R6/lib/x86_64-pc-linux-gnu/(lib)zdlld(\.so|\.a)
+    /usr/X11R6/lib/x86_64-pc-linux-gnu/(lib)zlibd1(\.so|\.a)
+    /usr/X11R6/lib/x86_64-pc-linux-gnu/(lib)zlib1d(\.so|\.a)
+    /usr/X11R6/lib/x86_64-pc-linux-gnu/(lib)zlibstaticd(\.so|\.a)
@@ -1175,12 +1309,12 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /usr/pkg/lib/x86_64-linux-gnu/lib/(lib)zd(\.so|\.a)
-    /usr/pkg/lib/x86_64-linux-gnu/lib/(lib)zlibd(\.so|\.a)
-    /usr/pkg/lib/x86_64-linux-gnu/lib/(lib)zdlld(\.so|\.a)
-    /usr/pkg/lib/x86_64-linux-gnu/lib/(lib)zlibd1(\.so|\.a)
-    /usr/pkg/lib/x86_64-linux-gnu/lib/(lib)zlib1d(\.so|\.a)
-    /usr/pkg/lib/x86_64-linux-gnu/lib/(lib)zlibstaticd(\.so|\.a)
-    /usr/pkg/lib/x86_64-linux-gnu/(lib)zd(\.so|\.a)
-    /usr/pkg/lib/x86_64-linux-gnu/(lib)zlibd(\.so|\.a)
-    /usr/pkg/lib/x86_64-linux-gnu/(lib)zdlld(\.so|\.a)
-    /usr/pkg/lib/x86_64-linux-gnu/(lib)zlibd1(\.so|\.a)
-    /usr/pkg/lib/x86_64-linux-gnu/(lib)zlib1d(\.so|\.a)
-    /usr/pkg/lib/x86_64-linux-gnu/(lib)zlibstaticd(\.so|\.a)
+    /usr/pkg/lib/x86_64-pc-linux-gnu/lib/(lib)zd(\.so|\.a)
+    /usr/pkg/lib/x86_64-pc-linux-gnu/lib/(lib)zlibd(\.so|\.a)
+    /usr/pkg/lib/x86_64-pc-linux-gnu/lib/(lib)zdlld(\.so|\.a)
+    /usr/pkg/lib/x86_64-pc-linux-gnu/lib/(lib)zlibd1(\.so|\.a)
+    /usr/pkg/lib/x86_64-pc-linux-gnu/lib/(lib)zlib1d(\.so|\.a)
+    /usr/pkg/lib/x86_64-pc-linux-gnu/lib/(lib)zlibstaticd(\.so|\.a)
+    /usr/pkg/lib/x86_64-pc-linux-gnu/(lib)zd(\.so|\.a)
+    /usr/pkg/lib/x86_64-pc-linux-gnu/(lib)zlibd(\.so|\.a)
+    /usr/pkg/lib/x86_64-pc-linux-gnu/(lib)zdlld(\.so|\.a)
+    /usr/pkg/lib/x86_64-pc-linux-gnu/(lib)zlibd1(\.so|\.a)
+    /usr/pkg/lib/x86_64-pc-linux-gnu/(lib)zlib1d(\.so|\.a)
+    /usr/pkg/lib/x86_64-pc-linux-gnu/(lib)zlibstaticd(\.so|\.a)
@@ -1211,12 +1345,12 @@ CMake Debug Log at /usr/share/cmake-3.21
-    /opt/lib/x86_64-linux-gnu/lib/(lib)zd(\.so|\.a)
-    /opt/lib/x86_64-linux-gnu/lib/(lib)zlibd(\.so|\.a)
-    /opt/lib/x86_64-linux-gnu/lib/(lib)zdlld(\.so|\.a)
-    /opt/lib/x86_64-linux-gnu/lib/(lib)zlibd1(\.so|\.a)
-    /opt/lib/x86_64-linux-gnu/lib/(lib)zlib1d(\.so|\.a)
-    /opt/lib/x86_64-linux-gnu/lib/(lib)zlibstaticd(\.so|\.a)
-    /opt/lib/x86_64-linux-gnu/(lib)zd(\.so|\.a)
-    /opt/lib/x86_64-linux-gnu/(lib)zlibd(\.so|\.a)
-    /opt/lib/x86_64-linux-gnu/(lib)zdlld(\.so|\.a)
-    /opt/lib/x86_64-linux-gnu/(lib)zlibd1(\.so|\.a)
-    /opt/lib/x86_64-linux-gnu/(lib)zlib1d(\.so|\.a)
-    /opt/lib/x86_64-linux-gnu/(lib)zlibstaticd(\.so|\.a)
+    /opt/lib/x86_64-pc-linux-gnu/lib/(lib)zd(\.so|\.a)
+    /opt/lib/x86_64-pc-linux-gnu/lib/(lib)zlibd(\.so|\.a)
+    /opt/lib/x86_64-pc-linux-gnu/lib/(lib)zdlld(\.so|\.a)
+    /opt/lib/x86_64-pc-linux-gnu/lib/(lib)zlibd1(\.so|\.a)
+    /opt/lib/x86_64-pc-linux-gnu/lib/(lib)zlib1d(\.so|\.a)
+    /opt/lib/x86_64-pc-linux-gnu/lib/(lib)zlibstaticd(\.so|\.a)
+    /opt/lib/x86_64-pc-linux-gnu/(lib)zd(\.so|\.a)
+    /opt/lib/x86_64-pc-linux-gnu/(lib)zlibd(\.so|\.a)
+    /opt/lib/x86_64-pc-linux-gnu/(lib)zdlld(\.so|\.a)
+    /opt/lib/x86_64-pc-linux-gnu/(lib)zlibd1(\.so|\.a)
+    /opt/lib/x86_64-pc-linux-gnu/(lib)zlib1d(\.so|\.a)
+    /opt/lib/x86_64-pc-linux-gnu/(lib)zlibstaticd(\.so|\.a)
@@ -1290 +1424 @@ Call Stack (most recent call first):
--- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.11") 
+-- Could NOT find ZLIB (missing: ZLIB_LIBRARY) (found version "1.2.11")
@@ -1299,0 +1434 @@ CMake Debug Log at CMakeLists.txt:5 (fin
+  The module is considered not found due to ZLIB_FOUND being FALSE.

[MaskRay]

In D107799#3027607, @lebedev.ri wrote:

Hello. This broke the world for me.
Both @sylvestre.ledru's llvm-main package for debian sid (https://apt.llvm.org/)
and clean local builds are broken:

What's your build? I am using Debian testing and switching to LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=on works for me well.

Are you doing something special/weird? If so, I don't think it deserves a revert. Please revert your revert.

[sylvestre.ledru]

if you are talking about apt.llvm.org, I think I fixed most of the issues now.

It was caused by a few errors on my side and changes because of this patch - see https://reviews.llvm.org/D89013
For example, the triple not being the same as Debian's.

[lebedev.ri]

TLDR: it reproduces if there is some runtime library installed (e.g. libc++-14-dev),
then there is /usr/lib/llvm-14/lib/x86_64-pc-linux-gnu/,
which is i guess picked by cmake as the triple to use.

mv /usr/lib/llvm-14/lib/x86_64-pc-linux-gnu /usr/lib/llvm-14/lib/x86_64-linux-gnu hides the issue,
which highlights that the fix is as simple as understanding why clang is using x86_64-pc-linux-gnu triple on debian.
Please do not reland, the bug is real.

Thanks.

[MaskRay]

In D107799#3028818, @lebedev.ri wrote:

TLDR: it reproduces if there is some runtime library installed (e.g. libc++-14-dev),
then there is /usr/lib/llvm-14/lib/x86_64-pc-linux-gnu/,
which is i guess picked by cmake as the triple to use.

mv /usr/lib/llvm-14/lib/x86_64-pc-linux-gnu /usr/lib/llvm-14/lib/x86_64-linux-gnu hides the issue,
which highlights that the fix is as simple as understanding why clang is using x86_64-pc-linux-gnu triple on debian.
Please do not reland, the bug is real.

Thanks.

I am 95% confident that the bug lies on Debian side: it configures something special so that the triple shifts from the normal x86_64-linux-gnu (Debian multiarch, gcc -dumpmachine output) or x86_64-unknown-linux-gnu (many other Linux distros; vanilla gcc built on Debian/Ubuntu) to x86_64-pc-linux-gnu.

I may have a hard time making debootstrap work to reproduce your environment:

% sudo debootstrap --arch amd64 unstable . http://deb.debian.org/debian
...
I: Unpacking the base system...
W: Failure trying to run: chroot "/tmp/chroot/." dpkg --force-overwrite --force-confold --skip-same-version --install /var/cache/apt/archives/libapparmor1_3.0.3-2_amd64.deb /var/cache/apt/archives/libargon2-1_0~20171227-0.2_amd64.deb /var/cache/apt/archives/libcryptsetup12_2%3a2.4.0-1_amd64.deb /var/cache/apt/archives/libip4tc2_1.8.7-1_amd64.deb /var/cache/apt/archives/libjson-c5_0.15-2_amd64.deb /var/cache/apt/archives/libkmod2_29-1_amd64.deb /var/cache/apt/archives/dmsetup_2%3a1.02.175-2.1_amd64.deb /var/cache/apt/archives/libdevmapper1.02.1_2%3a1.02.175-2.1_amd64.deb /var/cache/apt/archives/systemd_247.9-3_amd64.deb
W: See /tmp/chroot/./debootstrap/debootstrap.log for details (possibly the package systemd is at fault)

Advice to the next Clang version on Debian, consider -DLLVM_HOST_TRIPLE=x86_64-linux-gnu instead of pc if D110663 is accepted.

---

During investigation, I find one possible source of pc:

sh llvm/cmake/config.guess => x86_64-unknown-linux-gnu
linux32 sh llvm/cmake/config.guess => i686-pc-linux-gnu

But I cannot find anything else in llvm-project which may lead to pc.

[lebedev.ri]

In D107799#3028862, @MaskRay wrote:

In D107799#3028818, @lebedev.ri wrote:

TLDR: it reproduces if there is some runtime library installed (e.g. libc++-14-dev),
then there is /usr/lib/llvm-14/lib/x86_64-pc-linux-gnu/,
which is i guess picked by cmake as the triple to use.

mv /usr/lib/llvm-14/lib/x86_64-pc-linux-gnu /usr/lib/llvm-14/lib/x86_64-linux-gnu hides the issue,
which highlights that the fix is as simple as understanding why clang is using x86_64-pc-linux-gnu triple on debian.
Please do not reland, the bug is real.

Thanks.

I am 95% confident that the bug lies on Debian side:

I'm 99.99995% sure that things were working before this change, so it is to blame, not distro.

it configures something special so that the triple shifts from the normal x86_64-linux-gnu (Debian multiarch, gcc -dumpmachine output) or x86_64-unknown-linux-gnu (many other Linux distros; vanilla gcc built on Debian/Ubuntu) to x86_64-pc-linux-gnu.

Does it? As far as i can tell gcc's triple is correct, and in llvm's cmake *_TRIPLE are all x86_64-unknown-linux-gnu

$ gcc -dumpmachine
x86_64-linux-gnu

So i have no idea why clang says it is targeting x86_64-pc-linux-gnu.

I may have a hard time making debootstrap work to reproduce your environment:

% sudo debootstrap --arch amd64 unstable . http://deb.debian.org/debian
...
I: Unpacking the base system...
W: Failure trying to run: chroot "/tmp/chroot/." dpkg --force-overwrite --force-confold --skip-same-version --install /var/cache/apt/archives/libapparmor1_3.0.3-2_amd64.deb /var/cache/apt/archives/libargon2-1_0~20171227-0.2_amd64.deb /var/cache/apt/archives/libcryptsetup12_2%3a2.4.0-1_amd64.deb /var/cache/apt/archives/libip4tc2_1.8.7-1_amd64.deb /var/cache/apt/archives/libjson-c5_0.15-2_amd64.deb /var/cache/apt/archives/libkmod2_29-1_amd64.deb /var/cache/apt/archives/dmsetup_2%3a1.02.175-2.1_amd64.deb /var/cache/apt/archives/libdevmapper1.02.1_2%3a1.02.175-2.1_amd64.deb /var/cache/apt/archives/systemd_247.9-3_amd64.deb
W: See /tmp/chroot/./debootstrap/debootstrap.log for details (possibly the package systemd is at fault)

It works for me,
Basically, create fresh debian unstable chroot, install clang + libc++-dev from https://apt.llvm.org/ (NOT THE ONES FROM DEBIAN ITSELF!),
install cmake + zlib1g-dev, and see my initial comment for the actual reproduction.

Advice to the next Clang version on Debian, consider -DLLVM_HOST_TRIPLE=x86_64-linux-gnu instead of pc if D110663 is accepted.

I would very much like for LLVM to work on debian, of all things, out of the box :S

---

During investigation, I find one possible source of pc:

sh llvm/cmake/config.guess => x86_64-unknown-linux-gnu
linux32 sh llvm/cmake/config.guess => i686-pc-linux-gnu

But I cannot find anything else in llvm-project which may lead to pc.

Yeah, i'm not sure why that is happening yet.

[sylvestre.ledru]

I agree with @lebedev.ri : it is causing significant regressions which aren't trivial to debug for someone building firefox

[tstellar]

llvm/CMakeLists.txt

674–679

Doesn't clang need to be updated to find the compiler-rt libs in the per-target dirs? It currently only looks in CLANG_RESOURCE_DIR from what I can tell.

[manojgupta]

Assuming CMAKE_SYSTEM_NAME implies the machine, clang is being built on,
it may also break baremetal targets since clang does not currently handle per ABI runtime dirs for baremetal targets
e.g. see the problem in https://reviews.llvm.org/D98452

[MaskRay]

The runtime builds use -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=off.
Non-runtime builds for libc++/libc++abi/libunwind have been deprecated (D112724).
Over time, it seems clear that non-runtime builds for them will more and more likely break.
I think it's important for non-runtime and runtime builds to be similar in the file hierarchy (-DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=on) so that eventually we can converge to one single supported scheme.

I don't really track development in runtime builds. But it seems many changes have been made.
I will still try relanding the -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=on change. I use Debian testing and I don't see any breakage.
If people are using Debian and find things break, please help out and give detailed reproduce.

[mstorsjo]

In D107799#3555458, @MaskRay wrote:

The runtime builds use -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=off.
Non-runtime builds for libc++/libc++abi/libunwind have been deprecated (D112724).

This statement is a bit inaccurate. What you refer to as "runtime builds" can mean two different things:

• Pointing CMake at <monorepo>/llvm and including the runtimes with -DLLVM_ENABLE_RUNTIMES (so it builds llvm+clang and then uses the just-built clang to build the runtimes)
• Pointing CMake at <monorepo/runtimes and building libcxx/etc with any compiler you like (as long as it's new enough)

Both of them are still supported for libc++/libcxx+abi/libunwind. The former defaults to LLVM_ENABLE_PER_TARGET_RUNTIME_DIR on (as far as I know), the latter doesn't.

[manojgupta]

Yes, LLVM_ENABLE_RUNTIMES and LLVM_ENABLE_PER_TARGET_RUNTIME_DIR are two separate things.

[MaskRay]

In D107799#3556057, @manojgupta wrote:

Yes, LLVM_ENABLE_RUNTIMES and LLVM_ENABLE_PER_TARGET_RUNTIME_DIR are two separate things.

They are separate, but LLVM_ENABLE_RUNTIMES implies LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=on for non-AIX.

I haven't figured out what runtimes/ is yet.

[phosek]

In D107799#3557780, @MaskRay wrote:

In D107799#3556057, @manojgupta wrote:

Yes, LLVM_ENABLE_RUNTIMES and LLVM_ENABLE_PER_TARGET_RUNTIME_DIR are two separate things.

They are separate, but LLVM_ENABLE_RUNTIMES implies LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=on for non-AIX.

I haven't figured out what runtimes/ is yet.

runtimes/ is what we now refer to the runtimes build: it's a way to build runtimes in a single CMake build, sharing CMake modules, common configuration bits, etc.
llvm/runtimes/ is what we now refer to the bootstrapping build: it's a way to build a complete toolchain by first building Clang and LLVM tools, and then using those to build runtimes (using the runtimes build), potentially for multiple targets (using the runtimes build multiple times, once for each target).

The directory naming is a bit confusing and we should probably consider renaming llvm/runtimes/ to bootstrap/ (or perhaps clang/bootstrap/) to make it clearer.

To make things even more confusing, there's also clang/runtime which is the predecessor of the bootstrapping build. I don't know if anyone is still using it, we might be able to delete it.

[glandium]

This is causing problems finding system zlib when compiling clang with clang, because when using the target directories for runtimes, cmake determines the target name to match that, and then proceeds to try to find zlib.h in /usr/include/$target, which doesn't work when $target doesn't match the system multi-arch triple, which it doesn't on Debian systems because the runtime directory and the cmake target is e.g. x86_64-unknown-linux-gnu, and the system multiarch triple is x86_64-linux-gnu...

[glandium]

I suspect even eliminating the machine part of the target, there could still be problems with i386 vs i686.

[glandium]

IOW, nothing changed since https://reviews.llvm.org/D107799#3027607

[MaskRay]

In D107799#3565520, @glandium wrote:

This is causing problems finding system zlib when compiling clang with clang, because when using the target directories for runtimes, cmake determines the target name to match that, and then proceeds to try to find zlib.h in /usr/include/$target, which doesn't work when $target doesn't match the system multi-arch triple, which it doesn't on Debian systems because the runtime directory and the cmake target is e.g. x86_64-unknown-linux-gnu, and the system multiarch triple is x86_64-linux-gnu...

Please give a reproduce instruction.

I am using Debian testing x86-64 and cmake -GNinja -Sllvm -B/tmp/RelA -DLLVM_ENABLE_PROJECTS='clang;clang-tools-extra;flang;lldb;lld;compiler-rt;openmp;mlir;polly;cross-project-tests' -DLLVM_ENABLE_RUNTIMES='libcxx;libcxxabi;libunwind' -DCMAKE_CXX_COMPILER=$HOME/Stable/bin/clang++ -DCMAKE_C_COMPILER=$HOME/Stable/bin/clang works fine. The upstream debian build bots looks happy.
lld/test/ELF/compress-debug-sections.s is a zlib test and it works fine.

Here is another build at /tmp/out/custom1. -L/usr/lib/x86_64-linux-gnu is there.

% /tmp/out/custom1/bin/clang a.c '-###' |& sed -E 's/ "?-[iIL]/\n&/g' 
clang version 15.0.0
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /tmp/out/custom1/bin
 "/tmp/out/custom1/bin/clang-15" "-cc1" "-triple" "x86_64-unknown-linux-gnu" "-emit-obj" "-mrelax-all" "--mrelax-relocations" "-disable-free" "-clear-ast-before-backend" "-disable-llvm-verifier" "-discard-value-names" "-main-file-name" "a.c" "-mrelocation-model" "pic" "-pic-level" "2" "-pic-is-pie" "-mframe-pointer=all" "-fmath-errno" "-ffp-contract=on" "-fno-rounding-math" "-mconstructor-aliases" "-funwind-tables=2" "-target-cpu" "x86-64" "-tune-cpu" "generic" "-mllvm" "-treat-scalable-fixed-error-as-warning" "-debugger-tuning=gdb" "-fcoverage-compilation-dir=/tmp/c" "-resource-dir" "/tmp/out/custom1/lib/clang/15.0.0"
 "-internal-isystem" "/tmp/out/custom1/lib/clang/15.0.0/include"
 "-internal-isystem" "/usr/local/include"
 "-internal-isystem" "/usr/lib/gcc/x86_64-linux-gnu/12/../../../../x86_64-linux-gnu/include"
 "-internal-externc-isystem" "/usr/include/x86_64-linux-gnu"
 "-internal-externc-isystem" "/include"
 "-internal-externc-isystem" "/usr/include" "-fdebug-compilation-dir=/tmp/c" "-ferror-limit" "19" "-fgnuc-version=4.2.1" "-faddrsig" "-D__GCC_HAVE_DWARF2_CFI_ASM=1" "-o" "/tmp/a-59cd08.o" "-x" "c" "a.c"
 "/usr/local/bin/ld" "-pie" "--eh-frame-hdr" "-m" "elf_x86_64" "-dynamic-linker" "/lib64/ld-linux-x86-64.so.2" "-o" "a.out" "/lib/x86_64-linux-gnu/Scrt1.o" "/lib/x86_64-linux-gnu/crti.o" "/usr/lib/gcc/x86_64-linux-gnu/12/crtbeginS.o"
 "-L/usr/lib/gcc/x86_64-linux-gnu/12"
 "-L/usr/lib/gcc/x86_64-linux-gnu/12/../../../../lib64"
 "-L/lib/x86_64-linux-gnu"
 "-L/lib/../lib64"
 "-L/usr/lib/x86_64-linux-gnu"
 "-L/usr/lib/../lib64"
 "-L/lib"
 "-L/usr/lib" "/tmp/a-59cd08.o" "-lgcc" "--as-needed" "-lgcc_s" "--no-as-needed" "-lc" "-lgcc" "--as-needed" "-lgcc_s" "--no-as-needed" "/usr/lib/gcc/x86_64-linux-gnu/12/crtendS.o" "/lib/x86_64-linux-gnu/crtn.o"

If you have some builds which now have problems, you will need to specify -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=off now and figure out a long-term sustainable solution.

I suspect even eliminating the machine part of the target, there could still be problems with i386 vs i686.

My -DLLVM_TARGETS_TO_BUILD='X86' -DCMAKE_BUILD_TYPE=Release -DLLVM_DEFAULT_TARGET_TRIPLE=i386-unknown-linux-gnu build works fine on Debian testing x86-64.
/tmp/out/custom1/bin/clang -fsanitize=address a.c '-###' uses something like /tmp/out/custom1/lib/clang/15.0.0/lib/i386-unknown-linux-gnu/libclang_rt.asan.a and adds -L/usr/lib/i386-linux-gnu.

[glandium]

What is your $HOME/Stable/bin/clang? it has to be a clang with the patch applied.

[MaskRay]

In D107799#3565613, @glandium wrote:

What is your $HOME/Stable/bin/clang? it has to be a clang with the patch applied.

I use prebuilt clang from curl -s https://raw.githubusercontent.com/chromium/chromium/main/tools/clang/scripts/update.py | python3 - --output-dir=~/Stable.

But it is irrelevant here. Using GCC (cmake -GNinja -Sllvm -B/tmp/out/custom2 -DLLVM_ENABLE_PROJECTS='clang;clang-tools-extra;flang;lldb;lld;compiler-rt;openmp;mlir;polly;cross-project-tests' -DLLVM_ENABLE_RUNTIMES='libcxx;libcxxabi;libunwind' -DCMAKE_CXX_COMPILER=g++ -DCMAKE_C_COMPILER=gcc) works fine, too.

[glandium]

But it is irrelevant here

It is relevant, I'm telling you that's precisely the setup that's broken. That it builds with GCC *is* irrelevant.

[MaskRay]

In D107799#3565680, @glandium wrote:

But it is irrelevant here

It is relevant, I'm telling you that's precisely the setup that's broken. That it builds with GCC *is* irrelevant.

Please give the full command line as requested, not just describe what happened which others may not be able to comprehend.
I have given my cmake command line and here is additional information:

-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.11")
...
-- Found LibXml2: /usr/lib/x86_64-linux-gnu/libxml2.so (found version "2.9.13")

If I use -DCMAKE_INSTALL_PREFIX=/tmp/install/custom2, the ninja install installed clang can find libc++ and asan runtime as well:

% /tmp/install/custom2/bin/clang++ --stdlib=libc++ -fsanitize=address a.cc -Wl,-t
...
/tmp/install/custom2/lib/clang/15.0.0/lib/x86_64-unknown-linux-gnu/libclang_rt.asan_cxx.a(ubsan_type_hash_win.cpp.o)
/tmp/a-952d5e.o
/tmp/install/custom2/bin/../lib/x86_64-unknown-linux-gnu/libc++.so.1
/tmp/install/custom2/bin/../lib/x86_64-unknown-linux-gnu/libc++abi.so
/lib/x86_64-linux-gnu/libm.so.6
...
/usr/lib/x86_64-linux-gnu/libc_nonshared.a(elf-init.oS)
/usr/lib/x86_64-linux-gnu/libc_nonshared.a(atexit.oS)
/lib64/ld-linux-x86-64.so.2
/lib/x86_64-linux-gnu/libgcc_s.so.1
/usr/lib/gcc/x86_64-linux-gnu/11/crtendS.o
/lib/x86_64-linux-gnu/crtn.o

[glandium]

Minimal reproducer:

• git checkout 311f7839602344ca347816146edb68c0ffaaa060
• cmake llvm -B obj -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/tmp/llvm/stage1 -DLLVM_TARGETS_TO_BUILD=X86 -DLLVM_ENABLE_PROJECTS="clang" -DLLVM_ENABLE_RUNTIMES="compiler-rt;libcxx;libcxxabi"
• ninja -C obj install
• cmake llvm -B obj2 -G Ninja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS="clang" -DCMAKE_C_COMPILER=/tmp/llvm/stage1/bin/clang

The key to the failure is $clang/lib/x86_64-unknown-linux-gnu existing (even empty), which happens when enabling libcxx/libcxxabi.

[MaskRay]

In D107799#3569323, @glandium wrote:

Minimal reproducer:

• git checkout 311f7839602344ca347816146edb68c0ffaaa060
• cmake llvm -B obj -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/tmp/llvm/stage1 -DLLVM_TARGETS_TO_BUILD=X86 -DLLVM_ENABLE_PROJECTS="clang" -DLLVM_ENABLE_RUNTIMES="compiler-rt;libcxx;libcxxabi"
• ninja -C obj install
• cmake llvm -B obj2 -G Ninja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS="clang" -DCMAKE_C_COMPILER=/tmp/llvm/stage1/bin/clang

The key to the failure is $clang/lib/x86_64-unknown-linux-gnu existing (even empty), which happens when enabling libcxx/libcxxabi.

This instructions work for me on a Debian testing amd64 machine. ninja -C /tmp/out/obj2 passes.
I added -DCMAKE_CXX_COMPILER=/tmp/llvm/stage1/bin/clang++ for the -B obj2 command line, though.

[glandium]

Do you mean it found zlib?

[MaskRay]

In D107799#3572539, @glandium wrote:

Do you mean it found zlib?

Yes.

% ldd /tmp/out/obj2/bin/clang | grep libz
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007efd91747000)
% /tmp/out/obj2/bin/clang -g -gz -c a.cc # works

[glandium]

I don't know how this works for you. CMake gets a list of implicit link dirs from parsing the ld command that clang -v outputs when linking. When $clang/lib/x86_64-unknown-linux-gnu exists, that's part of that output. Then, CMake determines the arch it later uses when finding libraries by matching architectures in each of the implicit link dirs, and $clang/lib/x86_64-unknown-linux-gnu is the first match. From there [it uses x86_64-unknown-linux-gnu](https://github.com/Kitware/CMake/blob/master/Source/cmSearchPath.cxx#L180), and of course it doesn't find libraries in /usr/lib/x86_64-linux-gnu because it never looks there in find_library. That can even affect pkg-config, because it uses that same derived architecture to find pkgconfig directories. When $clang/lib/x86_64-unknown-linux-gnu doesn't exist, /lib/x86_64-linux-gnu is the first match, and it uses x86_64-linux-gnu as arch, and that works.

[DavidSpickett]

FYI this broke our armv7 and v8 bots https://lab.llvm.org/buildbot/#/builders/178/builds/2293 (as it did last time it landed).

I'll attempt to revive the patch to fix it I had from last time.

[thakis]

https://github.com/llvm/llvm-project/commit/716e27bc9ad4699bbc82318834e938bcc9682cf8 turned this on for arm linux systems. Having this on by default in some configs on linux and off on others seems like it'd be very confusing. Let's please either have it on or off uniformly per OS?

[DavidSpickett]

Having this on by default in some configs on linux and off on others seems like it'd be very confusing. Let's please either have it on or off uniformly per OS?

I wasn't sure this met the revert bar but sure suits me: https://github.com/llvm/llvm-project/commit/90c9d41c8acc34fb68958f373eb23f53c1b1c840

I am working to make it work but mostly from the point of view of not having broken builds. I don't have background on what this is trying to achieve overall and if arm/armhf will even fit that vision at all.

So @MaskRay if you can summarise the goal and/or I can provide you with instructions to reproduce the armhf config and you can try to fix it up yourself.

[glandium]

In D107799#3598787, @thakis wrote:

https://github.com/llvm/llvm-project/commit/716e27bc9ad4699bbc82318834e938bcc9682cf8 turned this on for arm linux systems. Having this on by default in some configs on linux and off on others seems like it'd be very confusing. Let's please either have it on or off uniformly per OS?

It's worth noting that darwin is also different, where it uses "darwin" instead of a target triplet, presumably so that the libs there can be universal binaries.

[MaskRay]

In D107799#3598878, @DavidSpickett wrote:

Having this on by default in some configs on linux and off on others seems like it'd be very confusing. Let's please either have it on or off uniformly per OS?

I wasn't sure this met the revert bar but sure suits me: https://github.com/llvm/llvm-project/commit/90c9d41c8acc34fb68958f373eb23f53c1b1c840

I am working to make it work but mostly from the point of view of not having broken builds. I don't have background on what this is trying to achieve overall and if arm/armhf will even fit that vision at all.

I think defaulting arm/armhf to LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=off is fine. So I reverted your revert.

So @MaskRay if you can summarise the goal and/or I can provide you with instructions to reproduce the armhf config and you can try to fix it up yourself.

See the summary for the goal. It is more important for LLVM_ENABLE_PROJECTS and LLVM_ENABLE_RUNTIMES to have the same behavior, and push target maintainers to fix CMake.
(Note that LLVM_ENABLE_PROJECTS=libcxx;libcxxabi;libunwind has been deprecated for quite a while now.)

Your arm failure link likely just exposed an issue that LLVM_ENABLE_RUNTIMES=compiler-rt (and probably libcxx;libcxxabi;libunwind) likely doesn't work (well) with arm/armhf.
Assuming that you maintain arm/armhf builds, you probably need to investigate how to make them work. I don't have an arm/armhf machine.

[glandium]

@MaskRay This Dockerfile reproduces it with Debian testing: https://gist.github.com/glandium/6c130dee608f9585b425c4a40a084d27

[MaskRay]

In D107799#3725086, @glandium wrote:

@MaskRay This Dockerfile reproduces it with Debian testing: https://gist.github.com/glandium/6c130dee608f9585b425c4a40a084d27

Thanks for the instructions. This involves -DLLVM_ENABLE_RUNTIMES builds (which defaults to LLVM_ENABLE_PER_TARGET_RUNTIME_DIR_default=on on Linux long before this patch).
I have created https://github.com/llvm/llvm-project/issues/57259 to discuss the Debian issue. I am unable to reproduce on my Debian testing machine (with many more files than your Dockerfile).

llvm/CMakeLists.txt

 
 set(LLVM_TARGETS_TO_BUILD
    ${LLVM_TARGETS_TO_BUILD}
    ${LLVM_EXPERIMENTAL_TARGETS_TO_BUILD})
 list(REMOVE_DUPLICATES LLVM_TARGETS_TO_BUILD)
 
 option(LLVM_ENABLE_PIC "Build Position-Independent Code" ON)
 option(LLVM_ENABLE_MODULES "Compile with C++ modules enabled." OFF)
 if(${CMAKE_SYSTEM_NAME} MATCHES "Darwin")
   option(LLVM_ENABLE_MODULE_DEBUGGING "Compile with -gmodules." ON)
 else()
   option(LLVM_ENABLE_MODULE_DEBUGGING "Compile with -gmodules." OFF)
 endif()
 option(LLVM_ENABLE_LOCAL_SUBMODULE_VISIBILITY "Compile with -fmodules-local-submodule-visibility." ON)
 option(LLVM_ENABLE_LIBCXX "Use libc++ if available." OFF)
 option(LLVM_STATIC_LINK_CXX_STDLIB "Statically link the standard library." OFF)
 option(LLVM_ENABLE_LLD "Use lld as C and C++ linker." OFF)

 if(CMAKE_HOST_APPLE AND APPLE)
   include(UseLibtool)
 endif()
 
 # Override the default target with an environment variable named by LLVM_TARGET_TRIPLE_ENV.
 set(LLVM_TARGET_TRIPLE_ENV CACHE STRING "The name of environment variable to override default target. Disabled by blank.")
 mark_as_advanced(LLVM_TARGET_TRIPLE_ENV)
 
-set(LLVM_ENABLE_PER_TARGET_RUNTIME_DIR OFF CACHE BOOL
+if(CMAKE_SYSTEM_NAME MATCHES "Linux")
+  set(LLVM_ENABLE_PER_TARGET_RUNTIME_DIR_default ON)
+else()
+  set(LLVM_ENABLE_PER_TARGET_RUNTIME_DIR_default OFF)
+endif()
+set(LLVM_ENABLE_PER_TARGET_RUNTIME_DIR ${LLVM_ENABLE_PER_TARGET_RUNTIME_DIR_default} CACHE BOOL
   "Enable per-target runtimes directory")
 
 set(LLVM_PROFDATA_FILE "" CACHE FILEPATH
   "Profiling data file to use when compiling in order to improve runtime performance.")
 
 # All options referred to from HandleLLVMOptions have to be specified
 # BEFORE this include, otherwise options will not be correctly set on
 # first cmake run