This is an archive of the discontinued LLVM Phabricator instance.

[Driver] Push multiarch path setup to individual drivers
ClosedPublic

Authored by phosek on Apr 23 2021, 12:48 PM.

Details

Reviewers
MaskRay
Group Reviewers
Restricted Project
Commits
rGb4537c3f51bc: [Driver] Push multiarch path setup to individual drivers
Summary

Different platforms use different rules for multiarch triples so
it's difficult to provide a single method for all platforms. We
instead move the getMultiarchTriple to the ToolChain class and let
individual platforms override it and provide their custom logic.

Diff Detail

Event Timeline

phosek created this revision.Apr 23 2021, 12:48 PM
phosek requested review of this revision.Apr 23 2021, 12:48 PM
Herald added a project: Restricted Project. · View Herald TranscriptApr 23 2021, 12:48 PM
MaskRay added a comment.EditedApr 23 2021, 8:35 PM

Different platforms use different rules for multiarch triples so it's difficult to provide a single method for all platforms.

Guess this is a Linux/Hurd/Fuchsia specific thing. I don't know much about the runtime build but I hope the Linux hierarchy is not set in stone.

Can you dump a diff how addPathIfExists(D, getStdlibPath(), Paths); and a similar call will change -L paths in the linking stage?
You can use clang ... '-###' |& sed -E 's/ "?-L/\n&/g; s/ "?-[iI]/\n&/g' to break -L into multiple lines.

Different platforms use different rules for multiarch triples so it's difficult to provide a single method for all platforms.

Guess this is a Linux/Hurd/Fuchsia specific thing. I don't know much about the runtime build but I hope the Linux hierarchy is not set in stone.

Can you dump a diff how addPathIfExists(D, getStdlibPath(), Paths); and a similar call will change -L paths in the linking stage?
You can use clang ... '-###' |& sed -E 's/ "?-L/\n&/g; s/ "?-[iI]/\n&/g' to break -L into multiple lines.

There's no difference, current behavior is already covered by tests and those are still passing. The difference is that ToolChain::getRuntimePath() and ToolChain::getStdlibPath() currently try various possible spellings of triples whereas with this patch that's no longer needed because each platform controls its own spelling so we save a few syscalls.

I checked and the multiarch runtimes layout is currently only used by Fuchsia, Linux and WebAssembly, other platforms use their own custom logic. If other platforms decide to pickup this layout as well, they can follow the same pattern.

MaskRay accepted this revision.Apr 25 2021, 12:07 AM

Different platforms use different rules for multiarch triples so it's difficult to provide a single method for all platforms.

Guess this is a Linux/Hurd/Fuchsia specific thing. I don't know much about the runtime build but I hope the Linux hierarchy is not set in stone.

Can you dump a diff how addPathIfExists(D, getStdlibPath(), Paths); and a similar call will change -L paths in the linking stage?
You can use clang ... '-###' |& sed -E 's/ "?-L/\n&/g; s/ "?-[iI]/\n&/g' to break -L into multiple lines.

There's no difference, current behavior is already covered by tests and those are still passing. The difference is that ToolChain::getRuntimePath() and ToolChain::getStdlibPath() currently try various possible spellings of triples whereas with this patch that's no longer needed because each platform controls its own spelling so we save a few syscalls.

I checked and the multiarch runtimes layout is currently only used by Fuchsia, Linux and WebAssembly, other platforms use their own custom logic. If other platforms decide to pickup this layout as well, they can follow the same pattern.

Hope you can dump the current behavior.

This revision is now accepted and ready to land.Apr 25 2021, 12:07 AM
This revision was landed with ongoing or failed builds.Apr 26 2021, 10:18 PM
This revision was automatically updated to reflect the committed changes.
Herald added a project: Restricted Project. · View Herald TranscriptApr 26 2021, 10:18 PM
Herald added a reviewer: Restricted Project. · View Herald Transcript
bjope added a subscriber: bjope.Apr 27 2021, 1:16 PM

Got some problems with my builds after this commit.

My cmake commands looks like this (well, this is after having tried to fix the problem by replacing x86_64-unknown-linux-gnu by x86_64-linux-gnu as I figured that might be needed after looking at the changes in this patch):

cmake /repo/uabbpet/llvm-upstream/llvm --debug-trycompile -G Ninja -DCMAKE_MAKE_PROGRAM=/app/ninja/1.8.2/SLED11-64/bin/ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DCMAKE_INSTALL_PREFIX=/compiler-clang -DLLVM_APPEND_VC_REV=OFF  -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_PROJECTS='clang;clang-tools-extra;lld' -DLLVM_BUILTIN_TARGETS=default -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_ENABLE_RUNTIMES='compiler-rt;libcxx;libcxxabi;libunwind' -DRUNTIMES_x86_64-linux-gnu_SANITIZER_USE_STATIC_LLVM_UNWINDER=ON -DRUNTIMES_x86_64-linux-gnu_SANITIZER_USE_STATIC_CXX_ABI=ON -DRUNTIMES_x86_64-linux-gnu_COMPILER_RT_USE_BUILTINS_LIBRARY=ON -DRUNTIMES_x86_64-linux-gnu_COMPILER_RT_CAN_EXECUTE_TESTS=OFF -DRUNTIMES_x86_64-linux-gnu_LIBCXX_HAS_ATOMIC_LIB=0 -DRUNTIMES_x86_64-linux-gnu_COMPILER_RT_BUILD_XRAY=OFF -DLLVM_RUNTIME_TARGETS='x86_64-linux-gnu' -DLLVM_ENABLE_LIBPFM=OFF -DCLANG_TOOLING_BUILD_AST_INTROSPECTION=OFF -DCLANG_ROUND_TRIP_CC1_ARGS=OFF

But my build fails, and it seems to be related to the message -- Failed to find compiler-rt builtins library for x86_64-linux-gnu, and later I get

-- Compiler-RT supported architectures: x86_64
CMake Error at /repo/uabbpet/llvm-upstream/compiler-rt/cmake/Modules/AddCompilerRT.cmake:258 (message):
  Cannot find builtins library for the target architecture
Call Stack (most recent call first):
  /repo/uabbpet/llvm-upstream/compiler-rt/lib/stats/CMakeLists.txt:23 (add_compiler_rt_runtime)

It tries to find libclang using a path like this my_build_dir/lib/clang/13.0.0/lib/linux/libclang_rt.builtins-x86_64.a (which is what I get if manually run my_build_dir/bin/clang --rtlib=compiler-rt -print-libgcc-file-name as well). But what I get when building is my_build_dir/lib/clang/13.0.0/lib/x86_64-unknown-linux-gnu/libclang_rt.builtins.a and `my_build_dir/lib/clang/13.0.0/lib/i386-unknown-linux-gnu/libclang_rt.builtins.a.

Not sure exactly what I'm doing wrong here. Looks like some kind of mismatch. I assume the layout with separate dirs for the libs is due to -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON, but the just built clang does not seem to know about that. Should it know about that? Is it no longer allowed to use -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON in combination with COMPILER_RT_USE_BUILTINS_LIBRARY=ON?

Or maybe the problem is that we use -DLLVM_BUILTIN_TARGETS=default?

Got some problems with my builds after this commit.

My cmake commands looks like this (well, this is after having tried to fix the problem by replacing x86_64-unknown-linux-gnu by x86_64-linux-gnu as I figured that might be needed after looking at the changes in this patch):

cmake /repo/uabbpet/llvm-upstream/llvm --debug-trycompile -G Ninja -DCMAKE_MAKE_PROGRAM=/app/ninja/1.8.2/SLED11-64/bin/ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DCMAKE_INSTALL_PREFIX=/compiler-clang -DLLVM_APPEND_VC_REV=OFF  -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_PROJECTS='clang;clang-tools-extra;lld' -DLLVM_BUILTIN_TARGETS=default -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_ENABLE_RUNTIMES='compiler-rt;libcxx;libcxxabi;libunwind' -DRUNTIMES_x86_64-linux-gnu_SANITIZER_USE_STATIC_LLVM_UNWINDER=ON -DRUNTIMES_x86_64-linux-gnu_SANITIZER_USE_STATIC_CXX_ABI=ON -DRUNTIMES_x86_64-linux-gnu_COMPILER_RT_USE_BUILTINS_LIBRARY=ON -DRUNTIMES_x86_64-linux-gnu_COMPILER_RT_CAN_EXECUTE_TESTS=OFF -DRUNTIMES_x86_64-linux-gnu_LIBCXX_HAS_ATOMIC_LIB=0 -DRUNTIMES_x86_64-linux-gnu_COMPILER_RT_BUILD_XRAY=OFF -DLLVM_RUNTIME_TARGETS='x86_64-linux-gnu' -DLLVM_ENABLE_LIBPFM=OFF -DCLANG_TOOLING_BUILD_AST_INTROSPECTION=OFF -DCLANG_ROUND_TRIP_CC1_ARGS=OFF

But my build fails, and it seems to be related to the message -- Failed to find compiler-rt builtins library for x86_64-linux-gnu, and later I get

-- Compiler-RT supported architectures: x86_64
CMake Error at /repo/uabbpet/llvm-upstream/compiler-rt/cmake/Modules/AddCompilerRT.cmake:258 (message):
  Cannot find builtins library for the target architecture
Call Stack (most recent call first):
  /repo/uabbpet/llvm-upstream/compiler-rt/lib/stats/CMakeLists.txt:23 (add_compiler_rt_runtime)

It tries to find libclang using a path like this my_build_dir/lib/clang/13.0.0/lib/linux/libclang_rt.builtins-x86_64.a (which is what I get if manually run my_build_dir/bin/clang --rtlib=compiler-rt -print-libgcc-file-name as well). But what I get when building is my_build_dir/lib/clang/13.0.0/lib/x86_64-unknown-linux-gnu/libclang_rt.builtins.a and `my_build_dir/lib/clang/13.0.0/lib/i386-unknown-linux-gnu/libclang_rt.builtins.a.

Not sure exactly what I'm doing wrong here. Looks like some kind of mismatch. I assume the layout with separate dirs for the libs is due to -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON, but the just built clang does not seem to know about that. Should it know about that? Is it no longer allowed to use -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON in combination with COMPILER_RT_USE_BUILTINS_LIBRARY=ON?

Or maybe the problem is that we use -DLLVM_BUILTIN_TARGETS=default?

The use of default is the issue, the problem is that the target triple "guessed" by config.guess doesn't necessarily match the multiarch triple used by Clang. I think that ideally we would stop relying on LLVM_DEFAULT_TARGET_TRIPLE and instead we would get the path directly from Clang. I can look into it but that might take a while to implement so we should probably revert the change for now?

I'm a bit surprised that you're using -DLLVM_BUILTIN_TARGETS=default but -DLLVM_RUNTIME_TARGETS='x86_64-linux-gnu', usually I'd expect these to match.

bjope added a comment.Apr 27 2021, 1:43 PM

Or maybe the problem is that we use -DLLVM_BUILTIN_TARGETS=default?

The use of default is the issue, the problem is that the target triple "guessed" by config.guess doesn't necessarily match the multiarch triple used by Clang. I think that ideally we would stop relying on LLVM_DEFAULT_TARGET_TRIPLE and instead we would get the path directly from Clang. I can look into it but that might take a while to implement so we should probably revert the change for now?

I'm a bit surprised that you're using -DLLVM_BUILTIN_TARGETS=default but -DLLVM_RUNTIME_TARGETS='x86_64-linux-gnu', usually I'd expect these to match.

Well, maybe that is for historical reasons. Or our downstream branch we use more targets, but I guess we got some problems on the main branch at some time and limited LLVM_RUNTIME_TARGETS to only do x86_64 (while normally we also build i386). And LLVM_BUILTIN_TARGETS=default happens to produce both x86_64 and i386 builtins.

Anyway I've tried changing to LLVM_BUILTIN_TARGETS='x86_64-linux-gnu' (and also LLVM_BUILTIN_TARGETS='i386-linux-gnu;x86_64-linux-gnu') and that seems to work fine. So I think I'm back on track. Thanks!
(Crossing my fingers and hopefully I'll get things working on the downstream branch with additional downstream targets.)

MaskRay added a comment.EditedApr 27 2021, 3:38 PM

FWIW I have played -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=on

I see lib/clang/13.0.0/lib/x86_64-unknown-linux-gnu/libclang_rt.*. The directory name is x86_64-unknown-linux-gnu, which actually looks good to me.
We don't necessarily copy the GCC x86_64-linux-gnu multiarch name. Preserving the default target triple as-is gives users more choices (e.g. if they want to make use of the vendor part to have multiple builds with the same architecture).

I thought this was NFC:(

Yeah, personally I think the previous behavior (lib/clang/13.0.0/lib/x86_64-unknown-linux-gnu/libclang_rt.asan.a) is slightly better than the new lib/clang/13.0.0/lib/x86_64-linux-gnu/libclang_rt.asan.a.
multiarch paths are suitable for GCC specific directories (lib/$triple, libstdc++ aarch64-linux-gnu/include/c++/10, etc) but llvm specific libraries don't necessarily use that.

I thought this was NFC:(

Yeah, personally I think the previous behavior (lib/clang/13.0.0/lib/x86_64-unknown-linux-gnu/libclang_rt.asan.a) is slightly better than the new lib/clang/13.0.0/lib/x86_64-linux-gnu/libclang_rt.asan.a.
multiarch paths are suitable for GCC specific directories (lib/$triple, libstdc++ aarch64-linux-gnu/include/c++/10, etc) but llvm specific libraries don't necessarily use that.

I should have been clear, sorry about that. I think there are three separate issues:

  1. Rather than trying various triple versions, we should use one canonical spelling when searching for Clang's standard and runtime libraries. That means less system calls and better error messages, this is what this change was primarily intended to do.
  2. Following from the previous point, we need to decide which canonical spelling to use. Since we're trying to use the multiarch layout, I was following the Debian documentation which is closest thing to a spec I'm aware of. The alternative would be to use the normalized target, the main disadvantage being that we would be diverging from existing quasi standard.
  3. -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON gives you a different result depending on how you spell the targets in -DLLVM_BUILTIN_TARGETS= and -DLLVM_RUNTIME_TARGETS=, this is what @bjope ran into and I'd consider that a bug since the directory layout is an implementation detail and we should be able to change it without users having to change their build.

I'm working on a solution for #3 and I should have a patch ready soon. I'm curious about your thoughts on #2.

MaskRay added a comment.EditedApr 27 2021, 7:56 PM

I thought this was NFC:(

Yeah, personally I think the previous behavior (lib/clang/13.0.0/lib/x86_64-unknown-linux-gnu/libclang_rt.asan.a) is slightly better than the new lib/clang/13.0.0/lib/x86_64-linux-gnu/libclang_rt.asan.a.
multiarch paths are suitable for GCC specific directories (lib/$triple, libstdc++ aarch64-linux-gnu/include/c++/10, etc) but llvm specific libraries don't necessarily use that.

I should have been clear, sorry about that. I think there are three separate issues:

  1. Rather than trying various triple versions, we should use one canonical spelling when searching for Clang's standard and runtime libraries. That means less system calls and better error messages, this is what this change was primarily intended to do.
  2. Following from the previous point, we need to decide which canonical spelling to use. Since we're trying to use the multiarch layout, I was following the Debian documentation which is closest thing to a spec I'm aware of. The alternative would be to use the normalized target, the main disadvantage being that we would be diverging from existing quasi standard.
  3. -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON gives you a different result depending on how you spell the targets in -DLLVM_BUILTIN_TARGETS= and -DLLVM_RUNTIME_TARGETS=, this is what @bjope ran into and I'd consider that a bug since the directory layout is an implementation detail and we should be able to change it without users having to change their build.

I'm working on a solution for #3 and I should have a patch ready soon. I'm curious about your thoughts on #2.

multiarch is not consistent within GCC, either (also note that not all GCC installations are configured with multiarch. --print-multiarch prints an empty line).
I have tried various GCC installations and made a summary https://maskray.me/blog/2021-03-28-compiler-driver-and-cross-compilation : even if the multiarch triple for x86_64 Linux is x86_64-linux-gnu, there are a few x86_64-pc-linux-gnu paths. (You can configure and get x86_64-pc-linux-gnu-gcc or x86_64-unknown-linux-gnu-gcc)
For musl systems (*-linux-musl), multiarch doesn't make sense. If we use multiarch, we'll need to invent *-linux-musl for them.

For some GCC related paths, we have no choice but to detect their paths. For entirely clang internal stuff (runtime library paths), I actually appreciate that we have a useful, un-normalized 'vendor' part.
It gives users a choice if they want to have x86_64-pc[12345]-linux-gnu. This argument is weak, but the other arguments are probably stronger. My viewpoint is: if we can avoid multiarch, just avoid it.


I haven't figured out how to do a proper runtime build...

-DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=on -DLLVM_RUNTIME_TARGETS=default -DLLVM_ENABLE_RUNTIMES='compiler-rt;libcxx;libunwind' gives me

CMake Error at /usr/share/cmake-3.18/Modules/ExternalProject.cmake:3152 (add_custom_target):                                                                                                                                                                                                
  add_custom_target cannot create target "builtins" because another target                                                                                                                                                                                                                  
  with the same name already exists.  The existing target is a custom target                                                                                                                                                                                                                
  created in source directory                                                                                                                                                                                                                                                               
  "/home/maskray/llvm/compiler-rt/lib/builtins".  See                                                                                                                                                                                                                      
  documentation for policy CMP0002 for more details.                                                                                                                                                                                                                                        
Call Stack (most recent call first):                                                                                                                                                                                                                                                        
  cmake/modules/LLVMExternalProjectUtils.cmake:283 (ExternalProject_Add)                                                                                                                                                                                                                    
  runtimes/CMakeLists.txt:81 (llvm_ExternalProject_Add)                                                                                                                                                                                                                                     
  runtimes/CMakeLists.txt:134 (builtin_default_target)

multiarch is not consistent within GCC, either (also note that not all GCC installations are configured with multiarch. --print-multiarch prints an empty line).
I have tried various GCC installations and made a summary https://maskray.me/blog/2021-03-28-compiler-driver-and-cross-compilation : even if the multiarch triple for x86_64 Linux is x86_64-linux-gnu, there are a few x86_64-pc-linux-gnu paths. (You can configure and get x86_64-pc-linux-gnu-gcc or x86_64-unknown-linux-gnu-gcc)
For musl systems (*-linux-musl), multiarch doesn't make sense. If we use multiarch, we'll need to invent *-linux-musl for them.

For some GCC related paths, we have no choice but to detect their paths. For entirely clang internal stuff (runtime library paths), I actually appreciate that we have a useful, un-normalized 'vendor' part.
It gives users a choice if they want to have x86_64-pc[12345]-linux-gnu. This argument is weak, but the other arguments are probably stronger. My viewpoint is: if we can avoid multiarch, just avoid it.

SGTM, I'll partially revert this change and switch back to using normalized triples if that's fine with you.

I haven't figured out how to do a proper runtime build...

-DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=on -DLLVM_RUNTIME_TARGETS=default -DLLVM_ENABLE_RUNTIMES='compiler-rt;libcxx;libunwind' gives me

CMake Error at /usr/share/cmake-3.18/Modules/ExternalProject.cmake:3152 (add_custom_target):                                                                                                                                                                                                
  add_custom_target cannot create target "builtins" because another target                                                                                                                                                                                                                  
  with the same name already exists.  The existing target is a custom target                                                                                                                                                                                                                
  created in source directory                                                                                                                                                                                                                                                               
  "/home/maskray/llvm/compiler-rt/lib/builtins".  See                                                                                                                                                                                                                      
  documentation for policy CMP0002 for more details.                                                                                                                                                                                                                                        
Call Stack (most recent call first):                                                                                                                                                                                                                                                        
  cmake/modules/LLVMExternalProjectUtils.cmake:283 (ExternalProject_Add)                                                                                                                                                                                                                    
  runtimes/CMakeLists.txt:81 (llvm_ExternalProject_Add)                                                                                                                                                                                                                                     
  runtimes/CMakeLists.txt:134 (builtin_default_target)

You also need -DLLVM_BUILTIN_TARGETS=default.

multiarch is not consistent within GCC, either (also note that not all GCC installations are configured with multiarch. --print-multiarch prints an empty line).
I have tried various GCC installations and made a summary https://maskray.me/blog/2021-03-28-compiler-driver-and-cross-compilation : even if the multiarch triple for x86_64 Linux is x86_64-linux-gnu, there are a few x86_64-pc-linux-gnu paths. (You can configure and get x86_64-pc-linux-gnu-gcc or x86_64-unknown-linux-gnu-gcc)
For musl systems (*-linux-musl), multiarch doesn't make sense. If we use multiarch, we'll need to invent *-linux-musl for them.

For some GCC related paths, we have no choice but to detect their paths. For entirely clang internal stuff (runtime library paths), I actually appreciate that we have a useful, un-normalized 'vendor' part.
It gives users a choice if they want to have x86_64-pc[12345]-linux-gnu. This argument is weak, but the other arguments are probably stronger. My viewpoint is: if we can avoid multiarch, just avoid it.

SGTM, I'll partially revert this change and switch back to using normalized triples if that's fine with you.

LG, thanks!

I haven't figured out how to do a proper runtime build...

-DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=on -DLLVM_RUNTIME_TARGETS=default -DLLVM_ENABLE_RUNTIMES='compiler-rt;libcxx;libunwind' gives me

CMake Error at /usr/share/cmake-3.18/Modules/ExternalProject.cmake:3152 (add_custom_target):                                                                                                                                                                                                
  add_custom_target cannot create target "builtins" because another target                                                                                                                                                                                                                  
  with the same name already exists.  The existing target is a custom target                                                                                                                                                                                                                
  created in source directory                                                                                                                                                                                                                                                               
  "/home/maskray/llvm/compiler-rt/lib/builtins".  See                                                                                                                                                                                                                      
  documentation for policy CMP0002 for more details.                                                                                                                                                                                                                                        
Call Stack (most recent call first):                                                                                                                                                                                                                                                        
  cmake/modules/LLVMExternalProjectUtils.cmake:283 (ExternalProject_Add)                                                                                                                                                                                                                    
  runtimes/CMakeLists.txt:81 (llvm_ExternalProject_Add)                                                                                                                                                                                                                                     
  runtimes/CMakeLists.txt:134 (builtin_default_target)

You also need -DLLVM_BUILTIN_TARGETS=default.

Yes, -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=on -DLLVM_RUNTIME_TARGETS=default -DLLVM_ENABLE_RUNTIMES='compiler-rt;libcxx;libunwind' -DLLVM_BUILTIN_TARGETS=default -DLLVM_ENABLE_PROJECTS='clang;lld' works.