This is an archive of the discontinued LLVM Phabricator instance.

[compiler-rt][CMake] Properly set COMPILER_RT_HAS_LLD
AcceptedPublic

Authored by aeubanks on Feb 23 2023, 11:24 AM.

Details

Summary

LLVM_TOOL_LLD_BUILD is a relic of the pre-monorepo times. This causes us to never set COMPILER_RT_HAS_LLD.

Instead, set it from the runtimes build if lld is being built and lld is used as the compiler-rt linker.

Mark a test that requires libstdc++ as requiring Android, as other platforms may not have a libstdc++ lying around.

Diff Detail

Event Timeline

aeubanks created this revision.Feb 23 2023, 11:24 AM
Herald added a project: Restricted Project. · View Herald TranscriptFeb 23 2023, 11:24 AM
Herald added subscribers: Enna1, dberris. · View Herald Transcript
aeubanks requested review of this revision.Feb 23 2023, 11:24 AM
Herald added a project: Restricted Project. · View Herald TranscriptFeb 23 2023, 11:24 AM
Herald added a subscriber: Restricted Project. · View Herald Transcript

I'm not if lld will always be built before invoking the compiler-rt sub-cmake

hmm looks like some tests are failing because they're using a system lld, should this only be set if we're building lld?

MaskRay added a comment.EditedFeb 27 2023, 8:36 PM

It seems reasonable that COMPILER_RT_HAS_LLD is only set to true when we build lld in LLVM_ENABLE_PROJECTS builds or lld is available in a runtimes build.
I.e. don't consider COMPILER_RT_HAS_LLD true in LLVM_ENABLE_PROJECTS builds when lld comes from the system.

It seems reasonable that COMPILER_RT_HAS_LLD is only set to true when we build lld in LLVM_ENABLE_PROJECTS builds or lld is available in a runtimes build.
I.e. don't consider COMPILER_RT_HAS_LLD true in LLVM_ENABLE_PROJECTS builds when lld comes from the system.

I agree, with system LLD we have no control over the version.

how would you detect if a built lld is available in a runtimes build?

aeubanks updated this revision to Diff 519750.May 4 2023, 11:56 PM
aeubanks edited the summary of this revision. (Show Details)

update

Herald added a project: Restricted Project. · View Herald TranscriptMay 4 2023, 11:56 PM

Is this correct? If lld is listed in LLVM_ENABLE_PROJECTS it does not necessarily
mean that it can target all architectures for which the libraries are being built.
Perhaps it should instead check RUNTIMES_<triple>_LLVM_ENABLE_LLD or something like that?

aeubanks updated this revision to Diff 520445.May 8 2023, 11:04 AM
aeubanks edited the summary of this revision. (Show Details)

also check if we're using lld

MaskRay accepted this revision.May 30 2023, 11:33 AM

A bit of rubber stamp as I don't understand CMake well enough...

This revision is now accepted and ready to land.May 30 2023, 11:33 AM

This still does not look right to me. The fact that lld is on the LLVM_ENABLE_PROJECT list does not mean that it is used for linking runtime libraries.
I add lld to the list because I use it to link x86 binaries, but I build runtimes for a different architecture and use a different linker for that.
This should be a per-target check somehow. AIUI it should check some RUNTIME_<triple>_ variable, presumably RUNTIME_<triple>_LLVM_USE_LINKER or RUNTIME_<triple>_LLVM_ENABLE_LLD.

This still does not look right to me. The fact that lld is on the LLVM_ENABLE_PROJECT list does not mean that it is used for linking runtime libraries.
I add lld to the list because I use it to link x86 binaries, but I build runtimes for a different architecture and use a different linker for that.
This should be a per-target check somehow. AIUI it should check some RUNTIME_<triple>_ variable, presumably RUNTIME_<triple>_LLVM_USE_LINKER or RUNTIME_<triple>_LLVM_ENABLE_LLD.

the compiler-rt/CMakeLists.txt change does check LLVM_USE_LINKER. the RUNTIME_*_LLVM_USE_LINKER gets translated to LLVM_USE_LINKER by the time compiler-rt/CMakeLists.txt is invoked

This still does not look right to me. The fact that lld is on the LLVM_ENABLE_PROJECT list does not mean that it is used for linking runtime libraries.
I add lld to the list because I use it to link x86 binaries, but I build runtimes for a different architecture and use a different linker for that.
This should be a per-target check somehow. AIUI it should check some RUNTIME_<triple>_ variable, presumably RUNTIME_<triple>_LLVM_USE_LINKER or RUNTIME_<triple>_LLVM_ENABLE_LLD.

the compiler-rt/CMakeLists.txt change does check LLVM_USE_LINKER. the RUNTIME_*_LLVM_USE_LINKER gets translated to LLVM_USE_LINKER by the time compiler-rt/CMakeLists.txt is invoked

Right, thanks. Then I have no objection here.

I'm confused about these lines:

llvm/runtimes/CMakeLists.txt:250: PASSTHROUGH_PREFIXES LLVM_ENABLE_RUNTIMES
llvm/runtimes/CMakeLists.txt:251: LLVM_USE_LINKER
llvm/runtimes/CMakeLists.txt:326: list(APPEND ${name}_extra_args -DLLVM_USE_LINKER=${LLVM_USE_LINKER})

But I guess this is a different story…

This revision was automatically updated to reflect the committed changes.

I missed the latest revision so I apologize about the late comment. I'm fine with this change but I'm worried whether this strategy is going to scale. If we need a similar variable in other runtimes, are we going to introduce more variables like ${name}_HAS_TRUNK_LLD=TRUE, etc? I think we need a better long-term strategy that doesn't require special casing individual runtimes. Perhaps we could just passthrough LLVM_ENABLE_PROJECTS into the runtimes build and then do the "lld" IN_LIST LLVM_ENABLE_PROJECTS check in individual runtimes?

I missed the latest revision so I apologize about the late comment. I'm fine with this change but I'm worried whether this strategy is going to scale. If we need a similar variable in other runtimes, are we going to introduce more variables like ${name}_HAS_TRUNK_LLD=TRUE, etc? I think we need a better long-term strategy that doesn't require special casing individual runtimes. Perhaps we could just passthrough LLVM_ENABLE_PROJECTS into the runtimes build and then do the "lld" IN_LIST LLVM_ENABLE_PROJECTS check in individual runtimes?

passing through LLVM_ENABLE_PROJECTS seems good, I'll look at that

haowei added a subscriber: haowei.May 31 2023, 10:59 AM

Hi, this patch breaks runtime tests HWAddressSanitizer-x86_64 :: TestCases/sizes.cpp , Error message:

Script:
--
: 'RUN: at line 3';      /b/s/w/ir/x/w/staging/llvm_build/./bin/clang  --driver-mode=g++   -Wthread-safety -Wthread-safety-reference -Wthread-safety-beta   -gline-tables-only -fsanitize=hwaddress -fuse-ld=lld -fsanitize-hwaddress-experimental-aliasing -mllvm -hwasan-generate-tags-with-calls=1 -mllvm -hwasan-globals -mllvm -hwasan-use-short-granules -mllvm -hwasan-instrument-landing-pads=0 -mllvm -hwasan-instrument-personality-functions /b/s/w/ir/x/w/llvm-llvm-project/compiler-rt/test/hwasan/TestCases/sizes.cpp -nostdlib++ -lstdc++ -o /b/s/w/ir/x/w/staging/llvm_build/runtimes/runtimes-x86_64-unknown-linux-gnu-bins/compiler-rt/test/hwasan/X86_64/TestCases/Output/sizes.cpp.tmp
: 'RUN: at line 4';   env HWASAN_OPTIONS=disable_allocator_tagging=1:random_tags=0:fail_without_syscall_abi=0:allocator_may_return_null=0 not  /b/s/w/ir/x/w/staging/llvm_build/runtimes/runtimes-x86_64-unknown-linux-gnu-bins/compiler-rt/test/hwasan/X86_64/TestCases/Output/sizes.cpp.tmp malloc 2>&1          | FileCheck /b/s/w/ir/x/w/llvm-llvm-project/compiler-rt/test/hwasan/TestCases/sizes.cpp --check-prefix=CHECK-max
: 'RUN: at line 5';   env HWASAN_OPTIONS=disable_allocator_tagging=1:random_tags=0:fail_without_syscall_abi=0:allocator_may_return_null=1      /b/s/w/ir/x/w/staging/llvm_build/runtimes/runtimes-x86_64-unknown-linux-gnu-bins/compiler-rt/test/hwasan/X86_64/TestCases/Output/sizes.cpp.tmp malloc 2>&1
: 'RUN: at line 6';   env HWASAN_OPTIONS=disable_allocator_tagging=1:random_tags=0:fail_without_syscall_abi=0:allocator_may_return_null=0 not  /b/s/w/ir/x/w/staging/llvm_build/runtimes/runtimes-x86_64-unknown-linux-gnu-bins/compiler-rt/test/hwasan/X86_64/TestCases/Output/sizes.cpp.tmp malloc max 2>&1      | FileCheck /b/s/w/ir/x/w/llvm-llvm-project/compiler-rt/test/hwasan/TestCases/sizes.cpp --check-prefix=CHECK-max
: 'RUN: at line 7';   env HWASAN_OPTIONS=disable_allocator_tagging=1:random_tags=0:fail_without_syscall_abi=0:allocator_may_return_null=1      /b/s/w/ir/x/w/staging/llvm_build/runtimes/runtimes-x86_64-unknown-linux-gnu-bins/compiler-rt/test/hwasan/X86_64/TestCases/Output/sizes.cpp.tmp malloc max 2>&1
: 'RUN: at line 8';   env HWASAN_OPTIONS=disable_allocator_tagging=1:random_tags=0:fail_without_syscall_abi=0:allocator_may_return_null=0 not  /b/s/w/ir/x/w/staging/llvm_build/runtimes/runtimes-x86_64-unknown-linux-gnu-bins/compiler-rt/test/hwasan/X86_64/TestCases/Output/sizes.cpp.tmp calloc 2>&1          | FileCheck /b/s/w/ir/x/w/llvm-llvm-project/compiler-rt/test/hwasan/TestCases/sizes.cpp --check-prefix=CHECK-calloc
: 'RUN: at line 9';   env HWASAN_OPTIONS=disable_allocator_tagging=1:random_tags=0:fail_without_syscall_abi=0:allocator_may_return_null=1      /b/s/w/ir/x/w/staging/llvm_build/runtimes/runtimes-x86_64-unknown-linux-gnu-bins/compiler-rt/test/hwasan/X86_64/TestCases/Output/sizes.cpp.tmp calloc 2>&1
: 'RUN: at line 10';   env HWASAN_OPTIONS=disable_allocator_tagging=1:random_tags=0:fail_without_syscall_abi=0:allocator_may_return_null=0 not  /b/s/w/ir/x/w/staging/llvm_build/runtimes/runtimes-x86_64-unknown-linux-gnu-bins/compiler-rt/test/hwasan/X86_64/TestCases/Output/sizes.cpp.tmp reallocarray 2>&1    | FileCheck /b/s/w/ir/x/w/llvm-llvm-project/compiler-rt/test/hwasan/TestCases/sizes.cpp --check-prefix=CHECK-reallocarray
: 'RUN: at line 11';   env HWASAN_OPTIONS=disable_allocator_tagging=1:random_tags=0:fail_without_syscall_abi=0:allocator_may_return_null=1      /b/s/w/ir/x/w/staging/llvm_build/runtimes/runtimes-x86_64-unknown-linux-gnu-bins/compiler-rt/test/hwasan/X86_64/TestCases/Output/sizes.cpp.tmp reallocarray 2>&1
: 'RUN: at line 12';   env HWASAN_OPTIONS=disable_allocator_tagging=1:random_tags=0:fail_without_syscall_abi=0:allocator_may_return_null=0 not  /b/s/w/ir/x/w/staging/llvm_build/runtimes/runtimes-x86_64-unknown-linux-gnu-bins/compiler-rt/test/hwasan/X86_64/TestCases/Output/sizes.cpp.tmp new 2>&1             | FileCheck /b/s/w/ir/x/w/llvm-llvm-project/compiler-rt/test/hwasan/TestCases/sizes.cpp --check-prefix=CHECK-max
: 'RUN: at line 13';   env HWASAN_OPTIONS=disable_allocator_tagging=1:random_tags=0:fail_without_syscall_abi=0:allocator_may_return_null=1 not  /b/s/w/ir/x/w/staging/llvm_build/runtimes/runtimes-x86_64-unknown-linux-gnu-bins/compiler-rt/test/hwasan/X86_64/TestCases/Output/sizes.cpp.tmp new 2>&1             | FileCheck /b/s/w/ir/x/w/llvm-llvm-project/compiler-rt/test/hwasan/TestCases/sizes.cpp --check-prefix=CHECK-oom
: 'RUN: at line 14';   env HWASAN_OPTIONS=disable_allocator_tagging=1:random_tags=0:fail_without_syscall_abi=0:allocator_may_return_null=0 not  /b/s/w/ir/x/w/staging/llvm_build/runtimes/runtimes-x86_64-unknown-linux-gnu-bins/compiler-rt/test/hwasan/X86_64/TestCases/Output/sizes.cpp.tmp new max 2>&1         | FileCheck /b/s/w/ir/x/w/llvm-llvm-project/compiler-rt/test/hwasan/TestCases/sizes.cpp --check-prefix=CHECK-max
: 'RUN: at line 15';   env HWASAN_OPTIONS=disable_allocator_tagging=1:random_tags=0:fail_without_syscall_abi=0:allocator_may_return_null=1 not  /b/s/w/ir/x/w/staging/llvm_build/runtimes/runtimes-x86_64-unknown-linux-gnu-bins/compiler-rt/test/hwasan/X86_64/TestCases/Output/sizes.cpp.tmp new max 2>&1         | FileCheck /b/s/w/ir/x/w/llvm-llvm-project/compiler-rt/test/hwasan/TestCases/sizes.cpp --check-prefix=CHECK-oom
: 'RUN: at line 16';   env HWASAN_OPTIONS=disable_allocator_tagging=1:random_tags=0:fail_without_syscall_abi=0:allocator_may_return_null=0 not  /b/s/w/ir/x/w/staging/llvm_build/runtimes/runtimes-x86_64-unknown-linux-gnu-bins/compiler-rt/test/hwasan/X86_64/TestCases/Output/sizes.cpp.tmp new-nothrow 2>&1     | FileCheck /b/s/w/ir/x/w/llvm-llvm-project/compiler-rt/test/hwasan/TestCases/sizes.cpp --check-prefix=CHECK-max
: 'RUN: at line 17';   env HWASAN_OPTIONS=disable_allocator_tagging=1:random_tags=0:fail_without_syscall_abi=0:allocator_may_return_null=1      /b/s/w/ir/x/w/staging/llvm_build/runtimes/runtimes-x86_64-unknown-linux-gnu-bins/compiler-rt/test/hwasan/X86_64/TestCases/Output/sizes.cpp.tmp new-nothrow 2>&1
: 'RUN: at line 18';   env HWASAN_OPTIONS=disable_allocator_tagging=1:random_tags=0:fail_without_syscall_abi=0:allocator_may_return_null=0 not  /b/s/w/ir/x/w/staging/llvm_build/runtimes/runtimes-x86_64-unknown-linux-gnu-bins/compiler-rt/test/hwasan/X86_64/TestCases/Output/sizes.cpp.tmp new-nothrow max 2>&1 | FileCheck /b/s/w/ir/x/w/llvm-llvm-project/compiler-rt/test/hwasan/TestCases/sizes.cpp --check-prefix=CHECK-max
: 'RUN: at line 19';   env HWASAN_OPTIONS=disable_allocator_tagging=1:random_tags=0:fail_without_syscall_abi=0:allocator_may_return_null=1      /b/s/w/ir/x/w/staging/llvm_build/runtimes/runtimes-x86_64-unknown-linux-gnu-bins/compiler-rt/test/hwasan/X86_64/TestCases/Output/sizes.cpp.tmp new-nothrow max 2>&1
: 'RUN: at line 20';                                                    /b/s/w/ir/x/w/staging/llvm_build/runtimes/runtimes-x86_64-unknown-linux-gnu-bins/compiler-rt/test/hwasan/X86_64/TestCases/Output/sizes.cpp.tmp usable 2>&1
--
Exit Code: 1

Command Output (stderr):
--
ld.lld: error: unable to find library -lstdc++
clang: error: linker command failed with exit code 1 (use -v to see invocation)

--

I ran a test build without this change and it passed.
Could you look into this, and if it takes a while, could you revert this change please?

@haowei could you post the lld command before and after the patch? bin/llvm-lit -a runtimes/runtimes-bins/compiler-rt/test/hwasan/X86_64/TestCases/sizes.cpp

@haowei could you post the lld command before and after the patch? bin/llvm-lit -a runtimes/runtimes-bins/compiler-rt/test/hwasan/X86_64/TestCases/sizes.cpp

The test run with revert was conducted on a bot, I don't have shell access to manually ran any command nor do I have access to the build directory, sorry.

The task with your change reverted: https://luci-milo.appspot.com/raw/build/logs.chromium.org/fuchsia/led/haowei_google.com/44a1b91abf6fccb72bd3e88d382fb350d3647485d851446b6364cbe9008c9837/+/build.proto

The task failed with your change: https://ci.chromium.org/ui/p/fuchsia/builders/toolchain.ci/clang-linux-x64/b8779565760229543425/overview

I could manually verify it on my workstation and try digging into the lld invocation in this unit test. But it would take some time. If you prefer that, could you revert the change while we investigating the issue?

I missed the latest revision so I apologize about the late comment. I'm fine with this change but I'm worried whether this strategy is going to scale. If we need a similar variable in other runtimes, are we going to introduce more variables like ${name}_HAS_TRUNK_LLD=TRUE, etc? I think we need a better long-term strategy that doesn't require special casing individual runtimes. Perhaps we could just passthrough LLVM_ENABLE_PROJECTS into the runtimes build and then do the "lld" IN_LIST LLVM_ENABLE_PROJECTS check in individual runtimes?

passing through LLVM_ENABLE_PROJECTS gives me a bunch of

CMake Error at /usr/local/google/home/aeubanks/repos/llvm-project/llvm/cmake/modules/AddLLVM.cmake:1929 (add_dependencies):
  The dependency target "lld" of target "check-runtimes" does not exist.
Call Stack (most recent call first):
  /usr/local/google/home/aeubanks/repos/llvm-project/llvm/cmake/modules/AddLLVM.cmake:1969 (add_lit_target)
  CMakeLists.txt:252 (umbrella_lit_testsuite_end)
aeubanks reopened this revision.May 31 2023, 11:28 AM

@haowei could you post the lld command before and after the patch? bin/llvm-lit -a runtimes/runtimes-bins/compiler-rt/test/hwasan/X86_64/TestCases/sizes.cpp

The test run with revert was conducted on a bot, I don't have shell access to manually ran any command nor do I have access to the build directory, sorry.

The task with your change reverted: https://luci-milo.appspot.com/raw/build/logs.chromium.org/fuchsia/led/haowei_google.com/44a1b91abf6fccb72bd3e88d382fb350d3647485d851446b6364cbe9008c9837/+/build.proto

The task failed with your change: https://ci.chromium.org/ui/p/fuchsia/builders/toolchain.ci/clang-linux-x64/b8779565760229543425/overview

I could manually verify it on my workstation and try digging into the lld invocation in this unit test. But it would take some time. If you prefer that, could you revert the change while we investigating the issue?

I've reverted for now. My guess is that most typical bots have libstdc++ lying around but yours doesn't? Although I have no idea why this change would affect that?

This revision is now accepted and ready to land.May 31 2023, 11:28 AM

@haowei could you post the lld command before and after the patch? bin/llvm-lit -a runtimes/runtimes-bins/compiler-rt/test/hwasan/X86_64/TestCases/sizes.cpp

The test run with revert was conducted on a bot, I don't have shell access to manually ran any command nor do I have access to the build directory, sorry.

The task with your change reverted: https://luci-milo.appspot.com/raw/build/logs.chromium.org/fuchsia/led/haowei_google.com/44a1b91abf6fccb72bd3e88d382fb350d3647485d851446b6364cbe9008c9837/+/build.proto

The task failed with your change: https://ci.chromium.org/ui/p/fuchsia/builders/toolchain.ci/clang-linux-x64/b8779565760229543425/overview

I could manually verify it on my workstation and try digging into the lld invocation in this unit test. But it would take some time. If you prefer that, could you revert the change while we investigating the issue?

I've reverted for now. My guess is that most typical bots have libstdc++ lying around but yours doesn't? Although I have no idea why this change would affect that?

Yeah, the bot's doesn't have libc++ installed from system. It does have a sysroot prebuilt but everything ran from clang/llvm build should be self contained.

I will start a A/B build locally right away. But since my workstation has libstdc++, it probably won't reproduce the failure, but I can see what command line have changed.

I think I understood why your change trigger the failure on the bots, without your patch, this test is unsupported, see:

UNSUPPORTED: HWAddressSanitizer-x86_64 :: TestCases/sizes.cpp (1365 of 13292)

from log output: https://logs.chromium.org/logs/fuchsia/buildbucket/cr-buildbucket/8779557369614664945/+/u/clang/test/stdout

aeubanks updated this revision to Diff 527439.Jun 1 2023, 8:51 AM

mark test as requiring android

aeubanks updated this revision to Diff 527442.Jun 1 2023, 8:52 AM
aeubanks edited the summary of this revision. (Show Details)

update commit message

UNSUPPORTED: HWAddressSanitizer-x86_64 :: TestCases/sizes.cpp (1365 of 13292)

ah makes sense, compiler-rt/test/hwasan/lit.cfg.py does check for this

Yeah, the bot's doesn't have libc++ installed from system. It does have a sysroot prebuilt but everything ran from clang/llvm build should be self contained.

Did you mean libstdc++? IIUC compiler-rt tests will often compile with system headers. I think libstdc++ is typically expected to be on Linux bots. But I've marked that test as REQUIRES: android so it's more targeted.

aeubanks reopened this revision.Jun 6 2023, 4:05 PM
This revision is now accepted and ready to land.Jun 6 2023, 4:05 PM
vitalybuka added inline comments.
compiler-rt/CMakeLists.txt
740

What is LLVM_ENABLE_LLD?

vitalybuka added inline comments.Jun 22 2023, 4:07 PM
compiler-rt/CMakeLists.txt
740

Looks like it's covered

compiler-rt/test/hwasan/TestCases/sizes.cpp
22

Please add comment like:
FIXME: Allocation sizes adjusted to trigger errors on Android. Support general Linux.

vitalybuka accepted this revision.Jun 22 2023, 4:09 PM