This is an archive of the discontinued LLVM Phabricator instance.

[compiler-rt][CMake][arm64] Use a custom target for symlinking LSE sources
ClosedPublic

Authored by tambre on Jul 19 2021, 1:24 PM.

Details

Summary

On Apple platforms the builtins may be built for both arm64 and arm64e.
With Makefile generators separate targets are built using Make sub-invocations.
This causes a race when creating the symlink which may sometimes fail.

Work around this by using a custom target that the builtin targets depend on.
This causes any sub-invocations to depend on the symlinks having been created before.

Mailing list thread: https://lists.llvm.org/pipermail/llvm-dev/2021-July/151822.html

Diff Detail

Event Timeline

steven_wu created this revision.Jul 19 2021, 1:24 PM
steven_wu requested review of this revision.Jul 19 2021, 1:24 PM
Herald added a project: Restricted Project. · View Herald TranscriptJul 19 2021, 1:24 PM
Herald added a subscriber: Restricted Project. · View Herald Transcript
steven_wu edited the summary of this revision. (Show Details)Jul 19 2021, 1:26 PM
thakis added inline comments.Jul 19 2021, 5:26 PM
compiler-rt/lib/builtins/CMakeLists.txt
534

Could you pull this out of the loop and keep it a custom command instead?

Doing more stuff in the build and less during cmake time is multicore- and incremental-build-friendlier.

See this mailing list thread for the background. Please also link it in the commit description.

Per the description here it seems that different architectures do get different build directories, so the symlink creation shouldn't conflict. As I noted in the thread, I would prefer to understand the root cause before applying a workaround or a fix.
Per my current understanding the issue shouldn't be possible and might well be a bug in CMake's Makefile generator since the issue was reported to not occur with Ninja.

If someone could provide me the build directory for the failing build (with the relevant runtimes builds generated) I would be able to understand the issue better.
Also, what CMake version is the issue encountered with?

See this mailing list thread for the background. Please also link it in the commit description.

Per the description here it seems that different architectures do get different build directories, so the symlink creation shouldn't conflict. As I noted in the thread, I would prefer to understand the root cause before applying a workaround or a fix.
Per my current understanding the issue shouldn't be possible and might well be a bug in CMake's Makefile generator since the issue was reported to not occur with Ninja.

If someone could provide me the build directory for the failing build (with the relevant runtimes builds generated) I would be able to understand the issue better.
Also, what CMake version is the issue encountered with?

Thanks for providing the background. I don't know enough about the makefile generator to know if this is a bug or a limitation. I am on CMake 3.19 and I think the original reporter is on 3.20. You can reproduce the issue running build with Xcode 12+/macOS 11+SDK on macOS (I am not sure how you cross compile for macOS if you are on other platform). I don't think you need to do anything special, just enable clang and compiler-rt in LLVM_ENABLE_PROJECTS with makefile generator.

Like I mentioned in the email thread, create_symlink command in both: projects/compiler-rt/lib/builtins/CMakeFiles/clang_rt.builtins_arm64_osx.dir/build.make and projects/compiler-rt/lib/builtins/CMakeFiles/clang_rt.builtins_arm64e_osx.dir/build.make. I can probably create a dump of makefiles generated from cmake if needed.

The other option is probably just use copy instead of symlink when build for darwin. I don't see copy file will error on race conditions.

The configured directory is rather large so let me give some more detail from the makefile generated. The custom target is generated in two of the makefiles I mentioned above, like following:

projects/compiler-rt/lib/builtins/outline_atomic_helpers.dir/outline_atomic_cas1_1.S:
        @$(CMAKE_COMMAND) -E cmake_echo_color --switch=$(COLOR) --blue --bold --progress-dir=/Users/steven/makefile_build/CMakeFiles --progress-num=$(CMAKE_PROGRESS_1) "Generating outline_atomic_helpers.dir/outline_atomic_cas1_1.S"
        cd /Users/steven/makefile_build/projects/compiler-rt/lib/builtins && /usr/local/bin/cmake -E create_symlink /Users/steven/llvm-project/compiler-rt/lib/builtins/aarch64/lse.S /Users/steven/makefile_build/projects/compiler-rt/lib/builtins/outline_atomic_helpers.dir/outline_atomic_cas1_1.S

When you build the clang builtin library, it does:

$ make -j 32 clang_rt.osx VERBOSE=1
...
/usr/bin/make  -f projects/compiler-rt/lib/builtins/CMakeFiles/clang_rt.builtins_arm64_osx.dir/build.make projects/compiler-rt/lib/        builtins/CMakeFiles/clang_rt.builtins_arm64_osx.dir/depend
/usr/bin/make  -f projects/compiler-rt/lib/builtins/CMakeFiles/clang_rt.builtins_arm64e_osx.dir/build.make projects/compiler-rt/lib/       builtins/CMakeFiles/clang_rt.builtins_arm64e_osx.dir/depend
...
CMake Error: failed to create symbolic link '/Users/steven/makefile_build/projects/compiler-rt/lib/builtins/outline_atomic_helpers.dir/outline_atomic_ldset1_3.S': file already exists
``
steven_wu edited the summary of this revision. (Show Details)Jul 20 2021, 9:12 AM

The configured directory is rather large so let me give some more detail from the makefile generated. The custom target is generated in two of the makefiles I mentioned above, like following:

projects/compiler-rt/lib/builtins/outline_atomic_helpers.dir/outline_atomic_cas1_1.S:
        @$(CMAKE_COMMAND) -E cmake_echo_color --switch=$(COLOR) --blue --bold --progress-dir=/Users/steven/makefile_build/CMakeFiles --progress-num=$(CMAKE_PROGRESS_1) "Generating outline_atomic_helpers.dir/outline_atomic_cas1_1.S"
        cd /Users/steven/makefile_build/projects/compiler-rt/lib/builtins && /usr/local/bin/cmake -E create_symlink /Users/steven/llvm-project/compiler-rt/lib/builtins/aarch64/lse.S /Users/steven/makefile_build/projects/compiler-rt/lib/builtins/outline_atomic_helpers.dir/outline_atomic_cas1_1.S

When you build the clang builtin library, it does:

$ make -j 32 clang_rt.osx VERBOSE=1
...
/usr/bin/make  -f projects/compiler-rt/lib/builtins/CMakeFiles/clang_rt.builtins_arm64_osx.dir/build.make projects/compiler-rt/lib/        builtins/CMakeFiles/clang_rt.builtins_arm64_osx.dir/depend
/usr/bin/make  -f projects/compiler-rt/lib/builtins/CMakeFiles/clang_rt.builtins_arm64e_osx.dir/build.make projects/compiler-rt/lib/       builtins/CMakeFiles/clang_rt.builtins_arm64e_osx.dir/depend
...
CMake Error: failed to create symbolic link '/Users/steven/makefile_build/projects/compiler-rt/lib/builtins/outline_atomic_helpers.dir/outline_atomic_ldset1_3.S': file already exists
``

Thanks, I now understand why this issue happens. With Make each target gets a Make subinvocation, while in Ninja they're generated into a single buildfile and can be correctly scheduled.
This issue only happens with targets that can build multiple sub-architectures at the same time (ARM32, MIPS, PPC64) possibly causing the symlinking to run at the sametime. On Apple this includes ARM64 and x86-64, making this a bit easier to hit.

The add_custom_target() solution noted in the mailing list seems the most appropriate to force ordering but still keep parallelism as noted by @thakis. Let's go with that.

steven_wu added a comment.EditedJul 20 2021, 12:52 PM

The configured directory is rather large so let me give some more detail from the makefile generated. The custom target is generated in two of the makefiles I mentioned above, like following:

projects/compiler-rt/lib/builtins/outline_atomic_helpers.dir/outline_atomic_cas1_1.S:
        @$(CMAKE_COMMAND) -E cmake_echo_color --switch=$(COLOR) --blue --bold --progress-dir=/Users/steven/makefile_build/CMakeFiles --progress-num=$(CMAKE_PROGRESS_1) "Generating outline_atomic_helpers.dir/outline_atomic_cas1_1.S"
        cd /Users/steven/makefile_build/projects/compiler-rt/lib/builtins && /usr/local/bin/cmake -E create_symlink /Users/steven/llvm-project/compiler-rt/lib/builtins/aarch64/lse.S /Users/steven/makefile_build/projects/compiler-rt/lib/builtins/outline_atomic_helpers.dir/outline_atomic_cas1_1.S

When you build the clang builtin library, it does:

$ make -j 32 clang_rt.osx VERBOSE=1
...
/usr/bin/make  -f projects/compiler-rt/lib/builtins/CMakeFiles/clang_rt.builtins_arm64_osx.dir/build.make projects/compiler-rt/lib/        builtins/CMakeFiles/clang_rt.builtins_arm64_osx.dir/depend
/usr/bin/make  -f projects/compiler-rt/lib/builtins/CMakeFiles/clang_rt.builtins_arm64e_osx.dir/build.make projects/compiler-rt/lib/       builtins/CMakeFiles/clang_rt.builtins_arm64e_osx.dir/depend
...
CMake Error: failed to create symbolic link '/Users/steven/makefile_build/projects/compiler-rt/lib/builtins/outline_atomic_helpers.dir/outline_atomic_ldset1_3.S': file already exists
``

Thanks, I now understand why this issue happens. With Make each target gets a Make subinvocation, while in Ninja they're generated into a single buildfile and can be correctly scheduled.
This issue only happens with targets that can build multiple sub-architectures at the same time (ARM32, MIPS, PPC64) possibly causing the symlinking to run at the sametime. On Apple this includes ARM64 and x86-64, making this a bit easier to hit.

The add_custom_target() solution noted in the mailing list seems the most appropriate to force ordering but still keep parallelism as noted by @thakis. Let's go with that.

I don't know how to use add_custom_target() to fix this. Maybe you can propose a patch?

Edit: a bit more context for why I don't know how to do that. You can't add multiple custom_target of the same name to enforce it only happens once. Using target of different name will only make problem worse since we now have to execute the command multiple times even with ninja.

tambre commandeered this revision.Jul 21 2021, 11:25 AM
tambre edited reviewers, added: steven_wu; removed: tambre.
tambre updated this revision to Diff 360536.Jul 21 2021, 11:26 AM

Custom target approach

tambre updated this revision to Diff 360538.Jul 21 2021, 11:31 AM

Fix whitespace changes

tambre retitled this revision from [compiler-rt][CMake] create aarch64 helper asm during configure time to [compiler-rt][CMake][arm64] Use a custom target for symlinking LSE sources.Jul 21 2021, 11:31 AM
tambre edited the summary of this revision. (Show Details)
tambre marked an inline comment as done.

I've commandeered this and used the custom target approach to ensure that all builtin Make sub-invocations depend on the symlinks having been created before.
Makefiles seem to now be generated in the correct way.

@steven_wu Could you try if this works?

steven_wu requested changes to this revision.Jul 22 2021, 8:32 AM

No, this doesn't work:

CMake Error at /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:420 (add_custom_target):
  add_custom_target cannot create target "lse_builtin_symlinks" because
  another target with the same name already exists.  The existing target is a
  custom target created in source directory
  "/Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins".  See
  documentation for policy CMP0002 for more details.
Call Stack (most recent call first):
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins/CMakeLists.txt:674 (darwin_add_builtin_libraries)


-- For ios_arm64e builtins preferring aarch64/fp_mode.c to fp_mode.c
CMake Error at /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:420 (add_custom_target):
  add_custom_target cannot create target "lse_builtin_symlinks" because
  another target with the same name already exists.  The existing target is a
  custom target created in source directory
  "/Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins".  See
  documentation for policy CMP0002 for more details.
Call Stack (most recent call first):
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins/CMakeLists.txt:674 (darwin_add_builtin_libraries)

That is exactly the error I got when I tried custom_target in a slightly different approach. lse_builtin_symlinks target is created twice by arm64 and arm64e builtin.

This revision now requires changes to proceed.Jul 22 2021, 8:32 AM

The error says ios_arm64e because it duplicates for every single os arm64 combination. osx_arm64, osx_arm64e, ios_arm64, ios_arm64e, etc.

tambre updated this revision to Diff 361109.Jul 23 2021, 12:23 AM

Prevent duplicate targets, simplify code

No, this doesn't work:

CMake Error at /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:420 (add_custom_target):
  add_custom_target cannot create target "lse_builtin_symlinks" because
  another target with the same name already exists.  The existing target is a
  custom target created in source directory
  "/Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins".  See
  documentation for policy CMP0002 for more details.
Call Stack (most recent call first):
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins/CMakeLists.txt:674 (darwin_add_builtin_libraries)


-- For ios_arm64e builtins preferring aarch64/fp_mode.c to fp_mode.c
CMake Error at /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:420 (add_custom_target):
  add_custom_target cannot create target "lse_builtin_symlinks" because
  another target with the same name already exists.  The existing target is a
  custom target created in source directory
  "/Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins".  See
  documentation for policy CMP0002 for more details.
Call Stack (most recent call first):
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins/CMakeLists.txt:674 (darwin_add_builtin_libraries)

That is exactly the error I got when I tried custom_target in a slightly different approach. lse_builtin_symlinks target is created twice by arm64 and arm64e builtin.

Thanks for testing. I think it should be fixed now, please try again.

tambre updated this revision to Diff 361110.Jul 23 2021, 12:27 AM

Remove whitespace changes

thakis accepted this revision.Jul 23 2021, 6:00 AM

I haven't checked if it works, but it has roughly the right shape imho. So if this does solve the problem, this lg, thanks for the patch :)

Now I got this error when generating makefile:

-- Configuring done
CMake Error at /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:275 (add_library):
  Cannot find source file:

    /Users/steven/dev/monorepo/makefile_build/projects/compiler-rt/lib/builtins/outline_atomic_helpers.dir/outline_atomic_cas1_1.S

  Tried extensions .c .C .c++ .cc .cpp .cxx .cu .m .M .mm .h .hh .h++ .hm
  .hpp .hxx .in .txx .f .F .for .f77 .f90 .f95 .f03 .ispc
Call Stack (most recent call first):
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:456 (darwin_add_builtin_library)
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins/CMakeLists.txt:674 (darwin_add_builtin_libraries)


CMake Error at /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:275 (add_library):
  No SOURCES given to target: clang_rt.cc_kext_arm64e_ios
Call Stack (most recent call first):
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:456 (darwin_add_builtin_library)
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins/CMakeLists.txt:674 (darwin_add_builtin_libraries)


CMake Error at /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:275 (add_library):
  No SOURCES given to target: clang_rt.builtins_arm64_osx
Call Stack (most recent call first):
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:431 (darwin_add_builtin_library)
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins/CMakeLists.txt:674 (darwin_add_builtin_libraries)


CMake Error at /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:275 (add_library):
  No SOURCES given to target: clang_rt.builtins_arm64e_ios
Call Stack (most recent call first):
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:431 (darwin_add_builtin_library)
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins/CMakeLists.txt:674 (darwin_add_builtin_libraries)


CMake Error at /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:275 (add_library):
  No SOURCES given to target: clang_rt.builtins_arm64_ios
Call Stack (most recent call first):
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:431 (darwin_add_builtin_library)
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins/CMakeLists.txt:674 (darwin_add_builtin_libraries)


CMake Error at /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:275 (add_library):
  No SOURCES given to target: clang_rt.cc_kext_arm64_ios
Call Stack (most recent call first):
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:456 (darwin_add_builtin_library)
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins/CMakeLists.txt:674 (darwin_add_builtin_libraries)


CMake Error at /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:275 (add_library):
  No SOURCES given to target: clang_rt.cc_kext_arm64e_osx
Call Stack (most recent call first):
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:456 (darwin_add_builtin_library)
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins/CMakeLists.txt:674 (darwin_add_builtin_libraries)


CMake Error at /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:275 (add_library):
  No SOURCES given to target: clang_rt.builtins_arm64e_osx
Call Stack (most recent call first):
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:431 (darwin_add_builtin_library)
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins/CMakeLists.txt:674 (darwin_add_builtin_libraries)


CMake Error at /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:275 (add_library):
  No SOURCES given to target: clang_rt.cc_kext_arm64_osx
Call Stack (most recent call first):
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:456 (darwin_add_builtin_library)
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins/CMakeLists.txt:674 (darwin_add_builtin_libraries)


CMake Generate step failed.  Build files cannot be regenerated correctly.
tambre updated this revision to Diff 361430.Jul 24 2021, 3:22 AM

Fix missing dependency for Darwin kexts

Now I got this error when generating makefile:

Please try again now.

I still get the same error. 😿

tambre updated this revision to Diff 361611.Jul 26 2021, 3:50 AM

Fix Darwin target creation check

I still get the same error. 😿

Please try again.

steven_wu accepted this revision.Jul 26 2021, 9:39 AM

Works now. Thanks!

Looking at the makefile and ninja file created after this CMake change and it looks good. LGTM.

This revision is now accepted and ready to land.Jul 26 2021, 9:39 AM

Oh, wait, I do see this warning in the build and I am not sure if this will be a problem or not:

Dependee "/Users/steven/dev/monorepo/makefile_build/projects/compiler-rt/lib/builtins/CMakeFiles/lse_builtin_symlinks.dir/DependInfo.cmake" is newer than depender "/Users/steven/dev/monorepo/makefile_build/projects/compiler-rt/lib/builtins/CMakeFiles/lse_builtin_symlinks.dir/depend.internal".

Oh, wait, I do see this warning in the build and I am not sure if this will be a problem or not:

Dependee "/Users/steven/dev/monorepo/makefile_build/projects/compiler-rt/lib/builtins/CMakeFiles/lse_builtin_symlinks.dir/DependInfo.cmake" is newer than depender "/Users/steven/dev/monorepo/makefile_build/projects/compiler-rt/lib/builtins/CMakeFiles/lse_builtin_symlinks.dir/depend.internal".

That simply seems to be CMake's Make generator's verbose output in cases when dependencies need to be regenerated (in this case the symlinks). I think this should be fine, so I'll land this.

This revision was landed with ongoing or failed builds.Jul 26 2021, 11:42 AM
This revision was automatically updated to reflect the committed changes.