[compiler-rt][CMake][arm64] Use a custom target for symlinking LSE sources

[tambre]

Summary

On Apple platforms the builtins may be built for both arm64 and arm64e.
With Makefile generators separate targets are built using Make sub-invocations.
This causes a race when creating the symlink which may sometimes fail.

Work around this by using a custom target that the builtin targets depend on.
This causes any sub-invocations to depend on the symlinks having been created before.

Mailing list thread: https://lists.llvm.org/pipermail/llvm-dev/2021-July/151822.html

[thakis]

compiler-rt/lib/builtins/CMakeLists.txt

534

Could you pull this out of the loop and keep it a custom command instead?

Doing more stuff in the build and less during cmake time is multicore- and incremental-build-friendlier.

[tambre]

See this mailing list thread for the background. Please also link it in the commit description.

Per the description here it seems that different architectures do get different build directories, so the symlink creation shouldn't conflict. As I noted in the thread, I would prefer to understand the root cause before applying a workaround or a fix.
Per my current understanding the issue shouldn't be possible and might well be a bug in CMake's Makefile generator since the issue was reported to not occur with Ninja.

If someone could provide me the build directory for the failing build (with the relevant runtimes builds generated) I would be able to understand the issue better.
Also, what CMake version is the issue encountered with?

[steven_wu]

In D106305#2889456, @tambre wrote:

See this mailing list thread for the background. Please also link it in the commit description.

Per the description here it seems that different architectures do get different build directories, so the symlink creation shouldn't conflict. As I noted in the thread, I would prefer to understand the root cause before applying a workaround or a fix.
Per my current understanding the issue shouldn't be possible and might well be a bug in CMake's Makefile generator since the issue was reported to not occur with Ninja.

If someone could provide me the build directory for the failing build (with the relevant runtimes builds generated) I would be able to understand the issue better.
Also, what CMake version is the issue encountered with?

Thanks for providing the background. I don't know enough about the makefile generator to know if this is a bug or a limitation. I am on CMake 3.19 and I think the original reporter is on 3.20. You can reproduce the issue running build with Xcode 12+/macOS 11+SDK on macOS (I am not sure how you cross compile for macOS if you are on other platform). I don't think you need to do anything special, just enable clang and compiler-rt in LLVM_ENABLE_PROJECTS with makefile generator.

Like I mentioned in the email thread, create_symlink command in both: projects/compiler-rt/lib/builtins/CMakeFiles/clang_rt.builtins_arm64_osx.dir/build.make and projects/compiler-rt/lib/builtins/CMakeFiles/clang_rt.builtins_arm64e_osx.dir/build.make. I can probably create a dump of makefiles generated from cmake if needed.

The other option is probably just use copy instead of symlink when build for darwin. I don't see copy file will error on race conditions.

[steven_wu]

The configured directory is rather large so let me give some more detail from the makefile generated. The custom target is generated in two of the makefiles I mentioned above, like following:

projects/compiler-rt/lib/builtins/outline_atomic_helpers.dir/outline_atomic_cas1_1.S:
        @$(CMAKE_COMMAND) -E cmake_echo_color --switch=$(COLOR) --blue --bold --progress-dir=/Users/steven/makefile_build/CMakeFiles --progress-num=$(CMAKE_PROGRESS_1) "Generating outline_atomic_helpers.dir/outline_atomic_cas1_1.S"
        cd /Users/steven/makefile_build/projects/compiler-rt/lib/builtins && /usr/local/bin/cmake -E create_symlink /Users/steven/llvm-project/compiler-rt/lib/builtins/aarch64/lse.S /Users/steven/makefile_build/projects/compiler-rt/lib/builtins/outline_atomic_helpers.dir/outline_atomic_cas1_1.S

When you build the clang builtin library, it does:

$ make -j 32 clang_rt.osx VERBOSE=1
...
/usr/bin/make  -f projects/compiler-rt/lib/builtins/CMakeFiles/clang_rt.builtins_arm64_osx.dir/build.make projects/compiler-rt/lib/        builtins/CMakeFiles/clang_rt.builtins_arm64_osx.dir/depend
/usr/bin/make  -f projects/compiler-rt/lib/builtins/CMakeFiles/clang_rt.builtins_arm64e_osx.dir/build.make projects/compiler-rt/lib/       builtins/CMakeFiles/clang_rt.builtins_arm64e_osx.dir/depend
...
CMake Error: failed to create symbolic link '/Users/steven/makefile_build/projects/compiler-rt/lib/builtins/outline_atomic_helpers.dir/outline_atomic_ldset1_3.S': file already exists
``

[tambre]

In D106305#2890733, @steven_wu wrote:

The configured directory is rather large so let me give some more detail from the makefile generated. The custom target is generated in two of the makefiles I mentioned above, like following:

projects/compiler-rt/lib/builtins/outline_atomic_helpers.dir/outline_atomic_cas1_1.S:
        @$(CMAKE_COMMAND) -E cmake_echo_color --switch=$(COLOR) --blue --bold --progress-dir=/Users/steven/makefile_build/CMakeFiles --progress-num=$(CMAKE_PROGRESS_1) "Generating outline_atomic_helpers.dir/outline_atomic_cas1_1.S"
        cd /Users/steven/makefile_build/projects/compiler-rt/lib/builtins && /usr/local/bin/cmake -E create_symlink /Users/steven/llvm-project/compiler-rt/lib/builtins/aarch64/lse.S /Users/steven/makefile_build/projects/compiler-rt/lib/builtins/outline_atomic_helpers.dir/outline_atomic_cas1_1.S

When you build the clang builtin library, it does:

$ make -j 32 clang_rt.osx VERBOSE=1
...
/usr/bin/make  -f projects/compiler-rt/lib/builtins/CMakeFiles/clang_rt.builtins_arm64_osx.dir/build.make projects/compiler-rt/lib/        builtins/CMakeFiles/clang_rt.builtins_arm64_osx.dir/depend
/usr/bin/make  -f projects/compiler-rt/lib/builtins/CMakeFiles/clang_rt.builtins_arm64e_osx.dir/build.make projects/compiler-rt/lib/       builtins/CMakeFiles/clang_rt.builtins_arm64e_osx.dir/depend
...
CMake Error: failed to create symbolic link '/Users/steven/makefile_build/projects/compiler-rt/lib/builtins/outline_atomic_helpers.dir/outline_atomic_ldset1_3.S': file already exists
``

Thanks, I now understand why this issue happens. With Make each target gets a Make subinvocation, while in Ninja they're generated into a single buildfile and can be correctly scheduled.
This issue only happens with targets that can build multiple sub-architectures at the same time (ARM32, MIPS, PPC64) possibly causing the symlinking to run at the sametime. On Apple this includes ARM64 and x86-64, making this a bit easier to hit.

The add_custom_target() solution noted in the mailing list seems the most appropriate to force ordering but still keep parallelism as noted by @thakis. Let's go with that.

[steven_wu]

In D106305#2891318, @tambre wrote:

In D106305#2890733, @steven_wu wrote:

The configured directory is rather large so let me give some more detail from the makefile generated. The custom target is generated in two of the makefiles I mentioned above, like following:

projects/compiler-rt/lib/builtins/outline_atomic_helpers.dir/outline_atomic_cas1_1.S:
        @$(CMAKE_COMMAND) -E cmake_echo_color --switch=$(COLOR) --blue --bold --progress-dir=/Users/steven/makefile_build/CMakeFiles --progress-num=$(CMAKE_PROGRESS_1) "Generating outline_atomic_helpers.dir/outline_atomic_cas1_1.S"
        cd /Users/steven/makefile_build/projects/compiler-rt/lib/builtins && /usr/local/bin/cmake -E create_symlink /Users/steven/llvm-project/compiler-rt/lib/builtins/aarch64/lse.S /Users/steven/makefile_build/projects/compiler-rt/lib/builtins/outline_atomic_helpers.dir/outline_atomic_cas1_1.S

When you build the clang builtin library, it does:

$ make -j 32 clang_rt.osx VERBOSE=1
...
/usr/bin/make  -f projects/compiler-rt/lib/builtins/CMakeFiles/clang_rt.builtins_arm64_osx.dir/build.make projects/compiler-rt/lib/        builtins/CMakeFiles/clang_rt.builtins_arm64_osx.dir/depend
/usr/bin/make  -f projects/compiler-rt/lib/builtins/CMakeFiles/clang_rt.builtins_arm64e_osx.dir/build.make projects/compiler-rt/lib/       builtins/CMakeFiles/clang_rt.builtins_arm64e_osx.dir/depend
...
CMake Error: failed to create symbolic link '/Users/steven/makefile_build/projects/compiler-rt/lib/builtins/outline_atomic_helpers.dir/outline_atomic_ldset1_3.S': file already exists
``

Thanks, I now understand why this issue happens. With Make each target gets a Make subinvocation, while in Ninja they're generated into a single buildfile and can be correctly scheduled.
This issue only happens with targets that can build multiple sub-architectures at the same time (ARM32, MIPS, PPC64) possibly causing the symlinking to run at the sametime. On Apple this includes ARM64 and x86-64, making this a bit easier to hit.

The add_custom_target() solution noted in the mailing list seems the most appropriate to force ordering but still keep parallelism as noted by @thakis. Let's go with that.

I don't know how to use add_custom_target() to fix this. Maybe you can propose a patch?

Edit: a bit more context for why I don't know how to do that. You can't add multiple custom_target of the same name to enforce it only happens once. Using target of different name will only make problem worse since we now have to execute the command multiple times even with ninja.

[tambre]

Custom target approach

[tambre]

Fix whitespace changes

[tambre]

I've commandeered this and used the custom target approach to ensure that all builtin Make sub-invocations depend on the symlinks having been created before.
Makefiles seem to now be generated in the correct way.

@steven_wu Could you try if this works?

[steven_wu]

No, this doesn't work:

CMake Error at /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:420 (add_custom_target):
  add_custom_target cannot create target "lse_builtin_symlinks" because
  another target with the same name already exists.  The existing target is a
  custom target created in source directory
  "/Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins".  See
  documentation for policy CMP0002 for more details.
Call Stack (most recent call first):
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins/CMakeLists.txt:674 (darwin_add_builtin_libraries)


-- For ios_arm64e builtins preferring aarch64/fp_mode.c to fp_mode.c
CMake Error at /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:420 (add_custom_target):
  add_custom_target cannot create target "lse_builtin_symlinks" because
  another target with the same name already exists.  The existing target is a
  custom target created in source directory
  "/Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins".  See
  documentation for policy CMP0002 for more details.
Call Stack (most recent call first):
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins/CMakeLists.txt:674 (darwin_add_builtin_libraries)

That is exactly the error I got when I tried custom_target in a slightly different approach. lse_builtin_symlinks target is created twice by arm64 and arm64e builtin.

[steven_wu]

The error says ios_arm64e because it duplicates for every single os arm64 combination. osx_arm64, osx_arm64e, ios_arm64, ios_arm64e, etc.

[tambre]

Prevent duplicate targets, simplify code

[tambre]

In D106305#2896719, @steven_wu wrote:

No, this doesn't work:

CMake Error at /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:420 (add_custom_target):
  add_custom_target cannot create target "lse_builtin_symlinks" because
  another target with the same name already exists.  The existing target is a
  custom target created in source directory
  "/Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins".  See
  documentation for policy CMP0002 for more details.
Call Stack (most recent call first):
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins/CMakeLists.txt:674 (darwin_add_builtin_libraries)


-- For ios_arm64e builtins preferring aarch64/fp_mode.c to fp_mode.c
CMake Error at /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:420 (add_custom_target):
  add_custom_target cannot create target "lse_builtin_symlinks" because
  another target with the same name already exists.  The existing target is a
  custom target created in source directory
  "/Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins".  See
  documentation for policy CMP0002 for more details.
Call Stack (most recent call first):
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins/CMakeLists.txt:674 (darwin_add_builtin_libraries)

That is exactly the error I got when I tried custom_target in a slightly different approach. lse_builtin_symlinks target is created twice by arm64 and arm64e builtin.

Thanks for testing. I think it should be fixed now, please try again.

[tambre]

Remove whitespace changes

[thakis]

I haven't checked if it works, but it has roughly the right shape imho. So if this does solve the problem, this lg, thanks for the patch :)

[steven_wu]

Now I got this error when generating makefile:

-- Configuring done
CMake Error at /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:275 (add_library):
  Cannot find source file:

    /Users/steven/dev/monorepo/makefile_build/projects/compiler-rt/lib/builtins/outline_atomic_helpers.dir/outline_atomic_cas1_1.S

  Tried extensions .c .C .c++ .cc .cpp .cxx .cu .m .M .mm .h .hh .h++ .hm
  .hpp .hxx .in .txx .f .F .for .f77 .f90 .f95 .f03 .ispc
Call Stack (most recent call first):
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:456 (darwin_add_builtin_library)
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins/CMakeLists.txt:674 (darwin_add_builtin_libraries)


CMake Error at /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:275 (add_library):
  No SOURCES given to target: clang_rt.cc_kext_arm64e_ios
Call Stack (most recent call first):
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:456 (darwin_add_builtin_library)
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins/CMakeLists.txt:674 (darwin_add_builtin_libraries)


CMake Error at /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:275 (add_library):
  No SOURCES given to target: clang_rt.builtins_arm64_osx
Call Stack (most recent call first):
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:431 (darwin_add_builtin_library)
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins/CMakeLists.txt:674 (darwin_add_builtin_libraries)


CMake Error at /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:275 (add_library):
  No SOURCES given to target: clang_rt.builtins_arm64e_ios
Call Stack (most recent call first):
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:431 (darwin_add_builtin_library)
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins/CMakeLists.txt:674 (darwin_add_builtin_libraries)


CMake Error at /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:275 (add_library):
  No SOURCES given to target: clang_rt.builtins_arm64_ios
Call Stack (most recent call first):
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:431 (darwin_add_builtin_library)
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins/CMakeLists.txt:674 (darwin_add_builtin_libraries)


CMake Error at /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:275 (add_library):
  No SOURCES given to target: clang_rt.cc_kext_arm64_ios
Call Stack (most recent call first):
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:456 (darwin_add_builtin_library)
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins/CMakeLists.txt:674 (darwin_add_builtin_libraries)


CMake Error at /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:275 (add_library):
  No SOURCES given to target: clang_rt.cc_kext_arm64e_osx
Call Stack (most recent call first):
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:456 (darwin_add_builtin_library)
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins/CMakeLists.txt:674 (darwin_add_builtin_libraries)


CMake Error at /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:275 (add_library):
  No SOURCES given to target: clang_rt.builtins_arm64e_osx
Call Stack (most recent call first):
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:431 (darwin_add_builtin_library)
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins/CMakeLists.txt:674 (darwin_add_builtin_libraries)


CMake Error at /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:275 (add_library):
  No SOURCES given to target: clang_rt.cc_kext_arm64_osx
Call Stack (most recent call first):
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake:456 (darwin_add_builtin_library)
  /Users/steven/dev/monorepo/llvm-project/compiler-rt/lib/builtins/CMakeLists.txt:674 (darwin_add_builtin_libraries)


CMake Generate step failed.  Build files cannot be regenerated correctly.

[tambre]

Fix missing dependency for Darwin kexts

[tambre]

In D106305#2900380, @steven_wu wrote:

Now I got this error when generating makefile:

Please try again now.

[steven_wu]

I still get the same error. 😿

[tambre]

Fix Darwin target creation check

[tambre]

In D106305#2903507, @steven_wu wrote:

I still get the same error. 😿

Please try again.

[steven_wu]

Works now. Thanks!

Looking at the makefile and ninja file created after this CMake change and it looks good. LGTM.

[steven_wu]

Oh, wait, I do see this warning in the build and I am not sure if this will be a problem or not:

Dependee "/Users/steven/dev/monorepo/makefile_build/projects/compiler-rt/lib/builtins/CMakeFiles/lse_builtin_symlinks.dir/DependInfo.cmake" is newer than depender "/Users/steven/dev/monorepo/makefile_build/projects/compiler-rt/lib/builtins/CMakeFiles/lse_builtin_symlinks.dir/depend.internal".

[tambre]

In D106305#2904687, @steven_wu wrote:

Oh, wait, I do see this warning in the build and I am not sure if this will be a problem or not:

Dependee "/Users/steven/dev/monorepo/makefile_build/projects/compiler-rt/lib/builtins/CMakeFiles/lse_builtin_symlinks.dir/DependInfo.cmake" is newer than depender "/Users/steven/dev/monorepo/makefile_build/projects/compiler-rt/lib/builtins/CMakeFiles/lse_builtin_symlinks.dir/depend.internal".

That simply seems to be CMake's Make generator's verbose output in cases when dependencies need to be regenerated (in this case the symlinks). I think this should be fine, so I'll land this.

compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake

macro(darwin_add_builtin_libraries)
   set(PROFILE_SOURCES ../profile/InstrProfiling.c
                       ../profile/InstrProfilingBuffer.c
                       ../profile/InstrProfilingPlatformDarwin.c
                       ../profile/InstrProfilingWriter.c
                       ../profile/InstrProfilingInternal.c
                       ../profile/InstrProfilingVersionVar.c)
   foreach (os ${ARGN})
     list_intersect(DARWIN_BUILTIN_ARCHS DARWIN_${os}_BUILTIN_ARCHS BUILTIN_SUPPORTED_ARCH)
+
+    if((arm64 IN_LIST DARWIN_BUILTIN_ARCHS OR arm64e IN_LIST DARWIN_BUILTIN_ARCHS) AND NOT TARGET lse_builtin_symlinks)
+      add_custom_target(
+        lse_builtin_symlinks
+        BYPRODUCTS ${lse_builtins}
+        ${arm64_lse_commands}
+      )
+
+      set(deps_arm64 lse_builtin_symlinks)
+      set(deps_arm64e lse_builtin_symlinks)
+    endif()
+
     foreach (arch ${DARWIN_BUILTIN_ARCHS})
       darwin_find_excluded_builtins_list(${arch}_${os}_EXCLUDED_BUILTINS
                               OS ${os}
                               ARCH ${arch}
                               MIN_VERSION ${DARWIN_${os}_BUILTIN_MIN_VER})
 
       darwin_filter_builtin_sources(filtered_sources
         ${os}_${arch}
         EXCLUDE ${arch}_${os}_EXCLUDED_BUILTINS
         ${${arch}_SOURCES})
 
       darwin_add_builtin_library(clang_rt builtins
                               OS ${os}
                               ARCH ${arch}
+                              DEPS ${deps_${arch}}
                               SOURCES ${filtered_sources}
                               CFLAGS ${CFLAGS} -arch ${arch}
                               PARENT_TARGET builtins)
     endforeach()
 
     # Don't build cc_kext libraries for simulator platforms
     if(NOT DARWIN_${os}_SKIP_CC_KEXT)
       foreach (arch ${DARWIN_BUILTIN_ARCHS})
         # By not specifying MIN_VERSION this only reads the OS and OS-arch lists.
         # We don't want to filter out the builtins that are present in libSystem
         # because kexts can't link libSystem.
         darwin_find_excluded_builtins_list(${arch}_${os}_EXCLUDED_BUILTINS
                               OS ${os}
                               ARCH ${arch})
 
         darwin_filter_builtin_sources(filtered_sources
           cc_kext_${os}_${arch}
           EXCLUDE ${arch}_${os}_EXCLUDED_BUILTINS
           ${${arch}_SOURCES})
 
         # In addition to the builtins cc_kext includes some profile sources
         darwin_add_builtin_library(clang_rt cc_kext
                                 OS ${os}
                                 ARCH ${arch}
+                                DEPS ${deps_${arch}}
                                 SOURCES ${filtered_sources} ${PROFILE_SOURCES}
                                 CFLAGS ${CFLAGS} -arch ${arch} -mkernel
                                 DEFS KERNEL_USE
                                 INCLUDE_DIRS ../../include
                                 PARENT_TARGET builtins)
       endforeach()
       set(archive_name clang_rt.cc_kext_${os})
       if(${os} STREQUAL "osx")

compiler-rt/lib/builtins/CMakeLists.txt

else()
   set(COMPILER_RT_LINK_OR_COPY copy)
 endif()
 
 foreach(pat cas swp ldadd ldclr ldeor ldset)
   foreach(size 1 2 4 8 16)
     foreach(model 1 2 3 4)
       if(pat STREQUAL "cas" OR NOT size STREQUAL "16")
         set(helper_asm "${OA_HELPERS_DIR}/outline_atomic_${pat}${size}_${model}.S")
-        add_custom_command(
-          OUTPUT ${helper_asm}
-          COMMAND ${CMAKE_COMMAND} -E ${COMPILER_RT_LINK_OR_COPY} "${CMAKE_CURRENT_SOURCE_DIR}/aarch64/lse.S" "${helper_asm}"
-        )
+        list(APPEND lse_builtins "${helper_asm}")
+        list(APPEND arm64_lse_commands COMMAND ${CMAKE_COMMAND} -E ${COMPILER_RT_LINK_OR_COPY} "${CMAKE_CURRENT_SOURCE_DIR}/aarch64/lse.S" "${helper_asm}")
         set_source_files_properties("${helper_asm}"
           PROPERTIES
           COMPILE_DEFINITIONS "L_${pat};SIZE=${size};MODEL=${model}"
           INCLUDE_DIRECTORIES "${CMAKE_CURRENT_SOURCE_DIR}"
         )
         list(APPEND aarch64_SOURCES "${helper_asm}")
       endif()
     endforeach(model)
if (CAN_TARGET_${arch})
       endif()
 
       # For RISCV32, we must force enable int128 for compiling long
       # double routines.
       if("${arch}" STREQUAL "riscv32")
         list(APPEND BUILTIN_CFLAGS_${arch} -fforce-enable-int128)
       endif()
 
+      if(arch STREQUAL "aarch64")
+        add_custom_target(
+          lse_builtin_symlinks
+          BYPRODUCTS ${lse_builtins}
+          ${arm64_lse_commands}
+        )
+
+        set(deps_aarch64 lse_builtin_symlinks)
+      endif()
+
       add_compiler_rt_runtime(clang_rt.builtins
                               STATIC
                               ARCHS ${arch}
+                              DEPS ${deps_${arch}}
                               SOURCES ${${arch}_SOURCES}
                               DEFS ${BUILTIN_DEFS}
                               CFLAGS ${BUILTIN_CFLAGS_${arch}}
                               PARENT_TARGET builtins)
     endif ()
   endforeach ()
 endif ()