[OpenMP][CMake] Use in-project clang as CUDA->IR compiler for new DeviceRTL.

[Meinersbur]

Summary

Use the in-project clang, llvm-link and opt if available and unless CMake cache variables specify to use a different compiler. This applies D101265 to the new DeviceRTL's CMakeLists.txt which was copied before D101265 was applied.

Fixes the openmp-offloading-cuda-runtime builder failing since D110006.

[jdoerfert]

I like this. @ye-luo @tianshilei1992 @grokos any thoughts?

[ye-luo]

openmp/libomptarget/DeviceRTL/CMakeLists.txt

63–69

I got empty ${llvm_link} on my machine and use ${compiler_dir}/llvm-link solves the problem.

[Meinersbur]

openmp/libomptarget/DeviceRTL/CMakeLists.txt

63–69

Thanks for noticing. I removed ${llvm_link} to avoid confusion with ${bc_linker}. Missed this occurrence.

[Meinersbur]

• Replaced missed occurance of ${llvm_link}

[tianshilei1992]

Generally looks better than previous patch. Thanks for that. This patch assumes those targets must be there in runtime build, which might not be the case. I think it could be better if we first check if those targets are there. If not, raise an error directly, saying runtime build requires clang, etc.

openmp/libomptarget/DeviceRTL/CMakeLists.txt

46

This is the case with LLVM_ENABLE_RUNTIMES=openmp.

This is not always the case. Standalone build also falls to this branch.

75

${compiler_dir}/opt appears three times. Can we set a variable for that?

[Meinersbur]

In D110251#3015450, @tianshilei1992 wrote:

Generally looks better than previous patch. Thanks for that. This patch assumes those targets must be there in runtime build, which might not be the case. I think it could be better if we first check if those targets are there.

The clang target does not exist at this point since the CMake script for openmp is processed before clang.

If not, raise an error directly, saying runtime build requires clang, etc.

The error already exists: "Not building deviceRTL: clang not found", unless a CUDA->IR compiler is explicitly specified with LIBOMPTARGET_NVPTX_CUDA_COMPILER .

[Meinersbur]

• Use temporary variable for candidates

[tianshilei1992]

openmp/libomptarget/DeviceRTL/CMakeLists.txt

41

LLVM_TOOL_CLANG_BUILD seems already obsolete. I searched the whole LLVM project but didn't find where it is defined. The only two places it is still being used is in polly and libomptarget.

[Meinersbur]

openmp/libomptarget/DeviceRTL/CMakeLists.txt

41

It is defined here: https://github.com/llvm/llvm-project/blob/e21b0ba8c9378bca01d2311be4e1b6ccd3397bc4/llvm/CMakeLists.txt#L141

It is deprecated to be set by the user, as the comment before is explaining. It is still an integral part of the entire external project mechanism: https://github.com/llvm/llvm-project/blob/e21b0ba8c9378bca01d2311be4e1b6ccd3397bc4/llvm/cmake/modules/AddLLVM.cmake#L1355

There are not many literal LLVM_TOOL_CLANG_BUILD strings because its implementation is mostly generic for all tools, not just clang.

[tianshilei1992]

LGTM

openmp/libomptarget/DeviceRTL/CMakeLists.txt

 # an LLVM linker.
 set(LIBOMPTARGET_NVPTX_CUDA_COMPILER "" CACHE STRING
   "Location of a CUDA compiler capable of emitting LLVM bitcode.")
 set(LIBOMPTARGET_NVPTX_BC_LINKER "" CACHE STRING
   "Location of a linker capable of linking LLVM bitcode objects.")
 
 if (NOT LIBOMPTARGET_NVPTX_CUDA_COMPILER STREQUAL "")
   set(cuda_compiler ${LIBOMPTARGET_NVPTX_CUDA_COMPILER})
+elseif (LLVM_TOOL_CLANG_BUILD AND NOT CMAKE_CROSSCOMPILING)
+  # Compile the deviceRTL with the clang that is built in the project.
+  set(cuda_compiler "$<TARGET_FILE:clang>")
 elseif(${CMAKE_C_COMPILER_ID} STREQUAL "Clang")
+  # Compile the device runtime with the compiler that OpenMP is built with.
+  # This is the case with LLVM_ENABLE_RUNTIMES=openmp.
+  # FIXME: This is unreliable; the compiler can be on older version of clang
+  # that does not support compiling CUDA, or only an older version of it. The
+  # risk is especially high on sytems where clang is the default compiler
+  # (MacOS, BSDs). LLVM_ENABLE_RUNTIMES=openmp should itself set
+  # LIBOMPTARGET_NVPTX_CUDA_COMPILER instead.
   set(cuda_compiler ${CMAKE_C_COMPILER})
 else()
-  libomptarget_say("Not building NVPTX deviceRTL: clang not found")
+  libomptarget_say("Not building deviceRTL: clang not found")
   return()
 endif()
 
 # Get compiler directory to try to locate a suitable linker.
 get_filename_component(compiler_dir ${cuda_compiler} DIRECTORY)
-set(llvm_link "${compiler_dir}/llvm-link")
-set(opt "${compiler_dir}/opt")
 
 if (NOT LIBOMPTARGET_NVPTX_BC_LINKER STREQUAL "")
   set(bc_linker ${LIBOMPTARGET_NVPTX_BC_LINKER})
-elseif (EXISTS ${llvm_link})
-  set(bc_linker ${llvm_link})
+elseif (EXISTS "${compiler_dir}/llvm-link" AND NOT IS_DIRECTORY "${compiler_dir}/llvm-link")
+  # Try to use the linker consistent with the CUDA compiler unless explicitly
+  # set to a different linker.
+  set(bc_linker "${compiler_dir}/llvm-link")
+elseif (NOT OPENMP_STANDALONE_BUILD AND NOT CMAKE_CROSSCOMPILING)
+  # Use the linker also built in the same project.
+  set(bc_linker "$<TARGET_FILE:llvm-link>")
 else()
-  libomptarget_say("Not building NVPTX deviceRTL: llvm-link not found")
+  libomptarget_say("Not building deviceRTL: llvm-link not found")
+  return()
+endif()
+
+if (EXISTS "${compiler_dir}/opt" AND NOT IS_DIRECTORY "${compiler_dir}/opt")
+  # Try to use the opt consistent with the CUDA compiler.
+  set(opt "${compiler_dir}/opt")
+elseif (NOT OPENMP_STANDALONE_BUILD AND NOT CMAKE_CROSSCOMPILING)
+  # Use opt that is also built in the same project.
+  set(opt "$<TARGET_FILE:opt>")
+else()
+  libomptarget_say("Not building deviceRTL: opt not found")
   return()
 endif()
 
 # TODO: This part needs to be refined when libomptarget is going to support
 # Windows!
 # TODO: This part can also be removed if we can change the clang driver to make
 # it support device only compilation.
 if(CMAKE_HOST_SYSTEM_PROCESSOR MATCHES "x86_64")

 if (DEFINED LIBOMPTARGET_NVPTX_MAX_SM)
   set(MAX_SM_DEFINITION "-DMAX_SM=${LIBOMPTARGET_NVPTX_MAX_SM}")
 endif()
 
 # Activate RTL message dumps if requested by the user.
 set(LIBOMPTARGET_DEVICE_DEBUG FALSE CACHE BOOL
   "Activate NVPTX device RTL debug messages.")
 
-libomptarget_say("Building CUDA LLVM bitcode offloading device RTL.")
+if ("${cuda_compiler}" STREQUAL "$<TARGET_FILE:clang>")
+  libomptarget_say("Building LLVM bitcode offloading device RTL using in-tree clang.")
+else ()
+  libomptarget_say("Building LLVM bitcode offloading device RTL using ${cuda_compiler}")
+endif ()
 
 set(src_files
   ${source_directory}/Configuration.cpp
   ${source_directory}/Debug.cpp
   ${source_directory}/Kernel.cpp
   ${source_directory}/Mapping.cpp
   ${source_directory}/Misc.cpp
   ${source_directory}/Parallelism.cpp
foreach(src ${src_files})
     add_custom_command(OUTPUT ${outfile}
       COMMAND ${cuda_compiler} ${bc_flags}
         ${cuda_flags} ${MAX_SM_DEFINITION} ${infile} -o ${outfile}
       DEPENDS ${infile}
       IMPLICIT_DEPENDS CXX ${infile}
       COMMENT "Building LLVM bitcode ${outfile}"
       VERBATIM
     )
+    if("${cuda_compiler}" STREQUAL "$<TARGET_FILE:clang>")
+      # Add a file-level dependency to ensure that clang is up-to-date.
+      # By default, add_custom_command only builds clang if the
+      # executable is missing.
+      add_custom_command(OUTPUT ${outfile}
+        DEPENDS clang
+        APPEND
+      )
+    endif()
     set_property(DIRECTORY APPEND PROPERTY ADDITIONAL_MAKE_CLEAN_FILES ${outfile})
 
     list(APPEND bc_files ${outfile})
   endforeach()
 
   set(bclib_name "libomptarget-new-nvptx-sm_${sm}.bc")
 
   # Link to a bitcode library.
   add_custom_command(OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/${bclib_name}
       COMMAND ${bc_linker}
         -o ${CMAKE_CURRENT_BINARY_DIR}/${bclib_name} ${bc_files}
       DEPENDS ${bc_files}
       COMMENT "Linking LLVM bitcode ${bclib_name}"
   )
+  if("${bc_linker}" STREQUAL "$<TARGET_FILE:llvm-link>")
+    # Add a file-level dependency to ensure that llvm-link is up-to-date.
+    # By default, add_custom_command only builds llvm-link if the
+    # executable is missing.
+    add_custom_command(OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/${bclib_name}
+      DEPENDS llvm-link
+      APPEND
+    )
+  endif()
 
   add_custom_command(OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/${bclib_name}_opt
       COMMAND ${opt} ${link_opt_flags} ${CMAKE_CURRENT_BINARY_DIR}/${bclib_name}
                       -o ${CMAKE_CURRENT_BINARY_DIR}/${bclib_name}
       DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/${bclib_name}
       COMMENT "Optimizing LLVM bitcode ${bclib_name}"
   )
+  if("${opt}" STREQUAL "$<TARGET_FILE:opt>")
+    # Add a file-level dependency to ensure that opt is up-to-date.
+    # By default, add_custom_command only builds opt if the
+    # executable is missing.
+    add_custom_command(OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/${bclib_name}_opt
+      DEPENDS opt
+      APPEND
+    )
+  endif()
   set_property(DIRECTORY APPEND PROPERTY ADDITIONAL_MAKE_CLEAN_FILES ${bclib_name})
 
   set(bclib_target_name "omptarget-new-nvptx-sm_${sm}-bc")
 
   add_custom_target(${bclib_target_name} ALL DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/${bclib_name}_opt)
   add_dependencies(omptarget-new-nvptx-bc ${bclib_target_name})
   add_dependencies(${bclib_target_name} opt llvm-link)
 
   # Copy library to destination.
   add_custom_command(TARGET ${bclib_target_name} POST_BUILD
                     COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_CURRENT_BINARY_DIR}/${bclib_name}
                     ${LIBOMPTARGET_LIBRARY_DIR})
 
   # Install bitcode library under the lib destination folder.
   install(FILES ${CMAKE_CURRENT_BINARY_DIR}/${bclib_name} DESTINATION "${OPENMP_INSTALL_LIBDIR}")
 endforeach()