[OpenMP][CMake] Use in-project clang as CUDA->IR compiler.

[Meinersbur]

Summary

If available, use the clang that is already built in the same project as CUDA compiler unless another executable is explicitly defined. This also ensures the generated deviceRTL IR will be consistent with the version of Clang.

The change in add_subdirectory order is required to ensure that if clang is part of the project build, its target exists before openmp is included. Alternatively, LLVM_TOOL_CLANG_BUILD can could be used to determine whether clang build is enabled (Not sure how reliable it is).

This patch is required to reliably test OpenMP offloading in a buildbot without either a two-stage build (e.g. with LLVM_ENABLE_RUNTIMES) or a separately installed clang on the worker that will eventually become outdated.

See the current builder and the build with this patch applied.

[tianshilei1992]

I didn't get in what scenario we need this patch. Could you please expatiate it?

[Meinersbur]

• Add context
• Update README

[Meinersbur]

In D101265#2715632, @tianshilei1992 wrote:

I didn't get in what scenario we need this patch. Could you please expatiate it?

Not having to provide another sufficiently recent clang/llvm-link binary to pass with -DLIBOMPTARGET_NVPTX_CUDA_COMPILER/-DLIBOMPTARGET_NVPTX_BC_LINKER.

D95466 seems to have removed the checks for a working Clang at configure-time. If it was still there, this patch would have disabled them since the Clang in the repository with OpenMP would be the most recent and known to support all required features. This also makes it ideal default if available.

[tianshilei1992]

In D101265#2715702, @Meinersbur wrote:

In D101265#2715632, @tianshilei1992 wrote:

I didn't get in what scenario we need this patch. Could you please expatiate it?

Not having to provide another sufficiently recent clang/llvm-link binary to pass with -DLIBOMPTARGET_NVPTX_CUDA_COMPILER/-DLIBOMPTARGET_NVPTX_BC_LINKER.

If OpenMP is built via LLVM_ENABLE_RUNTIMES, the two variables you mentioned here are not needed. If OpenMP is built via LLVM_ENABLE_PROJECTS, it is *intentional* to build OpenMP w/o using the clang in recent build.

D95466 seems to have removed the checks for a working Clang at configure-time. If it was still there, this patch would have disabled them since the Clang in the repository with OpenMP would be the most recent and known to support all required features. This also makes it ideal default if available.

The code removed only checked whether a compiler can compile CUDA code. Since we already move to OpenMP style device runtime, it is not needed anymore. The problem here is, we have a requirement of minimum version of clang to work, and we lack that check.

[Meinersbur]

In D101265#2715714, @tianshilei1992 wrote:

If OpenMP is built via LLVM_ENABLE_PROJECTS, it is *intentional* to build OpenMP w/o using the clang in recent build.

Is this documented somewhere?

Would you require the buildbot to do a stage1 build first for a LLVM_ENABLE_PROJECTS=openmp build?
I don't see a reason for a stage1 build given that using the in-project clang is as simple. OPENMP_TEST_C_COMPILER/OPENMP_TEST_CXX_COMPILER uses it as well by default.

[tianshilei1992]

In D101265#2715749, @Meinersbur wrote:

In D101265#2715714, @tianshilei1992 wrote:

If OpenMP is built via LLVM_ENABLE_PROJECTS, it is *intentional* to build OpenMP w/o using the clang in recent build.

Is this documented somewhere?

You could refer to https://llvm.org/docs/CMake.html for LLVM_ENABLE_PROJECTS and https://llvm.org/docs/BuildingADistribution.html for LLVM_ENABLE_RUNTIMES.

Would you require the buildbot to do a stage1 build first for a LLVM_ENABLE_PROJECTS=openmp build?

It depends. If you want offloading feature, then it requires. If you don't want that, it doesn't. I suppose you are building LLVM with GCC. If you're building LLVM with LLVM, offloading support will also be enabled. The idea here is, the OpenMP is built using the same compiler to build LLVM. Basically the OpenMP project will be built in a same way as others. For example, for LLVM_ENABLE_PROJECTS=clang, we don't expect to build clang with the "recent built" clang, right? If you need it to be built with the recent build Clang, then LLVM_ENABLE_RUNTIMES is for that purpose.

I don't see a reason for a stage1 build given that using the in-project clang is as simple. OPENMP_TEST_C_COMPILER/OPENMP_TEST_CXX_COMPILER uses it as well by default.

Why not go with LLVM_ENABLE_RUNTIMES? It will build everything needed "all at once" (internally it is not. It first builds all parts except projects in LLVM_ENABLE_RUNTIMES, and then automatically invokes CMake configuration of those projects, set corresponding environment variables, and starts the build).

However, another potential direction can be, if we find OpenMP is in LLVM_ENABLE_PROJECTS, we "move" it to LLVM_ENABLE_RUNTIMES. This could arguably make more sense.

[Meinersbur]

In D101265#2715759, @tianshilei1992 wrote:

In D101265#2715749, @Meinersbur wrote:

In D101265#2715714, @tianshilei1992 wrote:

If OpenMP is built via LLVM_ENABLE_PROJECTS, it is *intentional* to build OpenMP w/o using the clang in recent build.

Is this documented somewhere?

You could refer to https://llvm.org/docs/CMake.html for LLVM_ENABLE_PROJECTS and https://llvm.org/docs/BuildingADistribution.html for LLVM_ENABLE_RUNTIMES.

The first link does not mention what is intended to build with what compiler.
The second link is for creating a distributable package, which does not apply here.

Would you require the buildbot to do a stage1 build first for a LLVM_ENABLE_PROJECTS=openmp build?

It depends. If you want offloading feature, then it requires. If you don't want that, it doesn't.

It is not required, it works without a stage1 build and this patch.

For example, for LLVM_ENABLE_PROJECTS=clang, we don't expect to build clang with the "recent built" clang, right?

We expect .td files to be built with recent built tablegen.

If you need it to be built with the recent build Clang, then LLVM_ENABLE_RUNTIMES is for that purpose.
Why not go with LLVM_ENABLE_RUNTIMES?

That is another configuration: http://meinersbur.de:8011/#/builders/143 (The LLVM_ENABLE_PROJECT configuration is http://meinersbur.de:8011/#/builders/142)

I would like to test both.

There are valid use cases for using a LLVM_ENABLE_PROJECT configuration. One is that it is a single build dir for using in an IDE or generating a single CMAKE_EXPORT_COMPILE_COMMANDS for use with tools such as clangd. Ninja can also better track dependencies. libomp(target?) is also used by non-Clang compilers such as icc and msvc.

I suppose you are building LLVM with GCC. If you're building LLVM with LLVM, offloading support will also be enabled. The idea here is, the OpenMP is built using the same compiler to build LLVM. Basically the OpenMP project will be built in a same way as others.

The OpenMP host code, but CMake does not configure a device RTL compiler. I would not expect it to use the host compiler for cross-compiling to the GPU. That's what a cross-compiler is for.

Another difference is while for assembly we have a well-defined ABI that ensures that outputs from different compilers are compatible, mixing BC files from different versions of LLVM might not be such a good idea. Sure, there is the AutoUpgrader, but it's probably one of the least tested components in LLVM. Then there are also future versions or third-party forks of clang who's BC files might not at all compatible with our clang, e.g. Apple's clang.

LLVM_ENABLE_RUNTIMES also seems badly documented. When it was first introduced, it did not work for a couple of years and I only found out relatively recently that it is supposed to work. And I am not the only one. A newcomer will first try LLVM_ENABLE_PROJECT=clang;openmp and be frustrated to see that it will not work with a cryptic error message

No library 'libomptarget-nvptx-sm_61.bc' found in the default clang lib directory or in LIBRARY_PATH. Please use --libomptarget-nvptx-bc-path to specify nvptx bitcode library.

These problems can all be avoided by using the "recent built" clang as a sensible default. You can still use another by explicitly setting LIBOMPTARGET_NVPTX_CUDA_COMPILER.

[jdoerfert]

openmp/libomptarget/deviceRTLs/nvptx/CMakeLists.txt

43

LLVM_ENABLE_RUNTIMES=openmp should itself set LIBOMPTARGET_NVPTX_CUDA_COMPILER instead.

That seems sensible. And then we could emit a warning/note here and above.

[tianshilei1992]

In D101265#2718317, @Meinersbur wrote:

In D101265#2715759, @tianshilei1992 wrote:

In D101265#2715749, @Meinersbur wrote:

In D101265#2715714, @tianshilei1992 wrote:

If OpenMP is built via LLVM_ENABLE_PROJECTS, it is *intentional* to build OpenMP w/o using the clang in recent build.

Is this documented somewhere?

You could refer to https://llvm.org/docs/CMake.html for LLVM_ENABLE_PROJECTS and https://llvm.org/docs/BuildingADistribution.html for LLVM_ENABLE_RUNTIMES.

The first link does not mention what is intended to build with what compiler.
The second link is for creating a distributable package, which does not apply here.

The two links show the difference between LLVM_ENABLE_RUNTIMES and LLVM_ENABLE_PROJECTS. It does mention that projects in LLVM_ENABLE_RUNTIMES are built by the recent built clang.

Would you require the buildbot to do a stage1 build first for a LLVM_ENABLE_PROJECTS=openmp build?

It depends. If you want offloading feature, then it requires. If you don't want that, it doesn't.

It is not required, it works without a stage1 build and this patch.

If you need offloading feature, it is required because OpenMP offloading features requires clang to be the compiler. Yes, libomptarget.so will be generated, but no deviceRTLs will be generated so the offloading is still unusable.

For example, for LLVM_ENABLE_PROJECTS=clang, we don't expect to build clang with the "recent built" clang, right?

We expect .td files to be built with recent built tablegen.

It's chicken egg problem. Let's say we have LLVM_ENABLE_PROJECTS=clang;openmp, and we are using GCC to build the whole LLVM. Let's call the clang generated here "the clang". If you expect OpenMP to be built by "the clang", what do you expect to build "the clang"? If GCC, why is OpenMP built by "the clang"? If "the clang", where is "the clang" at the first place? And even if we only have`LLVM_ENABLE_PROJECTS=openmp, why is OpenMP in this case built by GCC?

From my perspective, to keep the semantics consistent is important, and that's the only reason that I think all projects in LLVM_ENABLE_PROJECTS should be built by the same compiler as the one to build LLVM.

There are valid use cases for using a LLVM_ENABLE_PROJECT configuration. One is that it is a single build dir for using in an IDE or generating a single CMAKE_EXPORT_COMPILE_COMMANDS for use with tools such as clangd. Ninja can also better track dependencies. libomp(target?) is also used by non-Clang compilers such as icc and msvc.

I don't doubt that. To have LLVM_ENABLE_PROJECTS for IDE is totally fine, and I think even now w/o your patch, libomp and libomptarget can work with clangd except those plugins and deviceRTLs. Plugins and deviceRTLs will still be missing in the compilation database even with your patch if I understand correctly.

I suppose you are building LLVM with GCC. If you're building LLVM with LLVM, offloading support will also be enabled. The idea here is, the OpenMP is built using the same compiler to build LLVM. Basically the OpenMP project will be built in a same way as others.

The OpenMP host code, but CMake does not configure a device RTL compiler. I would not expect it to use the host compiler for cross-compiling to the GPU. That's what a cross-compiler is for.

Sure, you can, but by default it will use what CMake detects. If you don't want it, just specify the one.

Another difference is while for assembly we have a well-defined ABI that ensures that outputs from different compilers are compatible, mixing BC files from different versions of LLVM might not be such a good idea. Sure, there is the AutoUpgrader, but it's probably one of the least tested components in LLVM. Then there are also future versions or third-party forks of clang who's BC files might not at all compatible with our clang, e.g. Apple's clang.

That's unrelated to the problem we discuss here. The key point is, you propose to let LLVM_ENABLE_PROJECTS behave same as LLVM_ENABLE_RUNTIMES, and solely for OpenMP. If so, why do we even have this two arguments at the first place? Or let me put it in this way, is there any other project (such as lld) that when it is in LLVM_ENABLE_PROJECTS along with clang, it is actually built by the clang just built? If yes, I'll be totally fine with your change.

And BTW, using LLVM_ENABLE_RUNTIMES can already avoid all problems you mentioned here.

LLVM_ENABLE_RUNTIMES also seems badly documented. When it was first introduced, it did not work for a couple of years and I only found out relatively recently that it is supposed to work. And I am not the only one. A newcomer will first try LLVM_ENABLE_PROJECT=clang;openmp and be frustrated to see that it will not work with a cryptic error message

No library 'libomptarget-nvptx-sm_61.bc' found in the default clang lib directory or in LIBRARY_PATH. Please use --libomptarget-nvptx-bc-path to specify nvptx bitcode library.

These problems can all be avoided by using the "recent built" clang as a sensible default. You can still use another by explicitly setting LIBOMPTARGET_NVPTX_CUDA_COMPILER.

I agree with you that LLVM_ENABLE_RUNTIMES was broken before, but now it works, and it is in OpenMP Q&A (https://openmp.llvm.org/docs/SupportAndFAQ.html).

FWIW, the only problem for LLVM_ENABLE_RUNTIMES now is, if we run check-all, and if any test case in libomptarget fails, other checks will not be run. Since currently offloading x86-64 is broken, test cases for libomptarget never all pass, but that's another story and we need to fix the lit for libomptarget.

[protze.joachim]

llvm/CMakeLists.txt

971 ↗

(On Diff #340408)

Are you sure that this doesn't break cmake dependencies for compiler-rt?

openmp/libomptarget/deviceRTLs/nvptx/CMakeLists.txt

35

I think, you want to have:

set(cuda_compiler $<TARGET_FILE:clang>)

[protze.joachim]

In D101265#2724364, @tianshilei1992 wrote:

In D101265#2718317, @Meinersbur wrote:

In D101265#2715759, @tianshilei1992 wrote:

In D101265#2715749, @Meinersbur wrote:

In D101265#2715714, @tianshilei1992 wrote:

If OpenMP is built via LLVM_ENABLE_PROJECTS, it is *intentional* to build OpenMP w/o using the clang in recent build.

Is this documented somewhere?

You could refer to https://llvm.org/docs/CMake.html for LLVM_ENABLE_PROJECTS and https://llvm.org/docs/BuildingADistribution.html for LLVM_ENABLE_RUNTIMES.

The first link does not mention what is intended to build with what compiler.
The second link is for creating a distributable package, which does not apply here.

The two links show the difference between LLVM_ENABLE_RUNTIMES and LLVM_ENABLE_PROJECTS. It does mention that projects in LLVM_ENABLE_RUNTIMES are built by the recent built clang.

I think, what Michael really meant with "Is this documented somewhere?":
Is there user documentation, that LLVM_ENABLE_PROJECTS=openmp was bricked by llvm-12 and you shouldn't expect libomptarget to work properly unless moving to LLVM_ENABLE_RUNTIMES=openmp or doing two-stage build? Since this is crucial information, this should be BOLD in the OpenMP release notes.

Building LLVM_ENABLE_PROJECTS=openmp with the previous clang release used to work fine with single-stage build.

[Meinersbur]

In D101265#2724364, @tianshilei1992 wrote:

That's unrelated to the problem we discuss here. The key point is, you propose to let LLVM_ENABLE_PROJECTS behave same as LLVM_ENABLE_RUNTIMES, and solely for OpenMP. If so, why do we even have this two arguments at the first place? Or let me put it in this way, is there any other project (such as lld) that when it is in LLVM_ENABLE_PROJECTS along with clang, it is actually built by the clang just built? If yes, I'll be totally fine with your change.

OK, there seems to be a misconception. With this patch LLVM_ENABLE_PROJECTS and LLVM_ENABLE_RUNTIMES will still be different.
LLVM_ENABLE_PROJECTS will continue to compile all host code (libomp.so, libomptarget.so) with the compiler selected by CMake.
This only difference is that by default and if available, the LLVM_ENABLE_PROJECTS configuration also uses the just-built clang to cross-compile the deviceRTL to LLVM-IR. The host-compiler selected by CMake is for compiling to host-compatible assembly and unsuitable for cross-compilation for generation of LLVM bitcode.

Or let me put it in this way, is there any other project (such as lld) that when it is in LLVM_ENABLE_PROJECTS along with clang, it is actually built by the clang just built? If yes, I'll be totally fine with your change.

1. Thread-sanitizer (compiler-rt) compiles libcxx with $<TARGET_FILE:clang> to get a thread-sanitized C++ standard library.
2. libtooling uses just-built clang for AST introspection
3. Polly uses just-built clang-tidy to check whether its correctly formatted.
4. libc uses just-built clang-tidy for some linting.

llvm/CMakeLists.txt

971 ↗

(On Diff #340408)

I am not (for compiler-rt or other uses such as those mentioned in the response); due to its global consequences, this change might indeed be risky. Logically, compiler-rt itself is also a runtime and hence should logically be ordered the same way. I would check this further before committing to verify.

In the summary I mentioned an alternative that would work without this change (checking LLVM_TOOL_CLANG_BUILD instead, like Polly).

openmp/libomptarget/deviceRTLs/nvptx/CMakeLists.txt

35

Having it named clang will make CMake assume it is the clang target, and add dependency to the target and replace clang with that path of the output executable.
See https://cmake.org/cmake/help/latest/command/add_custom_target.html

If COMMAND specifies an executable target name (created by the add_executable() command), it will automatically be replaced by the location of the executable created at build time

With specifying a full path, CMake may not add the dependency to the clang target anymore (maybe it does, haven't checked) and would have to be defined manually.

However:

This target-level dependency does NOT add a file-level dependency that would cause the custom command to re-run whenever the executable is recompiled. List target names with the DEPENDS option to add such file-level dependencies.

Maybe we do want such a dependency to guarantee that the bitcode is always the latest
(But then it seems to be illogical that we allow this with pre-compiled clangs via LIBOMPTARGET_NVPTX_CUDA_COMPILER).

[Meinersbur]

• Rebase
• Fix typos, remove risky change, fix cross-compiling

[protze.joachim]

openmp/libomptarget/deviceRTLs/nvptx/CMakeLists.txt

35

Thanks for the clarification regarding the add_custom_command behavior

The only case I see where this really matters is, when you recompile in an existing build dir after switching branches.
In that case it makes sense to recompile the BC files, if the compiler is newer than the existing file.
So, I'd add cuda_compiler as an explicit dependency in the custom target below.

[tianshilei1992]

In D101265#2727998, @Meinersbur wrote:

In D101265#2724364, @tianshilei1992 wrote:

Or let me put it in this way, is there any other project (such as lld) that when it is in LLVM_ENABLE_PROJECTS along with clang, it is actually built by the clang just built? If yes, I'll be totally fine with your change.

1. Thread-sanitizer (compiler-rt) compiles libcxx with $<TARGET_FILE:clang> to get a thread-sanitized C++ standard library.
2. libtooling uses just-built clang for AST introspection
3. Polly uses just-built clang-tidy to check whether its correctly formatted.
4. libc uses just-built clang-tidy for some linting.

Thanks for the information. Now I'm okay with the change. @protze.joachim 's comment sounds reasonable. LIBOMPTARGET_NVPTX_CUDA_COMPILER is a path to clang so it's different from the target clang. llvm-link as well. Others LGTM.

[Meinersbur]

• Add file-level dependency to ensure most recent clang ("Joachim's suggestion")
• Use $<TARGET_FILE:clang> for just-built clang. The reason is that otherwise if the user specifies LIBOMPTARGET_NVPTX_CUDA_COMPILER=clang (could mean the clang in $PATH; we cannot avoid that CMake rewrites it to in-tree clang), CMake would error out saying that a target "clang" does not exist when adding the dependency.
• Add output message to notify user which clang is used. To make it clear to users that it might not be the host compiler determined by CMake.

openmp/README.rst

**LIBOMPTARGET_NVPTX_ENABLE_BCLIB** = ``ON|OFF``
   Enable CUDA LLVM bitcode offloading device RTL. This is used for link time
   optimization of the OMP runtime and application code. This option is enabled
   by default if the build system determines that `CMAKE_C_COMPILER` is able to
   compile and link the library.
 
 **LIBOMPTARGET_NVPTX_CUDA_COMPILER** = ``""``
   Location of a CUDA compiler capable of emitting LLVM bitcode. Currently only
   the Clang compiler is supported. This is only used when building the CUDA LLVM
-  bitcode offloading device RTL. If unspecified and the CMake C compiler is
-  Clang, then Clang is used.
+  bitcode offloading device RTL. If unspecified, either the Clang from the build
+  itself is used (i.e. an in-tree build with LLVM_ENABLE_PROJECTS including
+  clang), or the Clang compiler that the build uses as C compiler
+  (CMAKE_C_COMPILER; only if it is Clang). The latter is common for a
+  stage2-build or when using -DLLVM_ENABLE_RUNTIMES=openmp.
 
 **LIBOMPTARGET_NVPTX_BC_LINKER** = ``""``
   Location of a linker capable of linking LLVM bitcode objects. This is only
-  used when building the CUDA LLVM bitcode offloading device RTL. If unspecified
-  and the CMake C compiler is Clang and there exists a llvm-link binary in the
-  directory containing Clang, then this llvm-link binary is used.
+  used when building the CUDA LLVM bitcode offloading device RTL. If
+  unspecified, either the llvm-link in that same directory as
+  LIBOMPTARGET_NVPTX_CUDA_COMPILER is used, or the llvm-link from the
+  same build (available in an in-tree build).
 
 **LIBOMPTARGET_NVPTX_ALTERNATE_HOST_COMPILER** = ``""``
   Host compiler to use with NVCC. This compiler is not going to be used to
   produce any binary. Instead, this is used to overcome the input compiler
   checks done by NVCC. E.g. if using a default host compiler that is not
   compatible with NVCC, this option can be use to pass to NVCC a valid compiler
   to avoid the error.
 

openmp/libomptarget/deviceRTLs/nvptx/CMakeLists.txt

 # an LLVM linker.
 set(LIBOMPTARGET_NVPTX_CUDA_COMPILER "" CACHE STRING
   "Location of a CUDA compiler capable of emitting LLVM bitcode.")
 set(LIBOMPTARGET_NVPTX_BC_LINKER "" CACHE STRING
   "Location of a linker capable of linking LLVM bitcode objects.")
 
 if (NOT LIBOMPTARGET_NVPTX_CUDA_COMPILER STREQUAL "")
   set(cuda_compiler ${LIBOMPTARGET_NVPTX_CUDA_COMPILER})
+elseif (LLVM_TOOL_CLANG_BUILD AND NOT CMAKE_CROSSCOMPILING)
+  # Compile the deviceRTL with the clang that is built in the project.
+  set(cuda_compiler "$<TARGET_FILE:clang>")
 elseif(${CMAKE_C_COMPILER_ID} STREQUAL "Clang")
+  # Compile the device runtime with the compiler that OpenMP is built with.
+  # This is the case with LLVM_ENABLE_RUNTIMES=openmp.
+  # FIXME: This is unreliable; the compiler can be on older version of clang
+  # that does not support compiling CUDA, or only an older version of it. The
+  # risk is especially high on sytems where clang is the default compiler
+  # (MacOS, BSDs). LLVM_ENABLE_RUNTIMES=openmp should itself set
+  # LIBOMPTARGET_NVPTX_CUDA_COMPILER instead.
   set(cuda_compiler ${CMAKE_C_COMPILER})
 else()
   libomptarget_say("Not building NVPTX deviceRTL: clang not found")
   return()
 endif()
 
 # Get compiler directory to try to locate a suitable linker.
 get_filename_component(compiler_dir ${cuda_compiler} DIRECTORY)
 set(llvm_link "${compiler_dir}/llvm-link")
 
 if (NOT LIBOMPTARGET_NVPTX_BC_LINKER STREQUAL "")
   set(bc_linker ${LIBOMPTARGET_NVPTX_BC_LINKER})
 elseif (EXISTS ${llvm_link})
+  # Try to use the linker consistent with the CUDA compiler unless explicitly
+  # set to a different linker.
   set(bc_linker ${llvm_link})
+elseif (NOT OPENMP_STANDALONE_BUILD AND NOT CMAKE_CROSSCOMPILING)
+  # Use the linker also built in the same project.
+  set(bc_linker "$<TARGET_FILE:llvm-link>")
 else()
   libomptarget_say("Not building NVPTX deviceRTL: llvm-link not found")
   return()
 endif()
 
 # TODO: This part needs to be refined when libomptarget is going to support
 # Windows!
 # TODO: This part can also be removed if we can change the clang driver to make

 if (DEFINED LIBOMPTARGET_NVPTX_MAX_SM)
   set(MAX_SM_DEFINITION "-DMAX_SM=${LIBOMPTARGET_NVPTX_MAX_SM}")
 endif()
 
 # Activate RTL message dumps if requested by the user.
 set(LIBOMPTARGET_NVPTX_DEBUG FALSE CACHE BOOL
   "Activate NVPTX device RTL debug messages.")
 
-libomptarget_say("Building CUDA LLVM bitcode offloading device RTL.")
+if ("${cuda_compiler}" STREQUAL "$<TARGET_FILE:clang>")
+  libomptarget_say("Building CUDA LLVM bitcode offloading device RTL using ${cuda_compiler}")
+else ()
+  libomptarget_say("Building CUDA LLVM bitcode offloading device RTL using in-tree clang.")
+endif ()
 
 set(cuda_src_files
   ${devicertl_common_directory}/src/cancel.cu
   ${devicertl_common_directory}/src/critical.cu
   ${devicertl_common_directory}/src/data_sharing.cu
   ${devicertl_common_directory}/src/libcall.cu
   ${devicertl_common_directory}/src/loop.cu
   ${devicertl_common_directory}/src/omp_data.cu
foreach(src ${cuda_src_files})
     add_custom_command(OUTPUT ${outfile}
       COMMAND ${cuda_compiler} ${bc_flags}
         ${cuda_flags} ${MAX_SM_DEFINITION} ${infile} -o ${outfile}
       DEPENDS ${infile}
       IMPLICIT_DEPENDS CXX ${infile}
       COMMENT "Building LLVM bitcode ${outfile}"
       VERBATIM
     )
+    if("${cuda_compiler}" STREQUAL "$<TARGET_FILE:clang>")
+      # Add a file-level dependency to ensure that clang is up-to-date.
+      # By default, add_custom_command only builds clang if the
+      # executable is missing.
+      add_custom_command(OUTPUT ${outfile}
+        DEPENDS clang
+        APPEND
+      )
+    endif()
     set_property(DIRECTORY APPEND PROPERTY ADDITIONAL_MAKE_CLEAN_FILES ${outfile})
 
     list(APPEND bc_files ${outfile})
   endforeach()
 
   set(bclib_name "libomptarget-nvptx-sm_${sm}.bc")
 
   # Link to a bitcode library.
   add_custom_command(OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/${bclib_name}
       COMMAND ${bc_linker}
         -o ${CMAKE_CURRENT_BINARY_DIR}/${bclib_name} ${bc_files}
       DEPENDS ${bc_files}
       COMMENT "Linking LLVM bitcode ${bclib_name}"
   )
+  if("${bc_linker}" STREQUAL "$<TARGET_FILE:llvm-link>")
+    # Add a file-level dependency to ensure that llvm-link is up-to-date.
+    # By default, add_custom_command only builds llvm-link if the
+    # executable is missing.
+    add_custom_command(OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/${bclib_name}
+      DEPENDS llvm-link
+      APPEND
+    )
+  endif()
   set_property(DIRECTORY APPEND PROPERTY ADDITIONAL_MAKE_CLEAN_FILES ${bclib_name})
 
   set(bclib_target_name "omptarget-nvptx-sm_${sm}-bc")
 
   add_custom_target(${bclib_target_name} ALL DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/${bclib_name})
   add_dependencies(omptarget-nvptx-bc ${bclib_target_name})
 
   # Copy library to destination.