[OpenMP][libomptarget] Enable the compilation of multiple bc libraries for runtime inlining

[gtbercea]

Summary

Different NVIDIA GPUs support different compute capabilities. To enable the inlining of runtime functions and the best performance on different generations of NVIDIA GPUs, a bc library for each compute capability needs to be compiled. The same compiler build will then be usable in conjunction with multiple generations of NVIDIA GPUs.
To differentiate between versions of the same bc lib, the output file name will contain the compute capability ID.
Depends on D14254

[Hahnfeld]

Does the bitcode lib work with upstream Clang trunk?

[Hahnfeld]

Please rebase on top of the lastest changes in D14254.

Do we want to rename LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITY to LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES (plural) to reflect that the user can pass in a list? In any case, please document in README.rst!

[gtbercea]

Rebase on new master.

[Hahnfeld]

libomptarget/deviceRTLs/nvptx/CMakeLists.txt

63–64

This should definitely stay a CACHE variable. And I still propose to rename it to _CAPABILITIES...

203

Do we want to use eg 35 as suffix or full sm_35?

[gtbercea]

libomptarget/deviceRTLs/nvptx/CMakeLists.txt

63–64

Ah so you want me to keep this line. Sure I'll reinstate it.

Regarding the name ... this would lead to many people having to change their compiler build scripts. I'd like to avoid that.

203

35

[gtbercea]

libomptarget/deviceRTLs/nvptx/CMakeLists.txt

203

Actually I don't care which one we use. Depending on what we choose here I'll make clang agree with it :)

[gtbercea]

Reinstate cache variable.

[Hahnfeld]

On whatever name we decide, please document in README.rst that the user can pass multiple values.

libomptarget/deviceRTLs/nvptx/CMakeLists.txt

63–64

Maybe we can add some compatibility? Upstream only has it since a few days and others will need to change their scripts anyway.

set(default_capabilities 35)
if (DEFINED LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITY)
  set(default_capabilities ${LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITY})
  libomptarget_warning_say("LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITY is deprecated, please use LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES")
endif()
set(LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES ${default_capabilities} CACHE STRING
  "List of CUDA Compute Capabilities to be used to compile the NVPTX device RTL.")

203

I agree: The user will probably never see the name anyway. So let's just keep it compatible.

[gtbercea]

Fix documentation.

[gtbercea]

Fix documentation.

[gtbercea]

Good to go @Hahnfeld ?

[gtbercea]

Forgot to add @grokos as a reviewer. Fixed now.

[grokos]

README.rst

282–286

Maybe add a comment here that the single-capability option is deprecated and the user should use the next option to define compute capabilities (even if they want to define only one)?

[gtbercea]

Add deprecated message to documentation.

[Hahnfeld]

README.rst

282–286

I think we should just remove it: There was no (upstream) release with this variable and the only code checking for it is there for compatibility.

[gtbercea]

Remove LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITY

[gtbercea]

ping @Hahnfeld

[Hahnfeld]

Looks good, thanks.

In D41724#1003880, @gtbercea wrote:

ping @Hahnfeld

Generally you can't expect me to reply immediately...

[gtbercea]

Change name to include full compute capability name not just the number.

README.rst

**LIBOMPTARGET_NVPTX_BC_LINKER** = ``""``
   directory containing Clang, then this llvm-link binary is used.
 
 **LIBOMPTARGET_NVPTX_ALTERNATE_HOST_COMPILER** = ``""``
   Host compiler to use with NVCC. This compiler is not going to be used to
   produce any binary. Instead, this is used to overcome the input compiler
   checks done by NVCC. E.g. if using a default host compiler that is not
   compatible with NVCC, this option can be use to pass to NVCC a valid compiler
   to avoid the error.
 
- **LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITY** = ``35``
-  CUDA compute capability that should be supported by the NVPTX device RTL. E.g.
-  for compute capability 6.0, the option "60" should be used. Compute capability
-  3.5 is the minimum required.
+ **LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES** = ``35``
+  List of CUDA compute capabilities that should be supported by the NVPTX
+  device RTL. E.g. for compute capabilities 6.0 and 7.0, the option "60,70"
+  should be used. Compute capability 3.5 is the minimum required.
 
  **LIBOMPTARGET_NVPTX_DEBUG** = ``OFF|ON``
   Enable printing of debug messages from the NVPTX device RTL.
 
 Example Usages of CMake
 =======================
 
 Typical Invocations

libomptarget/deviceRTLs/nvptx/CMakeLists.txt

set(cuda_src_files
       src/sync.cu
       src/task.cu
   )
 
   set(omp_data_objects src/omp_data.cu)
 
   # Get the compute capability the user requested or use SM_35 by default.
   # SM_35 is what clang uses by default.
-  set(LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITY 35 CACHE STRING
-    "CUDA Compute Capability to be used to compile the NVPTX device RTL.")
-  set(CUDA_ARCH -arch sm_${LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITY})
+  set(default_capabilities 35)
+  if (DEFINED LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITY)
+    set(default_capabilities ${LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITY})
+    libomptarget_warning_say("LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITY is deprecated, please use LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES")
+  endif()
+  set(LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES ${default_capabilities} CACHE STRING
+    "List of CUDA Compute Capabilities to be used to compile the NVPTX device RTL.")
+  string(REPLACE "," ";" nvptx_sm_list ${LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES})
+
+  foreach(sm ${nvptx_sm_list})
+    set(CUDA_ARCH ${CUDA_ARCH} -gencode arch=compute_${sm},code=sm_${sm})
+  endforeach()
 
   # Activate RTL message dumps if requested by the user.
   set(LIBOMPTARGET_NVPTX_DEBUG FALSE CACHE BOOL
     "Activate NVPTX device RTL debug messages.")
   if(${LIBOMPTARGET_NVPTX_DEBUG})
     set(CUDA_DEBUG -DOMPTARGET_NVPTX_DEBUG=-1 -g --ptxas-options=-v)
   endif()
 
if(LIBOMPTARGET_NVPTX_SELECTED_CUDA_COMPILER AND LIBOMPTARGET_NVPTX_SELECTED_BC_LINKER)
       # to handle. Therefore, we use 'weak' instead. We are compiling only for the
       # device, so it should be equivalent.
       if(CUDA_VERSION_MAJOR EQUAL 9)
         set(CUDA_FLAGS ${CUDA_FLAGS} -Dnv_weak=weak)
       endif()
 
       # Get the compute capability the user requested or use SM_35 by default.
       set(CUDA_ARCH "")
-      set(CUDA_ARCH --cuda-gpu-arch=sm_${LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITY})
+      foreach(sm ${nvptx_sm_list})
+        set(CUDA_ARCH --cuda-gpu-arch=sm_${sm})
 
         # Compile cuda files to bitcode.
         set(bc_files "")
         foreach(src ${cuda_src_files})
           get_filename_component(infile ${src} ABSOLUTE)
           get_filename_component(outfile ${src} NAME)
 
-        add_custom_command(OUTPUT ${outfile}.bc
+          add_custom_command(OUTPUT ${outfile}-${sm}.bc
             COMMAND ${LIBOMPTARGET_NVPTX_SELECTED_CUDA_COMPILER} ${CUDA_FLAGS} ${CUDA_ARCH} ${CUDA_INCLUDES}
-            -c ${infile} -o ${outfile}.bc
+              -c ${infile} -o ${outfile}-${sm}.bc
             DEPENDS ${infile}
             IMPLICIT_DEPENDS CXX ${infile}
-          COMMENT "Building LLVM bitcode ${outfile}.bc"
+            COMMENT "Building LLVM bitcode ${outfile}-${sm}.bc"
             VERBATIM
           )
-        set_property(DIRECTORY APPEND PROPERTY ADDITIONAL_MAKE_CLEAN_FILES ${outfile}.bc)
+          set_property(DIRECTORY APPEND PROPERTY ADDITIONAL_MAKE_CLEAN_FILES ${outfile}-${sm}.bc)
 
-        list(APPEND bc_files ${outfile}.bc)
+          list(APPEND bc_files ${outfile}-${sm}.bc)
         endforeach()
 
         # Link to a bitcode library.
-      add_custom_command(OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/libomptarget-nvptx.bc
+        add_custom_command(OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/libomptarget-nvptx-${sm}.bc
             COMMAND ${LIBOMPTARGET_NVPTX_SELECTED_BC_LINKER}
-            -o ${CMAKE_CURRENT_BINARY_DIR}/libomptarget-nvptx.bc ${bc_files}
+              -o ${CMAKE_CURRENT_BINARY_DIR}/libomptarget-nvptx-${sm}.bc ${bc_files}
             DEPENDS ${bc_files}
-          COMMENT "Linking LLVM bitcode libomptarget-nvptx.bc"
+            COMMENT "Linking LLVM bitcode libomptarget-nvptx-${sm}.bc"
         )
-      set_property(DIRECTORY APPEND PROPERTY ADDITIONAL_MAKE_CLEAN_FILES libomptarget-nvptx.bc)
+        set_property(DIRECTORY APPEND PROPERTY ADDITIONAL_MAKE_CLEAN_FILES libomptarget-nvptx-${sm}.bc)
 
-      add_custom_target(omptarget-nvptx-bc ALL DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/libomptarget-nvptx.bc)
+        add_custom_target(omptarget-nvptx-${sm}-bc ALL DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/libomptarget-nvptx-${sm}.bc)
 
         # Copy library to destination.
-      add_custom_command(TARGET omptarget-nvptx-bc POST_BUILD
-                         COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_CURRENT_BINARY_DIR}/libomptarget-nvptx.bc
+        add_custom_command(TARGET omptarget-nvptx-${sm}-bc POST_BUILD
+                           COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_CURRENT_BINARY_DIR}/libomptarget-nvptx-${sm}.bc
                            $<TARGET_FILE_DIR:omptarget-nvptx>)
 
         # Install device RTL under the lib destination folder.
-      install(FILES ${CMAKE_CURRENT_BINARY_DIR}/libomptarget-nvptx.bc DESTINATION "lib")
-
+        install(FILES ${CMAKE_CURRENT_BINARY_DIR}/libomptarget-nvptx-${sm}.bc DESTINATION "lib")
+      endforeach()
     endif()
   endif()
 
 else()
   libomptarget_say("Not building CUDA offloading device RTL: CUDA tools not found in the system.")
 endif()