[libomptarget] Change nvcc compilation to use a unity build

[JonChesterfield]

Summary

[libomptarget] Change nvcc compilation to use a unity build

This allows nvcc to inline functions between what would otherwise be distinct
translation units, which in turn removes any runtime cost from implementing
functions in source files (as opposed to inline in headers).

This will then allow the circular dependencies in deviceRTL to be readily
broken and individual components more easily shared between architectures.

[JonChesterfield]

My test coverage for nvptx is quite poor so I'd be interested if this breaks downstream tests.

[jdoerfert]

Given that the build model I will advocate includes linking the device RTL into the application (as IR), I don't see missing incremental builds as a real drawback.

I think we should compile unity.cu also when we create a bitcode RTL. (This will simplify the cmake file, remove the need for llvm-link, ...)

openmp/libomptarget/deviceRTLs/nvptx/CMakeLists.txt

60

Why is this still needed, or put differently, why do we not compile the unity.cu into bitcode where this variable (cuda_src_files) is still used below?

62

same as above (I think)

[JonChesterfield]

My understanding of this cmake is that nvptx is built as both a static archive and as a llvm-link'ed bitcode archive. The former suggests a toolchain that may not be capable of LTO, the latter suggests a toolchain that definitely is. When llvm-link is available, so is opt.

I'd like to compile the translation units separately when we can. Incremental builds don't matter hugely as the build time is negligible. However separate compilation means we don't pick up spurious relationships between source files. E.g. if data_sharing.cu adds a static function that gets called from loop.cu, the concatenated build will work just fine but a standalone one wouldn't. It also means headers must be present in all compilation units that use them, instead of included from at least one unit that gets #included earlier.

Strongly in agreement with your build model of linking the deviceRTL with the application code. I see the header/source/library boundaries as useful for developers and necessary to erase at compile time.

[jdoerfert]

I see. I think we can keep both building modes for now. LGTM.

[JonChesterfield]

Cool. I view this change as pretty low risk - we can move to always using the unity build or back to separate compilation easily.

openmp/libomptarget/deviceRTLs/nvptx/CMakeLists.txt

set(cuda_src_files
       src/data_sharing.cu
       src/libcall.cu
       src/loop.cu
       src/omptarget-nvptx.cu
       src/parallel.cu
       src/reduction.cu
       src/sync.cu
       src/task.cu
   )
 
   set(omp_data_objects src/omp_data.cu)
 
   # Get the compute capability the user requested or use SM_35 by default.
   # SM_35 is what clang uses by default.
   set(default_capabilities 35)
   if (DEFINED LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITY)
     set(default_capabilities ${LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITY})
     libomptarget_warning_say("LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITY is deprecated, please use LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES")
   endif()
if(${LIBOMPTARGET_NVPTX_DEBUG})
     set(CUDA_DEBUG -DOMPTARGET_NVPTX_DEBUG=-1 -g --ptxas-options=-v)
   endif()
 
   # NVPTX runtime library has to be statically linked. Dynamic linking is not
   # yet supported by the CUDA toolchain on the device.
   set(BUILD_SHARED_LIBS OFF)
   set(CUDA_SEPARABLE_COMPILATION ON)
   list(APPEND CUDA_NVCC_FLAGS -I${devicertl_base_directory})
-  cuda_add_library(omptarget-nvptx STATIC ${cuda_src_files} ${omp_data_objects}
+  cuda_add_library(omptarget-nvptx STATIC unity.cu
       OPTIONS ${CUDA_ARCH} ${CUDA_DEBUG})
 
   # Install device RTL under the lib destination folder.
   install(TARGETS omptarget-nvptx ARCHIVE DESTINATION "${OPENMP_INSTALL_LIBDIR}")
 
   target_link_libraries(omptarget-nvptx ${CUDA_LIBRARIES})
 
 

openmp/libomptarget/deviceRTLs/nvptx/unity.cu

+//===------ unity.cu - Unity build of NVPTX deviceRTL ------------ CUDA -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Support compilers, specifically NVCC, which have not implemented link time
+// optimisation. This removes the runtime cost of moving inline functions into
+// source files in exchange for preventing efficient incremental builds.
+//
+//===----------------------------------------------------------------------===//
+
+#include "src/cancel.cu"
+#include "src/critical.cu"
+#include "src/data_sharing.cu"
+#include "src/libcall.cu"
+#include "src/loop.cu"
+#include "src/omp_data.cu"
+#include "src/omptarget-nvptx.cu"
+#include "src/parallel.cu"
+#include "src/reduction.cu"
+#include "src/sync.cu"
+#include "src/task.cu"