[openmp][devicertl] Freestanding nvptx via stub printf

[JonChesterfield]

Summary

Compiled nvptx devicertl as freestanding, breaking the
dependency on host glibc and gcc-multilibs. Thus build it by default.

Comes at the cost of #defining out printf. Tried mapping it onto
__builtin_printf but that gets transformed back to printf instead
of hitting the cuda/openmp lowering transform.

Printf could be preserved by one of:

• dropping all the standard headers and ffreestanding
• providing a header only printf implementation
• changing the compiler handling of printf

[JonChesterfield]

openmp/libomptarget/deviceRTLs/nvptx/src/target_impl.h

83

If we accept this approach, can subsequently delete the code paths around the now-dead printf. Or can leave this here as a vestigial debugging hook.

[grokos]

openmp/libomptarget/deviceRTLs/nvptx/CMakeLists.txt

12–13

Change the comment, "by default we will build"

[JonChesterfield]

• update comment

[jdoerfert]

I'm fine with this, @grokos ?

[grokos]

LGTM too.

[JonChesterfield]

• rebase

[ye-luo]

How soon will I be able to use printf? I feel loosing printf is a huge negative impact.

[JonChesterfield]

Do you use printf in the deviceRTL? Whether it exists in applications is orthogonal to this

[ye-luo]

In D108349#2961093, @JonChesterfield wrote:

Do you use printf in the deviceRTL? Whether it exists in applications is orthogonal to this

I don't. Does this affect using printf in the user offload region? I need to be sure users still can call printf without hassle.
if now user code need to figure out glibc and gcc-multilibs dependency when printf is used in the offload region. Then I'm afraid it is worse than libomptarget having this dependency.

[jdoerfert]

FWIW, I don't think this will introduce problems. If I'm wrong, we need to identify and resolve them rather than relying on gcc-multilib or something similar.

[JonChesterfield]

In D108349#2961131, @ye-luo wrote:

I don't. Does this affect using printf in the user offload region?

No. Printf in user openmp code (provided they didn't compile their own code as freestanding) gets mapped onto cuda printf and behaves identically (iiuc that means no output until the target region returns)

In D108349#2961131, @ye-luo wrote:

I need to be sure users still can call printf without hassle.
if now user code need to figure out glibc and gcc-multilibs dependency when printf is used in the offload region. Then I'm afraid it is worse than libomptarget having this dependency.

Also orthogonal. Users can have a variety of problems using 32 bit nvptx on 64 bit host and the error messages do not make it obvious that they're missing gcc-multilibs. They also need to arrange for there to be cuda somewhere and for the toolchain to find it. That seems to be difficult, e.g. I couldn't get clang to find the debian packaged cuda. So I can't say with any confidence that people will have a hassle free experience using printf via nvptx openmp.

However, that capability has no interaction with this patch. This drops the printf debugging calls from the runtime itself so that the runtime can build without gcc-multilibs, much like a different patch dropped the requirement for cuda to build the host runtime. The end user still needs all the cuda stuff but whoever is building the toolchain, e.g. as part of a linux distribution, won't need it any more.

[ye-luo]

In D108349#2961139, @jdoerfert wrote:

FWIW, I don't think this will introduce problems. If I'm wrong, we need to identify and resolve them rather than relying on gcc-multilib or something similar.

Cool. I frequently use printf for debugging and will report if it becomes an issue after landing the patch.

[ye-luo]

@JonChesterfield thank you for the clarification.

[JonChesterfield]

In D108349#2961142, @ye-luo wrote:

Cool. I frequently use printf for debugging

Right there with you :(

In D108349#2961142, @ye-luo wrote:

and will report if it becomes an issue after landing the patch.

Thanks!

openmp/libomptarget/deviceRTLs/nvptx/CMakeLists.txt

 ##===----------------------------------------------------------------------===##
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
 ##===----------------------------------------------------------------------===##
 #
 # Build the NVPTX (CUDA) Device RTL if the CUDA tools are available
 #
 ##===----------------------------------------------------------------------===##
 
-# By default we will not build NVPTX deviceRTL on a CUDA free system
-set(LIBOMPTARGET_BUILD_NVPTX_BCLIB FALSE CACHE BOOL
+# By default we will build NVPTX deviceRTL on a CUDA free system
+set(LIBOMPTARGET_BUILD_NVPTX_BCLIB TRUE CACHE BOOL
   "Whether build NVPTX deviceRTL on CUDA free system.")
 
 if (NOT (LIBOMPTARGET_DEP_CUDA_FOUND OR LIBOMPTARGET_BUILD_NVPTX_BCLIB))
   libomptarget_say("Not building NVPTX deviceRTL by default on CUDA free system.")
   return()
 endif()
 
 # Check if we can create an LLVM bitcode implementation of the runtime library
set(cuda_src_files
   ${devicertl_common_directory}/src/task.cu
   ${devicertl_common_directory}/src/shuffle.cpp
   src/target_impl.cu
 )
 
 # Set flags for LLVM Bitcode compilation.
 set(bc_flags -S -x c++ -O1 -std=c++14
              -mllvm -openmp-opt-disable
+             -ffreestanding
              -target nvptx64
              -Xclang -emit-llvm-bc
              -Xclang -aux-triple -Xclang ${aux_triple}
              -fopenmp -fopenmp-cuda-mode -Xclang -fopenmp-is-device
              -Xclang -target-feature -Xclang +ptx61
              -D__CUDACC__
              -I${devicertl_base_directory}
              -I${devicertl_common_directory}/include

openmp/libomptarget/deviceRTLs/nvptx/src/target_impl.h

 //===------------ target_impl.h - NVPTX OpenMP GPU options ------- CUDA -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Definitions of target specific functions
 //
 //===----------------------------------------------------------------------===//
 #ifndef _TARGET_IMPL_H_
 #define _TARGET_IMPL_H_
 
-#include <inttypes.h>
-#include <stdio.h>
-#include <stdlib.h>
-
 #include "nvptx_interface.h"
 
+#include <stddef.h>
+#include <stdint.h>
+
+// subset of inttypes.h
+#define PRId64 "ld"
+#define PRIu64 "lu"
+
 #define INLINE inline __attribute__((always_inline))
 #define NOINLINE __attribute__((noinline))
 #define ALIGN(N) __attribute__((aligned(N)))
 
 ////////////////////////////////////////////////////////////////////////////////
 // Kernel options
 ////////////////////////////////////////////////////////////////////////////////
 
enum DATA_SHARING_SIZES {
   // The size of the preallocated shared memory buffer per team
   DS_Shared_Memory_Size = 128,
 };
 
 enum : __kmpc_impl_lanemask_t {
   __kmpc_impl_all_lanes = ~(__kmpc_impl_lanemask_t)0
 };
 
+#define printf(...)
+
 #endif

openmp/libomptarget/deviceRTLs/nvptx/src/target_impl.cu

 EXTERN void __kmpc_impl_unset_lock(omp_lock_t *lock) {
   (void)__kmpc_atomic_exchange(lock, UNSET);
 }
 
 EXTERN int __kmpc_impl_test_lock(omp_lock_t *lock) {
   return __kmpc_atomic_add(lock, 0u);
 }
 
+extern "C" {
+void *malloc(size_t);
+void free(void *);
+}
+
 EXTERN void *__kmpc_impl_malloc(size_t x) { return malloc(x); }
 EXTERN void __kmpc_impl_free(void *x) { free(x); }
 
 #pragma omp end declare target