[OpenMP][NFC] Remove unused (always fixed) arguments

[jdoerfert]

Summary

There are various runtime calls in the device runtime with unused, or
always fixed, arguments. This is bad for all sorts of reasons. Clean up
two before as we match them in OpenMPOpt now.

[Hahnfeld]

This is definitely not NFC and breaks API compatibility (but apparently nobody cares anymore?).

[ABataev]

In D83268#2135081, @Hahnfeld wrote:

This is definitely not NFC and breaks API compatibility (but apparently nobody cares anymore?).

+1. Better to introduce new entry points and mark these ones as deprecated.

[jdoerfert]

In D83268#2135081, @Hahnfeld wrote:

This is definitely not NFC and breaks API compatibility (but apparently nobody cares anymore?).

This is the device RTL. I am not aware we (want to) keep the API stable. If we are, I'm not sure why:

• Dynamic linking (among other things) is not really an option so people that link against the device runtime (should) do so statically.
• Linking against an old device runtime with a new clang seems unreasonable to me. If you replace clang you must replace the static runtime as the new clang might use new functions.

In D83268#2135655, @ABataev wrote:

In D83268#2135081, @Hahnfeld wrote:

This is definitely not NFC and breaks API compatibility (but apparently nobody cares anymore?).

+1. Better to introduce new entry points and mark these ones as deprecated.

Same response as above. What is the use case here which we want to continue to support?

[ABataev]

In D83268#2135724, @jdoerfert wrote:

In D83268#2135081, @Hahnfeld wrote:

This is definitely not NFC and breaks API compatibility (but apparently nobody cares anymore?).

This is the device RTL. I am not aware we (want to) keep the API stable. If we are, I'm not sure why:

• Dynamic linking (among other things) is not really an option so people that link against the device runtime (should) do so statically.
• Linking against an old device runtime with a new clang seems unreasonable to me. If you replace clang you must replace the static runtime as the new clang might use new functions.

In D83268#2135655, @ABataev wrote:

In D83268#2135081, @Hahnfeld wrote:

This is definitely not NFC and breaks API compatibility (but apparently nobody cares anymore?).

+1. Better to introduce new entry points and mark these ones as deprecated.

Same response as above. What is the use case here which we want to continue to support?

Use of the new library with the previous version of the compiler.

[ABataev]

clang/lib/CodeGen/CGOpenMPRuntimeNVPTX.cpp

42

I think, instead the optimizer can try to detect if the runtime library is used by the kernel and switch this flag to 0 if no runtime calls are used in the kernel. For non-SPMD mode in most cases, the runtime is required, but in some cases, it can be disabled.

[JonChesterfield]

I'm not sure we have a consensus on api stability. Usually llvm allows mixing libraries and compilers from different sources, e.g. the various libunwind or compiler-rt vs libgcc. Libomptarget in general appears to be considered fixed and has external users (intel, maybe gcc).

The device runtime would be ill served by this default. This is the only openmp device runtime library which works with llvm. It's statically linked, usually as bitcode when performance is important. The code used to handle target offloading for nvptx is spread across the compiler and the runtime, probably not optimally.

I'm not familiar with the gcc-nvptx-openmp implementation. Reading through https://gcc.gnu.org/wiki/Offloading strongly suggests a totally independent implementation to this one. I don't think gcc can be using this runtime library for nvptx. It definitely doesn't for amdgcn. Proprietary compilers might be using this code, but we can have no duty of care to toolchains that use this code without telling us they're doing so.

Therefore the only backwards/forwards compatibility we can strive for is between different versions of clang and the device runtime. That seems potentially useful - one could use a release clang binary while working on the deviceRTL or vice versa. It's an expensive developer convenience though.

We would pay with things like the API rot fixed above. Introducing a faster lowering for an openmp construct would mean a redundant path through clang and some version checking to guess which runtime library we're targeting, which is not presently versioned. Likewise moving code between compiler and runtime becomes much more expensive to implement. Getting it wrong is inevitable given our test coverage.

I think the project is much better served by assuming that the runtime library used by clang is the one from the same hash in the monorepo. That leaves us free to fix debt and improve performance, at the price of needing to build clang from (near to) trunk while developing the rtl.

Perhaps we can embrace API stability later on, once we have things like versioning and a solid optimisation pipeline in place, especially if gcc wants to use the deviceRTL for nvptx. Now is too early.

[JonChesterfield]

Aside from the API stability concern this looks uncontentious. Removes dead arguments, generally makes things simpler. Thus LGTM.

@Hahnfeld @ABataev - are you sufficiently persuaded that preserving the current interface is not worth the development cost?

clang/lib/CodeGen/CGOpenMPRuntimeNVPTX.cpp

42

If we can detect that no runtime calls are used, we should be able to do better than passing a different argument. E.g. delete some setup calls.

Failing that, if we want to pass an argument which says 'actually don't do any work', it shouldn't be the same argument used to check whether the runtime has been initialised.

[ABataev]

In D83268#2135930, @JonChesterfield wrote:

Aside from the API stability concern this looks uncontentious. Removes dead arguments, generally makes things simpler. Thus LGTM.

@Hahnfeld @ABataev - are you sufficiently persuaded that preserving the current interface is not worth the development cost?

No, I'm not. Long before that, we relied on the API stability and already have some runtime calls marked as deprecated. Especially taking into account, that libomp can be built separately.

clang/lib/CodeGen/CGOpenMPRuntimeNVPTX.cpp

42

No, I don't think you can do this in all cases

[jdoerfert]

I don't think gcc can be using this runtime library for nvptx.

Yes, and: We are (going to) use clang specific intrinsics to avoid CUDA (soon).

Use of the new library with the previous version of the compiler.

Except that you cannot generally expect this to work. In our supported use case the library is kept as bitcode (LLVM-IR). Bitcode is not backward compatible. An old toolchain (clang, llvm-link, ...) cannot be fed new IR and be expected to work. So, we are already not able to give a stability guarantee here, why pretend we do. The bitcode runtime has to be kept in-sync with the toolchain.

clang/lib/CodeGen/CGOpenMPRuntimeNVPTX.cpp

42

We can detect all we want but switching it *does not have any effect*.

[JonChesterfield]

In D83268#2135938, @ABataev wrote:

@Hahnfeld @ABataev - are you sufficiently persuaded that preserving the current interface is not worth the development cost?

No, I'm not. Long before that, we relied on the API stability and already have some runtime calls marked as deprecated. Especially taking into account, that libomp can be built separately.

Yes, the existing v# naming and deprecated comments should also go.

What can libomp be built by separately? Nvcc and gcc don't use this runtime. That leaves us with downstream proprietary compilers derived from clang that are already stuck carrying extensive compatibility patches and usually ship as one large toolchain blob which only needs to be internally self consistent.

[jdoerfert]

Especially taking into account, that libomp can be built separately.

This is *not* affecting libomp in any way.

[ABataev]

In D83268#2135955, @jdoerfert wrote:

Especially taking into account, that libomp can be built separately.

This is *not* affecting libomp in any way.

libomptarget and device runtimes are part of libomp. If you're going to remove some params, you'll need to modify the runtime functions too, no?

[ABataev]

In D83268#2135954, @JonChesterfield wrote:

In D83268#2135938, @ABataev wrote:

@Hahnfeld @ABataev - are you sufficiently persuaded that preserving the current interface is not worth the development cost?

No, I'm not. Long before that, we relied on the API stability and already have some runtime calls marked as deprecated. Especially taking into account, that libomp can be built separately.

Yes, the existing v# naming and deprecated comments should also go.

What can libomp be built by separately? Nvcc and gcc don't use this runtime. That leaves us with downstream proprietary compilers derived from clang that are already stuck carrying extensive compatibility patches and usually ship as one large toolchain blob which only needs to be internally self consistent.

Answered already: the previous version of the compiler with the new version of the runtime.

[jdoerfert]

In D83268#2135989, @ABataev wrote:

In D83268#2135954, @JonChesterfield wrote:

What can libomp be built by separately? Nvcc and gcc don't use this runtime. That leaves us with downstream proprietary compilers derived from clang that are already stuck carrying extensive compatibility patches and usually ship as one large toolchain blob which only needs to be internally self consistent.

Answered already: the previous version of the compiler with the new version of the runtime.

Still cannot be expected to work: https://reviews.llvm.org/D83268#2135951
Are there other use cases?

[jdoerfert]

In D83268#2135988, @ABataev wrote:

In D83268#2135955, @jdoerfert wrote:

Especially taking into account, that libomp can be built separately.

This is *not* affecting libomp in any way.

libomptarget and device runtimes are part of libomp. If you're going to remove some params, you'll need to modify the runtime functions too, no?

No they are not.

[ABataev]

In D83268#2136021, @jdoerfert wrote:

In D83268#2135988, @ABataev wrote:

In D83268#2135955, @jdoerfert wrote:

Especially taking into account, that libomp can be built separately.

This is *not* affecting libomp in any way.

libomptarget and device runtimes are part of libomp. If you're going to remove some params, you'll need to modify the runtime functions too, no?

No they are not.

llvm-project/openmp/libomptarget

[ABataev]

In D83268#2135951, @jdoerfert wrote:

I don't think gcc can be using this runtime library for nvptx.

Yes, and: We are (going to) use clang specific intrinsics to avoid CUDA (soon).

Use of the new library with the previous version of the compiler.

Except that you cannot generally expect this to work. In our supported use case the library is kept as bitcode (LLVM-IR). Bitcode is not backward compatible. An old toolchain (clang, llvm-link, ...) cannot be fed new IR and be expected to work. So, we are already not able to give a stability guarantee here, why pretend we do. The bitcode runtime has to be kept in-sync with the toolchain.

There is still compatibility between clang10 and clang11. Or they are incompatible in LLVM IR level? Also, there was a mode (I don't remember if it was removed or not) where the runtime library could be linked as .a library, without LLVM IR inlining.

[jdoerfert]

In D83268#2136031, @ABataev wrote:

There is still compatibility between clang10 and clang11. Or they are incompatible in LLVM IR level?

That is the point. They might or might not be, right? There is no guarantee they are.

Also, there was a mode (I don't remember if it was removed or not) where the runtime library could be linked as .a library, without LLVM IR inlining.

That mode is deprecated.

[jdoerfert]

In D83268#2136029, @ABataev wrote:

llvm-project/openmp/libomptarget

Please use more words.

[ABataev]

In D83268#2136054, @jdoerfert wrote:

In D83268#2136029, @ABataev wrote:

llvm-project/openmp/libomptarget

Please use more words.

libomptarget is part of libomp

[ABataev]

In D83268#2136049, @jdoerfert wrote:

In D83268#2136031, @ABataev wrote:

There is still compatibility between clang10 and clang11. Or they are incompatible in LLVM IR level?

That is the point. They might or might not be, right? There is no guarantee they are.

Better to ask the users. Maybe, send an RFC to openmp-devs?

Also, there was a mode (I don't remember if it was removed or not) where the runtime library could be linked as .a library, without LLVM IR inlining.

That mode is deprecated.

[jdoerfert]

In D83268#2136055, @ABataev wrote:

In D83268#2136054, @jdoerfert wrote:

In D83268#2136029, @ABataev wrote:

llvm-project/openmp/libomptarget

Please use more words.

libomptarget is part of libomp

As mentioned before, no it is not.

Despite the similarity in name, libomp and libomptarget are distinct libraries, this was a conscious design choice.
FWIW, this patch does *not* modify libomptarget either. This modifies *the device runtime*, aka. libomptarget-nvptx-sm_XXX.bc.

[Hahnfeld]

In D83268#2135930, @JonChesterfield wrote:

Aside from the API stability concern this looks uncontentious. Removes dead arguments, generally makes things simpler. Thus LGTM.

@Hahnfeld @ABataev - are you sufficiently persuaded that preserving the current interface is not worth the development cost?

I'm neither, and I've long argued that being able to build the OpenMP runtime(s) without Clang trunk is an important use case. These arguments have gone largely unheard, so I'll not join this discussion once more.

[jdoerfert]

In D83268#2136060, @ABataev wrote:

Better to ask the users. Maybe, send an RFC to openmp-devs?

Sure: http://lists.llvm.org/pipermail/openmp-dev/2020-July/003531.html

[JonChesterfield]

__kmpc_spmd_kernel_init is always called with RequiresDataSharing == 0
Specifically, it's only called from clang, and emitSPMDEntryHeader unconditionally passes zero to it

I.e. I think there's more stuff that can be cleaned up in the theme of the above, suggest in later patches

[protze.joachim]

This patch breaks compilation of previously working code.

I added the following to openmp/libomptarget/test/offloading/offloading_success.c:

+// RUN: %libomptarget-compile-run-and-check-nvptx64-nvidia-cuda

which results in

# command stderr:
ptxas offloading_success-openmp-nvptx64-nvidia-cuda.s, line 173; error   : Call has wrong number of parameters
ptxas fatal   : Ptx assembly aborted due to errors
clang-12: error: ptxas command failed with exit code 255 (use -v to see invocation)

The file offloading_success-openmp-nvptx64-nvidia-cuda.s contains:

.func  (.param .b32 func_retval0) __kmpc_kernel_parallel
(
        .param .b64 __kmpc_kernel_parallel_param_0,
        .param .b32 __kmpc_kernel_parallel_param_1
)
;

The same file then calls the function with one argument:

call.uni (retval0), 
__kmpc_kernel_parallel, 
(
param0
);

For the clang 11 release, we should either fix the codegen or revert this patch.

[protze.joachim]

I carefully made sure, that the freshly built clang was used to execute the test. I opened https://bugs.llvm.org/show_bug.cgi?id=46836 to track the issue and made it release blocker.

[jdoerfert]

In D83268#2172748, @protze.joachim wrote:

I carefully made sure, that the freshly built clang was used to execute the test. I opened https://bugs.llvm.org/show_bug.cgi?id=46836 to track the issue and made it release blocker.

The question is if you picked up a fresly build device runtime as well.

clang/lib/CodeGen/CGOpenMPRuntimeNVPTX.cpp

enum OpenMPRTLFunctionNVPTX {
   /// Call to void __kmpc_kernel_deinit(int16_t IsOMPRuntimeInitialized);
   OMPRTL_NVPTX__kmpc_kernel_deinit,
   /// Call to void __kmpc_spmd_kernel_init(kmp_int32 thread_limit,
   /// int16_t RequiresOMPRuntime, int16_t RequiresDataSharing);
   OMPRTL_NVPTX__kmpc_spmd_kernel_init,
   /// Call to void __kmpc_spmd_kernel_deinit_v2(int16_t RequiresOMPRuntime);
   OMPRTL_NVPTX__kmpc_spmd_kernel_deinit_v2,
   /// Call to void __kmpc_kernel_prepare_parallel(void
-  /// *outlined_function, int16_t
-  /// IsOMPRuntimeInitialized);
+  /// *outlined_function);
   OMPRTL_NVPTX__kmpc_kernel_prepare_parallel,
-  /// Call to bool __kmpc_kernel_parallel(void **outlined_function,
-  /// int16_t IsOMPRuntimeInitialized);
+  /// Call to bool __kmpc_kernel_parallel(void **outlined_function);
   OMPRTL_NVPTX__kmpc_kernel_parallel,
   /// Call to void __kmpc_kernel_end_parallel();
   OMPRTL_NVPTX__kmpc_kernel_end_parallel,
   /// Call to void __kmpc_serialized_parallel(ident_t *loc, kmp_int32
   /// global_tid);
   OMPRTL_NVPTX__kmpc_serialized_parallel,
   /// Call to void __kmpc_end_serialized_parallel(ident_t *loc, kmp_int32
   /// global_tid);
void CGOpenMPRuntimeNVPTX::emitWorkerLoop(CodeGenFunction &CGF,
   Address WorkFn =
       CGF.CreateDefaultAlignTempAlloca(CGF.Int8PtrTy, /*Name=*/"work_fn");
   Address ExecStatus =
       CGF.CreateDefaultAlignTempAlloca(CGF.Int8Ty, /*Name=*/"exec_status");
   CGF.InitTempAlloca(ExecStatus, Bld.getInt8(/*C=*/0));
   CGF.InitTempAlloca(WorkFn, llvm::Constant::getNullValue(CGF.Int8PtrTy));
 
   // TODO: Optimize runtime initialization and pass in correct value.
-  llvm::Value *Args[] = {WorkFn.getPointer(),
-                         /*RequiresOMPRuntime=*/Bld.getInt16(1)};
+  llvm::Value *Args[] = {WorkFn.getPointer()};
   llvm::Value *Ret = CGF.EmitRuntimeCall(
       createNVPTXRuntimeFunction(OMPRTL_NVPTX__kmpc_kernel_parallel), Args);
   Bld.CreateStore(Bld.CreateZExt(Ret, CGF.Int8Ty), ExecStatus);
 
   // On termination condition (workid == 0), exit loop.
   llvm::Value *WorkID = Bld.CreateLoad(WorkFn);
   llvm::Value *ShouldTerminate = Bld.CreateIsNull(WorkID, "should_terminate");
   Bld.CreateCondBr(ShouldTerminate, ExitBB, SelectWorkersBB);
case OMPRTL_NVPTX__kmpc_spmd_kernel_deinit_v2: {
     llvm::Type *TypeParams[] = {CGM.Int16Ty};
     auto *FnTy =
         llvm::FunctionType::get(CGM.VoidTy, TypeParams, /*isVarArg*/ false);
     RTLFn = CGM.CreateRuntimeFunction(FnTy, "__kmpc_spmd_kernel_deinit_v2");
     break;
   }
   case OMPRTL_NVPTX__kmpc_kernel_prepare_parallel: {
     /// Build void __kmpc_kernel_prepare_parallel(
-    /// void *outlined_function, int16_t IsOMPRuntimeInitialized);
-    llvm::Type *TypeParams[] = {CGM.Int8PtrTy, CGM.Int16Ty};
+    /// void *outlined_function);
+    llvm::Type *TypeParams[] = {CGM.Int8PtrTy};
     auto *FnTy =
         llvm::FunctionType::get(CGM.VoidTy, TypeParams, /*isVarArg*/ false);
     RTLFn = CGM.CreateRuntimeFunction(FnTy, "__kmpc_kernel_prepare_parallel");
     break;
   }
   case OMPRTL_NVPTX__kmpc_kernel_parallel: {
-    /// Build bool __kmpc_kernel_parallel(void **outlined_function,
-    /// int16_t IsOMPRuntimeInitialized);
-    llvm::Type *TypeParams[] = {CGM.Int8PtrPtrTy, CGM.Int16Ty};
+    /// Build bool __kmpc_kernel_parallel(void **outlined_function);
+    llvm::Type *TypeParams[] = {CGM.Int8PtrPtrTy};
     llvm::Type *RetTy = CGM.getTypes().ConvertType(CGM.getContext().BoolTy);
     auto *FnTy =
         llvm::FunctionType::get(RetTy, TypeParams, /*isVarArg*/ false);
     RTLFn = CGM.CreateRuntimeFunction(FnTy, "__kmpc_kernel_parallel");
     break;
   }
   case OMPRTL_NVPTX__kmpc_kernel_end_parallel: {
     /// Build void __kmpc_kernel_end_parallel();
void CGOpenMPRuntimeNVPTX::emitNonSPMDParallelCall(
   auto &&L0ParallelGen = [this, CapturedVars, Fn](CodeGenFunction &CGF,
                                                   PrePostActionTy &Action) {
     CGBuilderTy &Bld = CGF.Builder;
     llvm::Function *WFn = WrapperFunctionsMap[Fn];
     assert(WFn && "Wrapper function does not exist!");
     llvm::Value *ID = Bld.CreateBitOrPointerCast(WFn, CGM.Int8PtrTy);
 
     // Prepare for parallel region. Indicate the outlined function.
-    llvm::Value *Args[] = {ID, /*RequiresOMPRuntime=*/Bld.getInt16(1)};
+    llvm::Value *Args[] = {ID};
     CGF.EmitRuntimeCall(
         createNVPTXRuntimeFunction(OMPRTL_NVPTX__kmpc_kernel_prepare_parallel),
         Args);
 
     // Create a private scope that will globalize the arguments
     // passed from the outside of the target region.
     CodeGenFunction::OMPPrivateScope PrivateArgScope(CGF);
 

clang/test/OpenMP/nvptx_data_sharing.cpp

 // SEQ: call void @__kmpc_get_team_static_memory(i16 0, i8* addrspacecast (i8 addrspace(3)* getelementptr inbounds ([[MEM_TY]], [[MEM_TY]] addrspace(3)* [[SHARED_GLOBAL_RD]], i32 0, i32 0, i32 0) to i8*), i64 [[SIZE]], i16 [[SHARED_MEM_FLAG]], i8** addrspacecast (i8* addrspace(3)* [[KERNEL_PTR]] to i8**))
 // SEQ: [[KERNEL_RD:%.+]] = load i8*, i8* addrspace(3)* [[KERNEL_PTR]],
 // SEQ: [[GLOBALSTACK:%.+]] = getelementptr inbounds i8, i8* [[KERNEL_RD]], i64 0
 // PAR: [[GLOBALSTACK:%.+]] = call i8* @__kmpc_data_sharing_push_stack(i{{32|64}} 8, i16 1)
 // CK1: [[GLOBALSTACK2:%.+]] = bitcast i8* [[GLOBALSTACK]] to %struct._globalized_locals_ty*
 // CK1: [[A:%.+]] = getelementptr inbounds %struct._globalized_locals_ty, %struct._globalized_locals_ty* [[GLOBALSTACK2]], i32 0, i32 0
 // CK1: [[B:%.+]] = getelementptr inbounds %struct._globalized_locals_ty, %struct._globalized_locals_ty* [[GLOBALSTACK2]], i32 0, i32 1
 // CK1: store i32 10, i32* [[A]]
-// CK1: call void @__kmpc_kernel_prepare_parallel({{.*}}, i16 1)
+// CK1: call void @__kmpc_kernel_prepare_parallel({{.*}})
 // CK1: call void @__kmpc_begin_sharing_variables(i8*** [[SHAREDARGS1]], i64 1)
 // CK1: [[SHARGSTMP1:%.+]] = load i8**, i8*** [[SHAREDARGS1]]
 // CK1: [[SHARGSTMP2:%.+]] = getelementptr inbounds i8*, i8** [[SHARGSTMP1]], i64 0
 // CK1: [[SHAREDVAR:%.+]] = bitcast i32* [[A]] to i8*
 // CK1: store i8* [[SHAREDVAR]], i8** [[SHARGSTMP2]]
 // CK1: call void @__kmpc_barrier_simple_spmd(%struct.ident_t* null, i32 0)
 // CK1: call void @__kmpc_barrier_simple_spmd(%struct.ident_t* null, i32 0)
 // CK1: call void @__kmpc_end_sharing_variables()
 // CK1: store i32 100, i32* [[B]]
-// CK1: call void @__kmpc_kernel_prepare_parallel({{.*}}, i16 1)
+// CK1: call void @__kmpc_kernel_prepare_parallel({{.*}})
 // CK1: call void @__kmpc_begin_sharing_variables(i8*** [[SHAREDARGS2]], i64 2)
 // CK1: [[SHARGSTMP3:%.+]] = load i8**, i8*** [[SHAREDARGS2]]
 // CK1: [[SHARGSTMP4:%.+]] = getelementptr inbounds i8*, i8** [[SHARGSTMP3]], i64 0
 // CK1: [[SHAREDVAR1:%.+]] = bitcast i32* [[B]] to i8*
 // CK1: store i8* [[SHAREDVAR1]], i8** [[SHARGSTMP4]]
 // CK1: [[SHARGSTMP12:%.+]] = getelementptr inbounds i8*, i8** [[SHARGSTMP3]], i64 1
 // CK1: [[SHAREDVAR2:%.+]] = bitcast i32* [[A]] to i8*
 // CK1: store i8* [[SHAREDVAR2]], i8** [[SHARGSTMP12]]

clang/test/OpenMP/nvptx_parallel_codegen.cpp

 // CHECK-DAG: [[OMP_EXEC_STATUS:%.+]] = alloca i8,
 // CHECK-DAG: [[OMP_WORK_FN:%.+]] = alloca i8*,
 // CHECK: store i8* null, i8** [[OMP_WORK_FN]],
 // CHECK: store i8 0, i8* [[OMP_EXEC_STATUS]],
 // CHECK: br label {{%?}}[[AWAIT_WORK:.+]]
 //
 // CHECK: [[AWAIT_WORK]]
 // CHECK: call void @__kmpc_barrier_simple_spmd(%struct.ident_t* null, i32 0) #[[#CONVERGENT:]]
-// CHECK: [[KPR:%.+]] = call i1 @__kmpc_kernel_parallel(i8** [[OMP_WORK_FN]]
+// CHECK: [[KPR:%.+]] = call i1 @__kmpc_kernel_parallel(i8** [[OMP_WORK_FN]])
 // CHECK: [[KPRB:%.+]] = zext i1 [[KPR]] to i8
 // store i8 [[KPRB]], i8* [[OMP_EXEC_STATUS]], align 1
 // CHECK: [[WORK:%.+]] = load i8*, i8** [[OMP_WORK_FN]],
 // CHECK: [[SHOULD_EXIT:%.+]] = icmp eq i8* [[WORK]], null
 // CHECK: br i1 [[SHOULD_EXIT]], label {{%?}}[[EXIT:.+]], label {{%?}}[[SEL_WORKERS:.+]]
 //
 // CHECK: [[SEL_WORKERS]]
 // CHECK: [[ST:%.+]] = load i8, i8* [[OMP_EXEC_STATUS]]

 // CHECK: [[IS_MASTER:%.+]] = icmp eq i32 [[CMTID]],
 // CHECK: br i1 [[IS_MASTER]], label {{%?}}[[MASTER:.+]], label {{%?}}[[EXIT]]
 //
 // CHECK: [[MASTER]]
 // CHECK-DAG: [[MNTH:%.+]] = call i32 @llvm.nvvm.read.ptx.sreg.ntid.x()
 // CHECK-DAG: [[MWS:%.+]] = call i32 @llvm.nvvm.read.ptx.sreg.warpsize()
 // CHECK: [[MTMP1:%.+]] = sub nuw i32 [[MNTH]], [[MWS]]
 // CHECK: call void @__kmpc_kernel_init(i32 [[MTMP1]]
-// CHECK: call void @__kmpc_kernel_prepare_parallel(i8* bitcast (void (i16, i32)* [[PARALLEL_FN1]]_wrapper to i8*),
+// CHECK: call void @__kmpc_kernel_prepare_parallel(i8* bitcast (void (i16, i32)* [[PARALLEL_FN1]]_wrapper to i8*))
 // CHECK: call void @__kmpc_barrier_simple_spmd(%struct.ident_t* null, i32 0)
 // CHECK: call void @__kmpc_barrier_simple_spmd(%struct.ident_t* null, i32 0)
 // CHECK: call void @__kmpc_serialized_parallel(
 // CHECK: {{call|invoke}} void [[PARALLEL_FN3:@.+]](
 // CHECK: call void @__kmpc_end_serialized_parallel(
-// CHECK: call void @__kmpc_kernel_prepare_parallel(i8* bitcast (void (i16, i32)* [[PARALLEL_FN2]]_wrapper to i8*),
+// CHECK: call void @__kmpc_kernel_prepare_parallel(i8* bitcast (void (i16, i32)* [[PARALLEL_FN2]]_wrapper to i8*))
 // CHECK: call void @__kmpc_barrier_simple_spmd(%struct.ident_t* null, i32 0)
 // CHECK: call void @__kmpc_barrier_simple_spmd(%struct.ident_t* null, i32 0)
 // CHECK-64-DAG: load i32, i32* [[REF_A]]
 // CHECK-32-DAG: load i32, i32* [[LOCAL_A]]
 // CHECK: br label {{%?}}[[TERMINATE:.+]]
 //
 // CHECK: [[TERMINATE]]
 // CHECK: call void @__kmpc_kernel_deinit(

 // CHECK-DAG: [[OMP_EXEC_STATUS:%.+]] = alloca i8,
 // CHECK-DAG: [[OMP_WORK_FN:%.+]] = alloca i8*,
 // CHECK: store i8* null, i8** [[OMP_WORK_FN]],
 // CHECK: store i8 0, i8* [[OMP_EXEC_STATUS]],
 // CHECK: br label {{%?}}[[AWAIT_WORK:.+]]
 //
 // CHECK: [[AWAIT_WORK]]
 // CHECK: call void @__kmpc_barrier_simple_spmd(%struct.ident_t* null, i32 0)
-// CHECK: [[KPR:%.+]] = call i1 @__kmpc_kernel_parallel(i8** [[OMP_WORK_FN]],
+// CHECK: [[KPR:%.+]] = call i1 @__kmpc_kernel_parallel(i8** [[OMP_WORK_FN]])
 // CHECK: [[KPRB:%.+]] = zext i1 [[KPR]] to i8
 // store i8 [[KPRB]], i8* [[OMP_EXEC_STATUS]], align 1
 // CHECK: [[WORK:%.+]] = load i8*, i8** [[OMP_WORK_FN]],
 // CHECK: [[SHOULD_EXIT:%.+]] = icmp eq i8* [[WORK]], null
 // CHECK: br i1 [[SHOULD_EXIT]], label {{%?}}[[EXIT:.+]], label {{%?}}[[SEL_WORKERS:.+]]
 //
 // CHECK: [[SEL_WORKERS]]
 // CHECK: [[ST:%.+]] = load i8, i8* [[OMP_EXEC_STATUS]]

 // CHECK: [[MTMP1:%.+]] = sub nuw i32 [[MNTH]], [[MWS]]
 // CHECK: call void @__kmpc_kernel_init(i32 [[MTMP1]]
 // CHECK-64: [[N:%.+]] = load i32, i32* [[REF_N]],
 // CHECK-32: [[N:%.+]] = load i32, i32* [[LOCAL_N]],
 // CHECK: [[CMP:%.+]] = icmp sgt i32 [[N]], 1000
 // CHECK: br i1 [[CMP]], label {{%?}}[[IF_THEN:.+]], label {{%?}}[[IF_ELSE:.+]]
 //
 // CHECK: [[IF_THEN]]
-// CHECK: call void @__kmpc_kernel_prepare_parallel(i8* bitcast (void (i16, i32)* [[PARALLEL_FN4]]_wrapper to i8*),
+// CHECK: call void @__kmpc_kernel_prepare_parallel(i8* bitcast (void (i16, i32)* [[PARALLEL_FN4]]_wrapper to i8*))
 // CHECK: call void @__kmpc_barrier_simple_spmd(%struct.ident_t* null, i32 0)
 // CHECK: call void @__kmpc_barrier_simple_spmd(%struct.ident_t* null, i32 0)
 // CHECK: br label {{%?}}[[IF_END:.+]]
 //
 // CHECK: [[IF_ELSE]]
 // CHECK: call void @__kmpc_serialized_parallel(
 // CHECK: {{call|invoke}} void [[PARALLEL_FN4]](
 // CHECK: call void @__kmpc_end_serialized_parallel(

clang/test/OpenMP/nvptx_target_codegen.cpp

 // CHECK: icmp ne i16 [[RES]], 0
 // CHECK: br i1
 
 // CHECK: call void @__kmpc_serialized_parallel(%struct.ident_t* [[UNKNOWN]], i32 [[GTID]])
 // CHECK: call void [[OUTLINED:@.+]](i32* [[ZERO_ADDR]], i32* [[BND_ZERO_ADDR]], i32* [[F_PTR]], double* %{{.+}})
 // CHECK: call void @__kmpc_end_serialized_parallel(%struct.ident_t* [[UNKNOWN]], i32 [[GTID]])
 // CHECK: br label
 
-// CHECK: call void @__kmpc_kernel_prepare_parallel(i8* bitcast (void (i16, i32)* @{{.+}} to i8*), i16 1)
+// CHECK: call void @__kmpc_kernel_prepare_parallel(i8* bitcast (void (i16, i32)* @{{.+}} to i8*))
 // CHECK: call void @__kmpc_begin_sharing_variables(i8*** [[SHARED_PTR:%.+]], i{{64|32}} 2)
 // CHECK: [[SHARED:%.+]] = load i8**, i8*** [[SHARED_PTR]],
 // CHECK: [[REF:%.+]] = getelementptr inbounds i8*, i8** [[SHARED]], i{{64|32}} 0
 // CHECK: [[F_REF:%.+]] = bitcast i32* [[F_PTR]] to i8*
 // CHECK: store i8* [[F_REF]], i8** [[REF]],
 // CHECK: call void @__kmpc_barrier_simple_spmd(%struct.ident_t* null, i32 0)
 // CHECK: call void @__kmpc_barrier_simple_spmd(%struct.ident_t* null, i32 0)
 // CHECK: call void @__kmpc_end_sharing_variables()

clang/test/OpenMP/nvptx_target_teams_codegen.cpp

int bar(int n){
   // CHECK-DAG: [[OMP_EXEC_STATUS:%.+]] = alloca i8,
   // CHECK-DAG: [[OMP_WORK_FN:%.+]] = alloca i8*,
   // CHECK: store i8* null, i8** [[OMP_WORK_FN]],
   // CHECK: store i8 0, i8* [[OMP_EXEC_STATUS]],
   // CHECK: br label {{%?}}[[AWAIT_WORK:.+]]
   //
   // CHECK: [[AWAIT_WORK]]
   // CHECK: call void @__kmpc_barrier_simple_spmd(%struct.ident_t* null, i32 0)
-  // CHECK: [[KPR:%.+]] = call i1 @__kmpc_kernel_parallel(i8** [[OMP_WORK_FN]], i16 1)
+  // CHECK: [[KPR:%.+]] = call i1 @__kmpc_kernel_parallel(i8** [[OMP_WORK_FN]])
   // CHECK: [[KPRB:%.+]] = zext i1 [[KPR]] to i8
   // store i8 [[KPRB]], i8* [[OMP_EXEC_STATUS]], align 1
   // CHECK: [[WORK:%.+]] = load i8*, i8** [[OMP_WORK_FN]],
   // CHECK: [[SHOULD_EXIT:%.+]] = icmp eq i8* [[WORK]], null
   // CHECK: br i1 [[SHOULD_EXIT]], label {{%?}}[[EXIT:.+]], label {{%?}}[[SEL_WORKERS:.+]]
   //
   // CHECK: [[SEL_WORKERS]]
   // CHECK: [[ST:%.+]] = load i8, i8* [[OMP_EXEC_STATUS]]
int bar(int n){
   // CHECK-DAG: [[OMP_EXEC_STATUS:%.+]] = alloca i8,
   // CHECK-DAG: [[OMP_WORK_FN:%.+]] = alloca i8*,
   // CHECK: store i8* null, i8** [[OMP_WORK_FN]],
   // CHECK: store i8 0, i8* [[OMP_EXEC_STATUS]],
   // CHECK: br label {{%?}}[[AWAIT_WORK:.+]]
   //
   // CHECK: [[AWAIT_WORK]]
   // CHECK: call void @__kmpc_barrier_simple_spmd(%struct.ident_t* null, i32 0)
-  // CHECK: [[KPR:%.+]] = call i1 @__kmpc_kernel_parallel(i8** [[OMP_WORK_FN]], i16 1)
+  // CHECK: [[KPR:%.+]] = call i1 @__kmpc_kernel_parallel(i8** [[OMP_WORK_FN]])
   // CHECK: [[KPRB:%.+]] = zext i1 [[KPR]] to i8
   // store i8 [[KPRB]], i8* [[OMP_EXEC_STATUS]], align 1
   // CHECK: [[WORK:%.+]] = load i8*, i8** [[OMP_WORK_FN]],
   // CHECK: [[SHOULD_EXIT:%.+]] = icmp eq i8* [[WORK]], null
   // CHECK: br i1 [[SHOULD_EXIT]], label {{%?}}[[EXIT:.+]], label {{%?}}[[SEL_WORKERS:.+]]
   //
   // CHECK: [[SEL_WORKERS]]
   // CHECK: [[ST:%.+]] = load i8, i8* [[OMP_EXEC_STATUS]]

clang/test/OpenMP/nvptx_target_teams_distribute_codegen.cpp

int bar(int n){
   // SEQ: call void @__kmpc_get_team_static_memory(i16 0, i8* addrspacecast (i8 addrspace(3)* getelementptr inbounds ([[MEM_TY]], [[MEM_TY]] addrspace(3)* @{{.+}}, i32 0, i32 0, i32 0) to i8*), i{{64|32}} [[SIZE]], i16 [[SHARED]], i8** addrspacecast (i8* addrspace(3)* [[BUF:@.+]] to i8**))
   // SEQ: [[PTR:%.+]] = load i8*, i8* addrspace(3)* [[BUF]],
   // SEQ: [[ADDR:%.+]] = getelementptr inbounds i8, i8* [[PTR]], i{{64|32}} 0
   // PAR: [[ADDR:%.+]] = call i8* @__kmpc_data_sharing_push_stack(i{{32|64}} 4, i16 1)
   // CHECK: [[RD:%.+]] = bitcast i8* [[ADDR]] to [[GLOB_TY:%.+]]*
   // CHECK: [[I_ADDR:%.+]] = getelementptr inbounds [[GLOB_TY]], [[GLOB_TY]]* [[RD]], i32 0, i32 0
   //
   // CHECK: call void @__kmpc_for_static_init_4(
-  // CHECK: call void @__kmpc_kernel_prepare_parallel(i8* bitcast (void (i16, i32)* @{{.+}} to i8*), i16 1)
+  // CHECK: call void @__kmpc_kernel_prepare_parallel(i8* bitcast (void (i16, i32)* @{{.+}} to i8*))
   // CHECK: call void @__kmpc_begin_sharing_variables(i8*** [[SHARED_VARS_PTR:%.+]], i{{64|32}} 1)
   // CHECK: [[SHARED_VARS_BUF:%.+]] = load i8**, i8*** [[SHARED_VARS_PTR]],
   // CHECK: [[VARS_BUF:%.+]] = getelementptr inbounds i8*, i8** [[SHARED_VARS_BUF]], i{{64|32}} 0
   // CHECK: [[I_ADDR_BC:%.+]] = bitcast i32* [[I_ADDR]] to i8*
   // CHECK: store i8* [[I_ADDR_BC]], i8** [[VARS_BUF]],
   // CHECK: call void @__kmpc_barrier_simple_spmd(%struct.ident_t* null, i32 0)
   // CHECK: call void @__kmpc_barrier_simple_spmd(%struct.ident_t* null, i32 0)
   // CHECK: call void @__kmpc_end_sharing_variables()
   // CHECK: call void @__kmpc_for_static_fini(
 #endif

llvm/include/llvm/Frontend/OpenMP/OMPKinds.def

 __OMP_RTL(__tgt_target_data_update_nowait, false, Void, Int64, Int32,
           VoidPtrPtr, VoidPtrPtr, Int64Ptr, Int64Ptr)
 __OMP_RTL(__tgt_mapper_num_components, false, Int64, VoidPtr)
 __OMP_RTL(__tgt_push_mapper_component, false, Void, VoidPtr, VoidPtr, VoidPtr,
           Int64, Int64)
 __OMP_RTL(__kmpc_task_allow_completion_event, false, VoidPtr, IdentPtr,
           /* Int */ Int32, /* kmp_task_t */ VoidPtr)
 
+/// Note that device runtime functions (in the following) do not necessarily
+/// need attributes as we expect to see the definitions.
+__OMP_RTL(__kmpc_kernel_parallel, false, Int1, VoidPtrPtr)
+__OMP_RTL(__kmpc_kernel_prepare_parallel, false, Void, VoidPtr)
+
 __OMP_RTL(__last, false, Void, )
 
 #undef __OMP_RTL
 #undef OMP_RTL
 
 #define EnumAttr(Kind) Attribute::get(Ctx, Attribute::AttrKind::Kind)
 #define AttributeSet(...)                                                      \
   AttributeSet::get(Ctx, ArrayRef<Attribute>({__VA_ARGS__}))

openmp/libomptarget/deviceRTLs/common/src/parallel.cu

if (NumThreads < WARPSIZE) {
     NumThreads = (NumThreads & ~((uint16_t)WARPSIZE - 1));
   }
 #endif
 
   return NumThreads;
 }
 
 // This routine is always called by the team master..
-EXTERN void __kmpc_kernel_prepare_parallel(void *WorkFn,
-                                           int16_t IsOMPRuntimeInitialized) {
+EXTERN void __kmpc_kernel_prepare_parallel(void *WorkFn) {
   PRINT0(LD_IO, "call to __kmpc_kernel_prepare_parallel\n");
-  ASSERT0(LT_FUSSY, IsOMPRuntimeInitialized, "Expected initialized runtime.");
 
   omptarget_nvptx_workFn = WorkFn;
 
   // This routine is only called by the team master.  The team master is
   // the first thread of the last warp.  It always has the logical thread
   // id of 0 (since it is a shadow for the first worker thread).
   const int threadId = 0;
   omptarget_nvptx_TaskDescr *currTaskDescr =
EXTERN void __kmpc_kernel_prepare_parallel(void *WorkFn) {
   threadsInTeam = NumThreads;
 }
 
 // All workers call this function.  Deactivate those not needed.
 // Fn - the outlined work function to execute.
 // returns True if this thread is active, else False.
 //
 // Only the worker threads call this routine.
-EXTERN bool __kmpc_kernel_parallel(void **WorkFn,
-                                   int16_t IsOMPRuntimeInitialized) {
+EXTERN bool __kmpc_kernel_parallel(void **WorkFn) {
   PRINT0(LD_IO | LD_PAR, "call to __kmpc_kernel_parallel\n");
 
-  ASSERT0(LT_FUSSY, IsOMPRuntimeInitialized, "Expected initialized runtime.");
-
   // Work function and arguments for L1 parallel region.
   *WorkFn = omptarget_nvptx_workFn;
 
   // If this is the termination signal from the master, quit early.
   if (!*WorkFn) {
     PRINT0(LD_IO | LD_PAR, "call to __kmpc_kernel_parallel finished\n");
     return false;
   }

openmp/libomptarget/deviceRTLs/interface.h

EXTERN int32_t __kmpc_cancel(kmp_Ident *loc, int32_t global_tid,
                              int32_t cancelVal);
 
 // non standard
 EXTERN void __kmpc_kernel_init(int ThreadLimit, int16_t RequiresOMPRuntime);
 EXTERN void __kmpc_kernel_deinit(int16_t IsOMPRuntimeInitialized);
 EXTERN void __kmpc_spmd_kernel_init(int ThreadLimit, int16_t RequiresOMPRuntime,
                                     int16_t RequiresDataSharing);
 EXTERN void __kmpc_spmd_kernel_deinit_v2(int16_t RequiresOMPRuntime);
-EXTERN void __kmpc_kernel_prepare_parallel(void *WorkFn,
-                                           int16_t IsOMPRuntimeInitialized);
-EXTERN bool __kmpc_kernel_parallel(void **WorkFn,
-                                   int16_t IsOMPRuntimeInitialized);
+EXTERN void __kmpc_kernel_prepare_parallel(void *WorkFn);
+EXTERN bool __kmpc_kernel_parallel(void **WorkFn);
 EXTERN void __kmpc_kernel_end_parallel();
 
 EXTERN void __kmpc_data_sharing_init_stack();
 EXTERN void __kmpc_data_sharing_init_stack_spmd();
 EXTERN void *__kmpc_data_sharing_coalesced_push_stack(size_t size,
     int16_t UseSharedMemory);
 EXTERN void *__kmpc_data_sharing_push_stack(size_t size, int16_t UseSharedMemory);
 EXTERN void __kmpc_data_sharing_pop_stack(void *a);