[AMDGPU] Use inreg for hint to preload kernel arguments

[kerbowa]

Summary

This patch is the first in a series that adds support for pre-loading
kernel arguments into SGPRs. The command-line argument
'amdgpu-kernarg-preload-count' is used to specify the number of
arguments sequentially from the first that we should attempt to preload,
the default is 0.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUAttributor.cpp

922

This isn't really gaining value by being in the attributor as it stands. Are you planning on a more sophisticated user analysis to select the starting point?

924

Assuming 1 register : 1 argument which is not the case.

Also the user SGPR count went up (in gfx10 I think?) so you should query the number for that

986

Don't need the arg_empty check, the implementation handles it anyway and it's uncommon

llvm/test/CodeGen/AMDGPU/preload-kernal-args-inreg-hints.ll

3

Use -passes

236

Needs some exotic types. i8, <2 x half>, <3 x half> odd bit sized scalars, odd bit sized element vectors, odd vectors, structs, arrays and nested structs

[arsenm]

llvm/test/CodeGen/AMDGPU/preload-kernal-args-inreg-hints.ll

236

We could probably get away with making the verifier reject aggregate arguments for amdgpu_kernel

[rampitec]

llvm/lib/Target/AMDGPU/AMDGPUAttributor.cpp

924

AFAIR it was increased to 32 in gfx9, then again was 16 in gfx10. I also do not think this is implemented.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUAttributor.cpp

924

We already have getMaxNumUserSGPRs but it's just hardcoded

[kerbowa]

llvm/lib/Target/AMDGPU/AMDGPUAttributor.cpp

922

Yes, that's the plan. Any heuristics of that sort will be implemented here.

924

InReg is just a hint so being super accurate with the number of available SGPRs isn't needed for correctness here. The intent is just to have it stop at the maximum, but if it goes over it doesn't matter. I wasn't aware that the number of available user SGPRs had changed i.e. at https://llvm.org/docs/AMDGPUUsage.html I just see 16, I've only ever heard the number 16 talked about. Anyway, I will change this to getMaxNumUserSGPRs as I look into what's actually supported more.

llvm/test/CodeGen/AMDGPU/preload-kernal-args-inreg-hints.ll

3

I think it won't work for amdgpu-attributor with old pass manager?

236

Kind of unnecessary here since none of that impacts the addition of inreg at this point, it will just reject preloading the argument later.

[kerbowa]

Use getMaxNumUserSGPRs.
Check incompatible attributes.

[kerbowa]

Add metadata to ensure that preloading is explicitly requested.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUAttributor.cpp

930

Could actually handle byref (and should, ideally all kernargs would use byref). If we consistently used byref, you could implement this by moving memory values to the arg list

939

Typo arguemnts

942

I don't understand why you need this metadata. You mark the arguments with inreg which provides the same information? You can still ignore the hint in the codegen

llvm/test/CodeGen/AMDGPU/preload-kernal-args-inreg-hints.ll

3

Apparently it hasn't been ported yet

[kerbowa]

Fix typo

[kerbowa]

llvm/lib/Target/AMDGPU/AMDGPUAttributor.cpp

930

I'm planning on adding it in a future revision but I don't think it's needed for this iteration of the changes.

942

I'm adding the metadata early to ensure that kernarg preloading is constrained to those that are explicitly using the CL flag. In case there are some frontend that may sneak inreg on some arguments. Will probably just until the feature is more mature or is enabled by default.

[kerbowa]

Cleanup.

[arsenm]

I don't want any usage of metadata, it's confusing and unnecessary. The layout should be fully deterministic from just the argument list and inreg, you don't need a parallel encoding scheme

llvm/lib/Target/AMDGPU/AMDGPUAttributor.cpp

942

This is an anti feature. Just respect inreg regardless of the source. You're just adding a huge source of fragility just in case a frontend tries to make use of an in progress feature

[kerbowa]

Remove metadata.

[kerbowa]

In D156852#4644070, @arsenm wrote:

I don't want any usage of metadata, it's confusing and unnecessary. The layout should be fully deterministic from just the argument list and inreg, you don't need a parallel encoding scheme

Thanks for the review Matt, I've removed the metadata system entirely.

llvm/lib/Target/AMDGPU/AMDGPUAttributor.cpp

 
 #include "AMDGPU.h"
 #include "GCNSubtarget.h"
 #include "Utils/AMDGPUBaseInfo.h"
 #include "llvm/Analysis/CycleAnalysis.h"
 #include "llvm/CodeGen/TargetPassConfig.h"
 #include "llvm/IR/IntrinsicsAMDGPU.h"
 #include "llvm/IR/IntrinsicsR600.h"
+#include "llvm/Support/CommandLine.h"
 #include "llvm/Target/TargetMachine.h"
 #include "llvm/Transforms/IPO/Attributor.h"
 
 #define DEBUG_TYPE "amdgpu-attributor"
 
 namespace llvm {
 void initializeCycleInfoWrapperPassPass(PassRegistry &);
 }
 
 using namespace llvm;
 
+static cl::opt<unsigned> KernargPreloadCount(
+    "amdgpu-kernarg-preload-count",
+    cl::desc("How many kernel arguments to preload onto SGPRs"), cl::init(0));
+
 #define AMDGPU_ATTRIBUTE(Name, Str) Name##_POS,
 
 enum ImplicitArgumentPositions {
   #include "AMDGPUAttributes.def"
   LAST_ARG_POS
 };
 
 #define AMDGPU_ATTRIBUTE(Name, Str) Name = 1 << Name##_POS,

 
 AAAMDWavesPerEU &AAAMDWavesPerEU::createForPosition(const IRPosition &IRP,
                                                     Attributor &A) {
   if (IRP.getPositionKind() == IRPosition::IRP_FUNCTION)
     return *new (A.Allocator) AAAMDWavesPerEU(IRP, A);
   llvm_unreachable("AAAMDWavesPerEU is only valid for function position");
 }
 
+static void addPreloadKernArgHint(Function &F, TargetMachine &TM) {
+  const GCNSubtarget &ST = TM.getSubtarget<GCNSubtarget>(F);
+  for (unsigned I = 0;
+       I < F.arg_size() &&
+       I < std::min(KernargPreloadCount.getValue(), ST.getMaxNumUserSGPRs());
+       ++I) {
+    Argument &Arg = *F.getArg(I);
+    // Check for incompatible attributes.
+    if (Arg.hasByRefAttr() || Arg.hasNestAttr())
+      break;
+    Arg.addAttr(Attribute::InReg);
+  }
+}
+
 class AMDGPUAttributor : public ModulePass {
 public:
   AMDGPUAttributor() : ModulePass(ID) {}
 
   /// doInitialization - Virtual method overridden by subclasses to do
   /// any necessary initialization before any pass is run.
   bool doInitialization(Module &) override {
     auto *TPC = getAnalysisIfAvailable<TargetPassConfig>();
     if (!TPC)
       report_fatal_error("TargetMachine is required");
 
     TM = &TPC->getTM<TargetMachine>();
     return false;
   }
 
bool runOnModule(Module &M) override {
     };
 
     Attributor A(Functions, InfoCache, AC);
 
     for (Function &F : M) {
       if (!F.isIntrinsic()) {
         A.getOrCreateAAFor<AAAMDAttributes>(IRPosition::function(F));
         A.getOrCreateAAFor<AAUniformWorkGroupSize>(IRPosition::function(F));
-        if (!AMDGPU::isEntryFunctionCC(F.getCallingConv())) {
+        CallingConv::ID CC = F.getCallingConv();
+        if (!AMDGPU::isEntryFunctionCC(CC)) {
           A.getOrCreateAAFor<AAAMDFlatWorkGroupSize>(IRPosition::function(F));
           A.getOrCreateAAFor<AAAMDWavesPerEU>(IRPosition::function(F));
+        } else if (CC == CallingConv::AMDGPU_KERNEL && !F.arg_empty()) {
+          addPreloadKernArgHint(F, *TM);
         }
       }
     }
 
     ChangeStatus Change = A.run();
     return Change == ChangeStatus::CHANGED;
   }
 

llvm/test/CodeGen/AMDGPU/preload-kernal-args-inreg-hints.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --function-signature
+; RUN: opt -mtriple=amdgcn-amd-amdhsa -mcpu=gfx90a -amdgpu-attributor -S < %s | FileCheck -check-prefix=NO-PRELOAD %s
+; RUN: opt -mtriple=amdgcn-amd-amdhsa -mcpu=gfx90a -amdgpu-kernarg-preload-count=1 -amdgpu-attributor -S < %s | FileCheck -check-prefix=PRELOAD-1 %s
+; RUN: opt -mtriple=amdgcn-amd-amdhsa -mcpu=gfx90a -amdgpu-kernarg-preload-count=3 -amdgpu-attributor -S < %s | FileCheck -check-prefix=PRELOAD-3 %s
+; RUN: opt -mtriple=amdgcn-amd-amdhsa -mcpu=gfx90a -amdgpu-kernarg-preload-count=16 -amdgpu-attributor -S < %s | FileCheck -check-prefix=PRELOAD-16 %s
+; RUN: opt -mtriple=amdgcn-amd-amdhsa -mcpu=gfx90a -amdgpu-kernarg-preload-count=20 -amdgpu-attributor -S < %s | FileCheck -check-prefix=PRELOAD-20 %s
+
+define amdgpu_kernel void @test_preload_hint_kernel_1(ptr %0) #0 {
+; NO-PRELOAD-LABEL: define {{[^@]+}}@test_preload_hint_kernel_1
+; NO-PRELOAD-SAME: (ptr [[TMP0:%.*]]) #[[ATTR0:[0-9]+]] {
+; NO-PRELOAD-NEXT:    ret void
+;
+; PRELOAD-1-LABEL: define {{[^@]+}}@test_preload_hint_kernel_1
+; PRELOAD-1-SAME: (ptr inreg [[TMP0:%.*]]) #[[ATTR0:[0-9]+]] {
+; PRELOAD-1-NEXT:    ret void
+;
+; PRELOAD-3-LABEL: define {{[^@]+}}@test_preload_hint_kernel_1
+; PRELOAD-3-SAME: (ptr inreg [[TMP0:%.*]]) #[[ATTR0:[0-9]+]] {
+; PRELOAD-3-NEXT:    ret void
+;
+; PRELOAD-16-LABEL: define {{[^@]+}}@test_preload_hint_kernel_1
+; PRELOAD-16-SAME: (ptr inreg [[TMP0:%.*]]) #[[ATTR0:[0-9]+]] {
+; PRELOAD-16-NEXT:    ret void
+;
+; PRELOAD-20-LABEL: define {{[^@]+}}@test_preload_hint_kernel_1
+; PRELOAD-20-SAME: (ptr inreg [[TMP0:%.*]]) #[[ATTR0:[0-9]+]] {
+; PRELOAD-20-NEXT:    ret void
+;
+  ret void
+}
+
+define amdgpu_kernel void @test_preload_hint_kernel_2(i32 %0, i64 %1) #0 {
+; NO-PRELOAD-LABEL: define {{[^@]+}}@test_preload_hint_kernel_2
+; NO-PRELOAD-SAME: (i32 [[TMP0:%.*]], i64 [[TMP1:%.*]]) #[[ATTR0]] {
+; NO-PRELOAD-NEXT:    ret void
+;
+; PRELOAD-1-LABEL: define {{[^@]+}}@test_preload_hint_kernel_2
+; PRELOAD-1-SAME: (i32 inreg [[TMP0:%.*]], i64 [[TMP1:%.*]]) #[[ATTR0]] {
+; PRELOAD-1-NEXT:    ret void
+;
+; PRELOAD-3-LABEL: define {{[^@]+}}@test_preload_hint_kernel_2
+; PRELOAD-3-SAME: (i32 inreg [[TMP0:%.*]], i64 inreg [[TMP1:%.*]]) #[[ATTR0]] {
+; PRELOAD-3-NEXT:    ret void
+;
+; PRELOAD-16-LABEL: define {{[^@]+}}@test_preload_hint_kernel_2
+; PRELOAD-16-SAME: (i32 inreg [[TMP0:%.*]], i64 inreg [[TMP1:%.*]]) #[[ATTR0]] {
+; PRELOAD-16-NEXT:    ret void
+;
+; PRELOAD-20-LABEL: define {{[^@]+}}@test_preload_hint_kernel_2
+; PRELOAD-20-SAME: (i32 inreg [[TMP0:%.*]], i64 inreg [[TMP1:%.*]]) #[[ATTR0]] {
+; PRELOAD-20-NEXT:    ret void
+;
+  ret void
+}
+
+define amdgpu_kernel void @test_preload_hint_kernel_4(i32 %0, i64 %1, <2 x float> %2, ptr %3) #0 {
+; NO-PRELOAD-LABEL: define {{[^@]+}}@test_preload_hint_kernel_4
+; NO-PRELOAD-SAME: (i32 [[TMP0:%.*]], i64 [[TMP1:%.*]], <2 x float> [[TMP2:%.*]], ptr [[TMP3:%.*]]) #[[ATTR0]] {
+; NO-PRELOAD-NEXT:    ret void
+;
+; PRELOAD-1-LABEL: define {{[^@]+}}@test_preload_hint_kernel_4
+; PRELOAD-1-SAME: (i32 inreg [[TMP0:%.*]], i64 [[TMP1:%.*]], <2 x float> [[TMP2:%.*]], ptr [[TMP3:%.*]]) #[[ATTR0]] {
+; PRELOAD-1-NEXT:    ret void
+;
+; PRELOAD-3-LABEL: define {{[^@]+}}@test_preload_hint_kernel_4
+; PRELOAD-3-SAME: (i32 inreg [[TMP0:%.*]], i64 inreg [[TMP1:%.*]], <2 x float> inreg [[TMP2:%.*]], ptr [[TMP3:%.*]]) #[[ATTR0]] {
+; PRELOAD-3-NEXT:    ret void
+;
+; PRELOAD-16-LABEL: define {{[^@]+}}@test_preload_hint_kernel_4
+; PRELOAD-16-SAME: (i32 inreg [[TMP0:%.*]], i64 inreg [[TMP1:%.*]], <2 x float> inreg [[TMP2:%.*]], ptr inreg [[TMP3:%.*]]) #[[ATTR0]] {
+; PRELOAD-16-NEXT:    ret void
+;
+; PRELOAD-20-LABEL: define {{[^@]+}}@test_preload_hint_kernel_4
+; PRELOAD-20-SAME: (i32 inreg [[TMP0:%.*]], i64 inreg [[TMP1:%.*]], <2 x float> inreg [[TMP2:%.*]], ptr inreg [[TMP3:%.*]]) #[[ATTR0]] {
+; PRELOAD-20-NEXT:    ret void
+;
+  ret void
+}
+
+define amdgpu_kernel void @test_preload_hint_kernel_18(i32 %0, i64 %1, <2 x float> %2, ptr %3, i32 %4, i32 %5, i32 %6, i32 %7, i32 %8, i32 %9, i32 %10, i32 %11, i32 %12, i32 %13, i32 %14, i32 %15, i32 %16, i32 %17) #0 {
+; NO-PRELOAD-LABEL: define {{[^@]+}}@test_preload_hint_kernel_18
+; NO-PRELOAD-SAME: (i32 [[TMP0:%.*]], i64 [[TMP1:%.*]], <2 x float> [[TMP2:%.*]], ptr [[TMP3:%.*]], i32 [[TMP4:%.*]], i32 [[TMP5:%.*]], i32 [[TMP6:%.*]], i32 [[TMP7:%.*]], i32 [[TMP8:%.*]], i32 [[TMP9:%.*]], i32 [[TMP10:%.*]], i32 [[TMP11:%.*]], i32 [[TMP12:%.*]], i32 [[TMP13:%.*]], i32 [[TMP14:%.*]], i32 [[TMP15:%.*]], i32 [[TMP16:%.*]], i32 [[TMP17:%.*]]) #[[ATTR0]] {
+; NO-PRELOAD-NEXT:    ret void
+;
+; PRELOAD-1-LABEL: define {{[^@]+}}@test_preload_hint_kernel_18
+; PRELOAD-1-SAME: (i32 inreg [[TMP0:%.*]], i64 [[TMP1:%.*]], <2 x float> [[TMP2:%.*]], ptr [[TMP3:%.*]], i32 [[TMP4:%.*]], i32 [[TMP5:%.*]], i32 [[TMP6:%.*]], i32 [[TMP7:%.*]], i32 [[TMP8:%.*]], i32 [[TMP9:%.*]], i32 [[TMP10:%.*]], i32 [[TMP11:%.*]], i32 [[TMP12:%.*]], i32 [[TMP13:%.*]], i32 [[TMP14:%.*]], i32 [[TMP15:%.*]], i32 [[TMP16:%.*]], i32 [[TMP17:%.*]]) #[[ATTR0]] {
+; PRELOAD-1-NEXT:    ret void
+;
+; PRELOAD-3-LABEL: define {{[^@]+}}@test_preload_hint_kernel_18
+; PRELOAD-3-SAME: (i32 inreg [[TMP0:%.*]], i64 inreg [[TMP1:%.*]], <2 x float> inreg [[TMP2:%.*]], ptr [[TMP3:%.*]], i32 [[TMP4:%.*]], i32 [[TMP5:%.*]], i32 [[TMP6:%.*]], i32 [[TMP7:%.*]], i32 [[TMP8:%.*]], i32 [[TMP9:%.*]], i32 [[TMP10:%.*]], i32 [[TMP11:%.*]], i32 [[TMP12:%.*]], i32 [[TMP13:%.*]], i32 [[TMP14:%.*]], i32 [[TMP15:%.*]], i32 [[TMP16:%.*]], i32 [[TMP17:%.*]]) #[[ATTR0]] {
+; PRELOAD-3-NEXT:    ret void
+;
+; PRELOAD-16-LABEL: define {{[^@]+}}@test_preload_hint_kernel_18
+; PRELOAD-16-SAME: (i32 inreg [[TMP0:%.*]], i64 inreg [[TMP1:%.*]], <2 x float> inreg [[TMP2:%.*]], ptr inreg [[TMP3:%.*]], i32 inreg [[TMP4:%.*]], i32 inreg [[TMP5:%.*]], i32 inreg [[TMP6:%.*]], i32 inreg [[TMP7:%.*]], i32 inreg [[TMP8:%.*]], i32 inreg [[TMP9:%.*]], i32 inreg [[TMP10:%.*]], i32 inreg [[TMP11:%.*]], i32 inreg [[TMP12:%.*]], i32 inreg [[TMP13:%.*]], i32 inreg [[TMP14:%.*]], i32 inreg [[TMP15:%.*]], i32 [[TMP16:%.*]], i32 [[TMP17:%.*]]) #[[ATTR0]] {
+; PRELOAD-16-NEXT:    ret void
+;
+; PRELOAD-20-LABEL: define {{[^@]+}}@test_preload_hint_kernel_18
+; PRELOAD-20-SAME: (i32 inreg [[TMP0:%.*]], i64 inreg [[TMP1:%.*]], <2 x float> inreg [[TMP2:%.*]], ptr inreg [[TMP3:%.*]], i32 inreg [[TMP4:%.*]], i32 inreg [[TMP5:%.*]], i32 inreg [[TMP6:%.*]], i32 inreg [[TMP7:%.*]], i32 inreg [[TMP8:%.*]], i32 inreg [[TMP9:%.*]], i32 inreg [[TMP10:%.*]], i32 inreg [[TMP11:%.*]], i32 inreg [[TMP12:%.*]], i32 inreg [[TMP13:%.*]], i32 inreg [[TMP14:%.*]], i32 inreg [[TMP15:%.*]], i32 [[TMP16:%.*]], i32 [[TMP17:%.*]]) #[[ATTR0]] {
+; PRELOAD-20-NEXT:    ret void
+;
+  ret void
+}
+
+define void @test_preload_hint_non_kernel_2(i32 %0, i64 %1) #0 {
+; NO-PRELOAD-LABEL: define {{[^@]+}}@test_preload_hint_non_kernel_2
+; NO-PRELOAD-SAME: (i32 [[TMP0:%.*]], i64 [[TMP1:%.*]]) #[[ATTR1:[0-9]+]] {
+; NO-PRELOAD-NEXT:    ret void
+;
+; PRELOAD-1-LABEL: define {{[^@]+}}@test_preload_hint_non_kernel_2
+; PRELOAD-1-SAME: (i32 [[TMP0:%.*]], i64 [[TMP1:%.*]]) #[[ATTR1:[0-9]+]] {
+; PRELOAD-1-NEXT:    ret void
+;
+; PRELOAD-3-LABEL: define {{[^@]+}}@test_preload_hint_non_kernel_2
+; PRELOAD-3-SAME: (i32 [[TMP0:%.*]], i64 [[TMP1:%.*]]) #[[ATTR1:[0-9]+]] {
+; PRELOAD-3-NEXT:    ret void
+;
+; PRELOAD-16-LABEL: define {{[^@]+}}@test_preload_hint_non_kernel_2
+; PRELOAD-16-SAME: (i32 [[TMP0:%.*]], i64 [[TMP1:%.*]]) #[[ATTR1:[0-9]+]] {
+; PRELOAD-16-NEXT:    ret void
+;
+; PRELOAD-20-LABEL: define {{[^@]+}}@test_preload_hint_non_kernel_2
+; PRELOAD-20-SAME: (i32 [[TMP0:%.*]], i64 [[TMP1:%.*]]) #[[ATTR1:[0-9]+]] {
+; PRELOAD-20-NEXT:    ret void
+;
+  ret void
+}
+
+define amdgpu_kernel void @test_preload_hint_kernel_1_call_func(ptr %0) #0 {
+; NO-PRELOAD-LABEL: define {{[^@]+}}@test_preload_hint_kernel_1_call_func
+; NO-PRELOAD-SAME: (ptr [[TMP0:%.*]]) #[[ATTR2:[0-9]+]] {
+; NO-PRELOAD-NEXT:    call void @func(ptr [[TMP0]])
+; NO-PRELOAD-NEXT:    ret void
+;
+; PRELOAD-1-LABEL: define {{[^@]+}}@test_preload_hint_kernel_1_call_func
+; PRELOAD-1-SAME: (ptr inreg [[TMP0:%.*]]) #[[ATTR2:[0-9]+]] {
+; PRELOAD-1-NEXT:    call void @func(ptr [[TMP0]])
+; PRELOAD-1-NEXT:    ret void
+;
+; PRELOAD-3-LABEL: define {{[^@]+}}@test_preload_hint_kernel_1_call_func
+; PRELOAD-3-SAME: (ptr inreg [[TMP0:%.*]]) #[[ATTR2:[0-9]+]] {
+; PRELOAD-3-NEXT:    call void @func(ptr [[TMP0]])
+; PRELOAD-3-NEXT:    ret void
+;
+; PRELOAD-16-LABEL: define {{[^@]+}}@test_preload_hint_kernel_1_call_func
+; PRELOAD-16-SAME: (ptr inreg [[TMP0:%.*]]) #[[ATTR2:[0-9]+]] {
+; PRELOAD-16-NEXT:    call void @func(ptr [[TMP0]])
+; PRELOAD-16-NEXT:    ret void
+;
+; PRELOAD-20-LABEL: define {{[^@]+}}@test_preload_hint_kernel_1_call_func
+; PRELOAD-20-SAME: (ptr inreg [[TMP0:%.*]]) #[[ATTR2:[0-9]+]] {
+; PRELOAD-20-NEXT:    call void @func(ptr [[TMP0]])
+; PRELOAD-20-NEXT:    ret void
+;
+  call void @func(ptr %0)
+  ret void
+}
+
+define amdgpu_kernel void @test_preload_hint_kernel_1_call_intrinsic(i16 %0) #0 {
+; NO-PRELOAD-LABEL: define {{[^@]+}}@test_preload_hint_kernel_1_call_intrinsic
+; NO-PRELOAD-SAME: (i16 [[TMP0:%.*]]) #[[ATTR2]] {
+; NO-PRELOAD-NEXT:    call void @llvm.amdgcn.set.prio(i16 [[TMP0]])
+; NO-PRELOAD-NEXT:    ret void
+;
+; PRELOAD-1-LABEL: define {{[^@]+}}@test_preload_hint_kernel_1_call_intrinsic
+; PRELOAD-1-SAME: (i16 inreg [[TMP0:%.*]]) #[[ATTR2]] {
+; PRELOAD-1-NEXT:    call void @llvm.amdgcn.set.prio(i16 [[TMP0]])
+; PRELOAD-1-NEXT:    ret void
+;
+; PRELOAD-3-LABEL: define {{[^@]+}}@test_preload_hint_kernel_1_call_intrinsic
+; PRELOAD-3-SAME: (i16 inreg [[TMP0:%.*]]) #[[ATTR2]] {
+; PRELOAD-3-NEXT:    call void @llvm.amdgcn.set.prio(i16 [[TMP0]])
+; PRELOAD-3-NEXT:    ret void
+;
+; PRELOAD-16-LABEL: define {{[^@]+}}@test_preload_hint_kernel_1_call_intrinsic
+; PRELOAD-16-SAME: (i16 inreg [[TMP0:%.*]]) #[[ATTR2]] {
+; PRELOAD-16-NEXT:    call void @llvm.amdgcn.set.prio(i16 [[TMP0]])
+; PRELOAD-16-NEXT:    ret void
+;
+; PRELOAD-20-LABEL: define {{[^@]+}}@test_preload_hint_kernel_1_call_intrinsic
+; PRELOAD-20-SAME: (i16 inreg [[TMP0:%.*]]) #[[ATTR2]] {
+; PRELOAD-20-NEXT:    call void @llvm.amdgcn.set.prio(i16 [[TMP0]])
+; PRELOAD-20-NEXT:    ret void
+;
+  call void @llvm.amdgcn.set.prio(i16 %0)
+  ret void
+}
+
+define spir_kernel void @test_preload_hint_kernel_1_spir_cc(ptr %0) #0 {
+; NO-PRELOAD-LABEL: define {{[^@]+}}@test_preload_hint_kernel_1_spir_cc
+; NO-PRELOAD-SAME: (ptr [[TMP0:%.*]]) #[[ATTR0]] {
+; NO-PRELOAD-NEXT:    ret void
+;
+; PRELOAD-1-LABEL: define {{[^@]+}}@test_preload_hint_kernel_1_spir_cc
+; PRELOAD-1-SAME: (ptr [[TMP0:%.*]]) #[[ATTR0]] {
+; PRELOAD-1-NEXT:    ret void
+;
+; PRELOAD-3-LABEL: define {{[^@]+}}@test_preload_hint_kernel_1_spir_cc
+; PRELOAD-3-SAME: (ptr [[TMP0:%.*]]) #[[ATTR0]] {
+; PRELOAD-3-NEXT:    ret void
+;
+; PRELOAD-16-LABEL: define {{[^@]+}}@test_preload_hint_kernel_1_spir_cc
+; PRELOAD-16-SAME: (ptr [[TMP0:%.*]]) #[[ATTR0]] {
+; PRELOAD-16-NEXT:    ret void
+;
+; PRELOAD-20-LABEL: define {{[^@]+}}@test_preload_hint_kernel_1_spir_cc
+; PRELOAD-20-SAME: (ptr [[TMP0:%.*]]) #[[ATTR0]] {
+; PRELOAD-20-NEXT:    ret void
+;
+  ret void
+}
+
+define amdgpu_kernel void @test_preload_hint_kernel_2_preexisting(i32 inreg %0, i64 %1) #0 {
+; NO-PRELOAD-LABEL: define {{[^@]+}}@test_preload_hint_kernel_2_preexisting
+; NO-PRELOAD-SAME: (i32 inreg [[TMP0:%.*]], i64 [[TMP1:%.*]]) #[[ATTR0]] {
+; NO-PRELOAD-NEXT:    ret void
+;
+; PRELOAD-1-LABEL: define {{[^@]+}}@test_preload_hint_kernel_2_preexisting
+; PRELOAD-1-SAME: (i32 inreg [[TMP0:%.*]], i64 [[TMP1:%.*]]) #[[ATTR0]] {
+; PRELOAD-1-NEXT:    ret void
+;
+; PRELOAD-3-LABEL: define {{[^@]+}}@test_preload_hint_kernel_2_preexisting
+; PRELOAD-3-SAME: (i32 inreg [[TMP0:%.*]], i64 inreg [[TMP1:%.*]]) #[[ATTR0]] {
+; PRELOAD-3-NEXT:    ret void
+;
+; PRELOAD-16-LABEL: define {{[^@]+}}@test_preload_hint_kernel_2_preexisting
+; PRELOAD-16-SAME: (i32 inreg [[TMP0:%.*]], i64 inreg [[TMP1:%.*]]) #[[ATTR0]] {
+; PRELOAD-16-NEXT:    ret void
+;
+; PRELOAD-20-LABEL: define {{[^@]+}}@test_preload_hint_kernel_2_preexisting
+; PRELOAD-20-SAME: (i32 inreg [[TMP0:%.*]], i64 inreg [[TMP1:%.*]]) #[[ATTR0]] {
+; PRELOAD-20-NEXT:    ret void
+;
+  ret void
+}
+
+define amdgpu_kernel void @test_preload_hint_kernel_incompatible_attributes(ptr addrspace(4) byref(i32) %0, ptr nest %1) {
+; NO-PRELOAD-LABEL: define {{[^@]+}}@test_preload_hint_kernel_incompatible_attributes
+; NO-PRELOAD-SAME: (ptr addrspace(4) byref(i32) [[TMP0:%.*]], ptr nest [[TMP1:%.*]]) #[[ATTR3:[0-9]+]] {
+; NO-PRELOAD-NEXT:    ret void
+;
+; PRELOAD-1-LABEL: define {{[^@]+}}@test_preload_hint_kernel_incompatible_attributes
+; PRELOAD-1-SAME: (ptr addrspace(4) byref(i32) [[TMP0:%.*]], ptr nest [[TMP1:%.*]]) #[[ATTR3:[0-9]+]] {
+; PRELOAD-1-NEXT:    ret void
+;
+; PRELOAD-3-LABEL: define {{[^@]+}}@test_preload_hint_kernel_incompatible_attributes
+; PRELOAD-3-SAME: (ptr addrspace(4) byref(i32) [[TMP0:%.*]], ptr nest [[TMP1:%.*]]) #[[ATTR3:[0-9]+]] {
+; PRELOAD-3-NEXT:    ret void
+;
+; PRELOAD-16-LABEL: define {{[^@]+}}@test_preload_hint_kernel_incompatible_attributes
+; PRELOAD-16-SAME: (ptr addrspace(4) byref(i32) [[TMP0:%.*]], ptr nest [[TMP1:%.*]]) #[[ATTR3:[0-9]+]] {
+; PRELOAD-16-NEXT:    ret void
+;
+; PRELOAD-20-LABEL: define {{[^@]+}}@test_preload_hint_kernel_incompatible_attributes
+; PRELOAD-20-SAME: (ptr addrspace(4) byref(i32) [[TMP0:%.*]], ptr nest [[TMP1:%.*]]) #[[ATTR3:[0-9]+]] {
+; PRELOAD-20-NEXT:    ret void
+;
+  ret void
+}
+
+declare void @func(ptr) #0
+declare void @llvm.amdgcn.set.prio(i16)
+
+attributes #0 = { nounwind }