[AMDGPU] add LDS f32 intrinsics

[dfukalov]

Summary

added llvm.amdgcn.atomic.{add|min|max}.f32 intrinsics
to allow generate ds_{add|min|max}[_rtn]_f32 instructions
needed for OpenCL float atomics in LDS

[rampitec]

LGTM

[arsenm]

Since it's atomic it needs to be added to isIntrinsicSourceOfDivergence. Also getTgtMemIntrinsic, and getAddrModeArguments (all with associated tests)

include/llvm/IR/IntrinsicsAMDGPU.td

303

Should this have an operand added for the ordering?

304

This is certainly not IntrNoReturn

[dfukalov]

include/llvm/IR/IntrinsicsAMDGPU.td

303

No, these intrinsics are created by request to be able to generate ds_{add|min|max}[_rtn]_f32 in case of OpenCL local memory atomics only. They work only for pointers to floats located in addrspace 3

304

You're right, I interpreted it as intrinsic that has no return value, diff updated

[arsenm]

include/llvm/IR/IntrinsicsAMDGPU.td

303

That doesn't change the ordering. Also needs an operand for volatile

[t-tye]

include/llvm/IR/IntrinsicsAMDGPU.td

303

How are the memory ordering, memory scope and volatile carried through so that those fields can be set in the Machine Memory Operand? All these properties are needed to generate the correct waitcnt in the memory legalizer (see AMDGPUUsage.rst section on memory model).

[arsenm]

include/llvm/IR/IntrinsicsAMDGPU.td

303

Those are supposed to be handled by the intrinsic callbacks I mentioned that need to be implemented (at least for volatile). I'm not sure anything really correctly considers the atomic scope for possibly atomic intrinsics.

The most similar case we have is for amdgcn_atomic_inc/dec.

[t-tye]

include/llvm/IR/IntrinsicsAMDGPU.td

300

Need to add the same fields as for AMDGPUAtomicIncIntrin, namely:

llvm_i32_ty, // ordering
llvm_i32_ty, // scope
llvm_i1_ty], // isVolatile

305

Need to add the same fields as for AMDGPUAtomicIncIntrin, namely:

llvm_i32_ty, // ordering
llvm_i32_ty, // scope
llvm_i1_ty], // isVolatile

lib/Target/AMDGPU/DSInstructions.td

619–628

Also, somehow a MachineMemoryOperand needs to be created with values from the ordering, scope and isVolatile LLVM IR instruction operands.

[arsenm]

This also should have an intermediate node added, so it is easy to add it to the various switches over memory operations (e.g. we look for all memory operations in PerformDAGCombine)

[dfukalov]

Am I right that since we should have almost the same processing as atomic inc intrinsics,
it would be better idea to define ds_add/min/max intrinsics the same way as AMDGPUAtomicIncIntrin (or unify them),
and then update AMDGPU BE to correctly process these ds_ instrisics the same ways as atomic inc/dec?

[dfukalov]

updates as requested by reviewers

[dfukalov]

ping

[arsenm]

include/llvm/IR/IntrinsicsAMDGPU.td

306

These should probably be named fadd.. etc to match the IR operations

lib/Target/AMDGPU/SIISelLowering.cpp

574–576 ↗

(On Diff #126743)

This will need to be rebased since I just changed these last week

6549–6552 ↗

(On Diff #126743)

Tests for these with this combine would be nice

test/CodeGen/AMDGPU/lds_atomic_f32.ll

1 ↗

(On Diff #126743)

Can you also add a pre-gfx9 run line, and check for the m0 initialization

[dfukalov]

diff updated according to latest comments

[arsenm]

LGTM except for the mystery name mangling in the test

test/CodeGen/AMDGPU/lds_atomic_f32.ll

4 ↗

(On Diff #129829)

These weren't declared as mangled, so why do these have the .f32? I'm kind of surprised this is accepted

include/llvm/IR/IntrinsicsAMDGPU.td

class AMDGPUAtomicIncIntrin : Intrinsic<[llvm_anyint_ty],
   llvm_i32_ty, // scope
   llvm_i1_ty], // isVolatile
   [IntrArgMemOnly, NoCapture<0>]
 >;
 
 def int_amdgcn_atomic_inc : AMDGPUAtomicIncIntrin;
 def int_amdgcn_atomic_dec : AMDGPUAtomicIncIntrin;
 
+class AMDGPUAtomicF32Intrin : Intrinsic<[llvm_float_ty],
+    [LLVMQualPointerType<llvm_float_ty, 3>, llvm_float_ty],
+    [IntrArgMemOnly, NoCapture<0>]
+>;
+
+class AMDGPUAtomicF32IntrinNORET : Intrinsic<[],
+    [LLVMQualPointerType<llvm_float_ty, 3>, llvm_float_ty],
+    [IntrArgMemOnly, NoCapture<0>]
+>;
+
+def int_amdgcn_ds_add_rtn_f32 : AMDGPUAtomicF32Intrin;
+def int_amdgcn_ds_min_rtn_f32 : AMDGPUAtomicF32Intrin;
+def int_amdgcn_ds_max_rtn_f32 : AMDGPUAtomicF32Intrin;
+def int_amdgcn_ds_add_f32 :  AMDGPUAtomicF32IntrinNORET;
+def int_amdgcn_ds_min_f32 :  AMDGPUAtomicF32IntrinNORET;
+def int_amdgcn_ds_max_f32 :  AMDGPUAtomicF32IntrinNORET;
+
 class AMDGPUImageLoad : Intrinsic <
   [llvm_anyfloat_ty], // vdata(VGPR)
   [llvm_anyint_ty,    // vaddr(VGPR)
    llvm_anyint_ty,    // rsrc(SGPR)
    llvm_i32_ty,       // dmask(imm)
    llvm_i1_ty,        // glc(imm)
    llvm_i1_ty,        // slc(imm)
    llvm_i1_ty,        // lwe(imm)

lib/Target/AMDGPU/DSInstructions.td

 def DS_DEC_U32        : DS_1A1D_NORET<"ds_dec_u32">;
 def DS_MIN_I32        : DS_1A1D_NORET<"ds_min_i32">;
 def DS_MAX_I32        : DS_1A1D_NORET<"ds_max_i32">;
 def DS_MIN_U32        : DS_1A1D_NORET<"ds_min_u32">;
 def DS_MAX_U32        : DS_1A1D_NORET<"ds_max_u32">;
 def DS_AND_B32        : DS_1A1D_NORET<"ds_and_b32">;
 def DS_OR_B32         : DS_1A1D_NORET<"ds_or_b32">;
 def DS_XOR_B32        : DS_1A1D_NORET<"ds_xor_b32">;
-def DS_ADD_F32        : DS_1A1D_NORET<"ds_add_f32">;
 def DS_MIN_F32        : DS_1A1D_NORET<"ds_min_f32">;
 def DS_MAX_F32        : DS_1A1D_NORET<"ds_max_f32">;
 
 let mayLoad = 0 in {
 def DS_WRITE_B8       : DS_1A1D_NORET<"ds_write_b8">;
 def DS_WRITE_B16      : DS_1A1D_NORET<"ds_write_b16">;
 def DS_WRITE_B32      : DS_1A1D_NORET<"ds_write_b32">;
 def DS_WRITE2_B32     : DS_1A2D_Off8_NORET<"ds_write2_b32">;

 }
 def DS_CMPST_B64      : DS_1A2D_NORET<"ds_cmpst_b64", VReg_64>;
 def DS_CMPST_F64      : DS_1A2D_NORET<"ds_cmpst_f64", VReg_64>;
 def DS_MIN_F64        : DS_1A1D_NORET<"ds_min_f64", VReg_64>;
 def DS_MAX_F64        : DS_1A1D_NORET<"ds_max_f64", VReg_64>;
 
 def DS_ADD_RTN_U32    : DS_1A1D_RET<"ds_add_rtn_u32">,
                         AtomicNoRet<"ds_add_u32", 1>;
-def DS_ADD_RTN_F32    : DS_1A1D_RET<"ds_add_rtn_f32">,
-                        AtomicNoRet<"ds_add_f32", 1>;
 def DS_SUB_RTN_U32    : DS_1A1D_RET<"ds_sub_rtn_u32">,
                         AtomicNoRet<"ds_sub_u32", 1>;
 def DS_RSUB_RTN_U32   : DS_1A1D_RET<"ds_rsub_rtn_u32">,
                         AtomicNoRet<"ds_rsub_u32", 1>;
 def DS_INC_RTN_U32    : DS_1A1D_RET<"ds_inc_rtn_u32">,
                         AtomicNoRet<"ds_inc_u32", 1>;
 def DS_DEC_RTN_U32    : DS_1A1D_RET<"ds_dec_rtn_u32">,
                         AtomicNoRet<"ds_dec_u32", 1>;

 
 let Uses = [EXEC] in {
 def DS_PERMUTE_B32  : DS_1A1D_PERMUTE <"ds_permute_b32",
                                        int_amdgcn_ds_permute>;
 def DS_BPERMUTE_B32 : DS_1A1D_PERMUTE <"ds_bpermute_b32",
                                        int_amdgcn_ds_bpermute>;
 }
 
+def DS_ADD_RTN_F32    : DS_1A1D_RET<"ds_add_rtn_f32">,
+                        AtomicNoRet<"ds_add_f32", 1>;
+def DS_ADD_F32        : DS_1A1D_NORET<"ds_add_f32">;
+
 } // let SubtargetPredicate = isVI
 
 //===----------------------------------------------------------------------===//
 // DS Patterns
 //===----------------------------------------------------------------------===//
 
 let Predicates = [isGCN] in {
 

 def : DSAtomicRetPat<DS_OR_RTN_B32, i32, si_atomic_load_or_local>;
 def : DSAtomicRetPat<DS_XOR_RTN_B32, i32, si_atomic_load_xor_local>;
 def : DSAtomicRetPat<DS_MIN_RTN_I32, i32, si_atomic_load_min_local>;
 def : DSAtomicRetPat<DS_MAX_RTN_I32, i32, si_atomic_load_max_local>;
 def : DSAtomicRetPat<DS_MIN_RTN_U32, i32, si_atomic_load_umin_local>;
 def : DSAtomicRetPat<DS_MAX_RTN_U32, i32, si_atomic_load_umax_local>;
 def : DSAtomicCmpXChg<DS_CMPST_RTN_B32, i32, si_atomic_cmp_swap_32_local>;
 
+multiclass DSAtomicPatF32<string op> {
+  def : Pat <
+    (!cast<Intrinsic>(NAME#"_rtn_f32") (DS1Addr1Offset i32:$ptr), f32:$value),
+    (!cast<DS_1A1D_RET>(op#"_RTN_F32") $ptr, $value, (i16 0), (i1 0))
+  >;
+  def : Pat <
+    (!cast<Intrinsic>(NAME#"_f32") (DS1Addr1Offset i32:$ptr), f32:$value),
+    (!cast<DS_1A1D_NORET>(op#"_F32") $ptr, $value, (i16 0), (i1 0))
+  >;
+}
+
+defm int_amdgcn_ds_add : DSAtomicPatF32<"DS_ADD">;
+defm int_amdgcn_ds_min : DSAtomicPatF32<"DS_MIN">;
+defm int_amdgcn_ds_max : DSAtomicPatF32<"DS_MAX">;
+
 // 64-bit atomics.
 def : DSAtomicRetPat<DS_WRXCHG_RTN_B64, i64, si_atomic_swap_local>;
 def : DSAtomicRetPat<DS_ADD_RTN_U64, i64, si_atomic_load_add_local>;
 def : DSAtomicRetPat<DS_SUB_RTN_U64, i64, si_atomic_load_sub_local>;
 def : DSAtomicRetPat<DS_INC_RTN_U64, i64, si_atomic_inc_local>;
 def : DSAtomicRetPat<DS_DEC_RTN_U64, i64, si_atomic_dec_local>;
 def : DSAtomicRetPat<DS_AND_RTN_B64, i64, si_atomic_load_and_local>;
 def : DSAtomicRetPat<DS_OR_RTN_B64, i64, si_atomic_load_or_local>;

test/CodeGen/AMDGPU/llvm.amdgcn.ds.f32.ll

+; RUN: llc -mtriple=amdgcn--amdhsa -mcpu=fiji -verify-machineinstrs < %s | FileCheck %s
+
+declare float @llvm.amdgcn.ds.add.rtn.f32(float addrspace(3)*, float) #0
+declare void @llvm.amdgcn.ds.add.f32(float addrspace(3)*, float) #0
+declare float @llvm.amdgcn.ds.min.rtn.f32(float addrspace(3)*, float) #0
+declare void @llvm.amdgcn.ds.min.f32(float addrspace(3)*, float) #0
+declare float @llvm.amdgcn.ds.max.rtn.f32(float addrspace(3)*, float) #0
+declare void @llvm.amdgcn.ds.max.f32(float addrspace(3)*, float) #0
+
+; FUNC-LABEL: {{^}}ds_f32:
+; CHECK: ds_add_f32 v{{[0-9]+}}, v{{[0-9]+}}
+; CHECK: ds_add_rtn_f32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
+; CHECK: ds_min_f32 v{{[0-9]+}}, v{{[0-9]+}}
+; CHECK: ds_min_rtn_f32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
+; CHECK: ds_max_f32 v{{[0-9]+}}, v{{[0-9]+}}
+; CHECK: ds_max_rtn_f32 v{{[0-9]+}}, v{{[0-9]+}}, v{{[0-9]+}}
+define amdgpu_kernel void @ds_f32(float addrspace(1)* %out, float addrspace(3)* %src1, float %src2) nounwind {
+  call void @llvm.amdgcn.ds.add.f32(float addrspace(3)* %src1, float %src2)
+  %res = call float @llvm.amdgcn.ds.add.rtn.f32(float addrspace(3)* %src1, float %src2)
+  call void @llvm.amdgcn.ds.min.f32(float addrspace(3)* %src1, float %res)
+  %res2 = call float @llvm.amdgcn.ds.min.rtn.f32(float addrspace(3)* %src1, float %res)
+  call void @llvm.amdgcn.ds.max.f32(float addrspace(3)* %src1, float %res2)
+  %res3 = call float @llvm.amdgcn.ds.max.rtn.f32(float addrspace(3)* %src1, float %res2)
+  store float %res3, float addrspace(1)* %out, align 4
+  ret void
+}
+
+attributes #0 = { nounwind argmemonly }