AMDGPU : Add intrinsics for compare with the full wavefront result, such as v_cmp_ne_i32, etc..

[wdng]

Summary

Add an LLVM intrinsic / Clang Builtin to expose the v_cmp_ne_i32 instruction.

[arsenm]

We should only do a general icmp, not an intrinsic for specific compare opcodes

[• tstellarAMD]

In D22482#487548, @arsenm wrote:

We should only do a general icmp, not an intrinsic for specific compare opcodes

Do you mean an LLVM IR icmp instruction or a generic intrinsic that takes a condition code as its third input?

[arsenm]

In D22482#488452, @tstellarAMD wrote:

In D22482#487548, @arsenm wrote:

We should only do a general icmp, not an intrinsic for specific compare opcodes

Do you mean an LLVM IR icmp instruction or a generic intrinsic that takes a condition code as its third input?

Yes

[wdng]

Use a general way to implement v_cmp_ne_i32.

[wdng]

Fixed a data type issue for fcmp intrinsic definition.

[arsenm]

include/llvm/IR/IntrinsicsAMDGPU.td

394

Remove the GCCBuiltins, they don't work with overloaded intrinsics

400

the 3rd parameter should be i32

lib/Target/AMDGPU/AMDGPUISelLowering.h

231

Needs a comment that this is setcc with the full mask result

lib/Target/AMDGPU/SIISelLowering.cpp

1655–1656

These should be put towards the end of the cases

1657

Variables should be capitalized and camel case.

What happens if the cond code is out of range? There should probably be a clamp

1658

Extra spaces between type and name

1659

This looks like it goes over 80 characters

lib/Target/AMDGPU/SIInstructions.td

2365–2366

This can just be a class. You can also try adding the pattern dag to the v_cmp instruction definition patterns list (although I'm not 100% sure if the multiple patterns actually work). A multiclass might help if you don't want to repeat for i32/i64

2372–2373

All compare types should be defined. Additionally i64 and the FP ones are missing

test/CodeGen/AMDGPU/llvm.amdgcn.icmp.ne.ll

6–12 ↗

(On Diff #64938)

There should be a test for every condition code and i32/i64

8 ↗

(On Diff #64938)

nounwind should also be an attribute group

9 ↗

(On Diff #64938)

Call site doesn't need the at tributes

[• tstellarAMD]

include/llvm/IR/IntrinsicsAMDGPU.td

400

Also, should the return type be i64 instead of double?

[wdng]

Code changes based on Matt's comments.

include/llvm/IR/IntrinsicsAMDGPU.td

400

Yes, code has been changed accordingly. Thanks!

[arsenm]

lib/Target/AMDGPU/AMDGPUISelLowering.h

231

Should have space after the //, and it should be capitalized and punctuated. Maybe clearer would be a compare with a result bit per item in the wavefront or something, mask result sounds more ambiguous maybe

lib/Target/AMDGPU/SIISelLowering.cpp

1909–1910

You should do the range check before the static_cast since I think it is undefined behavior to have an out of bounds enum value inserted.

This also won't work for fcmp, each should be handled in its own case with its own range check for the specific compare types' range

lib/Target/AMDGPU/SIInstructions.td

2374–2375

The unsigned should use the _U32 compare

2383–2384

Ditto

2385–2386

Ditto

2399–2411

This also needs to be done for the unordered compares

test/CodeGen/AMDGPU/llvm.amdgcn.fcmp.ll

102–104

Missing unordered compares

[wdng]

Changes based on Matt's comments.

[arsenm]

include/llvm/IR/IntrinsicsAMDGPU.td

392

You should remove this comment

397

And this one

lib/Target/AMDGPU/SIISelLowering.cpp

1907–1909

Instead of an assert, how about returning undef? this should also have a test. Same if the operand isn't really constant, you'll need to do the dyn_cast yourself

1918–1922

Should refer to FCmpInst

lib/Target/AMDGPU/SIInstructions.td

2413–2425

These are not the correct unordered comparison instructions, refer to the existing set of fcmp patterns for which to use

[• tstellarAMD]

lib/Target/AMDGPU/SIInstructions.td

2413–2425

Unordered compares should select the V_CMP_N* instructions. Take a look at the instruction definitions to see which condition matches to which instruction.

[arsenm]

test/CodeGen/AMDGPU/llvm.amdgcn.fcmp.ll

200

You should also add a test that uses fabs on the inputs to make sure that source modifiers are folded

[wdng]

1. Add dyn_cast for type converting.
2. Fixed not using correct FCmpInst type.
3. Fixed incorrectly use of unordered insutrctions
4. Added fabs as input for fcmp test.

[wdng]

Upload correct diff with cached LIT tests update.

[wdng]

Upload correct diff file.

[arsenm]

The title of the commit is also inaccurate, it should be intrinsics for compare with the full wavefront result

test/CodeGen/AMDGPU/llvm.amdgcn.fcmp.ll

201

Still missing these tests

[wdng]

test/CodeGen/AMDGPU/llvm.amdgcn.fcmp.ll

201

I have just created one "define void @v_fcmp_f32_oeq_with_fabs(i64 addrspace(1)* %out, float %src, float %a) #1" and put it one the top of tests. Should I write fabs tests for all fcmp comparisons?

[arsenm]

test/CodeGen/AMDGPU/llvm.amdgcn.fcmp.ll

7

Use the attribute group

11–16

The call site does not need the attribute specified. Can you also test the other operand? The check line should check the actual operands, this currently does not actually check much

[wdng]

Modified one LIT test to check operands.

[wdng]

Fixed corrupted diff patch.

[arsenm]

lib/Target/AMDGPU/SIISelLowering.cpp

1907

Space before =

1921

Ditto

test/CodeGen/AMDGPU/llvm.amdgcn.fcmp.ll

8

Missing test for invalid condition code value

10

You can move the |s outside of the regex and then you don't have to escape them

12

Don't need call site attributes

12–17

Still should test that both operands can have the source modifiers folded

30

it doesn't really matter, but there's no reason this test needs to under-align the stores, Fix these to be align 8 or remove the aligns

test/CodeGen/AMDGPU/llvm.amdgcn.icmp.ll

7

Missing test for invalid condition code value

[wdng]

Add illegal cond code LIT tests.

[arsenm]

lib/Target/AMDGPU/SIInstructions.td

2391–2392

Spaces before the types and the next (

test/CodeGen/AMDGPU/llvm.amdgcn.fcmp.ll

29

You should drop the suffix here to strengthen the test. It would be best to reduce to just v_cmp because something could commute the instruction

test/CodeGen/AMDGPU/llvm.amdgcn.icmp.ll

17

Ditto

[wdng]

1. Rename LIT test function names.
2. Add space between type and parenthesis.

[arsenm]

LGTM, you can drop the "such as" part from your commit message

include/llvm/IR/IntrinsicsAMDGPU.td

   GCCBuiltin<"__builtin_amdgcn_lerp">,
   Intrinsic<[llvm_i32_ty], [llvm_i32_ty, llvm_i32_ty, llvm_i32_ty], [IntrNoMem]>;
 
+// llvm.amdgcn.icmp
+def int_amdgcn_icmp :
+  Intrinsic<[llvm_i64_ty], [llvm_anyint_ty, LLVMMatchType<0>, llvm_i32_ty], 
+            [IntrNoMem, IntrConvergent]>;
+
+// llvm.amdgcn.fcmp
+def int_amdgcn_fcmp :
+  Intrinsic<[llvm_i64_ty], [llvm_anyfloat_ty, LLVMMatchType<0>, llvm_i32_ty], 
+            [IntrNoMem, IntrConvergent]>;
+
 //===----------------------------------------------------------------------===//
 // CI+ Intrinsics
 //===----------------------------------------------------------------------===//

lib/Target/AMDGPU/AMDGPUISelLowering.h

   DWORDADDR,
   FRACT,
   CLAMP,
+  SETCC,    //this is setcc with the full mask result
 
   // SIN_HW, COS_HW - f32 for SI, 1 ULP max error, valid from -100 pi to 100 pi.
   // Denormals handled on some parts.

lib/Target/AMDGPU/AMDGPUISelLowering.cpp

   NODE_NAME_CASE(RETURN)
   NODE_NAME_CASE(DWORDADDR)
   NODE_NAME_CASE(FRACT)
+  NODE_NAME_CASE(SETCC)
   NODE_NAME_CASE(CLAMP)
   NODE_NAME_CASE(COS_HW)
   NODE_NAME_CASE(SIN_HW)

lib/Target/AMDGPU/AMDGPUInstrInfo.td

 // out = (src1 > src0) ? 1 : 0
 def AMDGPUborrow : SDNode<"AMDGPUISD::BORROW", SDTIntBinOp, []>;
 
+def AMDGPUSetCCOp : SDTypeProfile<1, 3, [        // setcc
+  SDTCisVT<0, i64>, SDTCisSameAs<1, 2>, SDTCisVT<3, OtherVT>
+]>;
+
+def AMDGPUsetcc : SDNode<"AMDGPUISD::SETCC", AMDGPUSetCCOp>;
 
 def AMDGPUcvt_f32_ubyte0 : SDNode<"AMDGPUISD::CVT_F32_UBYTE0",
   SDTIntToFPOp, []>;

lib/Target/AMDGPU/SIISelLowering.cpp

 #include "llvm/CodeGen/MachineInstrBuilder.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/CodeGen/SelectionDAG.h"
+#include "llvm/CodeGen/Analysis.h"
 #include "llvm/IR/DiagnosticInfo.h"
 #include "llvm/IR/Function.h"
 

     return DAG.getNode(AMDGPUISD::DIV_SCALE, DL, Op->getVTList(), Src0,
                        Denominator, Numerator);
   }
+  case Intrinsic::amdgcn_icmp:
+  case Intrinsic::amdgcn_fcmp: {
+    ICmpInst::Predicate IcInput =
+     static_cast<ICmpInst::Predicate>(Op.getConstantOperandVal(3));
+    assert(ICmpInst::Predicate::FIRST_ICMP_PREDICATE <= IcInput &&
+         IcInput < ICmpInst::Predicate::BAD_ICMP_PREDICATE);
+    ISD::CondCode CCOpcode = getICmpCondCode(IcInput);
+    return DAG.getNode(AMDGPUISD::SETCC, DL, VT, Op.getOperand(1), 
+                       Op.getOperand(2), DAG.getCondCode(CCOpcode));
+  }
   default:
     return AMDGPUTargetLowering::LowerOperation(Op, DAG);
   }

lib/Target/AMDGPU/SIInstructions.td

   (DS_SWIZZLE_B32 $src, (as_i16imm $offset16), (i1 0))
 >;
 
+
+//===----------------------------------------------------------------------===//
+// V_ICMPIntrinsic Pattern.
+//===----------------------------------------------------------------------===//
+
+class ICMP_Pattern <PatLeaf cond, Instruction inst, ValueType vt> : Pat <
+  (AMDGPUsetcc vt:$src0, vt:$src1, cond),
+  (inst $src0, $src1)
+>;
+
+def : ICMP_Pattern <COND_EQ, V_CMP_EQ_I32_e64, i32>;
+def : ICMP_Pattern <COND_NE, V_CMP_NE_I32_e64, i32>;
+def : ICMP_Pattern <COND_UGT, V_CMP_GT_I32_e64, i32>;
+def : ICMP_Pattern <COND_UGE, V_CMP_GE_I32_e64, i32>;
+def : ICMP_Pattern <COND_ULT, V_CMP_LT_I32_e64, i32>;
+def : ICMP_Pattern <COND_ULE, V_CMP_LE_I32_e64, i32>;
+def : ICMP_Pattern <COND_SGT, V_CMP_GT_I32_e64, i32>;
+def : ICMP_Pattern <COND_SGE, V_CMP_GE_I32_e64, i32>;
+def : ICMP_Pattern <COND_SLT, V_CMP_LT_I32_e64, i32>;
+def : ICMP_Pattern <COND_SLE, V_CMP_LE_I32_e64, i32>;
+
+def : ICMP_Pattern <COND_EQ, V_CMP_EQ_I64_e64, i64>;
+def : ICMP_Pattern <COND_NE, V_CMP_NE_I64_e64, i64>;
+def : ICMP_Pattern <COND_UGT, V_CMP_GT_I64_e64, i64>;
+def : ICMP_Pattern <COND_UGE, V_CMP_GE_I64_e64, i64>;
+def : ICMP_Pattern <COND_ULT, V_CMP_LT_I64_e64, i64>;
+def : ICMP_Pattern <COND_ULE, V_CMP_LE_I64_e64, i64>;
+def : ICMP_Pattern <COND_SGT, V_CMP_GT_I64_e64, i64>;
+def : ICMP_Pattern <COND_SGE, V_CMP_GE_I64_e64, i64>;
+def : ICMP_Pattern <COND_SLT, V_CMP_LT_I64_e64, i64>;
+def : ICMP_Pattern <COND_SLE, V_CMP_LE_I64_e64, i64>;
+
+class FCMP_Pattern <PatLeaf cond, Instruction inst, ValueType vt> : Pat <
+  (i64(AMDGPUsetcc (vt(VOP3Mods vt:$src0, i32:$src0_modifiers)),
+                   (vt(VOP3Mods vt:$src1, i32:$src1_modifiers)), cond)),
+  (inst $src0_modifiers, $src0, $src1_modifiers, $src1, 
+        DSTCLAMP.NONE, DSTOMOD.NONE)
+>;
+
+def : FCMP_Pattern <COND_OEQ, V_CMP_EQ_F32_e64, f32>;
+def : FCMP_Pattern <COND_ONE, V_CMP_NEQ_F32_e64, f32>;
+def : FCMP_Pattern <COND_OGT, V_CMP_GT_F32_e64, f32>;
+def : FCMP_Pattern <COND_OGE, V_CMP_GE_F32_e64, f32>;
+def : FCMP_Pattern <COND_OLT, V_CMP_LT_F32_e64, f32>;
+def : FCMP_Pattern <COND_OLE, V_CMP_LE_F32_e64, f32>;
+
+def : FCMP_Pattern <COND_OEQ, V_CMP_EQ_F64_e64, f64>;
+def : FCMP_Pattern <COND_ONE, V_CMP_NEQ_F64_e64, f64>;
+def : FCMP_Pattern <COND_OGT, V_CMP_GT_F64_e64, f64>;
+def : FCMP_Pattern <COND_OGE, V_CMP_GE_F64_e64, f64>;
+def : FCMP_Pattern <COND_OLT, V_CMP_LT_F64_e64, f64>;
+def : FCMP_Pattern <COND_OLE, V_CMP_LE_F64_e64, f64>;
+
 //===----------------------------------------------------------------------===//
 // SMRD Patterns
 //===----------------------------------------------------------------------===//

test/CodeGen/AMDGPU/llvm.amdgcn.fcmp.ll

+; RUN: llc -march=amdgcn -verify-machineinstrs < %s | FileCheck -check-prefix=GCN %s
+; RUN: llc -march=amdgcn -mcpu=fiji -verify-machineinstrs < %s | FileCheck -check-prefix=GCN %s
+
+declare i64 @llvm.amdgcn.fcmp.f32(float, float, i32) #0
+declare i64 @llvm.amdgcn.fcmp.f64(double, double, i32) #0
+
+; GCN-LABEL: {{^}}v_fcmp_f32_eq:
+; GCN: v_cmp_eq_f32_e64
+define void @v_fcmp_f32_eq(i64 addrspace(1)* %out, float %src) #1 {
+  %result = call i64 @llvm.amdgcn.fcmp.f32(float %src, float 100.00, i32 32)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_fcmp_f32_ne:
+; GCN: v_cmp_neq_f32_e64
+define void @v_fcmp_f32_ne(i64 addrspace(1)* %out, float %src) #1 {
+  %result = call i64 @llvm.amdgcn.fcmp.f32(float %src, float 100.00, i32 33)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_fcmp_f32_ogt:
+; GCN: v_cmp_gt_f32_e64
+define void @v_fcmp_f32_ogt(i64 addrspace(1)* %out, float %src) #1 {
+  %result = call i64 @llvm.amdgcn.fcmp.f32(float %src, float 100.00, i32 38)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_fcmp_f32_oge:
+; GCN: v_cmp_ge_f32_e64
+define void @v_fcmp_f32_oge(i64 addrspace(1)* %out, float %src) #1 {
+  %result = call i64 @llvm.amdgcn.fcmp.f32(float %src, float 100.00, i32 39)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_fcmp_f32_olt:
+; GCN: v_cmp_lt_f32_e64
+define void @v_fcmp_f32_olt(i64 addrspace(1)* %out, float %src) #1 {
+  %result = call i64 @llvm.amdgcn.fcmp.f32(float %src, float 100.00, i32 40)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_fcmp_f32_ole:
+; GCN: v_cmp_le_f32_e64
+define void @v_fcmp_f32_ole(i64 addrspace(1)* %out, float %src) #1 {
+  %result = call i64 @llvm.amdgcn.fcmp.f32(float %src, float 100.00, i32 41)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_fcmp_f64_eq:
+; GCN: v_cmp_eq_f64_e64
+define void @v_fcmp_f64_eq(i64 addrspace(1)* %out, double %src) #1 {
+  %result = call i64 @llvm.amdgcn.fcmp.f64(double %src, double 100.00, i32 32)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_fcmp_f64_ne:
+; GCN: v_cmp_neq_f64_e64
+define void @v_fcmp_f64_ne(i64 addrspace(1)* %out, double %src) #1 {
+  %result = call i64 @llvm.amdgcn.fcmp.f64(double %src, double 100.00, i32 33)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_fcmp_f64_ogt:
+; GCN: v_cmp_gt_f64_e64
+define void @v_fcmp_f64_ogt(i64 addrspace(1)* %out, double %src) #1 {
+  %result = call i64 @llvm.amdgcn.fcmp.f64(double %src, double 100.00, i32 38)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_fcmp_f64_oge:
+; GCN: v_cmp_ge_f64_e64
+define void @v_fcmp_f64_oge(i64 addrspace(1)* %out, double %src) #1 {
+  %result = call i64 @llvm.amdgcn.fcmp.f64(double %src, double 100.00, i32 39)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_fcmp_f64_olt:
+; GCN: v_cmp_lt_f64_e64
+define void @v_fcmp_f64_olt(i64 addrspace(1)* %out, double %src) #1 {
+  %result = call i64 @llvm.amdgcn.fcmp.f64(double %src, double 100.00, i32 40)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_fcmp_f64_ole:
+; GCN: v_cmp_le_f64_e64
+define void @v_fcmp_f64_ole(i64 addrspace(1)* %out, double %src) #1 {
+  %result = call i64 @llvm.amdgcn.fcmp.f64(double %src, double 100.00, i32 41)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+
+
+
+
+attributes #0 = { nounwind readnone convergent }
+attributes #1 = { nounwind }

test/CodeGen/AMDGPU/llvm.amdgcn.icmp.ll

+; RUN: llc -march=amdgcn -verify-machineinstrs < %s | FileCheck -check-prefix=GCN %s
+; RUN: llc -march=amdgcn -mcpu=fiji -verify-machineinstrs < %s | FileCheck -check-prefix=GCN %s
+
+declare i64 @llvm.amdgcn.icmp.i32(i32, i32, i32) #0
+declare i64 @llvm.amdgcn.icmp.i64(i64, i64, i32) #0
+
+; GCN-LABEL: {{^}}v_icmp_i32_eq:
+; GCN: v_cmp_eq_i32_e64
+define void @v_icmp_i32_eq(i64 addrspace(1)* %out, i32 %src) #1 {
+  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 32)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_icmp_i32_ne:
+; GCN: v_cmp_ne_i32_e64
+define void @v_icmp_i32_ne(i64 addrspace(1)* %out, i32 %src) #1 {
+  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 33)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_icmp_i32_ugt:
+; GCN: v_cmp_gt_i32_e64
+define void @v_icmp_i32_ugt(i64 addrspace(1)* %out, i32 %src) #1 {
+  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 34)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_icmp_i32_uge:
+; GCN: v_cmp_ge_i32_e64
+define void @v_icmp_i32_uge(i64 addrspace(1)* %out, i32 %src) #1 {
+  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 35)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_icmp_i32_ult:
+; GCN: v_cmp_lt_i32_e64
+define void @v_icmp_i32_ult(i64 addrspace(1)* %out, i32 %src) #1 {
+  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 36)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_icmp_i32_ule:
+; GCN: v_cmp_le_i32_e64
+define void @v_icmp_i32_ule(i64 addrspace(1)* %out, i32 %src) #1 {
+  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 37)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_icmp_i32_sgt:
+; GCN: v_cmp_gt_i32_e64
+define void @v_icmp_i32_sgt(i64 addrspace(1)* %out, i32 %src) #1 {
+  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 38)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_icmp_i32_sge:
+; GCN: v_cmp_ge_i32_e64
+define void @v_icmp_i32_sge(i64 addrspace(1)* %out, i32 %src) #1 {
+  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 39)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_icmp_i32_slt:
+; GCN: v_cmp_lt_i32_e64
+define void @v_icmp_i32_slt(i64 addrspace(1)* %out, i32 %src) #1 {
+  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 40)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+; GCN-LABEL: {{^}}v_icmp_i32_sle:
+; GCN: v_cmp_le_i32_e64
+define void @v_icmp_i32_sle(i64 addrspace(1)* %out, i32 %src) #1 {
+  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 41)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_icmp_i64_eq:
+; GCN: v_cmp_eq_i64_e64
+define void @v_icmp_i64_eq(i64 addrspace(1)* %out, i64 %src) #1 {
+  %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 32)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_icmp_i64_ne:
+; GCN: v_cmp_ne_i64_e64
+define void @v_icmp_i64_ne(i64 addrspace(1)* %out, i64 %src) #1 {
+  %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 33)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_icmp_i64_ugt:
+; GCN: v_cmp_gt_i64_e64
+define void @v_icmp_i64_ugt(i64 addrspace(1)* %out, i64 %src) #1 {
+  %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 34)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_icmp_i64_uge:
+; GCN: v_cmp_ge_i64_e64
+define void @v_icmp_i64_uge(i64 addrspace(1)* %out, i64 %src) #1 {
+  %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 35)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_icmp_i64_ult:
+; GCN: v_cmp_lt_i64_e64
+define void @v_icmp_i64_ult(i64 addrspace(1)* %out, i64 %src) #1 {
+  %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 36)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_icmp_i64_ule:
+; GCN: v_cmp_le_i64_e64
+define void @v_icmp_i64_ule(i64 addrspace(1)* %out, i64 %src) #1 {
+  %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 37)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_icmp_i64_sgt:
+; GCN: v_cmp_gt_i64_e64
+define void @v_icmp_i64_sgt(i64 addrspace(1)* %out, i64 %src) #1 {
+  %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 38)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_icmp_i64_sge:
+; GCN: v_cmp_ge_i64_e64
+define void @v_icmp_i64_sge(i64 addrspace(1)* %out, i64 %src) #1 {
+  %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 39)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_icmp_i64_slt:
+; GCN: v_cmp_lt_i64_e64
+define void @v_icmp_i64_slt(i64 addrspace(1)* %out, i64 %src) #1 {
+  %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 40)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+; GCN-LABEL: {{^}}v_icmp_i64_sle:
+; GCN: v_cmp_le_i64_e64
+define void @v_icmp_i64_sle(i64 addrspace(1)* %out, i64 %src) #1 {
+  %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 41)
+  store i64 %result, i64 addrspace(1)* %out, align 4
+  ret void
+}
+
+attributes #0 = { nounwind readnone convergent }
+attributes #1 = { nounwind }