[AMDGPU][GISel] Add llvm.amdgcn.icmp selection

[Pierre-vh]

Summary

Add missing logic to select i16 variants and enable GISel testing.

[Pierre-vh]

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

1133–1143

The test won't pass on, say, hawaii because of lack of 16 bit insts. It'll fail to select.
I guess it's a separate issue where we should probably legalize the intrinsic (= widen its input to i32) for those archs?

1138

Note: not sure if "true" or "false" 16 bit insts matter.
I did both to be safe but I'd be curious to know if it's actually needed.

llvm/lib/Target/AMDGPU/SIInstructions.td

893 ↗

(On Diff #469577)

Help needed here, not sure how to get this one to work. I tried a lot of things, including COPY_TO_REGCLASS and nothing seems to do the trick.
Due to the simplicity of the pattern I'm wondering if it isn't better to just do it manually in the InstructionSelector rather than fight the GISel TableGen emitter?

[foad]

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

1133–1143

I think failing to select is the correct behaviour - or diagnose it as "unsupported" in the legalizer (we do this in some cases but not consistently). Generally intrinsics like this map one-to-one onto a machine instruction, and are not designed to be supported on subtargets that do not have the instruction.

[foad]

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

1138

I think both are needed here. The "true" ones will be used on GFX11 and the "fake"(!) ones on previous subtargets.

[arsenm]

llvm/lib/Target/AMDGPU/SIInstructions.td

893 ↗

(On Diff #469577)

What's the error with COPY_TO_REGCLASS?

[Pierre-vh]

llvm/lib/Target/AMDGPU/SIInstructions.td

893 ↗

(On Diff #469577)

IIRC with COPY_TO_REGCLASS it's a type issue, or if I put some regclass that works then it just doesn't match in the instruction selector.
I think my question is more: What regclass do I need to use there?

[Pierre-vh]

Rebase, fix test

[Joe_Nash]

llvm/lib/Target/AMDGPU/SIInstructions.td

893 ↗

(On Diff #469577)

Just a guess, but maybe the patterns need to be copied, one for wave32 and one for wave64? So on wave32 you do COPY_TO_REGCLASS to SReg_32_XM0_XEXEC, and for wave64 SReg_64_XM0_XEXEC

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.icmp.ll

0

I would prefer a check-prefix other than DAG. I believe this could be confused with https://llvm.org/docs/CommandGuide/FileCheck.html#the-check-dag-directive.

Can you also add a runline for gfx1100?

[foad]

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.icmp.ll

0

"SDAG" is quite common

[Pierre-vh]

llvm/lib/Target/AMDGPU/SIInstructions.td

893 ↗

(On Diff #469577)

The input pattern also needs a registerclass, that's the one I'm stuck on now
I tried (COPY_TO_REGCLASS SReg_1_XEXEC:$src, SReg_64_XEXEC) for Wave64 and it causes a "noreg" to be selected which crashes the backend

$vgpr1 = V_MOV_B32_e32 $noreg, implicit $exec, implicit killed $sgpr2, implicit $exec

VReg_1 cannot be used either, it says it's not a recognized class. I think it's not available in TableGen?

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.icmp.ll

0

For GFX11, is it fine if I just use that for the GFX run line (instead of no cpu) or is a third line needed?

[Joe_Nash]

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.icmp.ll

0

SDAG sounds good.

0

I don't think there is a need to be stingy with which tests are run. It seems good to test both no cpu (GFX6?) and gfx1100.

[Pierre-vh]

Comments

[Pierre-vh]

I think that for the icmp i1 case there's some problem in the RegBankSelect or the legalizer. I can't seem to find the right pattern.
My current best attempt, something like this:

def : Pat <
  (i64 (int_amdgcn_icmp i1:$src, (i1 0), (i32 33))),
  (COPY_TO_REGCLASS SCSrc_i1:$src, SReg_1_XEXEC) // Return the SGPRs representing i1 src
>;

def : Pat <
  (i32 (int_amdgcn_icmp i1:$src, (i1 0), (i32 33))),
  (COPY_TO_REGCLASS SCSrc_i1:$src, SReg_1_XEXEC) // Return the SGPRs representing i1 src
>;

Selects:

Selecting: 
  %23:sreg_64(s64) = G_INTRINSIC intrinsic(@llvm.amdgcn.icmp), %32:vgpr(s1), %33:vgpr(s1), 33
 Into:
  %23:sreg_64_xexec(s64) = COPY %32:vgpr(s1)

But later, even in wave64 mode, %32 gets constrained to a vgpr_32 and we get an impossible copy:

Selecting: 
  %32:vgpr(s1) = COPY %22:sgpr(s1)
Into:
  %32:vgpr_32(s1) = COPY %22:sreg_32(s1)

; later
  %31:sreg_32 = S_AND_B32 %27:sreg_32, %28:sreg_32, implicit-def $scc
  %32:vgpr_32 = COPY %31:sreg_32
  %23:sreg_64_xexec = COPY %32:vgpr_32

I'm really not sure how to deal with this. Thoughts?

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.icmp.ll

0

For GISel, with no CPU, I get

LLVM ERROR: cannot select: G_STORE %17:vgpr(s64), %9:sgpr(p1) :: (store (s64) into %ir.out.load, addrspace 1) (in function: v_icmp_i32_eq)

So I use GFX900 for both instead

[Pierre-vh]

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.icmp.ll

0

Also, for gfx 11 I had to use wave64 mode. All intrinsics in this test are the w64 variants (return i64 mask).
Should I add a w32 duplicate of the test (in a pre-patch)? Or maybe just duplicating a couple of functions in this test for w32 would be enough?

[Pierre-vh]

Add manual selection for i1 (I think it's currently incorrect, need feedback)
Add wave32 test

[Pierre-vh]

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

1295

This is obviously wrong as we only want to compare the first bit in each VGPR, but I don't know what instruction to use here yet.

Which instruction do I need to use here? V_CNDMASK ?

[Pierre-vh]

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

1295

CNDMASK doesn't work because it constrains the operands to 64 bits, but the i1 are widened to 32.
We need something that would do dst[threadI] = v0.i1, if I understand correctly?

[Pierre-vh]

Rename tests

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

1295

Where are you seeing i1 VGPRs? Those should just not happen. At one point I might have been trying to handle that, but I think it's basically wrong for it to see the selector

[Pierre-vh]

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

1295

I don't see i1 VGPRs, but vgpr(s1) as the arguments of the iccmp

[Joe_Nash]

llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-icmp.s16.mir

28

Why has the dst class been changed to sreg_32? These instructions should not write to m0 or exec

[Pierre-vh]

llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-icmp.s16.mir

28

Dst is constrained to getBoolRC. If we want xexec we need to use getWaveMaskRegClass instead
Not sure which one is technically correct? Isn't sreg_32 more correct because the dst is always a sgpr, and never exec/m0?

[arsenm]

LGTM except for trying to handle i1 intrinsics

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

1295

That shouldn't happen. The intrinsic shouldn't accept incoming i1 values. If you try to use i1 with these intrinsics, it should just fail selection

[Pierre-vh]

Don't support i1 inputs

Not sure how we want to handle this in a merged test given that the DAG supports them. I used global-isel-abort=2

[arsenm]

I'm surprised i1 works for DAG

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

case Intrinsic::amdgcn_wwm:
     return constrainCopyLikeIntrin(I, AMDGPU::STRICT_WWM);
   case Intrinsic::amdgcn_strict_wqm:
     return constrainCopyLikeIntrin(I, AMDGPU::STRICT_WQM);
   case Intrinsic::amdgcn_writelane:
     return selectWritelane(I);
   case Intrinsic::amdgcn_div_scale:
     return selectDivScale(I);
   case Intrinsic::amdgcn_icmp:
+    if (selectImpl(I, *CoverageInfo))
+      return true;
     return selectIntrinsicIcmp(I);
   case Intrinsic::amdgcn_ballot:
     return selectBallot(I);
   case Intrinsic::amdgcn_reloc_constant:
     return selectRelocConstant(I);
   case Intrinsic::amdgcn_groupstaticsize:
     return selectGroupStaticSize(I);
   case Intrinsic::returnaddress:
bool AMDGPUInstructionSelector::selectG_INTRINSIC(MachineInstr &I) const {
   case Intrinsic::amdgcn_smfmac_f32_32x32x32_fp8_bf8:
   case Intrinsic::amdgcn_smfmac_f32_32x32x32_fp8_fp8:
     return selectSMFMACIntrin(I);
   default:
     return selectImpl(I, *CoverageInfo);
   }
 }
 
-static int getV_CMPOpcode(CmpInst::Predicate P, unsigned Size) {
-  if (Size != 32 && Size != 64)
+static int getV_CMPOpcode(CmpInst::Predicate P, unsigned Size,
+                          const GCNSubtarget &ST) {
+  if (Size != 16 && Size != 32 && Size != 64)
+    return -1;
+
+  if (Size == 16 && !ST.has16BitInsts())
     return -1;
+
+  const auto Select = [&](unsigned S16Opc, unsigned TrueS16Opc, unsigned S32Opc,
+                          unsigned S64Opc) {
+    if (Size == 16)
+      return ST.hasTrue16BitInsts() ? TrueS16Opc : S16Opc;
+    if (Size == 32)
+      return S32Opc;
+    return S64Opc;
+  };
+
   switch (P) {
   default:
     llvm_unreachable("Unknown condition code!");
   case CmpInst::ICMP_NE:
-    return Size == 32 ? AMDGPU::V_CMP_NE_U32_e64 : AMDGPU::V_CMP_NE_U64_e64;
+    return Select(AMDGPU::V_CMP_NE_U16_e64, AMDGPU::V_CMP_NE_U16_t16_e64,
+                  AMDGPU::V_CMP_NE_U32_e64, AMDGPU::V_CMP_NE_U64_e64);
   case CmpInst::ICMP_EQ:
-    return Size == 32 ? AMDGPU::V_CMP_EQ_U32_e64 : AMDGPU::V_CMP_EQ_U64_e64;
+    return Select(AMDGPU::V_CMP_EQ_U16_e64, AMDGPU::V_CMP_EQ_U16_t16_e64,
+                  AMDGPU::V_CMP_EQ_U32_e64, AMDGPU::V_CMP_EQ_U64_e64);
   case CmpInst::ICMP_SGT:
-    return Size == 32 ? AMDGPU::V_CMP_GT_I32_e64 : AMDGPU::V_CMP_GT_I64_e64;
+    return Select(AMDGPU::V_CMP_GT_I16_e64, AMDGPU::V_CMP_GT_I16_t16_e64,
+                  AMDGPU::V_CMP_GT_I32_e64, AMDGPU::V_CMP_GT_I64_e64);
   case CmpInst::ICMP_SGE:
-    return Size == 32 ? AMDGPU::V_CMP_GE_I32_e64 : AMDGPU::V_CMP_GE_I64_e64;
+    return Select(AMDGPU::V_CMP_GE_I16_e64, AMDGPU::V_CMP_GE_I16_t16_e64,
+                  AMDGPU::V_CMP_GE_I32_e64, AMDGPU::V_CMP_GE_I64_e64);
   case CmpInst::ICMP_SLT:
-    return Size == 32 ? AMDGPU::V_CMP_LT_I32_e64 : AMDGPU::V_CMP_LT_I64_e64;
+    return Select(AMDGPU::V_CMP_LT_I16_e64, AMDGPU::V_CMP_LT_I16_t16_e64,
+                  AMDGPU::V_CMP_LT_I32_e64, AMDGPU::V_CMP_LT_I64_e64);
   case CmpInst::ICMP_SLE:
-    return Size == 32 ? AMDGPU::V_CMP_LE_I32_e64 : AMDGPU::V_CMP_LE_I64_e64;
+    return Select(AMDGPU::V_CMP_LE_I16_e64, AMDGPU::V_CMP_LE_I16_t16_e64,
+                  AMDGPU::V_CMP_LE_I32_e64, AMDGPU::V_CMP_LE_I64_e64);
   case CmpInst::ICMP_UGT:
-    return Size == 32 ? AMDGPU::V_CMP_GT_U32_e64 : AMDGPU::V_CMP_GT_U64_e64;
+    return Select(AMDGPU::V_CMP_GT_U16_e64, AMDGPU::V_CMP_GT_U16_t16_e64,
+                  AMDGPU::V_CMP_GT_U32_e64, AMDGPU::V_CMP_GT_U64_e64);
   case CmpInst::ICMP_UGE:
-    return Size == 32 ? AMDGPU::V_CMP_GE_U32_e64 : AMDGPU::V_CMP_GE_U64_e64;
+    return Select(AMDGPU::V_CMP_GE_U16_e64, AMDGPU::V_CMP_GE_U16_t16_e64,
+                  AMDGPU::V_CMP_GE_U32_e64, AMDGPU::V_CMP_GE_U64_e64);
   case CmpInst::ICMP_ULT:
-    return Size == 32 ? AMDGPU::V_CMP_LT_U32_e64 : AMDGPU::V_CMP_LT_U64_e64;
+    return Select(AMDGPU::V_CMP_LT_U16_e64, AMDGPU::V_CMP_LT_U16_t16_e64,
+                  AMDGPU::V_CMP_LT_U32_e64, AMDGPU::V_CMP_LT_U64_e64);
   case CmpInst::ICMP_ULE:
-    return Size == 32 ? AMDGPU::V_CMP_LE_U32_e64 : AMDGPU::V_CMP_LE_U64_e64;
+    return Select(AMDGPU::V_CMP_LE_U16_e64, AMDGPU::V_CMP_LE_U16_t16_e64,
+                  AMDGPU::V_CMP_LE_U32_e64, AMDGPU::V_CMP_LE_U64_e64);
   }
 }
 
 int AMDGPUInstructionSelector::getS_CMPOpcode(CmpInst::Predicate P,
                                               unsigned Size) const {
   if (Size == 64) {
     if (!STI.hasScalarCompareEq64())
       return -1;
BuildMI(*BB, &I, DL, TII.get(AMDGPU::COPY), CCReg)
       .addReg(AMDGPU::SCC);
     bool Ret =
         constrainSelectedInstRegOperands(*ICmp, TII, TRI, RBI) &&
         RBI.constrainGenericRegister(CCReg, AMDGPU::SReg_32RegClass, *MRI);
     I.eraseFromParent();
     return Ret;
   }
 
-  int Opcode = getV_CMPOpcode(Pred, Size);
+  int Opcode = getV_CMPOpcode(Pred, Size, *Subtarget);
   if (Opcode == -1)
     return false;
 
   MachineInstr *ICmp = BuildMI(*BB, &I, DL, TII.get(Opcode),
             I.getOperand(0).getReg())
             .add(I.getOperand(2))
             .add(I.getOperand(3));
   RBI.constrainGenericRegister(ICmp->getOperand(0).getReg(),
bool AMDGPUInstructionSelector::selectIntrinsicIcmp(MachineInstr &I) const {
   if (MRI->getType(Dst).getSizeInBits() != STI.getWavefrontSize())
     return false;
 
   MachineBasicBlock *BB = I.getParent();
   const DebugLoc &DL = I.getDebugLoc();
   Register SrcReg = I.getOperand(2).getReg();
   unsigned Size = RBI.getSizeInBits(SrcReg, *MRI, TRI);
 
+  // i1 inputs are not supported in GlobalISel.
+  if (Size == 1)
+    return false;
+
   auto Pred = static_cast<CmpInst::Predicate>(I.getOperand(4).getImm());
-  if (!ICmpInst::isIntPredicate(static_cast<ICmpInst::Predicate>(Pred))) {
-    MachineInstr *ICmp =
-        BuildMI(*BB, &I, DL, TII.get(AMDGPU::IMPLICIT_DEF), Dst);
-
-    if (!RBI.constrainGenericRegister(ICmp->getOperand(0).getReg(),
-                                      *TRI.getBoolRC(), *MRI))
-      return false;
+  if (!CmpInst::isIntPredicate(Pred)) {
     I.eraseFromParent();
-    return true;
+    return RBI.constrainGenericRegister(Dst, *TRI.getBoolRC(), *MRI);
   }
 
-  int Opcode = getV_CMPOpcode(Pred, Size);
+  int Opcode = getV_CMPOpcode(Pred, Size, *Subtarget);
   if (Opcode == -1)
     return false;
 
   MachineInstr *ICmp = BuildMI(*BB, &I, DL, TII.get(Opcode), Dst)
                            .add(I.getOperand(2))
                            .add(I.getOperand(3));
-  RBI.constrainGenericRegister(ICmp->getOperand(0).getReg(), *TRI.getBoolRC(),
-                               *MRI);
-  bool Ret = constrainSelectedInstRegOperands(*ICmp, TII, TRI, RBI);
+  RBI.constrainGenericRegister(Dst, *TRI.getBoolRC(), *MRI);
   I.eraseFromParent();
-  return Ret;
+  return constrainSelectedInstRegOperands(*ICmp, TII, TRI, RBI);
 }
 
 bool AMDGPUInstructionSelector::selectBallot(MachineInstr &I) const {
   MachineBasicBlock *BB = I.getParent();
   const DebugLoc &DL = I.getDebugLoc();
   Register DstReg = I.getOperand(0).getReg();
   const unsigned Size = MRI->getType(DstReg).getSizeInBits();
   const bool Is64 = Size == 64;

llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-icmp.s16.mir

body: |
   bb.0:
     liveins:  $sgpr0, $vgpr0
 
     ; WAVE64-LABEL: name: icmp_eq_s16_sv
     ; WAVE64: liveins: $sgpr0, $vgpr0
     ; WAVE64-NEXT: {{  $}}
     ; WAVE64-NEXT: [[COPY:%[0-9]+]]:sreg_32 = COPY $sgpr0
     ; WAVE64-NEXT: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr0
-    ; WAVE64-NEXT: [[V_CMP_EQ_U16_e64_:%[0-9]+]]:sreg_64_xexec = V_CMP_EQ_U16_e64 [[COPY]], [[COPY1]], implicit $exec
+    ; WAVE64-NEXT: [[V_CMP_EQ_U16_e64_:%[0-9]+]]:sreg_64 = V_CMP_EQ_U16_e64 [[COPY]], [[COPY1]], implicit $exec
     ; WAVE64-NEXT: S_ENDPGM 0, implicit [[V_CMP_EQ_U16_e64_]]
     ; WAVE32-LABEL: name: icmp_eq_s16_sv
     ; WAVE32: liveins: $sgpr0, $vgpr0
     ; WAVE32-NEXT: {{  $}}
     ; WAVE32-NEXT: [[COPY:%[0-9]+]]:sreg_32 = COPY $sgpr0
     ; WAVE32-NEXT: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr0
-    ; WAVE32-NEXT: [[V_CMP_EQ_U16_e64_:%[0-9]+]]:sreg_32_xm0_xexec = V_CMP_EQ_U16_e64 [[COPY]], [[COPY1]], implicit $exec
+    ; WAVE32-NEXT: [[V_CMP_EQ_U16_e64_:%[0-9]+]]:sreg_32 = V_CMP_EQ_U16_e64 [[COPY]], [[COPY1]], implicit $exec
     ; WAVE32-NEXT: S_ENDPGM 0, implicit [[V_CMP_EQ_U16_e64_]]
     ; GFX11-LABEL: name: icmp_eq_s16_sv
     ; GFX11: liveins: $sgpr0, $vgpr0
     ; GFX11-NEXT: {{  $}}
     ; GFX11-NEXT: [[COPY:%[0-9]+]]:sreg_32 = COPY $sgpr0
     ; GFX11-NEXT: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr0
-    ; GFX11-NEXT: [[V_CMP_EQ_U16_t16_e64_:%[0-9]+]]:sreg_32_xm0_xexec = V_CMP_EQ_U16_t16_e64 [[COPY]], [[COPY1]], implicit $exec
+    ; GFX11-NEXT: [[V_CMP_EQ_U16_t16_e64_:%[0-9]+]]:sreg_32 = V_CMP_EQ_U16_t16_e64 [[COPY]], [[COPY1]], implicit $exec
     ; GFX11-NEXT: S_ENDPGM 0, implicit [[V_CMP_EQ_U16_t16_e64_]]
     %0:sgpr(s32) = COPY $sgpr0
     %1:vgpr(s32) = COPY $vgpr0
     %2:sgpr(s16) = G_TRUNC %0
     %3:vgpr(s16) = G_TRUNC %1
     %4:vcc(s1) = G_ICMP intpred(eq), %2, %3
     S_ENDPGM 0, implicit %4
 ...
 
 ---
 
 name:            icmp_eq_s16_vs
 legalized:       true
 regBankSelected: true
 
 body: |
   bb.0:
     liveins:  $sgpr0, $vgpr0
 
     ; WAVE64-LABEL: name: icmp_eq_s16_vs
     ; WAVE64: liveins: $sgpr0, $vgpr0
     ; WAVE64-NEXT: {{  $}}
     ; WAVE64-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; WAVE64-NEXT: [[COPY1:%[0-9]+]]:sreg_32 = COPY $sgpr0
-    ; WAVE64-NEXT: [[V_CMP_EQ_U16_e64_:%[0-9]+]]:sreg_64_xexec = V_CMP_EQ_U16_e64 [[COPY]], [[COPY1]], implicit $exec
+    ; WAVE64-NEXT: [[V_CMP_EQ_U16_e64_:%[0-9]+]]:sreg_64 = V_CMP_EQ_U16_e64 [[COPY]], [[COPY1]], implicit $exec
     ; WAVE64-NEXT: S_ENDPGM 0, implicit [[V_CMP_EQ_U16_e64_]]
     ; WAVE32-LABEL: name: icmp_eq_s16_vs
     ; WAVE32: liveins: $sgpr0, $vgpr0
     ; WAVE32-NEXT: {{  $}}
     ; WAVE32-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; WAVE32-NEXT: [[COPY1:%[0-9]+]]:sreg_32 = COPY $sgpr0
-    ; WAVE32-NEXT: [[V_CMP_EQ_U16_e64_:%[0-9]+]]:sreg_32_xm0_xexec = V_CMP_EQ_U16_e64 [[COPY]], [[COPY1]], implicit $exec
+    ; WAVE32-NEXT: [[V_CMP_EQ_U16_e64_:%[0-9]+]]:sreg_32 = V_CMP_EQ_U16_e64 [[COPY]], [[COPY1]], implicit $exec
     ; WAVE32-NEXT: S_ENDPGM 0, implicit [[V_CMP_EQ_U16_e64_]]
     ; GFX11-LABEL: name: icmp_eq_s16_vs
     ; GFX11: liveins: $sgpr0, $vgpr0
     ; GFX11-NEXT: {{  $}}
     ; GFX11-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; GFX11-NEXT: [[COPY1:%[0-9]+]]:sreg_32 = COPY $sgpr0
-    ; GFX11-NEXT: [[V_CMP_EQ_U16_t16_e64_:%[0-9]+]]:sreg_32_xm0_xexec = V_CMP_EQ_U16_t16_e64 [[COPY]], [[COPY1]], implicit $exec
+    ; GFX11-NEXT: [[V_CMP_EQ_U16_t16_e64_:%[0-9]+]]:sreg_32 = V_CMP_EQ_U16_t16_e64 [[COPY]], [[COPY1]], implicit $exec
     ; GFX11-NEXT: S_ENDPGM 0, implicit [[V_CMP_EQ_U16_t16_e64_]]
     %0:vgpr(s32) = COPY $vgpr0
     %1:sgpr(s32) = COPY $sgpr0
     %2:vgpr(s16) = G_TRUNC %0
     %3:sgpr(s16) = G_TRUNC %1
     %4:vcc(s1) = G_ICMP intpred(eq), %2, %3
     S_ENDPGM 0, implicit %4
 ...
 
 ---
 
 name:            icmp_eq_s16_vv
 legalized:       true
 regBankSelected: true
 
 body: |
   bb.0:
     liveins:  $vgpr0, $vgpr1
 
     ; WAVE64-LABEL: name: icmp_eq_s16_vv
     ; WAVE64: liveins: $vgpr0, $vgpr1
     ; WAVE64-NEXT: {{  $}}
     ; WAVE64-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; WAVE64-NEXT: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr1
-    ; WAVE64-NEXT: [[V_CMP_EQ_U16_e64_:%[0-9]+]]:sreg_64_xexec = V_CMP_EQ_U16_e64 [[COPY]], [[COPY1]], implicit $exec
+    ; WAVE64-NEXT: [[V_CMP_EQ_U16_e64_:%[0-9]+]]:sreg_64 = V_CMP_EQ_U16_e64 [[COPY]], [[COPY1]], implicit $exec
     ; WAVE64-NEXT: S_ENDPGM 0, implicit [[V_CMP_EQ_U16_e64_]]
     ; WAVE32-LABEL: name: icmp_eq_s16_vv
     ; WAVE32: liveins: $vgpr0, $vgpr1
     ; WAVE32-NEXT: {{  $}}
     ; WAVE32-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; WAVE32-NEXT: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr1
-    ; WAVE32-NEXT: [[V_CMP_EQ_U16_e64_:%[0-9]+]]:sreg_32_xm0_xexec = V_CMP_EQ_U16_e64 [[COPY]], [[COPY1]], implicit $exec
+    ; WAVE32-NEXT: [[V_CMP_EQ_U16_e64_:%[0-9]+]]:sreg_32 = V_CMP_EQ_U16_e64 [[COPY]], [[COPY1]], implicit $exec
     ; WAVE32-NEXT: S_ENDPGM 0, implicit [[V_CMP_EQ_U16_e64_]]
     ; GFX11-LABEL: name: icmp_eq_s16_vv
     ; GFX11: liveins: $vgpr0, $vgpr1
     ; GFX11-NEXT: {{  $}}
     ; GFX11-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; GFX11-NEXT: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr1
-    ; GFX11-NEXT: [[V_CMP_EQ_U16_t16_e64_:%[0-9]+]]:sreg_32_xm0_xexec = V_CMP_EQ_U16_t16_e64 [[COPY]], [[COPY1]], implicit $exec
+    ; GFX11-NEXT: [[V_CMP_EQ_U16_t16_e64_:%[0-9]+]]:sreg_32 = V_CMP_EQ_U16_t16_e64 [[COPY]], [[COPY1]], implicit $exec
     ; GFX11-NEXT: S_ENDPGM 0, implicit [[V_CMP_EQ_U16_t16_e64_]]
     %0:vgpr(s32) = COPY $vgpr0
     %1:vgpr(s32) = COPY $vgpr1
     %2:vgpr(s16) = G_TRUNC %0
     %3:vgpr(s16) = G_TRUNC %1
     %4:vcc(s1) = G_ICMP intpred(eq), %2, %3
     S_ENDPGM 0, implicit %4
 ...
 
 ---
 
 name:            icmp_ne_s16_vv
 legalized:       true
 regBankSelected: true
 
 body: |
   bb.0:
     liveins:  $vgpr0, $vgpr1
 
     ; WAVE64-LABEL: name: icmp_ne_s16_vv
     ; WAVE64: liveins: $vgpr0, $vgpr1
     ; WAVE64-NEXT: {{  $}}
     ; WAVE64-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; WAVE64-NEXT: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr1
-    ; WAVE64-NEXT: [[V_CMP_NE_U16_e64_:%[0-9]+]]:sreg_64_xexec = V_CMP_NE_U16_e64 [[COPY]], [[COPY1]], implicit $exec
+    ; WAVE64-NEXT: [[V_CMP_NE_U16_e64_:%[0-9]+]]:sreg_64 = V_CMP_NE_U16_e64 [[COPY]], [[COPY1]], implicit $exec
     ; WAVE64-NEXT: S_ENDPGM 0, implicit [[V_CMP_NE_U16_e64_]]
     ; WAVE32-LABEL: name: icmp_ne_s16_vv
     ; WAVE32: liveins: $vgpr0, $vgpr1
     ; WAVE32-NEXT: {{  $}}
     ; WAVE32-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; WAVE32-NEXT: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr1
-    ; WAVE32-NEXT: [[V_CMP_NE_U16_e64_:%[0-9]+]]:sreg_32_xm0_xexec = V_CMP_NE_U16_e64 [[COPY]], [[COPY1]], implicit $exec
+    ; WAVE32-NEXT: [[V_CMP_NE_U16_e64_:%[0-9]+]]:sreg_32 = V_CMP_NE_U16_e64 [[COPY]], [[COPY1]], implicit $exec
     ; WAVE32-NEXT: S_ENDPGM 0, implicit [[V_CMP_NE_U16_e64_]]
     ; GFX11-LABEL: name: icmp_ne_s16_vv
     ; GFX11: liveins: $vgpr0, $vgpr1
     ; GFX11-NEXT: {{  $}}
     ; GFX11-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; GFX11-NEXT: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr1
-    ; GFX11-NEXT: [[V_CMP_NE_U16_t16_e64_:%[0-9]+]]:sreg_32_xm0_xexec = V_CMP_NE_U16_t16_e64 [[COPY]], [[COPY1]], implicit $exec
+    ; GFX11-NEXT: [[V_CMP_NE_U16_t16_e64_:%[0-9]+]]:sreg_32 = V_CMP_NE_U16_t16_e64 [[COPY]], [[COPY1]], implicit $exec
     ; GFX11-NEXT: S_ENDPGM 0, implicit [[V_CMP_NE_U16_t16_e64_]]
     %0:vgpr(s32) = COPY $vgpr0
     %1:vgpr(s32) = COPY $vgpr1
     %2:vgpr(s16) = G_TRUNC %0
     %3:vgpr(s16) = G_TRUNC %1
     %4:vcc(s1) = G_ICMP intpred(ne), %2, %3
     S_ENDPGM 0, implicit %4
 ...
 
 ---
 
 name:            icmp_slt_s16_vv
 legalized:       true
 regBankSelected: true
 
 body: |
   bb.0:
     liveins:  $vgpr0, $vgpr1
 
     ; WAVE64-LABEL: name: icmp_slt_s16_vv
     ; WAVE64: liveins: $vgpr0, $vgpr1
     ; WAVE64-NEXT: {{  $}}
     ; WAVE64-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; WAVE64-NEXT: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr1
-    ; WAVE64-NEXT: [[V_CMP_LT_I16_e64_:%[0-9]+]]:sreg_64_xexec = V_CMP_LT_I16_e64 [[COPY]], [[COPY1]], implicit $exec
+    ; WAVE64-NEXT: [[V_CMP_LT_I16_e64_:%[0-9]+]]:sreg_64 = V_CMP_LT_I16_e64 [[COPY]], [[COPY1]], implicit $exec
     ; WAVE64-NEXT: S_ENDPGM 0, implicit [[V_CMP_LT_I16_e64_]]
     ; WAVE32-LABEL: name: icmp_slt_s16_vv
     ; WAVE32: liveins: $vgpr0, $vgpr1
     ; WAVE32-NEXT: {{  $}}
     ; WAVE32-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; WAVE32-NEXT: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr1
-    ; WAVE32-NEXT: [[V_CMP_LT_I16_e64_:%[0-9]+]]:sreg_32_xm0_xexec = V_CMP_LT_I16_e64 [[COPY]], [[COPY1]], implicit $exec
+    ; WAVE32-NEXT: [[V_CMP_LT_I16_e64_:%[0-9]+]]:sreg_32 = V_CMP_LT_I16_e64 [[COPY]], [[COPY1]], implicit $exec
     ; WAVE32-NEXT: S_ENDPGM 0, implicit [[V_CMP_LT_I16_e64_]]
     ; GFX11-LABEL: name: icmp_slt_s16_vv
     ; GFX11: liveins: $vgpr0, $vgpr1
     ; GFX11-NEXT: {{  $}}
     ; GFX11-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; GFX11-NEXT: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr1
-    ; GFX11-NEXT: [[V_CMP_LT_I16_t16_e64_:%[0-9]+]]:sreg_32_xm0_xexec = V_CMP_LT_I16_t16_e64 [[COPY]], [[COPY1]], implicit $exec
+    ; GFX11-NEXT: [[V_CMP_LT_I16_t16_e64_:%[0-9]+]]:sreg_32 = V_CMP_LT_I16_t16_e64 [[COPY]], [[COPY1]], implicit $exec
     ; GFX11-NEXT: S_ENDPGM 0, implicit [[V_CMP_LT_I16_t16_e64_]]
     %0:vgpr(s32) = COPY $vgpr0
     %1:vgpr(s32) = COPY $vgpr1
     %2:vgpr(s16) = G_TRUNC %0
     %3:vgpr(s16) = G_TRUNC %1
     %4:vcc(s1) = G_ICMP intpred(slt), %2, %3
     S_ENDPGM 0, implicit %4
 ...
 
 ---
 
 name:            icmp_sle_s16_vv
 legalized:       true
 regBankSelected: true
 
 body: |
   bb.0:
     liveins:  $vgpr0, $vgpr1
 
     ; WAVE64-LABEL: name: icmp_sle_s16_vv
     ; WAVE64: liveins: $vgpr0, $vgpr1
     ; WAVE64-NEXT: {{  $}}
     ; WAVE64-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; WAVE64-NEXT: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr1
-    ; WAVE64-NEXT: [[V_CMP_LE_I16_e64_:%[0-9]+]]:sreg_64_xexec = V_CMP_LE_I16_e64 [[COPY]], [[COPY1]], implicit $exec
+    ; WAVE64-NEXT: [[V_CMP_LE_I16_e64_:%[0-9]+]]:sreg_64 = V_CMP_LE_I16_e64 [[COPY]], [[COPY1]], implicit $exec
     ; WAVE64-NEXT: S_ENDPGM 0, implicit [[V_CMP_LE_I16_e64_]]
     ; WAVE32-LABEL: name: icmp_sle_s16_vv
     ; WAVE32: liveins: $vgpr0, $vgpr1
     ; WAVE32-NEXT: {{  $}}
     ; WAVE32-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; WAVE32-NEXT: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr1
-    ; WAVE32-NEXT: [[V_CMP_LE_I16_e64_:%[0-9]+]]:sreg_32_xm0_xexec = V_CMP_LE_I16_e64 [[COPY]], [[COPY1]], implicit $exec
+    ; WAVE32-NEXT: [[V_CMP_LE_I16_e64_:%[0-9]+]]:sreg_32 = V_CMP_LE_I16_e64 [[COPY]], [[COPY1]], implicit $exec
     ; WAVE32-NEXT: S_ENDPGM 0, implicit [[V_CMP_LE_I16_e64_]]
     ; GFX11-LABEL: name: icmp_sle_s16_vv
     ; GFX11: liveins: $vgpr0, $vgpr1
     ; GFX11-NEXT: {{  $}}
     ; GFX11-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; GFX11-NEXT: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr1
-    ; GFX11-NEXT: [[V_CMP_LE_I16_t16_e64_:%[0-9]+]]:sreg_32_xm0_xexec = V_CMP_LE_I16_t16_e64 [[COPY]], [[COPY1]], implicit $exec
+    ; GFX11-NEXT: [[V_CMP_LE_I16_t16_e64_:%[0-9]+]]:sreg_32 = V_CMP_LE_I16_t16_e64 [[COPY]], [[COPY1]], implicit $exec
     ; GFX11-NEXT: S_ENDPGM 0, implicit [[V_CMP_LE_I16_t16_e64_]]
     %0:vgpr(s32) = COPY $vgpr0
     %1:vgpr(s32) = COPY $vgpr1
     %2:vgpr(s16) = G_TRUNC %0
     %3:vgpr(s16) = G_TRUNC %1
     %4:vcc(s1) = G_ICMP intpred(sle), %2, %3
     S_ENDPGM 0, implicit %4
 ...
 
 ---
 
 name:            icmp_ult_s16_vv
 legalized:       true
 regBankSelected: true
 
 body: |
   bb.0:
     liveins:  $vgpr0, $vgpr1
 
     ; WAVE64-LABEL: name: icmp_ult_s16_vv
     ; WAVE64: liveins: $vgpr0, $vgpr1
     ; WAVE64-NEXT: {{  $}}
     ; WAVE64-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; WAVE64-NEXT: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr1
-    ; WAVE64-NEXT: [[V_CMP_LT_U16_e64_:%[0-9]+]]:sreg_64_xexec = V_CMP_LT_U16_e64 [[COPY]], [[COPY1]], implicit $exec
+    ; WAVE64-NEXT: [[V_CMP_LT_U16_e64_:%[0-9]+]]:sreg_64 = V_CMP_LT_U16_e64 [[COPY]], [[COPY1]], implicit $exec
     ; WAVE64-NEXT: S_ENDPGM 0, implicit [[V_CMP_LT_U16_e64_]]
     ; WAVE32-LABEL: name: icmp_ult_s16_vv
     ; WAVE32: liveins: $vgpr0, $vgpr1
     ; WAVE32-NEXT: {{  $}}
     ; WAVE32-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; WAVE32-NEXT: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr1
-    ; WAVE32-NEXT: [[V_CMP_LT_U16_e64_:%[0-9]+]]:sreg_32_xm0_xexec = V_CMP_LT_U16_e64 [[COPY]], [[COPY1]], implicit $exec
+    ; WAVE32-NEXT: [[V_CMP_LT_U16_e64_:%[0-9]+]]:sreg_32 = V_CMP_LT_U16_e64 [[COPY]], [[COPY1]], implicit $exec
     ; WAVE32-NEXT: S_ENDPGM 0, implicit [[V_CMP_LT_U16_e64_]]
     ; GFX11-LABEL: name: icmp_ult_s16_vv
     ; GFX11: liveins: $vgpr0, $vgpr1
     ; GFX11-NEXT: {{  $}}
     ; GFX11-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; GFX11-NEXT: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr1
-    ; GFX11-NEXT: [[V_CMP_LT_U16_t16_e64_:%[0-9]+]]:sreg_32_xm0_xexec = V_CMP_LT_U16_t16_e64 [[COPY]], [[COPY1]], implicit $exec
+    ; GFX11-NEXT: [[V_CMP_LT_U16_t16_e64_:%[0-9]+]]:sreg_32 = V_CMP_LT_U16_t16_e64 [[COPY]], [[COPY1]], implicit $exec
     ; GFX11-NEXT: S_ENDPGM 0, implicit [[V_CMP_LT_U16_t16_e64_]]
     %0:vgpr(s32) = COPY $vgpr0
     %1:vgpr(s32) = COPY $vgpr1
     %2:vgpr(s16) = G_TRUNC %0
     %3:vgpr(s16) = G_TRUNC %1
     %4:vcc(s1) = G_ICMP intpred(ult), %2, %3
     S_ENDPGM 0, implicit %4
 ...
 
 ---
 
 name:            icmp_ule_s16_vv
 legalized:       true
 regBankSelected: true
 
 body: |
   bb.0:
     liveins:  $vgpr0, $vgpr1
 
     ; WAVE64-LABEL: name: icmp_ule_s16_vv
     ; WAVE64: liveins: $vgpr0, $vgpr1
     ; WAVE64-NEXT: {{  $}}
     ; WAVE64-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; WAVE64-NEXT: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr1
-    ; WAVE64-NEXT: [[V_CMP_LE_U16_e64_:%[0-9]+]]:sreg_64_xexec = V_CMP_LE_U16_e64 [[COPY]], [[COPY1]], implicit $exec
+    ; WAVE64-NEXT: [[V_CMP_LE_U16_e64_:%[0-9]+]]:sreg_64 = V_CMP_LE_U16_e64 [[COPY]], [[COPY1]], implicit $exec
     ; WAVE64-NEXT: S_ENDPGM 0, implicit [[V_CMP_LE_U16_e64_]]
     ; WAVE32-LABEL: name: icmp_ule_s16_vv
     ; WAVE32: liveins: $vgpr0, $vgpr1
     ; WAVE32-NEXT: {{  $}}
     ; WAVE32-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; WAVE32-NEXT: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr1
-    ; WAVE32-NEXT: [[V_CMP_LE_U16_e64_:%[0-9]+]]:sreg_32_xm0_xexec = V_CMP_LE_U16_e64 [[COPY]], [[COPY1]], implicit $exec
+    ; WAVE32-NEXT: [[V_CMP_LE_U16_e64_:%[0-9]+]]:sreg_32 = V_CMP_LE_U16_e64 [[COPY]], [[COPY1]], implicit $exec
     ; WAVE32-NEXT: S_ENDPGM 0, implicit [[V_CMP_LE_U16_e64_]]
     ; GFX11-LABEL: name: icmp_ule_s16_vv
     ; GFX11: liveins: $vgpr0, $vgpr1
     ; GFX11-NEXT: {{  $}}
     ; GFX11-NEXT: [[COPY:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; GFX11-NEXT: [[COPY1:%[0-9]+]]:vgpr_32 = COPY $vgpr1
-    ; GFX11-NEXT: [[V_CMP_LE_U16_t16_e64_:%[0-9]+]]:sreg_32_xm0_xexec = V_CMP_LE_U16_t16_e64 [[COPY]], [[COPY1]], implicit $exec
+    ; GFX11-NEXT: [[V_CMP_LE_U16_t16_e64_:%[0-9]+]]:sreg_32 = V_CMP_LE_U16_t16_e64 [[COPY]], [[COPY1]], implicit $exec
     ; GFX11-NEXT: S_ENDPGM 0, implicit [[V_CMP_LE_U16_t16_e64_]]
     %0:vgpr(s32) = COPY $vgpr0
     %1:vgpr(s32) = COPY $vgpr1
     %2:vgpr(s16) = G_TRUNC %0
     %3:vgpr(s16) = G_TRUNC %1
     %4:vcc(s1) = G_ICMP intpred(ule), %2, %3
     S_ENDPGM 0, implicit %4
 ...
 

llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.icmp.ll

-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -global-isel -global-isel-abort=1 -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs < %s | FileCheck -check-prefix=GFX10 %s
-; RUN: llc -global-isel -global-isel-abort=1 -march=amdgcn -mcpu=gfx1100 -amdgpu-enable-delay-alu=0 -verify-machineinstrs < %s | FileCheck -check-prefix=GFX11 %s
-
-define amdgpu_ps void @test_intr_icmp_eq_i64(i64 addrspace(1)* %out, i32 %src) #0 {
-; GFX10-LABEL: test_intr_icmp_eq_i64:
-; GFX10:       ; %bb.0:
-; GFX10-NEXT:    v_cmp_eq_u32_e64 s[0:1], 0x64, v2
-; GFX10-NEXT:    v_mov_b32_e32 v3, s1
-; GFX10-NEXT:    v_mov_b32_e32 v2, s0
-; GFX10-NEXT:    global_store_dwordx2 v[0:1], v[2:3], off
-; GFX10-NEXT:    s_endpgm
-;
-; GFX11-LABEL: test_intr_icmp_eq_i64:
-; GFX11:       ; %bb.0:
-; GFX11-NEXT:    v_cmp_eq_u32_e64 s[0:1], 0x64, v2
-; GFX11-NEXT:    v_mov_b32_e32 v3, s1
-; GFX11-NEXT:    v_mov_b32_e32 v2, s0
-; GFX11-NEXT:    global_store_b64 v[0:1], v[2:3], off
-; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
-; GFX11-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %src, i32 100, i32 32)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_ps void @test_intr_icmp_ne_i32(i32 addrspace(1)* %out, i32 %src) #1 {
-; GFX10-LABEL: test_intr_icmp_ne_i32:
-; GFX10:       ; %bb.0:
-; GFX10-NEXT:    v_cmp_ne_u32_e64 s0, 0x64, v2
-; GFX10-NEXT:    v_mov_b32_e32 v2, s0
-; GFX10-NEXT:    global_store_dword v[0:1], v2, off
-; GFX10-NEXT:    s_endpgm
-;
-; GFX11-LABEL: test_intr_icmp_ne_i32:
-; GFX11:       ; %bb.0:
-; GFX11-NEXT:    v_cmp_ne_u32_e64 s0, 0x64, v2
-; GFX11-NEXT:    v_mov_b32_e32 v2, s0
-; GFX11-NEXT:    global_store_b32 v[0:1], v2, off
-; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
-; GFX11-NEXT:    s_endpgm
-  %result = call i32 @llvm.amdgcn.icmp.i32.i32(i32 %src, i32 100, i32 33)
-  store i32 %result, i32 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_ps void @test_intr_icmp_i32_invalid_cc(i32 addrspace(1)* %out, i32 %src) #1 {
-; GFX10-LABEL: test_intr_icmp_i32_invalid_cc:
-; GFX10:       ; %bb.0:
-; GFX10-NEXT:    global_store_dword v[0:1], v0, off
-; GFX10-NEXT:    s_endpgm
-;
-; GFX11-LABEL: test_intr_icmp_i32_invalid_cc:
-; GFX11:       ; %bb.0:
-; GFX11-NEXT:    global_store_b32 v[0:1], v0, off
-; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
-; GFX11-NEXT:    s_endpgm
-  %result = call i32 @llvm.amdgcn.icmp.i32.i32(i32 %src, i32 100, i32 9999)
-  store i32 %result, i32 addrspace(1)* %out
-  ret void
-}
-
-declare i64 @llvm.amdgcn.icmp.i64.i32(i32, i32, i32)
-declare i32 @llvm.amdgcn.icmp.i32.i32(i32, i32, i32)
-attributes #0 = { "target-features"="+wavefrontsize64" }
-attributes #1 = { "target-features"="+wavefrontsize32" }

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.icmp.ll

-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -march=amdgcn -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX %s
-; RUN: llc -march=amdgcn -mcpu=fiji -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,VI %s
-
-declare i64 @llvm.amdgcn.icmp.i64.i32(i32, i32, i32)
-declare i32 @llvm.amdgcn.icmp.i32.i32(i32, i32, i32)
-declare i64 @llvm.amdgcn.icmp.i32(i32, i32, i32) #0
-declare i64 @llvm.amdgcn.icmp.i64(i64, i64, i32) #0
-declare i64 @llvm.amdgcn.icmp.i16(i16, i16, i32) #0
-declare i64 @llvm.amdgcn.icmp.i1(i1, i1, i32) #0
-
-define amdgpu_kernel void @v_icmp_i32_eq(i64 addrspace(1)* %out, i32 %src) {
-; GFX-LABEL: v_icmp_i32_eq:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_eq_u32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i32_eq:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_eq_u32_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 32)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_i32(i64 addrspace(1)* %out, i32 %src) {
-; GCN-LABEL: v_icmp_i32:
-; GCN:       ; %bb.0:
-; GCN-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 30)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_i32_ne(i64 addrspace(1)* %out, i32 %src) {
-; GFX-LABEL: v_icmp_i32_ne:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_ne_u32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i32_ne:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_ne_u32_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 33)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_i32_ugt(i64 addrspace(1)* %out, i32 %src) {
-; GFX-LABEL: v_icmp_i32_ugt:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_gt_u32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i32_ugt:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_gt_u32_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 34)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_i32_uge(i64 addrspace(1)* %out, i32 %src) {
-; GFX-LABEL: v_icmp_i32_uge:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_ge_u32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i32_uge:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_ge_u32_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 35)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_i32_ult(i64 addrspace(1)* %out, i32 %src) {
-; GFX-LABEL: v_icmp_i32_ult:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_lt_u32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i32_ult:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_lt_u32_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 36)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_i32_ule(i64 addrspace(1)* %out, i32 %src) {
-; GFX-LABEL: v_icmp_i32_ule:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_le_u32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i32_ule:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_le_u32_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 37)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_i32_sgt(i64 addrspace(1)* %out, i32 %src) #1 {
-; GFX-LABEL: v_icmp_i32_sgt:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_gt_i32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i32_sgt:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_gt_i32_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 38)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_i32_sge(i64 addrspace(1)* %out, i32 %src) {
-; GFX-LABEL: v_icmp_i32_sge:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_ge_i32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i32_sge:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_ge_i32_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 39)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_i32_slt(i64 addrspace(1)* %out, i32 %src) {
-; GFX-LABEL: v_icmp_i32_slt:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_lt_i32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i32_slt:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_lt_i32_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 40)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_i32_sle(i64 addrspace(1)* %out, i32 %src) {
-; GFX-LABEL: v_icmp_i32_sle:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_le_i32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i32_sle:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_le_i32_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 41)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_i64_eq(i64 addrspace(1)* %out, i64 %src) {
-; GFX-LABEL: v_icmp_i64_eq:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s7, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    v_mov_b32_e32 v1, 0
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_eq_u64_e64 s[2:3], s[2:3], v[0:1]
-; GFX-NEXT:    s_mov_b32 s6, -1
-; GFX-NEXT:    s_mov_b32 s4, s0
-; GFX-NEXT:    s_mov_b32 s5, s1
-; GFX-NEXT:    v_mov_b32_e32 v0, s2
-; GFX-NEXT:    v_mov_b32_e32 v1, s3
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i64_eq:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    v_mov_b32_e32 v1, 0
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_eq_u64_e64 s[2:3], s[2:3], v[0:1]
-; VI-NEXT:    v_mov_b32_e32 v2, s0
-; VI-NEXT:    v_mov_b32_e32 v0, s2
-; VI-NEXT:    v_mov_b32_e32 v3, s1
-; VI-NEXT:    v_mov_b32_e32 v1, s3
-; VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
-; VI-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 32)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_i64_ne(i64 addrspace(1)* %out, i64 %src) {
-; GFX-LABEL: v_icmp_i64_ne:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s7, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    v_mov_b32_e32 v1, 0
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_ne_u64_e64 s[2:3], s[2:3], v[0:1]
-; GFX-NEXT:    s_mov_b32 s6, -1
-; GFX-NEXT:    s_mov_b32 s4, s0
-; GFX-NEXT:    s_mov_b32 s5, s1
-; GFX-NEXT:    v_mov_b32_e32 v0, s2
-; GFX-NEXT:    v_mov_b32_e32 v1, s3
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i64_ne:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    v_mov_b32_e32 v1, 0
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_ne_u64_e64 s[2:3], s[2:3], v[0:1]
-; VI-NEXT:    v_mov_b32_e32 v2, s0
-; VI-NEXT:    v_mov_b32_e32 v0, s2
-; VI-NEXT:    v_mov_b32_e32 v3, s1
-; VI-NEXT:    v_mov_b32_e32 v1, s3
-; VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
-; VI-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 33)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_u64_ugt(i64 addrspace(1)* %out, i64 %src) {
-; GFX-LABEL: v_icmp_u64_ugt:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s7, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    v_mov_b32_e32 v1, 0
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_gt_u64_e64 s[2:3], s[2:3], v[0:1]
-; GFX-NEXT:    s_mov_b32 s6, -1
-; GFX-NEXT:    s_mov_b32 s4, s0
-; GFX-NEXT:    s_mov_b32 s5, s1
-; GFX-NEXT:    v_mov_b32_e32 v0, s2
-; GFX-NEXT:    v_mov_b32_e32 v1, s3
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_u64_ugt:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    v_mov_b32_e32 v1, 0
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_gt_u64_e64 s[2:3], s[2:3], v[0:1]
-; VI-NEXT:    v_mov_b32_e32 v2, s0
-; VI-NEXT:    v_mov_b32_e32 v0, s2
-; VI-NEXT:    v_mov_b32_e32 v3, s1
-; VI-NEXT:    v_mov_b32_e32 v1, s3
-; VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
-; VI-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 34)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_u64_uge(i64 addrspace(1)* %out, i64 %src) {
-; GFX-LABEL: v_icmp_u64_uge:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s7, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    v_mov_b32_e32 v1, 0
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_ge_u64_e64 s[2:3], s[2:3], v[0:1]
-; GFX-NEXT:    s_mov_b32 s6, -1
-; GFX-NEXT:    s_mov_b32 s4, s0
-; GFX-NEXT:    s_mov_b32 s5, s1
-; GFX-NEXT:    v_mov_b32_e32 v0, s2
-; GFX-NEXT:    v_mov_b32_e32 v1, s3
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_u64_uge:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    v_mov_b32_e32 v1, 0
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_ge_u64_e64 s[2:3], s[2:3], v[0:1]
-; VI-NEXT:    v_mov_b32_e32 v2, s0
-; VI-NEXT:    v_mov_b32_e32 v0, s2
-; VI-NEXT:    v_mov_b32_e32 v3, s1
-; VI-NEXT:    v_mov_b32_e32 v1, s3
-; VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
-; VI-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 35)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_u64_ult(i64 addrspace(1)* %out, i64 %src) {
-; GFX-LABEL: v_icmp_u64_ult:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s7, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    v_mov_b32_e32 v1, 0
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_lt_u64_e64 s[2:3], s[2:3], v[0:1]
-; GFX-NEXT:    s_mov_b32 s6, -1
-; GFX-NEXT:    s_mov_b32 s4, s0
-; GFX-NEXT:    s_mov_b32 s5, s1
-; GFX-NEXT:    v_mov_b32_e32 v0, s2
-; GFX-NEXT:    v_mov_b32_e32 v1, s3
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_u64_ult:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    v_mov_b32_e32 v1, 0
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_lt_u64_e64 s[2:3], s[2:3], v[0:1]
-; VI-NEXT:    v_mov_b32_e32 v2, s0
-; VI-NEXT:    v_mov_b32_e32 v0, s2
-; VI-NEXT:    v_mov_b32_e32 v3, s1
-; VI-NEXT:    v_mov_b32_e32 v1, s3
-; VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
-; VI-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 36)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_u64_ule(i64 addrspace(1)* %out, i64 %src) {
-; GFX-LABEL: v_icmp_u64_ule:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s7, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    v_mov_b32_e32 v1, 0
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_le_u64_e64 s[2:3], s[2:3], v[0:1]
-; GFX-NEXT:    s_mov_b32 s6, -1
-; GFX-NEXT:    s_mov_b32 s4, s0
-; GFX-NEXT:    s_mov_b32 s5, s1
-; GFX-NEXT:    v_mov_b32_e32 v0, s2
-; GFX-NEXT:    v_mov_b32_e32 v1, s3
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_u64_ule:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    v_mov_b32_e32 v1, 0
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_le_u64_e64 s[2:3], s[2:3], v[0:1]
-; VI-NEXT:    v_mov_b32_e32 v2, s0
-; VI-NEXT:    v_mov_b32_e32 v0, s2
-; VI-NEXT:    v_mov_b32_e32 v3, s1
-; VI-NEXT:    v_mov_b32_e32 v1, s3
-; VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
-; VI-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 37)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_i64_sgt(i64 addrspace(1)* %out, i64 %src) {
-; GFX-LABEL: v_icmp_i64_sgt:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s7, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    v_mov_b32_e32 v1, 0
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_gt_i64_e64 s[2:3], s[2:3], v[0:1]
-; GFX-NEXT:    s_mov_b32 s6, -1
-; GFX-NEXT:    s_mov_b32 s4, s0
-; GFX-NEXT:    s_mov_b32 s5, s1
-; GFX-NEXT:    v_mov_b32_e32 v0, s2
-; GFX-NEXT:    v_mov_b32_e32 v1, s3
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i64_sgt:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    v_mov_b32_e32 v1, 0
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_gt_i64_e64 s[2:3], s[2:3], v[0:1]
-; VI-NEXT:    v_mov_b32_e32 v2, s0
-; VI-NEXT:    v_mov_b32_e32 v0, s2
-; VI-NEXT:    v_mov_b32_e32 v3, s1
-; VI-NEXT:    v_mov_b32_e32 v1, s3
-; VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
-; VI-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 38)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_i64_sge(i64 addrspace(1)* %out, i64 %src) {
-; GFX-LABEL: v_icmp_i64_sge:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s7, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    v_mov_b32_e32 v1, 0
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_ge_i64_e64 s[2:3], s[2:3], v[0:1]
-; GFX-NEXT:    s_mov_b32 s6, -1
-; GFX-NEXT:    s_mov_b32 s4, s0
-; GFX-NEXT:    s_mov_b32 s5, s1
-; GFX-NEXT:    v_mov_b32_e32 v0, s2
-; GFX-NEXT:    v_mov_b32_e32 v1, s3
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i64_sge:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    v_mov_b32_e32 v1, 0
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_ge_i64_e64 s[2:3], s[2:3], v[0:1]
-; VI-NEXT:    v_mov_b32_e32 v2, s0
-; VI-NEXT:    v_mov_b32_e32 v0, s2
-; VI-NEXT:    v_mov_b32_e32 v3, s1
-; VI-NEXT:    v_mov_b32_e32 v1, s3
-; VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
-; VI-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 39)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_i64_slt(i64 addrspace(1)* %out, i64 %src) {
-; GFX-LABEL: v_icmp_i64_slt:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s7, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    v_mov_b32_e32 v1, 0
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_lt_i64_e64 s[2:3], s[2:3], v[0:1]
-; GFX-NEXT:    s_mov_b32 s6, -1
-; GFX-NEXT:    s_mov_b32 s4, s0
-; GFX-NEXT:    s_mov_b32 s5, s1
-; GFX-NEXT:    v_mov_b32_e32 v0, s2
-; GFX-NEXT:    v_mov_b32_e32 v1, s3
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i64_slt:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    v_mov_b32_e32 v1, 0
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_lt_i64_e64 s[2:3], s[2:3], v[0:1]
-; VI-NEXT:    v_mov_b32_e32 v2, s0
-; VI-NEXT:    v_mov_b32_e32 v0, s2
-; VI-NEXT:    v_mov_b32_e32 v3, s1
-; VI-NEXT:    v_mov_b32_e32 v1, s3
-; VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
-; VI-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 40)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_i64_sle(i64 addrspace(1)* %out, i64 %src) {
-; GFX-LABEL: v_icmp_i64_sle:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s7, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    v_mov_b32_e32 v1, 0
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_le_i64_e64 s[2:3], s[2:3], v[0:1]
-; GFX-NEXT:    s_mov_b32 s6, -1
-; GFX-NEXT:    s_mov_b32 s4, s0
-; GFX-NEXT:    s_mov_b32 s5, s1
-; GFX-NEXT:    v_mov_b32_e32 v0, s2
-; GFX-NEXT:    v_mov_b32_e32 v1, s3
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i64_sle:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    v_mov_b32_e32 v1, 0
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_le_i64_e64 s[2:3], s[2:3], v[0:1]
-; VI-NEXT:    v_mov_b32_e32 v2, s0
-; VI-NEXT:    v_mov_b32_e32 v0, s2
-; VI-NEXT:    v_mov_b32_e32 v3, s1
-; VI-NEXT:    v_mov_b32_e32 v1, s3
-; VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
-; VI-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 41)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_i16_eq(i64 addrspace(1)* %out, i16 %src) {
-; GFX-LABEL: v_icmp_i16_eq:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    s_and_b32 s2, s2, 0xffff
-; GFX-NEXT:    v_cmp_eq_u32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i16_eq:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_eq_u16_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 32)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_i16(i64 addrspace(1)* %out, i16 %src) {
-; GCN-LABEL: v_icmp_i16:
-; GCN:       ; %bb.0:
-; GCN-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 30)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_i16_ne(i64 addrspace(1)* %out, i16 %src) {
-; GFX-LABEL: v_icmp_i16_ne:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    s_and_b32 s2, s2, 0xffff
-; GFX-NEXT:    v_cmp_ne_u32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i16_ne:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_ne_u16_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 33)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_i16_ugt(i64 addrspace(1)* %out, i16 %src) {
-; GFX-LABEL: v_icmp_i16_ugt:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    s_and_b32 s2, s2, 0xffff
-; GFX-NEXT:    v_cmp_gt_u32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i16_ugt:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_gt_u16_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 34)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_i16_uge(i64 addrspace(1)* %out, i16 %src) {
-; GFX-LABEL: v_icmp_i16_uge:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    s_and_b32 s2, s2, 0xffff
-; GFX-NEXT:    v_cmp_ge_u32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i16_uge:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_ge_u16_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 35)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_i16_ult(i64 addrspace(1)* %out, i16 %src) {
-; GFX-LABEL: v_icmp_i16_ult:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    s_and_b32 s2, s2, 0xffff
-; GFX-NEXT:    v_cmp_lt_u32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i16_ult:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_lt_u16_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 36)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_i16_ule(i64 addrspace(1)* %out, i16 %src) {
-; GFX-LABEL: v_icmp_i16_ule:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    s_and_b32 s2, s2, 0xffff
-; GFX-NEXT:    v_cmp_le_u32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i16_ule:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_le_u16_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 37)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_i16_sgt(i64 addrspace(1)* %out, i16 %src) #1 {
-; GFX-LABEL: v_icmp_i16_sgt:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    s_sext_i32_i16 s2, s2
-; GFX-NEXT:    v_cmp_gt_i32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i16_sgt:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_gt_i16_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 38)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_i16_sge(i64 addrspace(1)* %out, i16 %src) {
-; GFX-LABEL: v_icmp_i16_sge:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    s_sext_i32_i16 s2, s2
-; GFX-NEXT:    v_cmp_ge_i32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i16_sge:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_ge_i16_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 39)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_i16_slt(i64 addrspace(1)* %out, i16 %src) {
-; GFX-LABEL: v_icmp_i16_slt:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    s_sext_i32_i16 s2, s2
-; GFX-NEXT:    v_cmp_lt_i32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i16_slt:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_lt_i16_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 40)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_i16_sle(i64 addrspace(1)* %out, i16 %src) {
-; GFX-LABEL: v_icmp_i16_sle:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    s_sext_i32_i16 s2, s2
-; GFX-NEXT:    v_cmp_le_i32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i16_sle:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_le_i16_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 41)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_kernel void @v_icmp_i1_ne0(i64 addrspace(1)* %out, i32 %a, i32 %b) {
-; GFX-LABEL: v_icmp_i1_ne0:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s7, 0xf000
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    s_cmp_gt_u32 s2, 1
-; GFX-NEXT:    s_cselect_b64 s[4:5], -1, 0
-; GFX-NEXT:    s_cmp_gt_u32 s3, 2
-; GFX-NEXT:    s_cselect_b64 s[2:3], -1, 0
-; GFX-NEXT:    s_and_b64 s[2:3], s[4:5], s[2:3]
-; GFX-NEXT:    s_mov_b32 s6, -1
-; GFX-NEXT:    s_mov_b32 s4, s0
-; GFX-NEXT:    s_mov_b32 s5, s1
-; GFX-NEXT:    v_mov_b32_e32 v0, s2
-; GFX-NEXT:    v_mov_b32_e32 v1, s3
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i1_ne0:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    s_cmp_gt_u32 s2, 1
-; VI-NEXT:    s_cselect_b64 s[4:5], -1, 0
-; VI-NEXT:    s_cmp_gt_u32 s3, 2
-; VI-NEXT:    s_cselect_b64 s[2:3], -1, 0
-; VI-NEXT:    s_and_b64 s[2:3], s[4:5], s[2:3]
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
-  %c0 = icmp ugt i32 %a, 1
-  %c1 = icmp ugt i32 %b, 2
-  %src = and i1 %c0, %c1
-  %result = call i64 @llvm.amdgcn.icmp.i1(i1 %src, i1 false, i32 33)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_ps void @test_intr_icmp_i32_invalid_cc(i32 addrspace(1)* %out, i32 %src) {
-; GCN-LABEL: test_intr_icmp_i32_invalid_cc:
-; GCN:       ; %bb.0:
-; GCN-NEXT:    s_endpgm
-  %result = call i32 @llvm.amdgcn.icmp.i32.i32(i32 %src, i32 100, i32 9999)
-  store i32 %result, i32 addrspace(1)* %out
-  ret void
-}
-
-attributes #0 = { nounwind readnone convergent }

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.icmp.w32.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -march=amdgcn -mcpu=gfx1100 -mattr="+wavefrontsize32" -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX11,SDAG-GFX11 %s
+; RUN: llc -march=amdgcn -mcpu=gfx1010 -mattr="+wavefrontsize32" -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX10,SDAG-GFX10 %s
+
+; RUN: llc -global-isel -global-isel-abort=2 -march=amdgcn -mcpu=gfx1100 -mattr="+wavefrontsize32" -verify-machineinstrs < %s 2>%t | FileCheck -check-prefixes=GCN,GFX11,GISEL-GFX11 %s
+; RUN: FileCheck --check-prefix=ERR %s < %t
+; RUN: llc -global-isel -global-isel-abort=2 -march=amdgcn -mcpu=gfx1010 -mattr="+wavefrontsize32" -verify-machineinstrs < %s 2>%t | FileCheck -check-prefixes=GCN,GFX10,GISEL-GFX10 %s
+; RUN: FileCheck --check-prefix=ERR %s < %t
+
+; Note: GlobalISel abort is disabled so we don't crash on i1 inputs.
+;  They are allowed in DAGISel but we (intentionally) don't support them
+;  in GlobalISel.
+
+; ERR: warning: Instruction selection used fallback path for v_icmp_i1_ne0
+
+declare i32 @llvm.amdgcn.icmp.i32(i32, i32, i32) #0
+declare i32 @llvm.amdgcn.icmp.i64(i64, i64, i32) #0
+declare i32 @llvm.amdgcn.icmp.i16(i16, i16, i32) #0
+declare i32 @llvm.amdgcn.icmp.i1(i1, i1, i32) #0
+
+define amdgpu_kernel void @v_icmp_i32_eq(i32 addrspace(1)* %out, i32 %src) {
+; SDAG-GFX11-LABEL: v_icmp_i32_eq:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_clause 0x1
+; SDAG-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; SDAG-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_cmp_eq_u32_e64 s2, 0x64, s2
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; SDAG-GFX11-NEXT:    global_store_b32 v0, v1, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_i32_eq:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_clause 0x1
+; SDAG-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; SDAG-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX10-NEXT:    v_cmp_eq_u32_e64 s0, 0x64, s4
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v1, s0
+; SDAG-GFX10-NEXT:    global_store_dword v0, v1, s[2:3]
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_i32_eq:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_clause 0x1
+; GISEL-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GISEL-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    v_cmp_eq_u32_e64 s2, 0x64, s2
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_i32_eq:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_clause 0x1
+; GISEL-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    v_cmp_eq_u32_e64 s0, 0x64, s4
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX10-NEXT:    global_store_dword v1, v0, s[2:3]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 32)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i32(i32 addrspace(1)* %out, i32 %src) {
+; SDAG-GFX11-LABEL: v_icmp_i32:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_i32:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_i32:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    global_store_b32 v0, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_i32:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    global_store_dword v0, v0, s[0:1]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 30)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i32_ne(i32 addrspace(1)* %out, i32 %src) {
+; SDAG-GFX11-LABEL: v_icmp_i32_ne:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_clause 0x1
+; SDAG-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; SDAG-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_cmp_ne_u32_e64 s2, 0x64, s2
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; SDAG-GFX11-NEXT:    global_store_b32 v0, v1, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_i32_ne:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_clause 0x1
+; SDAG-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; SDAG-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX10-NEXT:    v_cmp_ne_u32_e64 s0, 0x64, s4
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v1, s0
+; SDAG-GFX10-NEXT:    global_store_dword v0, v1, s[2:3]
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_i32_ne:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_clause 0x1
+; GISEL-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GISEL-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    v_cmp_ne_u32_e64 s2, 0x64, s2
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_i32_ne:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_clause 0x1
+; GISEL-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    v_cmp_ne_u32_e64 s0, 0x64, s4
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX10-NEXT:    global_store_dword v1, v0, s[2:3]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 33)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i32_ugt(i32 addrspace(1)* %out, i32 %src) {
+; SDAG-GFX11-LABEL: v_icmp_i32_ugt:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_clause 0x1
+; SDAG-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; SDAG-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_cmp_lt_u32_e64 s2, 0x64, s2
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; SDAG-GFX11-NEXT:    global_store_b32 v0, v1, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_i32_ugt:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_clause 0x1
+; SDAG-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; SDAG-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX10-NEXT:    v_cmp_lt_u32_e64 s0, 0x64, s4
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v1, s0
+; SDAG-GFX10-NEXT:    global_store_dword v0, v1, s[2:3]
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_i32_ugt:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_clause 0x1
+; GISEL-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GISEL-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    v_cmp_lt_u32_e64 s2, 0x64, s2
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_i32_ugt:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_clause 0x1
+; GISEL-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    v_cmp_lt_u32_e64 s0, 0x64, s4
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX10-NEXT:    global_store_dword v1, v0, s[2:3]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 34)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i32_uge(i32 addrspace(1)* %out, i32 %src) {
+; SDAG-GFX11-LABEL: v_icmp_i32_uge:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_clause 0x1
+; SDAG-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; SDAG-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_cmp_le_u32_e64 s2, 0x64, s2
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; SDAG-GFX11-NEXT:    global_store_b32 v0, v1, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_i32_uge:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_clause 0x1
+; SDAG-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; SDAG-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX10-NEXT:    v_cmp_le_u32_e64 s0, 0x64, s4
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v1, s0
+; SDAG-GFX10-NEXT:    global_store_dword v0, v1, s[2:3]
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_i32_uge:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_clause 0x1
+; GISEL-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GISEL-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    v_cmp_le_u32_e64 s2, 0x64, s2
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_i32_uge:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_clause 0x1
+; GISEL-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    v_cmp_le_u32_e64 s0, 0x64, s4
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX10-NEXT:    global_store_dword v1, v0, s[2:3]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 35)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i32_ult(i32 addrspace(1)* %out, i32 %src) {
+; SDAG-GFX11-LABEL: v_icmp_i32_ult:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_clause 0x1
+; SDAG-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; SDAG-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_cmp_gt_u32_e64 s2, 0x64, s2
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; SDAG-GFX11-NEXT:    global_store_b32 v0, v1, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_i32_ult:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_clause 0x1
+; SDAG-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; SDAG-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX10-NEXT:    v_cmp_gt_u32_e64 s0, 0x64, s4
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v1, s0
+; SDAG-GFX10-NEXT:    global_store_dword v0, v1, s[2:3]
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_i32_ult:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_clause 0x1
+; GISEL-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GISEL-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    v_cmp_gt_u32_e64 s2, 0x64, s2
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_i32_ult:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_clause 0x1
+; GISEL-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    v_cmp_gt_u32_e64 s0, 0x64, s4
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX10-NEXT:    global_store_dword v1, v0, s[2:3]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 36)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i32_ule(i32 addrspace(1)* %out, i32 %src) {
+; SDAG-GFX11-LABEL: v_icmp_i32_ule:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_clause 0x1
+; SDAG-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; SDAG-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_cmp_ge_u32_e64 s2, 0x64, s2
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; SDAG-GFX11-NEXT:    global_store_b32 v0, v1, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_i32_ule:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_clause 0x1
+; SDAG-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; SDAG-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX10-NEXT:    v_cmp_ge_u32_e64 s0, 0x64, s4
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v1, s0
+; SDAG-GFX10-NEXT:    global_store_dword v0, v1, s[2:3]
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_i32_ule:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_clause 0x1
+; GISEL-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GISEL-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    v_cmp_ge_u32_e64 s2, 0x64, s2
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_i32_ule:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_clause 0x1
+; GISEL-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    v_cmp_ge_u32_e64 s0, 0x64, s4
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX10-NEXT:    global_store_dword v1, v0, s[2:3]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 37)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i32_sgt(i32 addrspace(1)* %out, i32 %src) #1 {
+; SDAG-GFX11-LABEL: v_icmp_i32_sgt:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_clause 0x1
+; SDAG-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; SDAG-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_cmp_lt_i32_e64 s2, 0x64, s2
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; SDAG-GFX11-NEXT:    global_store_b32 v0, v1, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_i32_sgt:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_clause 0x1
+; SDAG-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; SDAG-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX10-NEXT:    v_cmp_lt_i32_e64 s0, 0x64, s4
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v1, s0
+; SDAG-GFX10-NEXT:    global_store_dword v0, v1, s[2:3]
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_i32_sgt:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_clause 0x1
+; GISEL-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GISEL-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    v_cmp_lt_i32_e64 s2, 0x64, s2
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_i32_sgt:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_clause 0x1
+; GISEL-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    v_cmp_lt_i32_e64 s0, 0x64, s4
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX10-NEXT:    global_store_dword v1, v0, s[2:3]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 38)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i32_sge(i32 addrspace(1)* %out, i32 %src) {
+; SDAG-GFX11-LABEL: v_icmp_i32_sge:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_clause 0x1
+; SDAG-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; SDAG-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_cmp_le_i32_e64 s2, 0x64, s2
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; SDAG-GFX11-NEXT:    global_store_b32 v0, v1, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_i32_sge:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_clause 0x1
+; SDAG-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; SDAG-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX10-NEXT:    v_cmp_le_i32_e64 s0, 0x64, s4
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v1, s0
+; SDAG-GFX10-NEXT:    global_store_dword v0, v1, s[2:3]
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_i32_sge:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_clause 0x1
+; GISEL-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GISEL-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    v_cmp_le_i32_e64 s2, 0x64, s2
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_i32_sge:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_clause 0x1
+; GISEL-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    v_cmp_le_i32_e64 s0, 0x64, s4
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX10-NEXT:    global_store_dword v1, v0, s[2:3]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 39)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i32_slt(i32 addrspace(1)* %out, i32 %src) {
+; SDAG-GFX11-LABEL: v_icmp_i32_slt:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_clause 0x1
+; SDAG-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; SDAG-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_cmp_gt_i32_e64 s2, 0x64, s2
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; SDAG-GFX11-NEXT:    global_store_b32 v0, v1, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_i32_slt:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_clause 0x1
+; SDAG-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; SDAG-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX10-NEXT:    v_cmp_gt_i32_e64 s0, 0x64, s4
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v1, s0
+; SDAG-GFX10-NEXT:    global_store_dword v0, v1, s[2:3]
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_i32_slt:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_clause 0x1
+; GISEL-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GISEL-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    v_cmp_gt_i32_e64 s2, 0x64, s2
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_i32_slt:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_clause 0x1
+; GISEL-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    v_cmp_gt_i32_e64 s0, 0x64, s4
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX10-NEXT:    global_store_dword v1, v0, s[2:3]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 40)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i32_sle(i32 addrspace(1)* %out, i32 %src) {
+; SDAG-GFX11-LABEL: v_icmp_i32_sle:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_clause 0x1
+; SDAG-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; SDAG-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_cmp_ge_i32_e64 s2, 0x64, s2
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; SDAG-GFX11-NEXT:    global_store_b32 v0, v1, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_i32_sle:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_clause 0x1
+; SDAG-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; SDAG-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX10-NEXT:    v_cmp_ge_i32_e64 s0, 0x64, s4
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v1, s0
+; SDAG-GFX10-NEXT:    global_store_dword v0, v1, s[2:3]
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_i32_sle:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_clause 0x1
+; GISEL-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GISEL-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    v_cmp_ge_i32_e64 s2, 0x64, s2
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_i32_sle:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_clause 0x1
+; GISEL-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    v_cmp_ge_i32_e64 s0, 0x64, s4
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX10-NEXT:    global_store_dword v1, v0, s[2:3]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 41)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i64_eq(i32 addrspace(1)* %out, i64 %src) {
+; SDAG-GFX11-LABEL: v_icmp_i64_eq:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    s_mov_b64 s[4:5], 0x64
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_cmp_eq_u64_e64 s2, s[2:3], s[4:5]
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; SDAG-GFX11-NEXT:    global_store_b32 v0, v1, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_i64_eq:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-GFX10-NEXT:    s_mov_b64 s[4:5], 0x64
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX10-NEXT:    v_cmp_eq_u64_e64 s2, s[2:3], s[4:5]
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v1, s2
+; SDAG-GFX10-NEXT:    global_store_dword v0, v1, s[0:1]
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_i64_eq:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    v_cmp_eq_u64_e64 s2, s[2:3], s[4:5]
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_i64_eq:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    v_cmp_eq_u64_e64 s2, s[2:3], s[4:5]
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX10-NEXT:    global_store_dword v1, v0, s[0:1]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 32)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i64_ne(i32 addrspace(1)* %out, i64 %src) {
+; SDAG-GFX11-LABEL: v_icmp_i64_ne:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    s_mov_b64 s[4:5], 0x64
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_cmp_ne_u64_e64 s2, s[2:3], s[4:5]
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; SDAG-GFX11-NEXT:    global_store_b32 v0, v1, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_i64_ne:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-GFX10-NEXT:    s_mov_b64 s[4:5], 0x64
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX10-NEXT:    v_cmp_ne_u64_e64 s2, s[2:3], s[4:5]
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v1, s2
+; SDAG-GFX10-NEXT:    global_store_dword v0, v1, s[0:1]
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_i64_ne:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    v_cmp_ne_u64_e64 s2, s[2:3], s[4:5]
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_i64_ne:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    v_cmp_ne_u64_e64 s2, s[2:3], s[4:5]
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX10-NEXT:    global_store_dword v1, v0, s[0:1]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 33)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_u64_ugt(i32 addrspace(1)* %out, i64 %src) {
+; SDAG-GFX11-LABEL: v_icmp_u64_ugt:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    s_mov_b64 s[4:5], 0x64
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_cmp_gt_u64_e64 s2, s[2:3], s[4:5]
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; SDAG-GFX11-NEXT:    global_store_b32 v0, v1, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_u64_ugt:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-GFX10-NEXT:    s_mov_b64 s[4:5], 0x64
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX10-NEXT:    v_cmp_gt_u64_e64 s2, s[2:3], s[4:5]
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v1, s2
+; SDAG-GFX10-NEXT:    global_store_dword v0, v1, s[0:1]
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_u64_ugt:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    v_cmp_gt_u64_e64 s2, s[2:3], s[4:5]
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_u64_ugt:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    v_cmp_gt_u64_e64 s2, s[2:3], s[4:5]
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX10-NEXT:    global_store_dword v1, v0, s[0:1]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 34)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_u64_uge(i32 addrspace(1)* %out, i64 %src) {
+; SDAG-GFX11-LABEL: v_icmp_u64_uge:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    s_mov_b64 s[4:5], 0x64
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_cmp_ge_u64_e64 s2, s[2:3], s[4:5]
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; SDAG-GFX11-NEXT:    global_store_b32 v0, v1, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_u64_uge:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-GFX10-NEXT:    s_mov_b64 s[4:5], 0x64
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX10-NEXT:    v_cmp_ge_u64_e64 s2, s[2:3], s[4:5]
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v1, s2
+; SDAG-GFX10-NEXT:    global_store_dword v0, v1, s[0:1]
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_u64_uge:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    v_cmp_ge_u64_e64 s2, s[2:3], s[4:5]
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_u64_uge:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    v_cmp_ge_u64_e64 s2, s[2:3], s[4:5]
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX10-NEXT:    global_store_dword v1, v0, s[0:1]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 35)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_u64_ult(i32 addrspace(1)* %out, i64 %src) {
+; SDAG-GFX11-LABEL: v_icmp_u64_ult:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    s_mov_b64 s[4:5], 0x64
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_cmp_lt_u64_e64 s2, s[2:3], s[4:5]
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; SDAG-GFX11-NEXT:    global_store_b32 v0, v1, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_u64_ult:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-GFX10-NEXT:    s_mov_b64 s[4:5], 0x64
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX10-NEXT:    v_cmp_lt_u64_e64 s2, s[2:3], s[4:5]
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v1, s2
+; SDAG-GFX10-NEXT:    global_store_dword v0, v1, s[0:1]
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_u64_ult:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    v_cmp_lt_u64_e64 s2, s[2:3], s[4:5]
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_u64_ult:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    v_cmp_lt_u64_e64 s2, s[2:3], s[4:5]
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX10-NEXT:    global_store_dword v1, v0, s[0:1]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 36)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_u64_ule(i32 addrspace(1)* %out, i64 %src) {
+; SDAG-GFX11-LABEL: v_icmp_u64_ule:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    s_mov_b64 s[4:5], 0x64
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_cmp_le_u64_e64 s2, s[2:3], s[4:5]
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; SDAG-GFX11-NEXT:    global_store_b32 v0, v1, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_u64_ule:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-GFX10-NEXT:    s_mov_b64 s[4:5], 0x64
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX10-NEXT:    v_cmp_le_u64_e64 s2, s[2:3], s[4:5]
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v1, s2
+; SDAG-GFX10-NEXT:    global_store_dword v0, v1, s[0:1]
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_u64_ule:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    v_cmp_le_u64_e64 s2, s[2:3], s[4:5]
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_u64_ule:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    v_cmp_le_u64_e64 s2, s[2:3], s[4:5]
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX10-NEXT:    global_store_dword v1, v0, s[0:1]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 37)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i64_sgt(i32 addrspace(1)* %out, i64 %src) {
+; SDAG-GFX11-LABEL: v_icmp_i64_sgt:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    s_mov_b64 s[4:5], 0x64
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_cmp_gt_i64_e64 s2, s[2:3], s[4:5]
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; SDAG-GFX11-NEXT:    global_store_b32 v0, v1, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_i64_sgt:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-GFX10-NEXT:    s_mov_b64 s[4:5], 0x64
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX10-NEXT:    v_cmp_gt_i64_e64 s2, s[2:3], s[4:5]
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v1, s2
+; SDAG-GFX10-NEXT:    global_store_dword v0, v1, s[0:1]
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_i64_sgt:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    v_cmp_gt_i64_e64 s2, s[2:3], s[4:5]
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_i64_sgt:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    v_cmp_gt_i64_e64 s2, s[2:3], s[4:5]
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX10-NEXT:    global_store_dword v1, v0, s[0:1]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 38)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i64_sge(i32 addrspace(1)* %out, i64 %src) {
+; SDAG-GFX11-LABEL: v_icmp_i64_sge:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    s_mov_b64 s[4:5], 0x64
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_cmp_ge_i64_e64 s2, s[2:3], s[4:5]
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; SDAG-GFX11-NEXT:    global_store_b32 v0, v1, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_i64_sge:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-GFX10-NEXT:    s_mov_b64 s[4:5], 0x64
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX10-NEXT:    v_cmp_ge_i64_e64 s2, s[2:3], s[4:5]
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v1, s2
+; SDAG-GFX10-NEXT:    global_store_dword v0, v1, s[0:1]
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_i64_sge:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    v_cmp_ge_i64_e64 s2, s[2:3], s[4:5]
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_i64_sge:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    v_cmp_ge_i64_e64 s2, s[2:3], s[4:5]
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX10-NEXT:    global_store_dword v1, v0, s[0:1]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 39)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i64_slt(i32 addrspace(1)* %out, i64 %src) {
+; SDAG-GFX11-LABEL: v_icmp_i64_slt:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    s_mov_b64 s[4:5], 0x64
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_cmp_lt_i64_e64 s2, s[2:3], s[4:5]
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; SDAG-GFX11-NEXT:    global_store_b32 v0, v1, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_i64_slt:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-GFX10-NEXT:    s_mov_b64 s[4:5], 0x64
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX10-NEXT:    v_cmp_lt_i64_e64 s2, s[2:3], s[4:5]
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v1, s2
+; SDAG-GFX10-NEXT:    global_store_dword v0, v1, s[0:1]
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_i64_slt:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    v_cmp_lt_i64_e64 s2, s[2:3], s[4:5]
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_i64_slt:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    v_cmp_lt_i64_e64 s2, s[2:3], s[4:5]
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX10-NEXT:    global_store_dword v1, v0, s[0:1]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 40)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i64_sle(i32 addrspace(1)* %out, i64 %src) {
+; SDAG-GFX11-LABEL: v_icmp_i64_sle:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    s_mov_b64 s[4:5], 0x64
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_cmp_le_i64_e64 s2, s[2:3], s[4:5]
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; SDAG-GFX11-NEXT:    global_store_b32 v0, v1, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_i64_sle:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-GFX10-NEXT:    s_mov_b64 s[4:5], 0x64
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX10-NEXT:    v_cmp_le_i64_e64 s2, s[2:3], s[4:5]
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v1, s2
+; SDAG-GFX10-NEXT:    global_store_dword v0, v1, s[0:1]
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_i64_sle:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    v_cmp_le_i64_e64 s2, s[2:3], s[4:5]
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_i64_sle:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    v_cmp_le_i64_e64 s2, s[2:3], s[4:5]
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX10-NEXT:    global_store_dword v1, v0, s[0:1]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 41)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i16_eq(i32 addrspace(1)* %out, i16 %src) {
+; SDAG-GFX11-LABEL: v_icmp_i16_eq:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_clause 0x1
+; SDAG-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; SDAG-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_cmp_eq_u16_e64 s2, 0x64, s2
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; SDAG-GFX11-NEXT:    global_store_b32 v0, v1, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_i16_eq:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_clause 0x1
+; SDAG-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; SDAG-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX10-NEXT:    v_cmp_eq_u16_e64 s0, 0x64, s4
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v1, s0
+; SDAG-GFX10-NEXT:    global_store_dword v0, v1, s[2:3]
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_i16_eq:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_clause 0x1
+; GISEL-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GISEL-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    v_cmp_eq_u16_e64 s2, 0x64, s2
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_i16_eq:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_clause 0x1
+; GISEL-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    v_cmp_eq_u16_e64 s0, 0x64, s4
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX10-NEXT:    global_store_dword v1, v0, s[2:3]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 32)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i16(i32 addrspace(1)* %out, i16 %src) {
+; SDAG-GFX11-LABEL: v_icmp_i16:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_i16:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_i16:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    global_store_b32 v0, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_i16:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    global_store_dword v0, v0, s[0:1]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 30)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i16_ne(i32 addrspace(1)* %out, i16 %src) {
+; SDAG-GFX11-LABEL: v_icmp_i16_ne:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_clause 0x1
+; SDAG-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; SDAG-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_cmp_ne_u16_e64 s2, 0x64, s2
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; SDAG-GFX11-NEXT:    global_store_b32 v0, v1, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_i16_ne:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_clause 0x1
+; SDAG-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; SDAG-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX10-NEXT:    v_cmp_ne_u16_e64 s0, 0x64, s4
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v1, s0
+; SDAG-GFX10-NEXT:    global_store_dword v0, v1, s[2:3]
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_i16_ne:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_clause 0x1
+; GISEL-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GISEL-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    v_cmp_ne_u16_e64 s2, 0x64, s2
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_i16_ne:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_clause 0x1
+; GISEL-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    v_cmp_ne_u16_e64 s0, 0x64, s4
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX10-NEXT:    global_store_dword v1, v0, s[2:3]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 33)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i16_ugt(i32 addrspace(1)* %out, i16 %src) {
+; SDAG-GFX11-LABEL: v_icmp_i16_ugt:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_clause 0x1
+; SDAG-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; SDAG-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_cmp_lt_u16_e64 s2, 0x64, s2
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; SDAG-GFX11-NEXT:    global_store_b32 v0, v1, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_i16_ugt:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_clause 0x1
+; SDAG-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; SDAG-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX10-NEXT:    v_cmp_lt_u16_e64 s0, 0x64, s4
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v1, s0
+; SDAG-GFX10-NEXT:    global_store_dword v0, v1, s[2:3]
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_i16_ugt:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_clause 0x1
+; GISEL-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GISEL-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    v_cmp_lt_u16_e64 s2, 0x64, s2
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_i16_ugt:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_clause 0x1
+; GISEL-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    v_cmp_lt_u16_e64 s0, 0x64, s4
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX10-NEXT:    global_store_dword v1, v0, s[2:3]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 34)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i16_uge(i32 addrspace(1)* %out, i16 %src) {
+; SDAG-GFX11-LABEL: v_icmp_i16_uge:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_clause 0x1
+; SDAG-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; SDAG-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_cmp_le_u16_e64 s2, 0x64, s2
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; SDAG-GFX11-NEXT:    global_store_b32 v0, v1, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_i16_uge:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_clause 0x1
+; SDAG-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; SDAG-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX10-NEXT:    v_cmp_le_u16_e64 s0, 0x64, s4
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v1, s0
+; SDAG-GFX10-NEXT:    global_store_dword v0, v1, s[2:3]
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_i16_uge:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_clause 0x1
+; GISEL-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GISEL-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    v_cmp_le_u16_e64 s2, 0x64, s2
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_i16_uge:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_clause 0x1
+; GISEL-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    v_cmp_le_u16_e64 s0, 0x64, s4
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX10-NEXT:    global_store_dword v1, v0, s[2:3]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 35)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i16_ult(i32 addrspace(1)* %out, i16 %src) {
+; SDAG-GFX11-LABEL: v_icmp_i16_ult:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_clause 0x1
+; SDAG-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; SDAG-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_cmp_gt_u16_e64 s2, 0x64, s2
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; SDAG-GFX11-NEXT:    global_store_b32 v0, v1, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_i16_ult:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_clause 0x1
+; SDAG-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; SDAG-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX10-NEXT:    v_cmp_gt_u16_e64 s0, 0x64, s4
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v1, s0
+; SDAG-GFX10-NEXT:    global_store_dword v0, v1, s[2:3]
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_i16_ult:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_clause 0x1
+; GISEL-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GISEL-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    v_cmp_gt_u16_e64 s2, 0x64, s2
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_i16_ult:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_clause 0x1
+; GISEL-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    v_cmp_gt_u16_e64 s0, 0x64, s4
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX10-NEXT:    global_store_dword v1, v0, s[2:3]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 36)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i16_ule(i32 addrspace(1)* %out, i16 %src) {
+; SDAG-GFX11-LABEL: v_icmp_i16_ule:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_clause 0x1
+; SDAG-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; SDAG-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_cmp_ge_u16_e64 s2, 0x64, s2
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; SDAG-GFX11-NEXT:    global_store_b32 v0, v1, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_i16_ule:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_clause 0x1
+; SDAG-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; SDAG-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX10-NEXT:    v_cmp_ge_u16_e64 s0, 0x64, s4
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v1, s0
+; SDAG-GFX10-NEXT:    global_store_dword v0, v1, s[2:3]
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_i16_ule:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_clause 0x1
+; GISEL-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GISEL-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    v_cmp_ge_u16_e64 s2, 0x64, s2
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_i16_ule:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_clause 0x1
+; GISEL-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    v_cmp_ge_u16_e64 s0, 0x64, s4
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX10-NEXT:    global_store_dword v1, v0, s[2:3]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 37)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i16_sgt(i32 addrspace(1)* %out, i16 %src) #1 {
+; SDAG-GFX11-LABEL: v_icmp_i16_sgt:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_clause 0x1
+; SDAG-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; SDAG-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_cmp_lt_i16_e64 s2, 0x64, s2
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; SDAG-GFX11-NEXT:    global_store_b32 v0, v1, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_i16_sgt:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_clause 0x1
+; SDAG-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; SDAG-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX10-NEXT:    v_cmp_lt_i16_e64 s0, 0x64, s4
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v1, s0
+; SDAG-GFX10-NEXT:    global_store_dword v0, v1, s[2:3]
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_i16_sgt:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_clause 0x1
+; GISEL-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GISEL-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    v_cmp_lt_i16_e64 s2, 0x64, s2
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_i16_sgt:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_clause 0x1
+; GISEL-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    v_cmp_lt_i16_e64 s0, 0x64, s4
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX10-NEXT:    global_store_dword v1, v0, s[2:3]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 38)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i16_sge(i32 addrspace(1)* %out, i16 %src) {
+; SDAG-GFX11-LABEL: v_icmp_i16_sge:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_clause 0x1
+; SDAG-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; SDAG-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_cmp_le_i16_e64 s2, 0x64, s2
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; SDAG-GFX11-NEXT:    global_store_b32 v0, v1, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_i16_sge:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_clause 0x1
+; SDAG-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; SDAG-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX10-NEXT:    v_cmp_le_i16_e64 s0, 0x64, s4
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v1, s0
+; SDAG-GFX10-NEXT:    global_store_dword v0, v1, s[2:3]
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_i16_sge:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_clause 0x1
+; GISEL-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GISEL-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    v_cmp_le_i16_e64 s2, 0x64, s2
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_i16_sge:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_clause 0x1
+; GISEL-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    v_cmp_le_i16_e64 s0, 0x64, s4
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX10-NEXT:    global_store_dword v1, v0, s[2:3]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 39)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i16_slt(i32 addrspace(1)* %out, i16 %src) {
+; SDAG-GFX11-LABEL: v_icmp_i16_slt:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_clause 0x1
+; SDAG-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; SDAG-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_cmp_gt_i16_e64 s2, 0x64, s2
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; SDAG-GFX11-NEXT:    global_store_b32 v0, v1, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_i16_slt:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_clause 0x1
+; SDAG-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; SDAG-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX10-NEXT:    v_cmp_gt_i16_e64 s0, 0x64, s4
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v1, s0
+; SDAG-GFX10-NEXT:    global_store_dword v0, v1, s[2:3]
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_i16_slt:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_clause 0x1
+; GISEL-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GISEL-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    v_cmp_gt_i16_e64 s2, 0x64, s2
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_i16_slt:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_clause 0x1
+; GISEL-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    v_cmp_gt_i16_e64 s0, 0x64, s4
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX10-NEXT:    global_store_dword v1, v0, s[2:3]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 40)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i16_sle(i32 addrspace(1)* %out, i16 %src) {
+; SDAG-GFX11-LABEL: v_icmp_i16_sle:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_clause 0x1
+; SDAG-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; SDAG-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; SDAG-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX11-NEXT:    v_cmp_ge_i16_e64 s2, 0x64, s2
+; SDAG-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; SDAG-GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; SDAG-GFX11-NEXT:    global_store_b32 v0, v1, s[0:1]
+; SDAG-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: v_icmp_i16_sle:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_clause 0x1
+; SDAG-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; SDAG-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; SDAG-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX10-NEXT:    v_cmp_ge_i16_e64 s0, 0x64, s4
+; SDAG-GFX10-NEXT:    v_mov_b32_e32 v1, s0
+; SDAG-GFX10-NEXT:    global_store_dword v0, v1, s[2:3]
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_i16_sle:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_clause 0x1
+; GISEL-GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GISEL-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    v_cmp_ge_i16_e64 s2, 0x64, s2
+; GISEL-GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1)
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX11-NEXT:    global_store_b32 v1, v0, s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: v_icmp_i16_sle:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    s_clause 0x1
+; GISEL-GFX10-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX10-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v1, 0
+; GISEL-GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX10-NEXT:    v_cmp_ge_i16_e64 s0, 0x64, s4
+; GISEL-GFX10-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX10-NEXT:    global_store_dword v1, v0, s[2:3]
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 41)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i1_ne0(i32 addrspace(1)* %out, i32 %a, i32 %b) {
+; GFX11-LABEL: v_icmp_i1_ne0:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    s_cmp_gt_u32 s2, 1
+; GFX11-NEXT:    s_cselect_b32 s2, -1, 0
+; GFX11-NEXT:    s_cmp_gt_u32 s3, 2
+; GFX11-NEXT:    s_cselect_b32 s3, -1, 0
+; GFX11-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX11-NEXT:    s_and_b32 s2, s2, s3
+; GFX11-NEXT:    v_dual_mov_b32 v0, 0 :: v_dual_mov_b32 v1, s2
+; GFX11-NEXT:    global_store_b32 v0, v1, s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; GFX10-LABEL: v_icmp_i1_ne0:
+; GFX10:       ; %bb.0:
+; GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX10-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    s_cmp_gt_u32 s2, 1
+; GFX10-NEXT:    s_cselect_b32 s2, -1, 0
+; GFX10-NEXT:    s_cmp_gt_u32 s3, 2
+; GFX10-NEXT:    s_cselect_b32 s3, -1, 0
+; GFX10-NEXT:    s_and_b32 s2, s2, s3
+; GFX10-NEXT:    v_mov_b32_e32 v1, s2
+; GFX10-NEXT:    global_store_dword v0, v1, s[0:1]
+; GFX10-NEXT:    s_endpgm
+  %c0 = icmp ugt i32 %a, 1
+  %c1 = icmp ugt i32 %b, 2
+  %src = and i1 %c0, %c1
+  %result = call i32 @llvm.amdgcn.icmp.i1(i1 %src, i1 false, i32 33)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_ps void @test_intr_icmp_i32_invalid_cc(i32 addrspace(1)* %out, i32 %src) {
+; SDAG-GFX11-LABEL: test_intr_icmp_i32_invalid_cc:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-GFX10-LABEL: test_intr_icmp_i32_invalid_cc:
+; SDAG-GFX10:       ; %bb.0:
+; SDAG-GFX10-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: test_intr_icmp_i32_invalid_cc:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    global_store_b32 v[0:1], v0, off
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-GFX10-LABEL: test_intr_icmp_i32_invalid_cc:
+; GISEL-GFX10:       ; %bb.0:
+; GISEL-GFX10-NEXT:    global_store_dword v[0:1], v0, off
+; GISEL-GFX10-NEXT:    s_endpgm
+  %result = call i32 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 9999)
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+attributes #0 = { nounwind readnone convergent }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GCN: {{.*}}

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.icmp.w64.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -march=amdgcn -mcpu=gfx1100 -mattr="+wavefrontsize64" -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX11,SDAG-GFX11 %s
+; RUN: llc -march=amdgcn -mcpu=fiji -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,VI,SDAG-VI %s
+; RUN: llc -march=amdgcn -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX9,SDAG-GFX9 %s
+
+; RUN: llc -global-isel -global-isel-abort=2 -march=amdgcn -mcpu=gfx1100 -mattr="+wavefrontsize64" -verify-machineinstrs < %s 2>%t | FileCheck -check-prefixes=GCN,GFX11,GISEL-GFX11 %s
+; RUN: FileCheck --check-prefix=ERR %s < %t
+; RUN: llc -global-isel -global-isel-abort=2 -march=amdgcn -mcpu=fiji -verify-machineinstrs < %s 2>%t | FileCheck -check-prefixes=GCN,VI,GISEL-VI %s
+; RUN: FileCheck --check-prefix=ERR %s < %t
+; RUN: llc -global-isel -global-isel-abort=2 -march=amdgcn -mcpu=gfx900 -verify-machineinstrs < %s 2>%t | FileCheck -check-prefixes=GCN,GFX9,GISEL-GFX9 %s
+; RUN: FileCheck --check-prefix=ERR %s < %t
+
+; Note: GlobalISel abort is disabled so we don't crash on i1 inputs.
+;  They are allowed in DAGISel but we (intentionally) don't support them
+;  in GlobalISel.
+
+; ERR: warning: Instruction selection used fallback path for v_icmp_i1_ne0
+
+declare i64 @llvm.amdgcn.icmp.i32(i32, i32, i32) #0
+declare i64 @llvm.amdgcn.icmp.i64(i64, i64, i32) #0
+declare i64 @llvm.amdgcn.icmp.i16(i16, i16, i32) #0
+declare i64 @llvm.amdgcn.icmp.i1(i1, i1, i32) #0
+
+define amdgpu_kernel void @v_icmp_i32_eq(i64 addrspace(1)* %out, i32 %src) {
+; GFX11-LABEL: v_icmp_i32_eq:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_clause 0x1
+; GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_cmp_eq_u32_e64 s[2:3], 0x64, s2
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    v_mov_b32_e32 v1, s3
+; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_i32_eq:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; SDAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    v_cmp_eq_u32_e64 s[2:3], s2, v0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; SDAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; SDAG-VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: v_icmp_i32_eq:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0x64
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_cmp_eq_u32_e64 s[0:1], s4, v0
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mov_b32_e32 v1, s1
+; GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GFX9-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i32_eq:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_eq_u32_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 32)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i32(i64 addrspace(1)* %out, i32 %src) {
+; SDAG-GFX11-LABEL: v_icmp_i32:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_i32:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_endpgm
+;
+; SDAG-GFX9-LABEL: v_icmp_i32:
+; SDAG-GFX9:       ; %bb.0:
+; SDAG-GFX9-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_i32:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    global_store_b64 v0, v[0:1], s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i32:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s1
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[0:1], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX9-LABEL: v_icmp_i32:
+; GISEL-GFX9:       ; %bb.0:
+; GISEL-GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v0, 0
+; GISEL-GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX9-NEXT:    global_store_dwordx2 v0, v[0:1], s[0:1]
+; GISEL-GFX9-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 30)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i32_ne(i64 addrspace(1)* %out, i32 %src) {
+; GFX11-LABEL: v_icmp_i32_ne:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_clause 0x1
+; GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_cmp_ne_u32_e64 s[2:3], 0x64, s2
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    v_mov_b32_e32 v1, s3
+; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_i32_ne:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; SDAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    v_cmp_ne_u32_e64 s[2:3], s2, v0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; SDAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; SDAG-VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: v_icmp_i32_ne:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0x64
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_cmp_ne_u32_e64 s[0:1], s4, v0
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mov_b32_e32 v1, s1
+; GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GFX9-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i32_ne:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_ne_u32_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 33)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i32_ugt(i64 addrspace(1)* %out, i32 %src) {
+; GFX11-LABEL: v_icmp_i32_ugt:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_clause 0x1
+; GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_cmp_lt_u32_e64 s[2:3], 0x64, s2
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    v_mov_b32_e32 v1, s3
+; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_i32_ugt:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; SDAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    v_cmp_gt_u32_e64 s[2:3], s2, v0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; SDAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; SDAG-VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: v_icmp_i32_ugt:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0x64
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_cmp_gt_u32_e64 s[0:1], s4, v0
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mov_b32_e32 v1, s1
+; GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GFX9-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i32_ugt:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_gt_u32_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 34)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i32_uge(i64 addrspace(1)* %out, i32 %src) {
+; GFX11-LABEL: v_icmp_i32_uge:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_clause 0x1
+; GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_cmp_le_u32_e64 s[2:3], 0x64, s2
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    v_mov_b32_e32 v1, s3
+; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_i32_uge:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; SDAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    v_cmp_ge_u32_e64 s[2:3], s2, v0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; SDAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; SDAG-VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: v_icmp_i32_uge:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0x64
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_cmp_ge_u32_e64 s[0:1], s4, v0
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mov_b32_e32 v1, s1
+; GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GFX9-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i32_uge:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_ge_u32_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 35)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i32_ult(i64 addrspace(1)* %out, i32 %src) {
+; GFX11-LABEL: v_icmp_i32_ult:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_clause 0x1
+; GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_cmp_gt_u32_e64 s[2:3], 0x64, s2
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    v_mov_b32_e32 v1, s3
+; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_i32_ult:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; SDAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    v_cmp_lt_u32_e64 s[2:3], s2, v0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; SDAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; SDAG-VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: v_icmp_i32_ult:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0x64
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_cmp_lt_u32_e64 s[0:1], s4, v0
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mov_b32_e32 v1, s1
+; GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GFX9-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i32_ult:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_lt_u32_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 36)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i32_ule(i64 addrspace(1)* %out, i32 %src) {
+; GFX11-LABEL: v_icmp_i32_ule:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_clause 0x1
+; GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_cmp_ge_u32_e64 s[2:3], 0x64, s2
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    v_mov_b32_e32 v1, s3
+; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_i32_ule:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; SDAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    v_cmp_le_u32_e64 s[2:3], s2, v0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; SDAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; SDAG-VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: v_icmp_i32_ule:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0x64
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_cmp_le_u32_e64 s[0:1], s4, v0
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mov_b32_e32 v1, s1
+; GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GFX9-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i32_ule:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_le_u32_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 37)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i32_sgt(i64 addrspace(1)* %out, i32 %src) #1 {
+; GFX11-LABEL: v_icmp_i32_sgt:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_clause 0x1
+; GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_cmp_lt_i32_e64 s[2:3], 0x64, s2
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    v_mov_b32_e32 v1, s3
+; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_i32_sgt:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; SDAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    v_cmp_gt_i32_e64 s[2:3], s2, v0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; SDAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; SDAG-VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: v_icmp_i32_sgt:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0x64
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_cmp_gt_i32_e64 s[0:1], s4, v0
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mov_b32_e32 v1, s1
+; GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GFX9-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i32_sgt:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_gt_i32_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 38)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i32_sge(i64 addrspace(1)* %out, i32 %src) {
+; GFX11-LABEL: v_icmp_i32_sge:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_clause 0x1
+; GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_cmp_le_i32_e64 s[2:3], 0x64, s2
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    v_mov_b32_e32 v1, s3
+; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_i32_sge:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; SDAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    v_cmp_ge_i32_e64 s[2:3], s2, v0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; SDAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; SDAG-VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: v_icmp_i32_sge:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0x64
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_cmp_ge_i32_e64 s[0:1], s4, v0
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mov_b32_e32 v1, s1
+; GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GFX9-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i32_sge:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_ge_i32_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 39)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i32_slt(i64 addrspace(1)* %out, i32 %src) {
+; GFX11-LABEL: v_icmp_i32_slt:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_clause 0x1
+; GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_cmp_gt_i32_e64 s[2:3], 0x64, s2
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    v_mov_b32_e32 v1, s3
+; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_i32_slt:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; SDAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    v_cmp_lt_i32_e64 s[2:3], s2, v0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; SDAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; SDAG-VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: v_icmp_i32_slt:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0x64
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_cmp_lt_i32_e64 s[0:1], s4, v0
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mov_b32_e32 v1, s1
+; GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GFX9-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i32_slt:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_lt_i32_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 40)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i32_sle(i64 addrspace(1)* %out, i32 %src) {
+; GFX11-LABEL: v_icmp_i32_sle:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_clause 0x1
+; GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_cmp_ge_i32_e64 s[2:3], 0x64, s2
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    v_mov_b32_e32 v1, s3
+; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_i32_sle:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; SDAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    v_cmp_le_i32_e64 s[2:3], s2, v0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; SDAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; SDAG-VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: v_icmp_i32_sle:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0x64
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_cmp_le_i32_e64 s[0:1], s4, v0
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mov_b32_e32 v1, s1
+; GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GFX9-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i32_sle:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_le_i32_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 41)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i64_eq(i64 addrspace(1)* %out, i64 %src) {
+; GFX11-LABEL: v_icmp_i64_eq:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; GFX11-NEXT:    s_mov_b64 s[4:5], 0x64
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_cmp_eq_u64_e64 s[2:3], s[2:3], s[4:5]
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    v_mov_b32_e32 v1, s3
+; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_i64_eq:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, 0
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    v_cmp_eq_u64_e64 s[2:3], s[2:3], v[0:1]
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s2
+; SDAG-VI-NEXT:    v_mov_b32_e32 v3, s1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s3
+; SDAG-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; SDAG-VI-NEXT:    s_endpgm
+;
+; SDAG-GFX9-LABEL: v_icmp_i64_eq:
+; SDAG-GFX9:       ; %bb.0:
+; SDAG-GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v1, 0
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX9-NEXT:    v_cmp_eq_u64_e64 s[2:3], s[2:3], v[0:1]
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v2, s2
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v3, s3
+; SDAG-GFX9-NEXT:    global_store_dwordx2 v1, v[2:3], s[0:1]
+; SDAG-GFX9-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i64_eq:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-VI-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_eq_u64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX9-LABEL: v_icmp_i64_eq:
+; GISEL-GFX9:       ; %bb.0:
+; GISEL-GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-GFX9-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX9-NEXT:    v_cmp_eq_u64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GISEL-GFX9-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 32)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i64_ne(i64 addrspace(1)* %out, i64 %src) {
+; GFX11-LABEL: v_icmp_i64_ne:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; GFX11-NEXT:    s_mov_b64 s[4:5], 0x64
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_cmp_ne_u64_e64 s[2:3], s[2:3], s[4:5]
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    v_mov_b32_e32 v1, s3
+; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_i64_ne:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, 0
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    v_cmp_ne_u64_e64 s[2:3], s[2:3], v[0:1]
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s2
+; SDAG-VI-NEXT:    v_mov_b32_e32 v3, s1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s3
+; SDAG-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; SDAG-VI-NEXT:    s_endpgm
+;
+; SDAG-GFX9-LABEL: v_icmp_i64_ne:
+; SDAG-GFX9:       ; %bb.0:
+; SDAG-GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v1, 0
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX9-NEXT:    v_cmp_ne_u64_e64 s[2:3], s[2:3], v[0:1]
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v2, s2
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v3, s3
+; SDAG-GFX9-NEXT:    global_store_dwordx2 v1, v[2:3], s[0:1]
+; SDAG-GFX9-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i64_ne:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-VI-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_ne_u64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX9-LABEL: v_icmp_i64_ne:
+; GISEL-GFX9:       ; %bb.0:
+; GISEL-GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-GFX9-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX9-NEXT:    v_cmp_ne_u64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GISEL-GFX9-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 33)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_u64_ugt(i64 addrspace(1)* %out, i64 %src) {
+; GFX11-LABEL: v_icmp_u64_ugt:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; GFX11-NEXT:    s_mov_b64 s[4:5], 0x64
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_cmp_gt_u64_e64 s[2:3], s[2:3], s[4:5]
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    v_mov_b32_e32 v1, s3
+; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_u64_ugt:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, 0
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    v_cmp_gt_u64_e64 s[2:3], s[2:3], v[0:1]
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s2
+; SDAG-VI-NEXT:    v_mov_b32_e32 v3, s1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s3
+; SDAG-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; SDAG-VI-NEXT:    s_endpgm
+;
+; SDAG-GFX9-LABEL: v_icmp_u64_ugt:
+; SDAG-GFX9:       ; %bb.0:
+; SDAG-GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v1, 0
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX9-NEXT:    v_cmp_gt_u64_e64 s[2:3], s[2:3], v[0:1]
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v2, s2
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v3, s3
+; SDAG-GFX9-NEXT:    global_store_dwordx2 v1, v[2:3], s[0:1]
+; SDAG-GFX9-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_u64_ugt:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-VI-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_gt_u64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX9-LABEL: v_icmp_u64_ugt:
+; GISEL-GFX9:       ; %bb.0:
+; GISEL-GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-GFX9-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX9-NEXT:    v_cmp_gt_u64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GISEL-GFX9-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 34)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_u64_uge(i64 addrspace(1)* %out, i64 %src) {
+; GFX11-LABEL: v_icmp_u64_uge:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; GFX11-NEXT:    s_mov_b64 s[4:5], 0x64
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_cmp_ge_u64_e64 s[2:3], s[2:3], s[4:5]
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    v_mov_b32_e32 v1, s3
+; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_u64_uge:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, 0
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    v_cmp_ge_u64_e64 s[2:3], s[2:3], v[0:1]
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s2
+; SDAG-VI-NEXT:    v_mov_b32_e32 v3, s1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s3
+; SDAG-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; SDAG-VI-NEXT:    s_endpgm
+;
+; SDAG-GFX9-LABEL: v_icmp_u64_uge:
+; SDAG-GFX9:       ; %bb.0:
+; SDAG-GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v1, 0
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX9-NEXT:    v_cmp_ge_u64_e64 s[2:3], s[2:3], v[0:1]
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v2, s2
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v3, s3
+; SDAG-GFX9-NEXT:    global_store_dwordx2 v1, v[2:3], s[0:1]
+; SDAG-GFX9-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_u64_uge:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-VI-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_ge_u64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX9-LABEL: v_icmp_u64_uge:
+; GISEL-GFX9:       ; %bb.0:
+; GISEL-GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-GFX9-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX9-NEXT:    v_cmp_ge_u64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GISEL-GFX9-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 35)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_u64_ult(i64 addrspace(1)* %out, i64 %src) {
+; GFX11-LABEL: v_icmp_u64_ult:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; GFX11-NEXT:    s_mov_b64 s[4:5], 0x64
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_cmp_lt_u64_e64 s[2:3], s[2:3], s[4:5]
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    v_mov_b32_e32 v1, s3
+; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_u64_ult:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, 0
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    v_cmp_lt_u64_e64 s[2:3], s[2:3], v[0:1]
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s2
+; SDAG-VI-NEXT:    v_mov_b32_e32 v3, s1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s3
+; SDAG-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; SDAG-VI-NEXT:    s_endpgm
+;
+; SDAG-GFX9-LABEL: v_icmp_u64_ult:
+; SDAG-GFX9:       ; %bb.0:
+; SDAG-GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v1, 0
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX9-NEXT:    v_cmp_lt_u64_e64 s[2:3], s[2:3], v[0:1]
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v2, s2
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v3, s3
+; SDAG-GFX9-NEXT:    global_store_dwordx2 v1, v[2:3], s[0:1]
+; SDAG-GFX9-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_u64_ult:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-VI-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_lt_u64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX9-LABEL: v_icmp_u64_ult:
+; GISEL-GFX9:       ; %bb.0:
+; GISEL-GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-GFX9-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX9-NEXT:    v_cmp_lt_u64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GISEL-GFX9-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 36)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_u64_ule(i64 addrspace(1)* %out, i64 %src) {
+; GFX11-LABEL: v_icmp_u64_ule:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; GFX11-NEXT:    s_mov_b64 s[4:5], 0x64
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_cmp_le_u64_e64 s[2:3], s[2:3], s[4:5]
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    v_mov_b32_e32 v1, s3
+; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_u64_ule:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, 0
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    v_cmp_le_u64_e64 s[2:3], s[2:3], v[0:1]
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s2
+; SDAG-VI-NEXT:    v_mov_b32_e32 v3, s1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s3
+; SDAG-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; SDAG-VI-NEXT:    s_endpgm
+;
+; SDAG-GFX9-LABEL: v_icmp_u64_ule:
+; SDAG-GFX9:       ; %bb.0:
+; SDAG-GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v1, 0
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX9-NEXT:    v_cmp_le_u64_e64 s[2:3], s[2:3], v[0:1]
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v2, s2
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v3, s3
+; SDAG-GFX9-NEXT:    global_store_dwordx2 v1, v[2:3], s[0:1]
+; SDAG-GFX9-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_u64_ule:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-VI-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_le_u64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX9-LABEL: v_icmp_u64_ule:
+; GISEL-GFX9:       ; %bb.0:
+; GISEL-GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-GFX9-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX9-NEXT:    v_cmp_le_u64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GISEL-GFX9-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 37)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i64_sgt(i64 addrspace(1)* %out, i64 %src) {
+; GFX11-LABEL: v_icmp_i64_sgt:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; GFX11-NEXT:    s_mov_b64 s[4:5], 0x64
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_cmp_gt_i64_e64 s[2:3], s[2:3], s[4:5]
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    v_mov_b32_e32 v1, s3
+; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_i64_sgt:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, 0
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    v_cmp_gt_i64_e64 s[2:3], s[2:3], v[0:1]
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s2
+; SDAG-VI-NEXT:    v_mov_b32_e32 v3, s1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s3
+; SDAG-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; SDAG-VI-NEXT:    s_endpgm
+;
+; SDAG-GFX9-LABEL: v_icmp_i64_sgt:
+; SDAG-GFX9:       ; %bb.0:
+; SDAG-GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v1, 0
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX9-NEXT:    v_cmp_gt_i64_e64 s[2:3], s[2:3], v[0:1]
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v2, s2
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v3, s3
+; SDAG-GFX9-NEXT:    global_store_dwordx2 v1, v[2:3], s[0:1]
+; SDAG-GFX9-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i64_sgt:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-VI-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_gt_i64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX9-LABEL: v_icmp_i64_sgt:
+; GISEL-GFX9:       ; %bb.0:
+; GISEL-GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-GFX9-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX9-NEXT:    v_cmp_gt_i64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GISEL-GFX9-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 38)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i64_sge(i64 addrspace(1)* %out, i64 %src) {
+; GFX11-LABEL: v_icmp_i64_sge:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; GFX11-NEXT:    s_mov_b64 s[4:5], 0x64
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_cmp_ge_i64_e64 s[2:3], s[2:3], s[4:5]
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    v_mov_b32_e32 v1, s3
+; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_i64_sge:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, 0
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    v_cmp_ge_i64_e64 s[2:3], s[2:3], v[0:1]
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s2
+; SDAG-VI-NEXT:    v_mov_b32_e32 v3, s1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s3
+; SDAG-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; SDAG-VI-NEXT:    s_endpgm
+;
+; SDAG-GFX9-LABEL: v_icmp_i64_sge:
+; SDAG-GFX9:       ; %bb.0:
+; SDAG-GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v1, 0
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX9-NEXT:    v_cmp_ge_i64_e64 s[2:3], s[2:3], v[0:1]
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v2, s2
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v3, s3
+; SDAG-GFX9-NEXT:    global_store_dwordx2 v1, v[2:3], s[0:1]
+; SDAG-GFX9-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i64_sge:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-VI-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_ge_i64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX9-LABEL: v_icmp_i64_sge:
+; GISEL-GFX9:       ; %bb.0:
+; GISEL-GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-GFX9-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX9-NEXT:    v_cmp_ge_i64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GISEL-GFX9-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 39)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i64_slt(i64 addrspace(1)* %out, i64 %src) {
+; GFX11-LABEL: v_icmp_i64_slt:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; GFX11-NEXT:    s_mov_b64 s[4:5], 0x64
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_cmp_lt_i64_e64 s[2:3], s[2:3], s[4:5]
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    v_mov_b32_e32 v1, s3
+; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_i64_slt:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, 0
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    v_cmp_lt_i64_e64 s[2:3], s[2:3], v[0:1]
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s2
+; SDAG-VI-NEXT:    v_mov_b32_e32 v3, s1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s3
+; SDAG-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; SDAG-VI-NEXT:    s_endpgm
+;
+; SDAG-GFX9-LABEL: v_icmp_i64_slt:
+; SDAG-GFX9:       ; %bb.0:
+; SDAG-GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v1, 0
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX9-NEXT:    v_cmp_lt_i64_e64 s[2:3], s[2:3], v[0:1]
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v2, s2
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v3, s3
+; SDAG-GFX9-NEXT:    global_store_dwordx2 v1, v[2:3], s[0:1]
+; SDAG-GFX9-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i64_slt:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-VI-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_lt_i64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX9-LABEL: v_icmp_i64_slt:
+; GISEL-GFX9:       ; %bb.0:
+; GISEL-GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-GFX9-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX9-NEXT:    v_cmp_lt_i64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GISEL-GFX9-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 40)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i64_sle(i64 addrspace(1)* %out, i64 %src) {
+; GFX11-LABEL: v_icmp_i64_sle:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; GFX11-NEXT:    s_mov_b64 s[4:5], 0x64
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_cmp_le_i64_e64 s[2:3], s[2:3], s[4:5]
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    v_mov_b32_e32 v1, s3
+; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_i64_sle:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, 0
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    v_cmp_le_i64_e64 s[2:3], s[2:3], v[0:1]
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s2
+; SDAG-VI-NEXT:    v_mov_b32_e32 v3, s1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s3
+; SDAG-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; SDAG-VI-NEXT:    s_endpgm
+;
+; SDAG-GFX9-LABEL: v_icmp_i64_sle:
+; SDAG-GFX9:       ; %bb.0:
+; SDAG-GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v1, 0
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-GFX9-NEXT:    v_cmp_le_i64_e64 s[2:3], s[2:3], v[0:1]
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v2, s2
+; SDAG-GFX9-NEXT:    v_mov_b32_e32 v3, s3
+; SDAG-GFX9-NEXT:    global_store_dwordx2 v1, v[2:3], s[0:1]
+; SDAG-GFX9-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i64_sle:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-VI-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_le_i64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX9-LABEL: v_icmp_i64_sle:
+; GISEL-GFX9:       ; %bb.0:
+; GISEL-GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-GFX9-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX9-NEXT:    v_cmp_le_i64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GISEL-GFX9-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 41)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i16_eq(i64 addrspace(1)* %out, i16 %src) {
+; GFX11-LABEL: v_icmp_i16_eq:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_clause 0x1
+; GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_cmp_eq_u16_e64 s[2:3], 0x64, s2
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    v_mov_b32_e32 v1, s3
+; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_i16_eq:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; SDAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    v_cmp_eq_u16_e64 s[2:3], s2, v0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; SDAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; SDAG-VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: v_icmp_i16_eq:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0x64
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_cmp_eq_u16_e64 s[0:1], s4, v0
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mov_b32_e32 v1, s1
+; GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GFX9-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i16_eq:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_eq_u16_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 32)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i16(i64 addrspace(1)* %out, i16 %src) {
+; SDAG-GFX11-LABEL: v_icmp_i16:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_i16:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_endpgm
+;
+; SDAG-GFX9-LABEL: v_icmp_i16:
+; SDAG-GFX9:       ; %bb.0:
+; SDAG-GFX9-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: v_icmp_i16:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GISEL-GFX11-NEXT:    v_mov_b32_e32 v0, 0
+; GISEL-GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX11-NEXT:    global_store_b64 v0, v[0:1], s[0:1]
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i16:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s1
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[0:1], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX9-LABEL: v_icmp_i16:
+; GISEL-GFX9:       ; %bb.0:
+; GISEL-GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-GFX9-NEXT:    v_mov_b32_e32 v0, 0
+; GISEL-GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX9-NEXT:    global_store_dwordx2 v0, v[0:1], s[0:1]
+; GISEL-GFX9-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 30)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i16_ne(i64 addrspace(1)* %out, i16 %src) {
+; GFX11-LABEL: v_icmp_i16_ne:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_clause 0x1
+; GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_cmp_ne_u16_e64 s[2:3], 0x64, s2
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    v_mov_b32_e32 v1, s3
+; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_i16_ne:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; SDAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    v_cmp_ne_u16_e64 s[2:3], s2, v0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; SDAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; SDAG-VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: v_icmp_i16_ne:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0x64
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_cmp_ne_u16_e64 s[0:1], s4, v0
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mov_b32_e32 v1, s1
+; GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GFX9-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i16_ne:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_ne_u16_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 33)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i16_ugt(i64 addrspace(1)* %out, i16 %src) {
+; GFX11-LABEL: v_icmp_i16_ugt:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_clause 0x1
+; GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_cmp_lt_u16_e64 s[2:3], 0x64, s2
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    v_mov_b32_e32 v1, s3
+; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_i16_ugt:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; SDAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    v_cmp_gt_u16_e64 s[2:3], s2, v0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; SDAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; SDAG-VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: v_icmp_i16_ugt:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0x64
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_cmp_gt_u16_e64 s[0:1], s4, v0
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mov_b32_e32 v1, s1
+; GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GFX9-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i16_ugt:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_gt_u16_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 34)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i16_uge(i64 addrspace(1)* %out, i16 %src) {
+; GFX11-LABEL: v_icmp_i16_uge:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_clause 0x1
+; GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_cmp_le_u16_e64 s[2:3], 0x64, s2
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    v_mov_b32_e32 v1, s3
+; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_i16_uge:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; SDAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    v_cmp_ge_u16_e64 s[2:3], s2, v0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; SDAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; SDAG-VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: v_icmp_i16_uge:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0x64
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_cmp_ge_u16_e64 s[0:1], s4, v0
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mov_b32_e32 v1, s1
+; GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GFX9-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i16_uge:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_ge_u16_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 35)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i16_ult(i64 addrspace(1)* %out, i16 %src) {
+; GFX11-LABEL: v_icmp_i16_ult:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_clause 0x1
+; GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_cmp_gt_u16_e64 s[2:3], 0x64, s2
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    v_mov_b32_e32 v1, s3
+; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_i16_ult:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; SDAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    v_cmp_lt_u16_e64 s[2:3], s2, v0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; SDAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; SDAG-VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: v_icmp_i16_ult:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0x64
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_cmp_lt_u16_e64 s[0:1], s4, v0
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mov_b32_e32 v1, s1
+; GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GFX9-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i16_ult:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_lt_u16_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 36)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i16_ule(i64 addrspace(1)* %out, i16 %src) {
+; GFX11-LABEL: v_icmp_i16_ule:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_clause 0x1
+; GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_cmp_ge_u16_e64 s[2:3], 0x64, s2
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    v_mov_b32_e32 v1, s3
+; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_i16_ule:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; SDAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    v_cmp_le_u16_e64 s[2:3], s2, v0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; SDAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; SDAG-VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: v_icmp_i16_ule:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0x64
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_cmp_le_u16_e64 s[0:1], s4, v0
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mov_b32_e32 v1, s1
+; GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GFX9-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i16_ule:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_le_u16_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 37)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i16_sgt(i64 addrspace(1)* %out, i16 %src) #1 {
+; GFX11-LABEL: v_icmp_i16_sgt:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_clause 0x1
+; GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_cmp_lt_i16_e64 s[2:3], 0x64, s2
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    v_mov_b32_e32 v1, s3
+; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_i16_sgt:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; SDAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    v_cmp_gt_i16_e64 s[2:3], s2, v0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; SDAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; SDAG-VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: v_icmp_i16_sgt:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0x64
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_cmp_gt_i16_e64 s[0:1], s4, v0
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mov_b32_e32 v1, s1
+; GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GFX9-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i16_sgt:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_gt_i16_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 38)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i16_sge(i64 addrspace(1)* %out, i16 %src) {
+; GFX11-LABEL: v_icmp_i16_sge:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_clause 0x1
+; GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_cmp_le_i16_e64 s[2:3], 0x64, s2
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    v_mov_b32_e32 v1, s3
+; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_i16_sge:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; SDAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    v_cmp_ge_i16_e64 s[2:3], s2, v0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; SDAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; SDAG-VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: v_icmp_i16_sge:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0x64
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_cmp_ge_i16_e64 s[0:1], s4, v0
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mov_b32_e32 v1, s1
+; GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GFX9-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i16_sge:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_ge_i16_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 39)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i16_slt(i64 addrspace(1)* %out, i16 %src) {
+; GFX11-LABEL: v_icmp_i16_slt:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_clause 0x1
+; GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_cmp_gt_i16_e64 s[2:3], 0x64, s2
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    v_mov_b32_e32 v1, s3
+; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_i16_slt:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; SDAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    v_cmp_lt_i16_e64 s[2:3], s2, v0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; SDAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; SDAG-VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: v_icmp_i16_slt:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0x64
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_cmp_lt_i16_e64 s[0:1], s4, v0
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mov_b32_e32 v1, s1
+; GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GFX9-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i16_slt:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_lt_i16_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 40)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i16_sle(i64 addrspace(1)* %out, i16 %src) {
+; GFX11-LABEL: v_icmp_i16_sle:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_clause 0x1
+; GFX11-NEXT:    s_load_b32 s2, s[0:1], 0x2c
+; GFX11-NEXT:    s_load_b64 s[0:1], s[0:1], 0x24
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    v_cmp_ge_i16_e64 s[2:3], 0x64, s2
+; GFX11-NEXT:    s_delay_alu instid0(VALU_DEP_1) | instskip(NEXT) | instid1(VALU_DEP_2)
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    v_mov_b32_e32 v1, s3
+; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: v_icmp_i16_sle:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; SDAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; SDAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; SDAG-VI-NEXT:    v_cmp_le_i16_e64 s[2:3], s2, v0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; SDAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; SDAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; SDAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; SDAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; SDAG-VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: v_icmp_i16_sle:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0x64
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_cmp_le_i16_e64 s[0:1], s4, v0
+; GFX9-NEXT:    v_mov_b32_e32 v0, s0
+; GFX9-NEXT:    v_mov_b32_e32 v1, s1
+; GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GFX9-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i16_sle:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_le_i16_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 41)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_kernel void @v_icmp_i1_ne0(i64 addrspace(1)* %out, i32 %a, i32 %b) {
+; GFX11-LABEL: v_icmp_i1_ne0:
+; GFX11:       ; %bb.0:
+; GFX11-NEXT:    s_load_b128 s[0:3], s[0:1], 0x24
+; GFX11-NEXT:    v_mov_b32_e32 v2, 0
+; GFX11-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX11-NEXT:    s_cmp_gt_u32 s2, 1
+; GFX11-NEXT:    s_cselect_b64 s[4:5], -1, 0
+; GFX11-NEXT:    s_cmp_gt_u32 s3, 2
+; GFX11-NEXT:    s_cselect_b64 s[2:3], -1, 0
+; GFX11-NEXT:    s_delay_alu instid0(SALU_CYCLE_1) | instskip(NEXT) | instid1(SALU_CYCLE_1)
+; GFX11-NEXT:    s_and_b64 s[2:3], s[4:5], s[2:3]
+; GFX11-NEXT:    v_mov_b32_e32 v0, s2
+; GFX11-NEXT:    v_mov_b32_e32 v1, s3
+; GFX11-NEXT:    global_store_b64 v2, v[0:1], s[0:1]
+; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GFX11-NEXT:    s_endpgm
+;
+; VI-LABEL: v_icmp_i1_ne0:
+; VI:       ; %bb.0:
+; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    s_cmp_gt_u32 s2, 1
+; VI-NEXT:    s_cselect_b64 s[4:5], -1, 0
+; VI-NEXT:    s_cmp_gt_u32 s3, 2
+; VI-NEXT:    s_cselect_b64 s[2:3], -1, 0
+; VI-NEXT:    s_and_b64 s[2:3], s[4:5], s[2:3]
+; VI-NEXT:    v_mov_b32_e32 v0, s0
+; VI-NEXT:    v_mov_b32_e32 v2, s2
+; VI-NEXT:    v_mov_b32_e32 v1, s1
+; VI-NEXT:    v_mov_b32_e32 v3, s3
+; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; VI-NEXT:    s_endpgm
+;
+; GFX9-LABEL: v_icmp_i1_ne0:
+; GFX9:       ; %bb.0:
+; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    s_cmp_gt_u32 s2, 1
+; GFX9-NEXT:    s_cselect_b64 s[4:5], -1, 0
+; GFX9-NEXT:    s_cmp_gt_u32 s3, 2
+; GFX9-NEXT:    s_cselect_b64 s[2:3], -1, 0
+; GFX9-NEXT:    s_and_b64 s[2:3], s[4:5], s[2:3]
+; GFX9-NEXT:    v_mov_b32_e32 v0, s2
+; GFX9-NEXT:    v_mov_b32_e32 v1, s3
+; GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GFX9-NEXT:    s_endpgm
+  %c0 = icmp ugt i32 %a, 1
+  %c1 = icmp ugt i32 %b, 2
+  %src = and i1 %c0, %c1
+  %result = call i64 @llvm.amdgcn.icmp.i1(i1 %src, i1 false, i32 33)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+define amdgpu_ps void @test_intr_icmp_i32_invalid_cc(i64 addrspace(1)* %out, i32 %src) {
+; SDAG-GFX11-LABEL: test_intr_icmp_i32_invalid_cc:
+; SDAG-GFX11:       ; %bb.0:
+; SDAG-GFX11-NEXT:    s_endpgm
+;
+; SDAG-VI-LABEL: test_intr_icmp_i32_invalid_cc:
+; SDAG-VI:       ; %bb.0:
+; SDAG-VI-NEXT:    s_endpgm
+;
+; SDAG-GFX9-LABEL: test_intr_icmp_i32_invalid_cc:
+; SDAG-GFX9:       ; %bb.0:
+; SDAG-GFX9-NEXT:    s_endpgm
+;
+; GISEL-GFX11-LABEL: test_intr_icmp_i32_invalid_cc:
+; GISEL-GFX11:       ; %bb.0:
+; GISEL-GFX11-NEXT:    global_store_b64 v[0:1], v[0:1], off
+; GISEL-GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
+; GISEL-GFX11-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: test_intr_icmp_i32_invalid_cc:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[0:1], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX9-LABEL: test_intr_icmp_i32_invalid_cc:
+; GISEL-GFX9:       ; %bb.0:
+; GISEL-GFX9-NEXT:    global_store_dwordx2 v[0:1], v[0:1], off
+; GISEL-GFX9-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 9999)
+  store i64 %result, i64 addrspace(1)* %out
+  ret void
+}
+
+attributes #0 = { nounwind readnone convergent }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GCN: {{.*}}