[AMDGPU][GISel] Add llvm.amdgcn.icmp selection

[Pierre-vh]

Summary

Add missing logic to select i16 variants and enable GISel testing.

[Pierre-vh]

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

1079

The test won't pass on, say, hawaii because of lack of 16 bit insts. It'll fail to select.
I guess it's a separate issue where we should probably legalize the intrinsic (= widen its input to i32) for those archs?

1084

Note: not sure if "true" or "false" 16 bit insts matter.
I did both to be safe but I'd be curious to know if it's actually needed.

llvm/lib/Target/AMDGPU/SIInstructions.td

893

Help needed here, not sure how to get this one to work. I tried a lot of things, including COPY_TO_REGCLASS and nothing seems to do the trick.
Due to the simplicity of the pattern I'm wondering if it isn't better to just do it manually in the InstructionSelector rather than fight the GISel TableGen emitter?

[foad]

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

1079

I think failing to select is the correct behaviour - or diagnose it as "unsupported" in the legalizer (we do this in some cases but not consistently). Generally intrinsics like this map one-to-one onto a machine instruction, and are not designed to be supported on subtargets that do not have the instruction.

[foad]

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

1084

I think both are needed here. The "true" ones will be used on GFX11 and the "fake"(!) ones on previous subtargets.

[arsenm]

llvm/lib/Target/AMDGPU/SIInstructions.td

893

What's the error with COPY_TO_REGCLASS?

[Pierre-vh]

llvm/lib/Target/AMDGPU/SIInstructions.td

893

IIRC with COPY_TO_REGCLASS it's a type issue, or if I put some regclass that works then it just doesn't match in the instruction selector.
I think my question is more: What regclass do I need to use there?

[Pierre-vh]

Rebase, fix test

[Joe_Nash]

llvm/lib/Target/AMDGPU/SIInstructions.td

893

Just a guess, but maybe the patterns need to be copied, one for wave32 and one for wave64? So on wave32 you do COPY_TO_REGCLASS to SReg_32_XM0_XEXEC, and for wave64 SReg_64_XM0_XEXEC

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.icmp.ll

2–6

I would prefer a check-prefix other than DAG. I believe this could be confused with https://llvm.org/docs/CommandGuide/FileCheck.html#the-check-dag-directive.

Can you also add a runline for gfx1100?

[foad]

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.icmp.ll

2–6

"SDAG" is quite common

[Pierre-vh]

llvm/lib/Target/AMDGPU/SIInstructions.td

893

The input pattern also needs a registerclass, that's the one I'm stuck on now
I tried (COPY_TO_REGCLASS SReg_1_XEXEC:$src, SReg_64_XEXEC) for Wave64 and it causes a "noreg" to be selected which crashes the backend

$vgpr1 = V_MOV_B32_e32 $noreg, implicit $exec, implicit killed $sgpr2, implicit $exec

VReg_1 cannot be used either, it says it's not a recognized class. I think it's not available in TableGen?

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.icmp.ll

2–6

For GFX11, is it fine if I just use that for the GFX run line (instead of no cpu) or is a third line needed?

[Joe_Nash]

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.icmp.ll

2–6

SDAG sounds good.

2–6

I don't think there is a need to be stingy with which tests are run. It seems good to test both no cpu (GFX6?) and gfx1100.

[Pierre-vh]

Comments

[Pierre-vh]

I think that for the icmp i1 case there's some problem in the RegBankSelect or the legalizer. I can't seem to find the right pattern.
My current best attempt, something like this:

def : Pat <
  (i64 (int_amdgcn_icmp i1:$src, (i1 0), (i32 33))),
  (COPY_TO_REGCLASS SCSrc_i1:$src, SReg_1_XEXEC) // Return the SGPRs representing i1 src
>;

def : Pat <
  (i32 (int_amdgcn_icmp i1:$src, (i1 0), (i32 33))),
  (COPY_TO_REGCLASS SCSrc_i1:$src, SReg_1_XEXEC) // Return the SGPRs representing i1 src
>;

Selects:

Selecting: 
  %23:sreg_64(s64) = G_INTRINSIC intrinsic(@llvm.amdgcn.icmp), %32:vgpr(s1), %33:vgpr(s1), 33
 Into:
  %23:sreg_64_xexec(s64) = COPY %32:vgpr(s1)

But later, even in wave64 mode, %32 gets constrained to a vgpr_32 and we get an impossible copy:

Selecting: 
  %32:vgpr(s1) = COPY %22:sgpr(s1)
Into:
  %32:vgpr_32(s1) = COPY %22:sreg_32(s1)

; later
  %31:sreg_32 = S_AND_B32 %27:sreg_32, %28:sreg_32, implicit-def $scc
  %32:vgpr_32 = COPY %31:sreg_32
  %23:sreg_64_xexec = COPY %32:vgpr_32

I'm really not sure how to deal with this. Thoughts?

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.icmp.ll

2–6

For GISel, with no CPU, I get

LLVM ERROR: cannot select: G_STORE %17:vgpr(s64), %9:sgpr(p1) :: (store (s64) into %ir.out.load, addrspace 1) (in function: v_icmp_i32_eq)

So I use GFX900 for both instead

[Pierre-vh]

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.icmp.ll

2–6

Also, for gfx 11 I had to use wave64 mode. All intrinsics in this test are the w64 variants (return i64 mask).
Should I add a w32 duplicate of the test (in a pre-patch)? Or maybe just duplicating a couple of functions in this test for w32 would be enough?

[Pierre-vh]

Add manual selection for i1 (I think it's currently incorrect, need feedback)
Add wave32 test

[Pierre-vh]

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

1242

This is obviously wrong as we only want to compare the first bit in each VGPR, but I don't know what instruction to use here yet.

Which instruction do I need to use here? V_CNDMASK ?

[Pierre-vh]

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

1242

CNDMASK doesn't work because it constrains the operands to 64 bits, but the i1 are widened to 32.
We need something that would do dst[threadI] = v0.i1, if I understand correctly?

[Pierre-vh]

Rename tests

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

1242

Where are you seeing i1 VGPRs? Those should just not happen. At one point I might have been trying to handle that, but I think it's basically wrong for it to see the selector

[Pierre-vh]

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

1242

I don't see i1 VGPRs, but vgpr(s1) as the arguments of the iccmp

[Joe_Nash]

llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-icmp.s16.mir

28 ↗

(On Diff #472619)

Why has the dst class been changed to sreg_32? These instructions should not write to m0 or exec

[Pierre-vh]

llvm/test/CodeGen/AMDGPU/GlobalISel/inst-select-icmp.s16.mir

28 ↗

(On Diff #472619)

Dst is constrained to getBoolRC. If we want xexec we need to use getWaveMaskRegClass instead
Not sure which one is technically correct? Isn't sreg_32 more correct because the dst is always a sgpr, and never exec/m0?

[arsenm]

LGTM except for trying to handle i1 intrinsics

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

1242

That shouldn't happen. The intrinsic shouldn't accept incoming i1 values. If you try to use i1 with these intrinsics, it should just fail selection

[Pierre-vh]

Don't support i1 inputs

Not sure how we want to handle this in a merged test given that the DAG supports them. I used global-isel-abort=2

[arsenm]

I'm surprised i1 works for DAG

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

case Intrinsic::amdgcn_wwm:
     return constrainCopyLikeIntrin(I, AMDGPU::STRICT_WWM);
   case Intrinsic::amdgcn_strict_wqm:
     return constrainCopyLikeIntrin(I, AMDGPU::STRICT_WQM);
   case Intrinsic::amdgcn_writelane:
     return selectWritelane(I);
   case Intrinsic::amdgcn_div_scale:
     return selectDivScale(I);
   case Intrinsic::amdgcn_icmp:
+    if (selectImpl(I, *CoverageInfo))
+      return true;
     return selectIntrinsicIcmp(I);
   case Intrinsic::amdgcn_ballot:
     return selectBallot(I);
   case Intrinsic::amdgcn_reloc_constant:
     return selectRelocConstant(I);
   case Intrinsic::amdgcn_groupstaticsize:
     return selectGroupStaticSize(I);
   case Intrinsic::returnaddress:
bool AMDGPUInstructionSelector::selectG_INTRINSIC(MachineInstr &I) const {
   case Intrinsic::amdgcn_smfmac_f32_32x32x32_fp8_bf8:
   case Intrinsic::amdgcn_smfmac_f32_32x32x32_fp8_fp8:
     return selectSMFMACIntrin(I);
   default:
     return selectImpl(I, *CoverageInfo);
   }
 }
 
-static int getV_CMPOpcode(CmpInst::Predicate P, unsigned Size) {
-  if (Size != 32 && Size != 64)
+static int getV_CMPOpcode(CmpInst::Predicate P, unsigned Size,
+                          const GCNSubtarget &ST) {
+  if (Size != 16 && Size != 32 && Size != 64)
+    return -1;
+
+  if (Size == 16 && !ST.has16BitInsts())
     return -1;
+
+  const auto Select = [&](unsigned S16Opc, unsigned TrueS16Opc, unsigned S32Opc,
+                          unsigned S64Opc) {
+    if (Size == 16)
+      return ST.hasTrue16BitInsts() ? TrueS16Opc : S16Opc;
+    if (Size == 32)
+      return S32Opc;
+    return S64Opc;
+  };
+
   switch (P) {
   default:
     llvm_unreachable("Unknown condition code!");
   case CmpInst::ICMP_NE:
-    return Size == 32 ? AMDGPU::V_CMP_NE_U32_e64 : AMDGPU::V_CMP_NE_U64_e64;
+    return Select(AMDGPU::V_CMP_NE_U16_e64, AMDGPU::V_CMP_NE_U16_t16_e64,
+                  AMDGPU::V_CMP_NE_U32_e64, AMDGPU::V_CMP_NE_U64_e64);
   case CmpInst::ICMP_EQ:
-    return Size == 32 ? AMDGPU::V_CMP_EQ_U32_e64 : AMDGPU::V_CMP_EQ_U64_e64;
+    return Select(AMDGPU::V_CMP_EQ_U16_e64, AMDGPU::V_CMP_EQ_U16_t16_e64,
+                  AMDGPU::V_CMP_EQ_U32_e64, AMDGPU::V_CMP_EQ_U64_e64);
   case CmpInst::ICMP_SGT:
-    return Size == 32 ? AMDGPU::V_CMP_GT_I32_e64 : AMDGPU::V_CMP_GT_I64_e64;
+    return Select(AMDGPU::V_CMP_GT_I16_e64, AMDGPU::V_CMP_GT_I16_t16_e64,
+                  AMDGPU::V_CMP_GT_I32_e64, AMDGPU::V_CMP_GT_I64_e64);
   case CmpInst::ICMP_SGE:
-    return Size == 32 ? AMDGPU::V_CMP_GE_I32_e64 : AMDGPU::V_CMP_GE_I64_e64;
+    return Select(AMDGPU::V_CMP_GE_I16_e64, AMDGPU::V_CMP_GE_I16_t16_e64,
+                  AMDGPU::V_CMP_GE_I32_e64, AMDGPU::V_CMP_GE_I64_e64);
   case CmpInst::ICMP_SLT:
-    return Size == 32 ? AMDGPU::V_CMP_LT_I32_e64 : AMDGPU::V_CMP_LT_I64_e64;
+    return Select(AMDGPU::V_CMP_LT_I16_e64, AMDGPU::V_CMP_LT_I16_t16_e64,
+                  AMDGPU::V_CMP_LT_I32_e64, AMDGPU::V_CMP_LT_I64_e64);
   case CmpInst::ICMP_SLE:
-    return Size == 32 ? AMDGPU::V_CMP_LE_I32_e64 : AMDGPU::V_CMP_LE_I64_e64;
+    return Select(AMDGPU::V_CMP_LE_I16_e64, AMDGPU::V_CMP_LE_I16_t16_e64,
+                  AMDGPU::V_CMP_LE_I32_e64, AMDGPU::V_CMP_LE_I64_e64);
   case CmpInst::ICMP_UGT:
-    return Size == 32 ? AMDGPU::V_CMP_GT_U32_e64 : AMDGPU::V_CMP_GT_U64_e64;
+    return Select(AMDGPU::V_CMP_GT_U16_e64, AMDGPU::V_CMP_GT_U16_t16_e64,
+                  AMDGPU::V_CMP_GT_U32_e64, AMDGPU::V_CMP_GT_U64_e64);
   case CmpInst::ICMP_UGE:
-    return Size == 32 ? AMDGPU::V_CMP_GE_U32_e64 : AMDGPU::V_CMP_GE_U64_e64;
+    return Select(AMDGPU::V_CMP_GE_U16_e64, AMDGPU::V_CMP_GE_U16_t16_e64,
+                  AMDGPU::V_CMP_GE_U32_e64, AMDGPU::V_CMP_GE_U64_e64);
   case CmpInst::ICMP_ULT:
-    return Size == 32 ? AMDGPU::V_CMP_LT_U32_e64 : AMDGPU::V_CMP_LT_U64_e64;
+    return Select(AMDGPU::V_CMP_LT_U16_e64, AMDGPU::V_CMP_LT_U16_t16_e64,
+                  AMDGPU::V_CMP_LT_U32_e64, AMDGPU::V_CMP_LT_U64_e64);
   case CmpInst::ICMP_ULE:
-    return Size == 32 ? AMDGPU::V_CMP_LE_U32_e64 : AMDGPU::V_CMP_LE_U64_e64;
+    return Select(AMDGPU::V_CMP_LE_U16_e64, AMDGPU::V_CMP_LE_U16_t16_e64,
+                  AMDGPU::V_CMP_LE_U32_e64, AMDGPU::V_CMP_LE_U64_e64);
   }
 }
 
 int AMDGPUInstructionSelector::getS_CMPOpcode(CmpInst::Predicate P,
                                               unsigned Size) const {
   if (Size == 64) {
     if (!STI.hasScalarCompareEq64())
       return -1;
BuildMI(*BB, &I, DL, TII.get(AMDGPU::COPY), CCReg)
       .addReg(AMDGPU::SCC);
     bool Ret =
         constrainSelectedInstRegOperands(*ICmp, TII, TRI, RBI) &&
         RBI.constrainGenericRegister(CCReg, AMDGPU::SReg_32RegClass, *MRI);
     I.eraseFromParent();
     return Ret;
   }
 
-  int Opcode = getV_CMPOpcode(Pred, Size);
+  int Opcode = getV_CMPOpcode(Pred, Size, *Subtarget);
   if (Opcode == -1)
     return false;
 
   MachineInstr *ICmp = BuildMI(*BB, &I, DL, TII.get(Opcode),
             I.getOperand(0).getReg())
             .add(I.getOperand(2))
             .add(I.getOperand(3));
   RBI.constrainGenericRegister(ICmp->getOperand(0).getReg(),
if (!ICmpInst::isIntPredicate(static_cast<ICmpInst::Predicate>(Pred))) {
 
     if (!RBI.constrainGenericRegister(ICmp->getOperand(0).getReg(),
                                       *TRI.getBoolRC(), *MRI))
       return false;
     I.eraseFromParent();
     return true;
   }
 
-  int Opcode = getV_CMPOpcode(Pred, Size);
+  int Opcode = getV_CMPOpcode(Pred, Size, *Subtarget);
   if (Opcode == -1)
     return false;
 
   MachineInstr *ICmp = BuildMI(*BB, &I, DL, TII.get(Opcode), Dst)
                            .add(I.getOperand(2))
                            .add(I.getOperand(3));
   RBI.constrainGenericRegister(ICmp->getOperand(0).getReg(), *TRI.getBoolRC(),
                                *MRI);
   bool Ret = constrainSelectedInstRegOperands(*ICmp, TII, TRI, RBI);
   I.eraseFromParent();
   return Ret;
 }
 

llvm/lib/Target/AMDGPU/SIInstructions.td

 
 def : Pat <
   (int_amdgcn_wqm_demote (i1 (not i1:$src))),
   (SI_DEMOTE_I1 SCSrc_i1:$src, -1)
 >;
 
   // TODO: we could add more variants for other types of conditionals
 
+// FIXME/HELP NEEDED: I can't get these patterns to be imported so v_icmp_i1_ne0 is commented
+//  out in the test.
+//   Skipped pattern: Dst MI def isn't a register class(COPY:{ *:[i64] } ?:{ *:[i1] }:$src)
 def : Pat <
   (i64 (int_amdgcn_icmp i1:$src, (i1 0), (i32 33))),
   (COPY $src) // Return the SGPRs representing i1 src
 >;
 
 def : Pat <
   (i32 (int_amdgcn_icmp i1:$src, (i1 0), (i32 33))),
   (COPY $src) // Return the SGPRs representing i1 src

llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.icmp.ll

-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -global-isel -global-isel-abort=1 -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs < %s | FileCheck -check-prefix=GFX10 %s
-; RUN: llc -global-isel -global-isel-abort=1 -march=amdgcn -mcpu=gfx1100 -amdgpu-enable-delay-alu=0 -verify-machineinstrs < %s | FileCheck -check-prefix=GFX11 %s
-
-define amdgpu_ps void @test_intr_icmp_eq_i64(i64 addrspace(1)* %out, i32 %src) #0 {
-; GFX10-LABEL: test_intr_icmp_eq_i64:
-; GFX10:       ; %bb.0:
-; GFX10-NEXT:    v_cmp_eq_u32_e64 s[0:1], 0x64, v2
-; GFX10-NEXT:    v_mov_b32_e32 v3, s1
-; GFX10-NEXT:    v_mov_b32_e32 v2, s0
-; GFX10-NEXT:    global_store_dwordx2 v[0:1], v[2:3], off
-; GFX10-NEXT:    s_endpgm
-;
-; GFX11-LABEL: test_intr_icmp_eq_i64:
-; GFX11:       ; %bb.0:
-; GFX11-NEXT:    v_cmp_eq_u32_e64 s[0:1], 0x64, v2
-; GFX11-NEXT:    v_mov_b32_e32 v3, s1
-; GFX11-NEXT:    v_mov_b32_e32 v2, s0
-; GFX11-NEXT:    global_store_b64 v[0:1], v[2:3], off
-; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
-; GFX11-NEXT:    s_endpgm
-  %result = call i64 @llvm.amdgcn.icmp.i64.i32(i32 %src, i32 100, i32 32)
-  store i64 %result, i64 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_ps void @test_intr_icmp_ne_i32(i32 addrspace(1)* %out, i32 %src) #1 {
-; GFX10-LABEL: test_intr_icmp_ne_i32:
-; GFX10:       ; %bb.0:
-; GFX10-NEXT:    v_cmp_ne_u32_e64 s0, 0x64, v2
-; GFX10-NEXT:    v_mov_b32_e32 v2, s0
-; GFX10-NEXT:    global_store_dword v[0:1], v2, off
-; GFX10-NEXT:    s_endpgm
-;
-; GFX11-LABEL: test_intr_icmp_ne_i32:
-; GFX11:       ; %bb.0:
-; GFX11-NEXT:    v_cmp_ne_u32_e64 s0, 0x64, v2
-; GFX11-NEXT:    v_mov_b32_e32 v2, s0
-; GFX11-NEXT:    global_store_b32 v[0:1], v2, off
-; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
-; GFX11-NEXT:    s_endpgm
-  %result = call i32 @llvm.amdgcn.icmp.i32.i32(i32 %src, i32 100, i32 33)
-  store i32 %result, i32 addrspace(1)* %out
-  ret void
-}
-
-define amdgpu_ps void @test_intr_icmp_i32_invalid_cc(i32 addrspace(1)* %out, i32 %src) #1 {
-; GFX10-LABEL: test_intr_icmp_i32_invalid_cc:
-; GFX10:       ; %bb.0:
-; GFX10-NEXT:    global_store_dword v[0:1], v0, off
-; GFX10-NEXT:    s_endpgm
-;
-; GFX11-LABEL: test_intr_icmp_i32_invalid_cc:
-; GFX11:       ; %bb.0:
-; GFX11-NEXT:    global_store_b32 v[0:1], v0, off
-; GFX11-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
-; GFX11-NEXT:    s_endpgm
-  %result = call i32 @llvm.amdgcn.icmp.i32.i32(i32 %src, i32 100, i32 9999)
-  store i32 %result, i32 addrspace(1)* %out
-  ret void
-}
-
-declare i64 @llvm.amdgcn.icmp.i64.i32(i32, i32, i32)
-declare i32 @llvm.amdgcn.icmp.i32.i32(i32, i32, i32)
-attributes #0 = { "target-features"="+wavefrontsize64" }
-attributes #1 = { "target-features"="+wavefrontsize32" }

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.icmp.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -march=amdgcn -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX %s
-; RUN: llc -march=amdgcn -mcpu=fiji -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,VI %s
+; RUN: llc -march=amdgcn -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX,DAG-GFX %s
+; RUN: llc -march=amdgcn -mcpu=fiji -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,VI,DAG-VI %s
+
+; RUN: llc -global-isel -march=amdgcn -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX,GISEL-GFX %s
+; RUN: llc -global-isel -march=amdgcn -mcpu=fiji -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,VI,GISEL-VI %s
 
-declare i64 @llvm.amdgcn.icmp.i64.i32(i32, i32, i32)
-declare i32 @llvm.amdgcn.icmp.i32.i32(i32, i32, i32)
 declare i64 @llvm.amdgcn.icmp.i32(i32, i32, i32) #0
 declare i64 @llvm.amdgcn.icmp.i64(i64, i64, i32) #0
 declare i64 @llvm.amdgcn.icmp.i16(i16, i16, i32) #0
 declare i64 @llvm.amdgcn.icmp.i1(i1, i1, i32) #0
 
 define amdgpu_kernel void @v_icmp_i32_eq(i64 addrspace(1)* %out, i32 %src) {
-; GFX-LABEL: v_icmp_i32_eq:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_eq_u32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i32_eq:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_eq_u32_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_i32_eq:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
+; DAG-GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
+; DAG-GFX-NEXT:    s_mov_b32 s3, 0xf000
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-GFX-NEXT:    v_cmp_eq_u32_e64 s[4:5], s2, v0
+; DAG-GFX-NEXT:    s_mov_b32 s2, -1
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, s4
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, s5
+; DAG-GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_i32_eq:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; DAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-VI-NEXT:    v_cmp_eq_u32_e64 s[2:3], s2, v0
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; DAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; DAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; DAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_i32_eq:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    v_cmp_eq_u32_e64 s[0:1], s4, v0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s1
+; GISEL-GFX-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i32_eq:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_eq_u32_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 32)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @v_icmp_i32(i64 addrspace(1)* %out, i32 %src) {
-; GCN-LABEL: v_icmp_i32:
-; GCN:       ; %bb.0:
-; GCN-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_i32:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_i32:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_i32:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    global_store_dwordx2 v0, v[0:1], s[0:1]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i32:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s1
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[0:1], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 30)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @v_icmp_i32_ne(i64 addrspace(1)* %out, i32 %src) {
-; GFX-LABEL: v_icmp_i32_ne:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_ne_u32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i32_ne:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_ne_u32_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_i32_ne:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
+; DAG-GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
+; DAG-GFX-NEXT:    s_mov_b32 s3, 0xf000
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-GFX-NEXT:    v_cmp_ne_u32_e64 s[4:5], s2, v0
+; DAG-GFX-NEXT:    s_mov_b32 s2, -1
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, s4
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, s5
+; DAG-GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_i32_ne:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; DAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-VI-NEXT:    v_cmp_ne_u32_e64 s[2:3], s2, v0
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; DAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; DAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; DAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_i32_ne:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    v_cmp_ne_u32_e64 s[0:1], s4, v0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s1
+; GISEL-GFX-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i32_ne:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_ne_u32_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 33)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @v_icmp_i32_ugt(i64 addrspace(1)* %out, i32 %src) {
-; GFX-LABEL: v_icmp_i32_ugt:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_gt_u32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i32_ugt:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_gt_u32_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_i32_ugt:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
+; DAG-GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
+; DAG-GFX-NEXT:    s_mov_b32 s3, 0xf000
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-GFX-NEXT:    v_cmp_gt_u32_e64 s[4:5], s2, v0
+; DAG-GFX-NEXT:    s_mov_b32 s2, -1
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, s4
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, s5
+; DAG-GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_i32_ugt:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; DAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-VI-NEXT:    v_cmp_gt_u32_e64 s[2:3], s2, v0
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; DAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; DAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; DAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_i32_ugt:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    v_cmp_gt_u32_e64 s[0:1], s4, v0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s1
+; GISEL-GFX-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i32_ugt:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_gt_u32_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 34)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @v_icmp_i32_uge(i64 addrspace(1)* %out, i32 %src) {
-; GFX-LABEL: v_icmp_i32_uge:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_ge_u32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i32_uge:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_ge_u32_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_i32_uge:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
+; DAG-GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
+; DAG-GFX-NEXT:    s_mov_b32 s3, 0xf000
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-GFX-NEXT:    v_cmp_ge_u32_e64 s[4:5], s2, v0
+; DAG-GFX-NEXT:    s_mov_b32 s2, -1
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, s4
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, s5
+; DAG-GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_i32_uge:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; DAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-VI-NEXT:    v_cmp_ge_u32_e64 s[2:3], s2, v0
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; DAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; DAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; DAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_i32_uge:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    v_cmp_ge_u32_e64 s[0:1], s4, v0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s1
+; GISEL-GFX-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i32_uge:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_ge_u32_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 35)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @v_icmp_i32_ult(i64 addrspace(1)* %out, i32 %src) {
-; GFX-LABEL: v_icmp_i32_ult:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_lt_u32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i32_ult:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_lt_u32_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_i32_ult:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
+; DAG-GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
+; DAG-GFX-NEXT:    s_mov_b32 s3, 0xf000
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-GFX-NEXT:    v_cmp_lt_u32_e64 s[4:5], s2, v0
+; DAG-GFX-NEXT:    s_mov_b32 s2, -1
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, s4
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, s5
+; DAG-GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_i32_ult:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; DAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-VI-NEXT:    v_cmp_lt_u32_e64 s[2:3], s2, v0
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; DAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; DAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; DAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_i32_ult:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    v_cmp_lt_u32_e64 s[0:1], s4, v0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s1
+; GISEL-GFX-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i32_ult:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_lt_u32_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 36)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @v_icmp_i32_ule(i64 addrspace(1)* %out, i32 %src) {
-; GFX-LABEL: v_icmp_i32_ule:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_le_u32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i32_ule:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_le_u32_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_i32_ule:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
+; DAG-GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
+; DAG-GFX-NEXT:    s_mov_b32 s3, 0xf000
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-GFX-NEXT:    v_cmp_le_u32_e64 s[4:5], s2, v0
+; DAG-GFX-NEXT:    s_mov_b32 s2, -1
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, s4
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, s5
+; DAG-GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_i32_ule:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; DAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-VI-NEXT:    v_cmp_le_u32_e64 s[2:3], s2, v0
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; DAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; DAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; DAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_i32_ule:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    v_cmp_le_u32_e64 s[0:1], s4, v0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s1
+; GISEL-GFX-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i32_ule:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_le_u32_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 37)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @v_icmp_i32_sgt(i64 addrspace(1)* %out, i32 %src) #1 {
-; GFX-LABEL: v_icmp_i32_sgt:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_gt_i32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i32_sgt:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_gt_i32_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_i32_sgt:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
+; DAG-GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
+; DAG-GFX-NEXT:    s_mov_b32 s3, 0xf000
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-GFX-NEXT:    v_cmp_gt_i32_e64 s[4:5], s2, v0
+; DAG-GFX-NEXT:    s_mov_b32 s2, -1
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, s4
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, s5
+; DAG-GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_i32_sgt:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; DAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-VI-NEXT:    v_cmp_gt_i32_e64 s[2:3], s2, v0
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; DAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; DAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; DAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_i32_sgt:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    v_cmp_gt_i32_e64 s[0:1], s4, v0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s1
+; GISEL-GFX-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i32_sgt:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_gt_i32_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 38)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @v_icmp_i32_sge(i64 addrspace(1)* %out, i32 %src) {
-; GFX-LABEL: v_icmp_i32_sge:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_ge_i32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i32_sge:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_ge_i32_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_i32_sge:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
+; DAG-GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
+; DAG-GFX-NEXT:    s_mov_b32 s3, 0xf000
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-GFX-NEXT:    v_cmp_ge_i32_e64 s[4:5], s2, v0
+; DAG-GFX-NEXT:    s_mov_b32 s2, -1
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, s4
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, s5
+; DAG-GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_i32_sge:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; DAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-VI-NEXT:    v_cmp_ge_i32_e64 s[2:3], s2, v0
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; DAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; DAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; DAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_i32_sge:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    v_cmp_ge_i32_e64 s[0:1], s4, v0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s1
+; GISEL-GFX-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i32_sge:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_ge_i32_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 39)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @v_icmp_i32_slt(i64 addrspace(1)* %out, i32 %src) {
-; GFX-LABEL: v_icmp_i32_slt:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_lt_i32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i32_slt:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_lt_i32_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_i32_slt:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
+; DAG-GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
+; DAG-GFX-NEXT:    s_mov_b32 s3, 0xf000
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-GFX-NEXT:    v_cmp_lt_i32_e64 s[4:5], s2, v0
+; DAG-GFX-NEXT:    s_mov_b32 s2, -1
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, s4
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, s5
+; DAG-GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_i32_slt:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; DAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-VI-NEXT:    v_cmp_lt_i32_e64 s[2:3], s2, v0
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; DAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; DAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; DAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_i32_slt:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    v_cmp_lt_i32_e64 s[0:1], s4, v0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s1
+; GISEL-GFX-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i32_slt:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_lt_i32_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 40)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @v_icmp_i32_sle(i64 addrspace(1)* %out, i32 %src) {
-; GFX-LABEL: v_icmp_i32_sle:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_le_i32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i32_sle:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_le_i32_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_i32_sle:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
+; DAG-GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
+; DAG-GFX-NEXT:    s_mov_b32 s3, 0xf000
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-GFX-NEXT:    v_cmp_le_i32_e64 s[4:5], s2, v0
+; DAG-GFX-NEXT:    s_mov_b32 s2, -1
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, s4
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, s5
+; DAG-GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_i32_sle:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; DAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-VI-NEXT:    v_cmp_le_i32_e64 s[2:3], s2, v0
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; DAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; DAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; DAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_i32_sle:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    v_cmp_le_i32_e64 s[0:1], s4, v0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s1
+; GISEL-GFX-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i32_sle:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_le_i32_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 41)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @v_icmp_i64_eq(i64 addrspace(1)* %out, i64 %src) {
-; GFX-LABEL: v_icmp_i64_eq:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s7, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    v_mov_b32_e32 v1, 0
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_eq_u64_e64 s[2:3], s[2:3], v[0:1]
-; GFX-NEXT:    s_mov_b32 s6, -1
-; GFX-NEXT:    s_mov_b32 s4, s0
-; GFX-NEXT:    s_mov_b32 s5, s1
-; GFX-NEXT:    v_mov_b32_e32 v0, s2
-; GFX-NEXT:    v_mov_b32_e32 v1, s3
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i64_eq:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    v_mov_b32_e32 v1, 0
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_eq_u64_e64 s[2:3], s[2:3], v[0:1]
-; VI-NEXT:    v_mov_b32_e32 v2, s0
-; VI-NEXT:    v_mov_b32_e32 v0, s2
-; VI-NEXT:    v_mov_b32_e32 v3, s1
-; VI-NEXT:    v_mov_b32_e32 v1, s3
-; VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
-; VI-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_i64_eq:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
+; DAG-GFX-NEXT:    s_mov_b32 s7, 0xf000
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, 0
+; DAG-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-GFX-NEXT:    v_cmp_eq_u64_e64 s[2:3], s[2:3], v[0:1]
+; DAG-GFX-NEXT:    s_mov_b32 s6, -1
+; DAG-GFX-NEXT:    s_mov_b32 s4, s0
+; DAG-GFX-NEXT:    s_mov_b32 s5, s1
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, s2
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, s3
+; DAG-GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_i64_eq:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, 0
+; DAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-VI-NEXT:    v_cmp_eq_u64_e64 s[2:3], s[2:3], v[0:1]
+; DAG-VI-NEXT:    v_mov_b32_e32 v2, s0
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, s2
+; DAG-VI-NEXT:    v_mov_b32_e32 v3, s1
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, s3
+; DAG-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_i64_eq:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-GFX-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    v_cmp_eq_u64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-GFX-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i64_eq:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-VI-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_eq_u64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 32)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @v_icmp_i64_ne(i64 addrspace(1)* %out, i64 %src) {
-; GFX-LABEL: v_icmp_i64_ne:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s7, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    v_mov_b32_e32 v1, 0
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_ne_u64_e64 s[2:3], s[2:3], v[0:1]
-; GFX-NEXT:    s_mov_b32 s6, -1
-; GFX-NEXT:    s_mov_b32 s4, s0
-; GFX-NEXT:    s_mov_b32 s5, s1
-; GFX-NEXT:    v_mov_b32_e32 v0, s2
-; GFX-NEXT:    v_mov_b32_e32 v1, s3
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i64_ne:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    v_mov_b32_e32 v1, 0
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_ne_u64_e64 s[2:3], s[2:3], v[0:1]
-; VI-NEXT:    v_mov_b32_e32 v2, s0
-; VI-NEXT:    v_mov_b32_e32 v0, s2
-; VI-NEXT:    v_mov_b32_e32 v3, s1
-; VI-NEXT:    v_mov_b32_e32 v1, s3
-; VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
-; VI-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_i64_ne:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
+; DAG-GFX-NEXT:    s_mov_b32 s7, 0xf000
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, 0
+; DAG-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-GFX-NEXT:    v_cmp_ne_u64_e64 s[2:3], s[2:3], v[0:1]
+; DAG-GFX-NEXT:    s_mov_b32 s6, -1
+; DAG-GFX-NEXT:    s_mov_b32 s4, s0
+; DAG-GFX-NEXT:    s_mov_b32 s5, s1
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, s2
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, s3
+; DAG-GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_i64_ne:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, 0
+; DAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-VI-NEXT:    v_cmp_ne_u64_e64 s[2:3], s[2:3], v[0:1]
+; DAG-VI-NEXT:    v_mov_b32_e32 v2, s0
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, s2
+; DAG-VI-NEXT:    v_mov_b32_e32 v3, s1
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, s3
+; DAG-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_i64_ne:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-GFX-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    v_cmp_ne_u64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-GFX-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i64_ne:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-VI-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_ne_u64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 33)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @v_icmp_u64_ugt(i64 addrspace(1)* %out, i64 %src) {
-; GFX-LABEL: v_icmp_u64_ugt:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s7, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    v_mov_b32_e32 v1, 0
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_gt_u64_e64 s[2:3], s[2:3], v[0:1]
-; GFX-NEXT:    s_mov_b32 s6, -1
-; GFX-NEXT:    s_mov_b32 s4, s0
-; GFX-NEXT:    s_mov_b32 s5, s1
-; GFX-NEXT:    v_mov_b32_e32 v0, s2
-; GFX-NEXT:    v_mov_b32_e32 v1, s3
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_u64_ugt:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    v_mov_b32_e32 v1, 0
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_gt_u64_e64 s[2:3], s[2:3], v[0:1]
-; VI-NEXT:    v_mov_b32_e32 v2, s0
-; VI-NEXT:    v_mov_b32_e32 v0, s2
-; VI-NEXT:    v_mov_b32_e32 v3, s1
-; VI-NEXT:    v_mov_b32_e32 v1, s3
-; VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
-; VI-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_u64_ugt:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
+; DAG-GFX-NEXT:    s_mov_b32 s7, 0xf000
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, 0
+; DAG-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-GFX-NEXT:    v_cmp_gt_u64_e64 s[2:3], s[2:3], v[0:1]
+; DAG-GFX-NEXT:    s_mov_b32 s6, -1
+; DAG-GFX-NEXT:    s_mov_b32 s4, s0
+; DAG-GFX-NEXT:    s_mov_b32 s5, s1
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, s2
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, s3
+; DAG-GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_u64_ugt:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, 0
+; DAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-VI-NEXT:    v_cmp_gt_u64_e64 s[2:3], s[2:3], v[0:1]
+; DAG-VI-NEXT:    v_mov_b32_e32 v2, s0
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, s2
+; DAG-VI-NEXT:    v_mov_b32_e32 v3, s1
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, s3
+; DAG-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_u64_ugt:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-GFX-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    v_cmp_gt_u64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-GFX-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_u64_ugt:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-VI-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_gt_u64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 34)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @v_icmp_u64_uge(i64 addrspace(1)* %out, i64 %src) {
-; GFX-LABEL: v_icmp_u64_uge:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s7, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    v_mov_b32_e32 v1, 0
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_ge_u64_e64 s[2:3], s[2:3], v[0:1]
-; GFX-NEXT:    s_mov_b32 s6, -1
-; GFX-NEXT:    s_mov_b32 s4, s0
-; GFX-NEXT:    s_mov_b32 s5, s1
-; GFX-NEXT:    v_mov_b32_e32 v0, s2
-; GFX-NEXT:    v_mov_b32_e32 v1, s3
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_u64_uge:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    v_mov_b32_e32 v1, 0
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_ge_u64_e64 s[2:3], s[2:3], v[0:1]
-; VI-NEXT:    v_mov_b32_e32 v2, s0
-; VI-NEXT:    v_mov_b32_e32 v0, s2
-; VI-NEXT:    v_mov_b32_e32 v3, s1
-; VI-NEXT:    v_mov_b32_e32 v1, s3
-; VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
-; VI-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_u64_uge:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
+; DAG-GFX-NEXT:    s_mov_b32 s7, 0xf000
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, 0
+; DAG-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-GFX-NEXT:    v_cmp_ge_u64_e64 s[2:3], s[2:3], v[0:1]
+; DAG-GFX-NEXT:    s_mov_b32 s6, -1
+; DAG-GFX-NEXT:    s_mov_b32 s4, s0
+; DAG-GFX-NEXT:    s_mov_b32 s5, s1
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, s2
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, s3
+; DAG-GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_u64_uge:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, 0
+; DAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-VI-NEXT:    v_cmp_ge_u64_e64 s[2:3], s[2:3], v[0:1]
+; DAG-VI-NEXT:    v_mov_b32_e32 v2, s0
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, s2
+; DAG-VI-NEXT:    v_mov_b32_e32 v3, s1
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, s3
+; DAG-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_u64_uge:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-GFX-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    v_cmp_ge_u64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-GFX-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_u64_uge:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-VI-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_ge_u64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 35)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @v_icmp_u64_ult(i64 addrspace(1)* %out, i64 %src) {
-; GFX-LABEL: v_icmp_u64_ult:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s7, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    v_mov_b32_e32 v1, 0
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_lt_u64_e64 s[2:3], s[2:3], v[0:1]
-; GFX-NEXT:    s_mov_b32 s6, -1
-; GFX-NEXT:    s_mov_b32 s4, s0
-; GFX-NEXT:    s_mov_b32 s5, s1
-; GFX-NEXT:    v_mov_b32_e32 v0, s2
-; GFX-NEXT:    v_mov_b32_e32 v1, s3
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_u64_ult:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    v_mov_b32_e32 v1, 0
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_lt_u64_e64 s[2:3], s[2:3], v[0:1]
-; VI-NEXT:    v_mov_b32_e32 v2, s0
-; VI-NEXT:    v_mov_b32_e32 v0, s2
-; VI-NEXT:    v_mov_b32_e32 v3, s1
-; VI-NEXT:    v_mov_b32_e32 v1, s3
-; VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
-; VI-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_u64_ult:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
+; DAG-GFX-NEXT:    s_mov_b32 s7, 0xf000
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, 0
+; DAG-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-GFX-NEXT:    v_cmp_lt_u64_e64 s[2:3], s[2:3], v[0:1]
+; DAG-GFX-NEXT:    s_mov_b32 s6, -1
+; DAG-GFX-NEXT:    s_mov_b32 s4, s0
+; DAG-GFX-NEXT:    s_mov_b32 s5, s1
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, s2
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, s3
+; DAG-GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_u64_ult:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, 0
+; DAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-VI-NEXT:    v_cmp_lt_u64_e64 s[2:3], s[2:3], v[0:1]
+; DAG-VI-NEXT:    v_mov_b32_e32 v2, s0
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, s2
+; DAG-VI-NEXT:    v_mov_b32_e32 v3, s1
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, s3
+; DAG-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_u64_ult:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-GFX-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    v_cmp_lt_u64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-GFX-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_u64_ult:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-VI-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_lt_u64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 36)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @v_icmp_u64_ule(i64 addrspace(1)* %out, i64 %src) {
-; GFX-LABEL: v_icmp_u64_ule:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s7, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    v_mov_b32_e32 v1, 0
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_le_u64_e64 s[2:3], s[2:3], v[0:1]
-; GFX-NEXT:    s_mov_b32 s6, -1
-; GFX-NEXT:    s_mov_b32 s4, s0
-; GFX-NEXT:    s_mov_b32 s5, s1
-; GFX-NEXT:    v_mov_b32_e32 v0, s2
-; GFX-NEXT:    v_mov_b32_e32 v1, s3
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_u64_ule:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    v_mov_b32_e32 v1, 0
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_le_u64_e64 s[2:3], s[2:3], v[0:1]
-; VI-NEXT:    v_mov_b32_e32 v2, s0
-; VI-NEXT:    v_mov_b32_e32 v0, s2
-; VI-NEXT:    v_mov_b32_e32 v3, s1
-; VI-NEXT:    v_mov_b32_e32 v1, s3
-; VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
-; VI-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_u64_ule:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
+; DAG-GFX-NEXT:    s_mov_b32 s7, 0xf000
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, 0
+; DAG-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-GFX-NEXT:    v_cmp_le_u64_e64 s[2:3], s[2:3], v[0:1]
+; DAG-GFX-NEXT:    s_mov_b32 s6, -1
+; DAG-GFX-NEXT:    s_mov_b32 s4, s0
+; DAG-GFX-NEXT:    s_mov_b32 s5, s1
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, s2
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, s3
+; DAG-GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_u64_ule:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, 0
+; DAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-VI-NEXT:    v_cmp_le_u64_e64 s[2:3], s[2:3], v[0:1]
+; DAG-VI-NEXT:    v_mov_b32_e32 v2, s0
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, s2
+; DAG-VI-NEXT:    v_mov_b32_e32 v3, s1
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, s3
+; DAG-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_u64_ule:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-GFX-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    v_cmp_le_u64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-GFX-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_u64_ule:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-VI-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_le_u64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 37)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @v_icmp_i64_sgt(i64 addrspace(1)* %out, i64 %src) {
-; GFX-LABEL: v_icmp_i64_sgt:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s7, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    v_mov_b32_e32 v1, 0
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_gt_i64_e64 s[2:3], s[2:3], v[0:1]
-; GFX-NEXT:    s_mov_b32 s6, -1
-; GFX-NEXT:    s_mov_b32 s4, s0
-; GFX-NEXT:    s_mov_b32 s5, s1
-; GFX-NEXT:    v_mov_b32_e32 v0, s2
-; GFX-NEXT:    v_mov_b32_e32 v1, s3
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i64_sgt:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    v_mov_b32_e32 v1, 0
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_gt_i64_e64 s[2:3], s[2:3], v[0:1]
-; VI-NEXT:    v_mov_b32_e32 v2, s0
-; VI-NEXT:    v_mov_b32_e32 v0, s2
-; VI-NEXT:    v_mov_b32_e32 v3, s1
-; VI-NEXT:    v_mov_b32_e32 v1, s3
-; VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
-; VI-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_i64_sgt:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
+; DAG-GFX-NEXT:    s_mov_b32 s7, 0xf000
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, 0
+; DAG-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-GFX-NEXT:    v_cmp_gt_i64_e64 s[2:3], s[2:3], v[0:1]
+; DAG-GFX-NEXT:    s_mov_b32 s6, -1
+; DAG-GFX-NEXT:    s_mov_b32 s4, s0
+; DAG-GFX-NEXT:    s_mov_b32 s5, s1
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, s2
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, s3
+; DAG-GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_i64_sgt:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, 0
+; DAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-VI-NEXT:    v_cmp_gt_i64_e64 s[2:3], s[2:3], v[0:1]
+; DAG-VI-NEXT:    v_mov_b32_e32 v2, s0
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, s2
+; DAG-VI-NEXT:    v_mov_b32_e32 v3, s1
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, s3
+; DAG-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_i64_sgt:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-GFX-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    v_cmp_gt_i64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-GFX-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i64_sgt:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-VI-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_gt_i64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 38)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @v_icmp_i64_sge(i64 addrspace(1)* %out, i64 %src) {
-; GFX-LABEL: v_icmp_i64_sge:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s7, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    v_mov_b32_e32 v1, 0
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_ge_i64_e64 s[2:3], s[2:3], v[0:1]
-; GFX-NEXT:    s_mov_b32 s6, -1
-; GFX-NEXT:    s_mov_b32 s4, s0
-; GFX-NEXT:    s_mov_b32 s5, s1
-; GFX-NEXT:    v_mov_b32_e32 v0, s2
-; GFX-NEXT:    v_mov_b32_e32 v1, s3
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i64_sge:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    v_mov_b32_e32 v1, 0
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_ge_i64_e64 s[2:3], s[2:3], v[0:1]
-; VI-NEXT:    v_mov_b32_e32 v2, s0
-; VI-NEXT:    v_mov_b32_e32 v0, s2
-; VI-NEXT:    v_mov_b32_e32 v3, s1
-; VI-NEXT:    v_mov_b32_e32 v1, s3
-; VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
-; VI-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_i64_sge:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
+; DAG-GFX-NEXT:    s_mov_b32 s7, 0xf000
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, 0
+; DAG-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-GFX-NEXT:    v_cmp_ge_i64_e64 s[2:3], s[2:3], v[0:1]
+; DAG-GFX-NEXT:    s_mov_b32 s6, -1
+; DAG-GFX-NEXT:    s_mov_b32 s4, s0
+; DAG-GFX-NEXT:    s_mov_b32 s5, s1
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, s2
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, s3
+; DAG-GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_i64_sge:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, 0
+; DAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-VI-NEXT:    v_cmp_ge_i64_e64 s[2:3], s[2:3], v[0:1]
+; DAG-VI-NEXT:    v_mov_b32_e32 v2, s0
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, s2
+; DAG-VI-NEXT:    v_mov_b32_e32 v3, s1
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, s3
+; DAG-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_i64_sge:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-GFX-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    v_cmp_ge_i64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-GFX-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i64_sge:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-VI-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_ge_i64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 39)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @v_icmp_i64_slt(i64 addrspace(1)* %out, i64 %src) {
-; GFX-LABEL: v_icmp_i64_slt:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s7, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    v_mov_b32_e32 v1, 0
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_lt_i64_e64 s[2:3], s[2:3], v[0:1]
-; GFX-NEXT:    s_mov_b32 s6, -1
-; GFX-NEXT:    s_mov_b32 s4, s0
-; GFX-NEXT:    s_mov_b32 s5, s1
-; GFX-NEXT:    v_mov_b32_e32 v0, s2
-; GFX-NEXT:    v_mov_b32_e32 v1, s3
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i64_slt:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    v_mov_b32_e32 v1, 0
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_lt_i64_e64 s[2:3], s[2:3], v[0:1]
-; VI-NEXT:    v_mov_b32_e32 v2, s0
-; VI-NEXT:    v_mov_b32_e32 v0, s2
-; VI-NEXT:    v_mov_b32_e32 v3, s1
-; VI-NEXT:    v_mov_b32_e32 v1, s3
-; VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
-; VI-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_i64_slt:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
+; DAG-GFX-NEXT:    s_mov_b32 s7, 0xf000
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, 0
+; DAG-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-GFX-NEXT:    v_cmp_lt_i64_e64 s[2:3], s[2:3], v[0:1]
+; DAG-GFX-NEXT:    s_mov_b32 s6, -1
+; DAG-GFX-NEXT:    s_mov_b32 s4, s0
+; DAG-GFX-NEXT:    s_mov_b32 s5, s1
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, s2
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, s3
+; DAG-GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_i64_slt:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, 0
+; DAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-VI-NEXT:    v_cmp_lt_i64_e64 s[2:3], s[2:3], v[0:1]
+; DAG-VI-NEXT:    v_mov_b32_e32 v2, s0
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, s2
+; DAG-VI-NEXT:    v_mov_b32_e32 v3, s1
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, s3
+; DAG-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_i64_slt:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-GFX-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    v_cmp_lt_i64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-GFX-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i64_slt:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-VI-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_lt_i64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 40)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @v_icmp_i64_sle(i64 addrspace(1)* %out, i64 %src) {
-; GFX-LABEL: v_icmp_i64_sle:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s7, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    v_mov_b32_e32 v1, 0
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    v_cmp_le_i64_e64 s[2:3], s[2:3], v[0:1]
-; GFX-NEXT:    s_mov_b32 s6, -1
-; GFX-NEXT:    s_mov_b32 s4, s0
-; GFX-NEXT:    s_mov_b32 s5, s1
-; GFX-NEXT:    v_mov_b32_e32 v0, s2
-; GFX-NEXT:    v_mov_b32_e32 v1, s3
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i64_sle:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    v_mov_b32_e32 v1, 0
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_le_i64_e64 s[2:3], s[2:3], v[0:1]
-; VI-NEXT:    v_mov_b32_e32 v2, s0
-; VI-NEXT:    v_mov_b32_e32 v0, s2
-; VI-NEXT:    v_mov_b32_e32 v3, s1
-; VI-NEXT:    v_mov_b32_e32 v1, s3
-; VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
-; VI-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_i64_sle:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
+; DAG-GFX-NEXT:    s_mov_b32 s7, 0xf000
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, 0
+; DAG-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-GFX-NEXT:    v_cmp_le_i64_e64 s[2:3], s[2:3], v[0:1]
+; DAG-GFX-NEXT:    s_mov_b32 s6, -1
+; DAG-GFX-NEXT:    s_mov_b32 s4, s0
+; DAG-GFX-NEXT:    s_mov_b32 s5, s1
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, s2
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, s3
+; DAG-GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_i64_sle:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, 0
+; DAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-VI-NEXT:    v_cmp_le_i64_e64 s[2:3], s[2:3], v[0:1]
+; DAG-VI-NEXT:    v_mov_b32_e32 v2, s0
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, s2
+; DAG-VI-NEXT:    v_mov_b32_e32 v3, s1
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, s3
+; DAG-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_i64_sle:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-GFX-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    v_cmp_le_i64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-GFX-NEXT:    global_store_dwordx2 v2, v[0:1], s[0:1]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i64_sle:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GISEL-VI-NEXT:    s_mov_b64 s[4:5], 0x64
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s4
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s5
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_le_i64_e64 s[2:3], s[2:3], v[0:1]
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i64(i64 %src, i64 100, i32 41)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @v_icmp_i16_eq(i64 addrspace(1)* %out, i16 %src) {
-; GFX-LABEL: v_icmp_i16_eq:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    s_and_b32 s2, s2, 0xffff
-; GFX-NEXT:    v_cmp_eq_u32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i16_eq:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_eq_u16_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_i16_eq:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
+; DAG-GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
+; DAG-GFX-NEXT:    s_mov_b32 s3, 0xf000
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-GFX-NEXT:    s_and_b32 s2, s2, 0xffff
+; DAG-GFX-NEXT:    v_cmp_eq_u32_e64 s[4:5], s2, v0
+; DAG-GFX-NEXT:    s_mov_b32 s2, -1
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, s4
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, s5
+; DAG-GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_i16_eq:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; DAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-VI-NEXT:    v_cmp_eq_u16_e64 s[2:3], s2, v0
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; DAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; DAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; DAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_i16_eq:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    v_cmp_eq_u16_e64 s[0:1], s4, v0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s1
+; GISEL-GFX-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i16_eq:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_eq_u16_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 32)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @v_icmp_i16(i64 addrspace(1)* %out, i16 %src) {
-; GCN-LABEL: v_icmp_i16:
-; GCN:       ; %bb.0:
-; GCN-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_i16:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_i16:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_i16:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    global_store_dwordx2 v0, v[0:1], s[0:1]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i16:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s1
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[0:1], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 30)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @v_icmp_i16_ne(i64 addrspace(1)* %out, i16 %src) {
-; GFX-LABEL: v_icmp_i16_ne:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    s_and_b32 s2, s2, 0xffff
-; GFX-NEXT:    v_cmp_ne_u32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i16_ne:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_ne_u16_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_i16_ne:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
+; DAG-GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
+; DAG-GFX-NEXT:    s_mov_b32 s3, 0xf000
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-GFX-NEXT:    s_and_b32 s2, s2, 0xffff
+; DAG-GFX-NEXT:    v_cmp_ne_u32_e64 s[4:5], s2, v0
+; DAG-GFX-NEXT:    s_mov_b32 s2, -1
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, s4
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, s5
+; DAG-GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_i16_ne:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; DAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-VI-NEXT:    v_cmp_ne_u16_e64 s[2:3], s2, v0
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; DAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; DAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; DAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_i16_ne:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    v_cmp_ne_u16_e64 s[0:1], s4, v0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s1
+; GISEL-GFX-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i16_ne:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_ne_u16_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 33)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @v_icmp_i16_ugt(i64 addrspace(1)* %out, i16 %src) {
-; GFX-LABEL: v_icmp_i16_ugt:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    s_and_b32 s2, s2, 0xffff
-; GFX-NEXT:    v_cmp_gt_u32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i16_ugt:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_gt_u16_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_i16_ugt:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
+; DAG-GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
+; DAG-GFX-NEXT:    s_mov_b32 s3, 0xf000
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-GFX-NEXT:    s_and_b32 s2, s2, 0xffff
+; DAG-GFX-NEXT:    v_cmp_gt_u32_e64 s[4:5], s2, v0
+; DAG-GFX-NEXT:    s_mov_b32 s2, -1
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, s4
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, s5
+; DAG-GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_i16_ugt:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; DAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-VI-NEXT:    v_cmp_gt_u16_e64 s[2:3], s2, v0
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; DAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; DAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; DAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_i16_ugt:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    v_cmp_gt_u16_e64 s[0:1], s4, v0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s1
+; GISEL-GFX-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i16_ugt:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_gt_u16_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 34)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @v_icmp_i16_uge(i64 addrspace(1)* %out, i16 %src) {
-; GFX-LABEL: v_icmp_i16_uge:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    s_and_b32 s2, s2, 0xffff
-; GFX-NEXT:    v_cmp_ge_u32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i16_uge:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_ge_u16_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_i16_uge:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
+; DAG-GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
+; DAG-GFX-NEXT:    s_mov_b32 s3, 0xf000
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-GFX-NEXT:    s_and_b32 s2, s2, 0xffff
+; DAG-GFX-NEXT:    v_cmp_ge_u32_e64 s[4:5], s2, v0
+; DAG-GFX-NEXT:    s_mov_b32 s2, -1
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, s4
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, s5
+; DAG-GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_i16_uge:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; DAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-VI-NEXT:    v_cmp_ge_u16_e64 s[2:3], s2, v0
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; DAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; DAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; DAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_i16_uge:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    v_cmp_ge_u16_e64 s[0:1], s4, v0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s1
+; GISEL-GFX-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i16_uge:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_ge_u16_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 35)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @v_icmp_i16_ult(i64 addrspace(1)* %out, i16 %src) {
-; GFX-LABEL: v_icmp_i16_ult:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    s_and_b32 s2, s2, 0xffff
-; GFX-NEXT:    v_cmp_lt_u32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i16_ult:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_lt_u16_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_i16_ult:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
+; DAG-GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
+; DAG-GFX-NEXT:    s_mov_b32 s3, 0xf000
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-GFX-NEXT:    s_and_b32 s2, s2, 0xffff
+; DAG-GFX-NEXT:    v_cmp_lt_u32_e64 s[4:5], s2, v0
+; DAG-GFX-NEXT:    s_mov_b32 s2, -1
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, s4
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, s5
+; DAG-GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_i16_ult:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; DAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-VI-NEXT:    v_cmp_lt_u16_e64 s[2:3], s2, v0
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; DAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; DAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; DAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_i16_ult:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    v_cmp_lt_u16_e64 s[0:1], s4, v0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s1
+; GISEL-GFX-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i16_ult:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_lt_u16_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 36)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @v_icmp_i16_ule(i64 addrspace(1)* %out, i16 %src) {
-; GFX-LABEL: v_icmp_i16_ule:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    s_and_b32 s2, s2, 0xffff
-; GFX-NEXT:    v_cmp_le_u32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i16_ule:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_le_u16_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_i16_ule:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
+; DAG-GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
+; DAG-GFX-NEXT:    s_mov_b32 s3, 0xf000
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-GFX-NEXT:    s_and_b32 s2, s2, 0xffff
+; DAG-GFX-NEXT:    v_cmp_le_u32_e64 s[4:5], s2, v0
+; DAG-GFX-NEXT:    s_mov_b32 s2, -1
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, s4
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, s5
+; DAG-GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_i16_ule:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; DAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-VI-NEXT:    v_cmp_le_u16_e64 s[2:3], s2, v0
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; DAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; DAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; DAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_i16_ule:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    v_cmp_le_u16_e64 s[0:1], s4, v0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s1
+; GISEL-GFX-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i16_ule:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_le_u16_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 37)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @v_icmp_i16_sgt(i64 addrspace(1)* %out, i16 %src) #1 {
-; GFX-LABEL: v_icmp_i16_sgt:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    s_sext_i32_i16 s2, s2
-; GFX-NEXT:    v_cmp_gt_i32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i16_sgt:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_gt_i16_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_i16_sgt:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
+; DAG-GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
+; DAG-GFX-NEXT:    s_mov_b32 s3, 0xf000
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-GFX-NEXT:    s_sext_i32_i16 s2, s2
+; DAG-GFX-NEXT:    v_cmp_gt_i32_e64 s[4:5], s2, v0
+; DAG-GFX-NEXT:    s_mov_b32 s2, -1
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, s4
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, s5
+; DAG-GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_i16_sgt:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; DAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-VI-NEXT:    v_cmp_gt_i16_e64 s[2:3], s2, v0
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; DAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; DAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; DAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_i16_sgt:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    v_cmp_gt_i16_e64 s[0:1], s4, v0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s1
+; GISEL-GFX-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i16_sgt:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_gt_i16_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 38)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @v_icmp_i16_sge(i64 addrspace(1)* %out, i16 %src) {
-; GFX-LABEL: v_icmp_i16_sge:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    s_sext_i32_i16 s2, s2
-; GFX-NEXT:    v_cmp_ge_i32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i16_sge:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_ge_i16_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_i16_sge:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
+; DAG-GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
+; DAG-GFX-NEXT:    s_mov_b32 s3, 0xf000
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-GFX-NEXT:    s_sext_i32_i16 s2, s2
+; DAG-GFX-NEXT:    v_cmp_ge_i32_e64 s[4:5], s2, v0
+; DAG-GFX-NEXT:    s_mov_b32 s2, -1
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, s4
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, s5
+; DAG-GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_i16_sge:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; DAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-VI-NEXT:    v_cmp_ge_i16_e64 s[2:3], s2, v0
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; DAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; DAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; DAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_i16_sge:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    v_cmp_ge_i16_e64 s[0:1], s4, v0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s1
+; GISEL-GFX-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i16_sge:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_ge_i16_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 39)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @v_icmp_i16_slt(i64 addrspace(1)* %out, i16 %src) {
-; GFX-LABEL: v_icmp_i16_slt:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    s_sext_i32_i16 s2, s2
-; GFX-NEXT:    v_cmp_lt_i32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i16_slt:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_lt_i16_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_i16_slt:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
+; DAG-GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
+; DAG-GFX-NEXT:    s_mov_b32 s3, 0xf000
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-GFX-NEXT:    s_sext_i32_i16 s2, s2
+; DAG-GFX-NEXT:    v_cmp_lt_i32_e64 s[4:5], s2, v0
+; DAG-GFX-NEXT:    s_mov_b32 s2, -1
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, s4
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, s5
+; DAG-GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_i16_slt:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; DAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-VI-NEXT:    v_cmp_lt_i16_e64 s[2:3], s2, v0
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; DAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; DAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; DAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_i16_slt:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    v_cmp_lt_i16_e64 s[0:1], s4, v0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s1
+; GISEL-GFX-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i16_slt:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_lt_i16_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 40)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @v_icmp_i16_sle(i64 addrspace(1)* %out, i16 %src) {
-; GFX-LABEL: v_icmp_i16_sle:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
-; GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s3, 0xf000
-; GFX-NEXT:    v_mov_b32_e32 v0, 0x64
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    s_sext_i32_i16 s2, s2
-; GFX-NEXT:    v_cmp_le_i32_e64 s[4:5], s2, v0
-; GFX-NEXT:    s_mov_b32 s2, -1
-; GFX-NEXT:    v_mov_b32_e32 v0, s4
-; GFX-NEXT:    v_mov_b32_e32 v1, s5
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i16_sle:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x64
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_cmp_le_i16_e64 s[2:3], s2, v0
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
+; DAG-GFX-LABEL: v_icmp_i16_sle:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_load_dword s2, s[0:1], 0xb
+; DAG-GFX-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
+; DAG-GFX-NEXT:    s_mov_b32 s3, 0xf000
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-GFX-NEXT:    s_sext_i32_i16 s2, s2
+; DAG-GFX-NEXT:    v_cmp_le_i32_e64 s[4:5], s2, v0
+; DAG-GFX-NEXT:    s_mov_b32 s2, -1
+; DAG-GFX-NEXT:    v_mov_b32_e32 v0, s4
+; DAG-GFX-NEXT:    v_mov_b32_e32 v1, s5
+; DAG-GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: v_icmp_i16_sle:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; DAG-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; DAG-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; DAG-VI-NEXT:    v_cmp_le_i16_e64 s[2:3], s2, v0
+; DAG-VI-NEXT:    v_mov_b32_e32 v0, s0
+; DAG-VI-NEXT:    v_mov_b32_e32 v2, s2
+; DAG-VI-NEXT:    v_mov_b32_e32 v1, s1
+; DAG-VI-NEXT:    v_mov_b32_e32 v3, s3
+; DAG-VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: v_icmp_i16_sle:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GISEL-GFX-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v2, 0
+; GISEL-GFX-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-GFX-NEXT:    v_cmp_le_i16_e64 s[0:1], s4, v0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v0, s0
+; GISEL-GFX-NEXT:    v_mov_b32_e32 v1, s1
+; GISEL-GFX-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: v_icmp_i16_sle:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GISEL-VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, 0x64
+; GISEL-VI-NEXT:    s_waitcnt lgkmcnt(0)
+; GISEL-VI-NEXT:    v_cmp_le_i16_e64 s[2:3], s2, v0
+; GISEL-VI-NEXT:    v_mov_b32_e32 v0, s2
+; GISEL-VI-NEXT:    v_mov_b32_e32 v3, s1
+; GISEL-VI-NEXT:    v_mov_b32_e32 v1, s3
+; GISEL-VI-NEXT:    v_mov_b32_e32 v2, s0
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
   %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 41)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
-define amdgpu_kernel void @v_icmp_i1_ne0(i64 addrspace(1)* %out, i32 %a, i32 %b) {
-; GFX-LABEL: v_icmp_i1_ne0:
-; GFX:       ; %bb.0:
-; GFX-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
-; GFX-NEXT:    s_mov_b32 s7, 0xf000
-; GFX-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX-NEXT:    s_cmp_gt_u32 s2, 1
-; GFX-NEXT:    s_cselect_b64 s[4:5], -1, 0
-; GFX-NEXT:    s_cmp_gt_u32 s3, 2
-; GFX-NEXT:    s_cselect_b64 s[2:3], -1, 0
-; GFX-NEXT:    s_and_b64 s[2:3], s[4:5], s[2:3]
-; GFX-NEXT:    s_mov_b32 s6, -1
-; GFX-NEXT:    s_mov_b32 s4, s0
-; GFX-NEXT:    s_mov_b32 s5, s1
-; GFX-NEXT:    v_mov_b32_e32 v0, s2
-; GFX-NEXT:    v_mov_b32_e32 v1, s3
-; GFX-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
-; GFX-NEXT:    s_endpgm
-;
-; VI-LABEL: v_icmp_i1_ne0:
-; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    s_cmp_gt_u32 s2, 1
-; VI-NEXT:    s_cselect_b64 s[4:5], -1, 0
-; VI-NEXT:    s_cmp_gt_u32 s3, 2
-; VI-NEXT:    s_cselect_b64 s[2:3], -1, 0
-; VI-NEXT:    s_and_b64 s[2:3], s[4:5], s[2:3]
-; VI-NEXT:    v_mov_b32_e32 v0, s0
-; VI-NEXT:    v_mov_b32_e32 v2, s2
-; VI-NEXT:    v_mov_b32_e32 v1, s1
-; VI-NEXT:    v_mov_b32_e32 v3, s3
-; VI-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
-; VI-NEXT:    s_endpgm
-  %c0 = icmp ugt i32 %a, 1
-  %c1 = icmp ugt i32 %b, 2
-  %src = and i1 %c0, %c1
-  %result = call i64 @llvm.amdgcn.icmp.i1(i1 %src, i1 false, i32 33)
+; FIXME: Can't get the DAG pattern to import properly
+;
+; define amdgpu_kernel void @v_icmp_i1_ne0(i64 addrspace(1)* %out, i32 %a, i32 %b) {
+;   %c0 = icmp ugt i32 %a, 1
+;   %c1 = icmp ugt i32 %b, 2
+;   %src = and i1 %c0, %c1
+;   %result = call i64 @llvm.amdgcn.icmp.i1(i1 %src, i1 false, i32 33)
+;   store i64 %result, i64 addrspace(1)* %out
+;   ret void
+; }
+
+define amdgpu_ps void @test_intr_icmp_i32_invalid_cc(i64 addrspace(1)* %out, i32 %src) {
+; DAG-GFX-LABEL: test_intr_icmp_i32_invalid_cc:
+; DAG-GFX:       ; %bb.0:
+; DAG-GFX-NEXT:    s_endpgm
+;
+; DAG-VI-LABEL: test_intr_icmp_i32_invalid_cc:
+; DAG-VI:       ; %bb.0:
+; DAG-VI-NEXT:    s_endpgm
+;
+; GISEL-GFX-LABEL: test_intr_icmp_i32_invalid_cc:
+; GISEL-GFX:       ; %bb.0:
+; GISEL-GFX-NEXT:    global_store_dwordx2 v[0:1], v[0:1], off
+; GISEL-GFX-NEXT:    s_endpgm
+;
+; GISEL-VI-LABEL: test_intr_icmp_i32_invalid_cc:
+; GISEL-VI:       ; %bb.0:
+; GISEL-VI-NEXT:    flat_store_dwordx2 v[0:1], v[0:1]
+; GISEL-VI-NEXT:    s_endpgm
+  %result = call i64 @llvm.amdgcn.icmp.i32(i32 %src, i32 100, i32 9999)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
-define amdgpu_ps void @test_intr_icmp_i32_invalid_cc(i32 addrspace(1)* %out, i32 %src) {
-; GCN-LABEL: test_intr_icmp_i32_invalid_cc:
-; GCN:       ; %bb.0:
-; GCN-NEXT:    s_endpgm
-  %result = call i32 @llvm.amdgcn.icmp.i32.i32(i32 %src, i32 100, i32 9999)
-  store i32 %result, i32 addrspace(1)* %out
-  ret void
-}
-
 attributes #0 = { nounwind readnone convergent }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GCN: {{.*}}
+; GFX: {{.*}}
+; VI: {{.*}}