AMDGPU/GlobalISel: Combine zext(trunc x) to x after RegBankSelect

[Petar.Avramovic]

Summary

RegBankSelect creates zext and trunc when it selects banks for uniform i1.
Add zext_trunc_fold from generic combiner to post RegBankSelect combiner.

[arsenm]

This should also apply to conditional branches, but the user doesn't actually matter. This is really a generic combine for (trunc (bool_ext_type bool_producer))

llvm/lib/Target/AMDGPU/AMDGPUCombine.td

44

The select part here isn't essential, but the matcher here wants a specific opcode. I guess you could bypass the generated combine matcher and just call this combine in a switch, or at least add the relevant G_BRCOND user

llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp

90

Changing the instruction without notifying the observer

llvm/test/CodeGen/AMDGPU/GlobalISel/combine-uniform-icmp-select.mir

37–62 ↗

(On Diff #319258)

Most of these instructions aren't relevant to the combine. You can also directly emit copies from 64-bit SGPRs even though we emit them normally as separate 32-bit copies

116–142 ↗

(On Diff #319258)

Ditto

[foad]

llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp

86

Can you call the variable ICmp instead of ICMP?

[foad]

llvm/lib/Target/AMDGPU/AMDGPUCombine.td

40

You don't need AMDGPURegBankCombinerHelper::UniformICmpSelectMatchInfo at all. You can just use Register. See D95645 for an example.

[Petar.Avramovic]

Addressed review comments.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp

76–77

I don't think this really needs a one use check. Consider the case where another use already exists without the intermediate casts:

%x:s32 = G_ICMP
%y:s1 = G_TRUNC %x
%z:s32 = G_ZEXT %y
%select0 = G_SELECT %x
%select1 = G_SELECT %z

[foad]

I don't understand why this needs to be AMDGPU-specific, and why it only works for uniform values, and why it only works inside G_SELECT and G_BRCOND.

Can't you have a generic combine that simplifies (zext (trunc x)) -> x if the types match and the high bits of x are known to be zero?

[Petar.Avramovic]

llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp

76–77

I wanted to keep this as simple as possible and cover most common case (single use).
Can we get mir like that? Regbankselect independently legalizes sgpr icmp and select. ICMP gets followed by G_TRUNC and G_SELECT has to G_ZEXT input condition. I would expect something like this:

%x:s32 = G_ICMP
%y:s1 = G_TRUNC %x
...
%z:s32 = G_ZEXT %y
%select1 = G_SELECT %z
...
%w:s32 = G_ZEXT %y
%select0 = G_SELECT %w

Maybe we could cover this case for select, are there more?
Afaik only trunc can end up having multiple uses. So we have to check G_TRUNC uses and find the one that Helper.dominates other uses. If this happens to be our G_ZEXT then move icmp before select and trunc after select, like this:

...
%z:s32 = G_ZEXT %y //dead
%x:s32 = G_ICMP
%select1 = G_SELECT %x
%y:s1 = G_TRUNC %x
...
%w:s32 = G_ZEXT %y
%select0 = G_SELECT %w

llvm/test/CodeGen/AMDGPU/GlobalISel/combine-move-uniform-icmp-with-one-use.mir

34–35

Combine targets specific case when there are instructions between uniform G_ICMP and G_SELECT/G_BRCOND. Zext and trunc are there but combining them has no effect on having to temporary save scc. We have to move icmp. Also since Zext and trunc have no effect on selected instruction we don't move them and just leave them to be removed as dead instructions.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp

76–77

If the builder is CSEing, you could end up with the same trunc used multiple times

[Petar.Avramovic]

Handle some cases with many uses. Adding icmp fold without move for the case when we can't move icmp because code looks nicer in the case with more than one use.

[Petar.Avramovic]

llvm/test/CodeGen/AMDGPU/GlobalISel/move-uniform-icmp.ll

24–26 ↗

(On Diff #320792)

This looks pretty much same as before without uniform_icmp combine.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp

123–126

I don't think this should be trying to find defs and move them. If we're CSEing, just creating the instruction you need would get the desired result

[Petar.Avramovic]

Use zext_trunc_fold from generic combiner to separately fold all cases of zext(trunc x) -> x made by regbankselect.
icmp move before select/brcond has to be aware of current state of MF since we run combines top-down and instructions (trunc) can be left without uses (zext was deleted by zext_trunc_fold)

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp

72

Opcode check first?

92

You shouldn't need to check dominance (and I don't see how this would ever not be the case)

103

This isn't changing the operands anymore? (I also think just creating the new instruction with the new operand is cleaner than modifying in place, doing it that way should fix the multiple use case too)

103–107

This really shouldn't be trying to move instructions. Do you even really need to erase the trunc? If it's dead it will be removed already

[Petar.Avramovic]

llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp

92

Trunc could have a use before the MI so we cant move icmp and trunc past that use.

103

zext_trunc_fold changes operands. This now only moves icmp (and trunc for multiple use).

103–107

Do you even really need to erase the trunc?

ICmp move breaks ssa in mir, trunc uses icmp before it is defined. I meant to delete trunc because this is the place we broke ssa, and we are aware of it. Leaving it to be deleted by something that eliminates dead instructions should work fine (I don't expect anything else to check where uses of this trunc are defined).

This really shouldn't be trying to move instructions.

What do you suggest, making new icmp (and trunc) or something else?

llvm/test/CodeGen/AMDGPU/GlobalISel/combine-move-uniform-icmp.mir

203 ↗

(On Diff #322084)

Trunc use before select.

213 ↗

(On Diff #322084)

After zext_trunc fold this select uses %17 G_ICMP instead of %22 G_ZEXT but we can't move icmp because %11:sgpr(s1) = G_TRUNC %17(s32) uses icmp and %18:sgpr(s32) = G_ANYEXT %11(s1) above uses trunc.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp

103–107

Yes, if you recreate the desired instruction it should automatically CSE as you need by the builder

[arsenm]

Actually, why is this patch necessary? The ZEXT+TRUNC handling takes care of this already?

[arsenm]

In D95432#2573420, @arsenm wrote:

Actually, why is this patch necessary? The ZEXT+TRUNC handling takes care of this already?

Oh right, the problem here is actually the intermediate SCC copies produced as a selection artifact.

I'm not sure treating this as a combine is the correct way to go about this. The DAG handles this with a scheduler to minimize physical register liveranges. We might be better treating this off as a scheduling issue for after selection, when we directly see the SCC defs.

[foad]

In D95432#2573444, @arsenm wrote:

In D95432#2573420, @arsenm wrote:

Actually, why is this patch necessary? The ZEXT+TRUNC handling takes care of this already?

Oh right, the problem here is actually the intermediate SCC copies produced as a selection artifact.

I'm not sure treating this as a combine is the correct way to go about this. The DAG handles this with a scheduler to minimize physical register liveranges. We might be better treating this off as a scheduling issue for after selection, when we directly see the SCC defs.

For the case I looked at (test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.end.cf.i64.ll) just running zext_trunc_fold as a post-regbankselect combine was enough to get rid of the SCC copies. So perhaps we should commit that first, and then worry about how to handle the remaining cases?

[arsenm]

In D95432#2581586, @foad wrote:

In D95432#2573444, @arsenm wrote:

In D95432#2573420, @arsenm wrote:

Actually, why is this patch necessary? The ZEXT+TRUNC handling takes care of this already?

Oh right, the problem here is actually the intermediate SCC copies produced as a selection artifact.

I'm not sure treating this as a combine is the correct way to go about this. The DAG handles this with a scheduler to minimize physical register liveranges. We might be better treating this off as a scheduling issue for after selection, when we directly see the SCC defs.

For the case I looked at (test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.end.cf.i64.ll) just running zext_trunc_fold as a post-regbankselect combine was enough to get rid of the SCC copies. So perhaps we should commit that first, and then worry about how to handle the remaining cases?

Yes, these are unrelated problems

[Petar.Avramovic]

Dropping icmp move for from this patch. Leaving zext_trunc_fold.
Zext is selected into AND with 1. zext_trunc_fold results in getting rid of the SCC copies when zext was the only instruction between icmp and select/branch.

[foad]

Looks good. I'm not sure we actually need to introduce the helper state class in this patch, do we? But I'll guess we'll need it later.

[Petar.Avramovic]

There is no need for helper state class.

llvm/lib/Target/AMDGPU/AMDGPUCombine.td

def cvt_f32_ubyteN : GICombineRule<
   (defs root:$cvt_f32_ubyteN, cvt_f32_ubyteN_matchdata:$matchinfo),
   (match (wip_match_opcode G_AMDGPU_CVT_F32_UBYTE0,
                            G_AMDGPU_CVT_F32_UBYTE1,
                            G_AMDGPU_CVT_F32_UBYTE2,
                            G_AMDGPU_CVT_F32_UBYTE3):$cvt_f32_ubyteN,
          [{ return PostLegalizerHelper.matchCvtF32UByteN(*${cvt_f32_ubyteN}, ${matchinfo}); }]),
   (apply [{ PostLegalizerHelper.applyCvtF32UByteN(*${cvt_f32_ubyteN}, ${matchinfo}); }])>;
 
+def move_uniform_icmp_w_one_use_matchdata : GIDefMatchData<"Register">;
+
+def move_uniform_icmp_w_one_use : GICombineRule<
+  (defs root:$i1_use, move_uniform_icmp_w_one_use_matchdata:$matchinfo),
+  (match (wip_match_opcode G_SELECT,
+                           G_BRCOND):$i1_use,
+         [{ return RegBankHelper.matchMoveUniformICmpWOneUse(*${i1_use}, ${matchinfo}); }]),
+  (apply [{ RegBankHelper.applyMoveUniformICmpWOneUse(*${i1_use}, ${matchinfo}); }])>;
+
 // Combines which should only apply on SI/VI
 def gfx6gfx7_combines : GICombineGroup<[fcmp_select_to_fmin_fmax_legacy]>;
 
 
 def AMDGPUPreLegalizerCombinerHelper: GICombinerHelper<
   "AMDGPUGenPreLegalizerCombinerHelper", [all_combines]> {
   let DisableRuleOption = "amdgpuprelegalizercombiner-disable-rule";
 }
 
 def AMDGPUPostLegalizerCombinerHelper: GICombinerHelper<
   "AMDGPUGenPostLegalizerCombinerHelper",
   [all_combines, gfx6gfx7_combines,
    uchar_to_float, cvt_f32_ubyteN]> {
   let DisableRuleOption = "amdgpupostlegalizercombiner-disable-rule";
   let StateClass = "AMDGPUPostLegalizerCombinerHelperState";
   let AdditionalArguments = [];
 }
 
 def AMDGPURegBankCombinerHelper : GICombinerHelper<
-  "AMDGPUGenRegBankCombinerHelper", []> {
+  "AMDGPUGenRegBankCombinerHelper", [move_uniform_icmp_w_one_use]> {
   let DisableRuleOption = "amdgpuregbankcombiner-disable-rule";
+  let StateClass = "AMDGPURegBankCombinerHelperState";
+  let AdditionalArguments = [];
 }

llvm/lib/Target/AMDGPU/AMDGPURegBankCombiner.cpp

 //=== lib/CodeGen/GlobalISel/AMDGPURegBankCombiner.cpp ---------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This pass does combining of machine instructions at the generic MI level,
 // after register banks are known.
 //
 //===----------------------------------------------------------------------===//
 
 #include "AMDGPU.h"
 #include "AMDGPULegalizerInfo.h"
+#include "AMDGPURegisterBankInfo.h"
 #include "GCNSubtarget.h"
 #include "llvm/CodeGen/GlobalISel/Combiner.h"
 #include "llvm/CodeGen/GlobalISel/CombinerHelper.h"
 #include "llvm/CodeGen/GlobalISel/CombinerInfo.h"
 #include "llvm/CodeGen/GlobalISel/GISelKnownBits.h"
 #include "llvm/CodeGen/GlobalISel/MIPatternMatch.h"
 #include "llvm/CodeGen/MachineDominators.h"
 #include "llvm/CodeGen/TargetPassConfig.h"
 #include "llvm/Target/TargetMachine.h"
 #define DEBUG_TYPE "amdgpu-regbank-combiner"
 
 using namespace llvm;
 using namespace MIPatternMatch;
 
+class AMDGPURegBankCombinerHelper {
+protected:
+  MachineIRBuilder &B;
+  MachineFunction &MF;
+  MachineRegisterInfo &MRI;
+  const RegisterBankInfo &RBI;
+  const TargetRegisterInfo &TRI;
+  CombinerHelper &Helper;
+  GISelChangeObserver &Observer;
+
+public:
+  AMDGPURegBankCombinerHelper(MachineIRBuilder &B, CombinerHelper &Helper,
+                              GISelChangeObserver &Observer)
+      : B(B), MF(B.getMF()), MRI(*B.getMRI()),
+        RBI(*MF.getSubtarget().getRegBankInfo()),
+        TRI(*MF.getSubtarget().getRegisterInfo()), Helper(Helper),
+        Observer(Observer){};
+
+  bool isSgprRegBank(Register Reg);
+
+  // select zext(trunc(icmp)) ? a : b -> select icmp ? a : b
+  // brcond zext(trunc(icmp)) %bb1, %bb2 -> brcond icmp %bb1, %bb2
+  bool matchMoveUniformICmpWOneUse(MachineInstr &MI, Register &ICmpDef);
+  void applyMoveUniformICmpWOneUse(MachineInstr &MI, Register &ICmpDef);
+  MachineOperand &getConditionOp(MachineInstr &MI);
+};
+
+bool AMDGPURegBankCombinerHelper::isSgprRegBank(Register Reg) {
+  return RBI.getRegBank(Reg, MRI, TRI)->getID() == AMDGPU::SGPRRegBankID;
+}
+
+MachineOperand &AMDGPURegBankCombinerHelper::getConditionOp(MachineInstr &MI) {
+  switch (MI.getOpcode()) {
+  default:
+    llvm_unreachable("Unexpected opcode!");
+  case TargetOpcode::G_SELECT:
+    return MI.getOperand(1);
+  case TargetOpcode::G_BRCOND:
+    return MI.getOperand(0);
+  }
+}
+
+bool AMDGPURegBankCombinerHelper::matchMoveUniformICmpWOneUse(
+    MachineInstr &MI, Register &ICmpDef) {
+  Register Condition = getConditionOp(MI).getReg();
+  if (!isSgprRegBank(Condition) || !MRI.hasOneNonDBGUse(Condition))
+    return false;
+
+  Register ZextSrc;
+  if (!mi_match(Condition, MRI, m_GZExt(m_Reg(ZextSrc))) ||
+      !MRI.hasOneNonDBGUse(ZextSrc))
+    return false;
+
+  Register TruncSrc;
+  if (!mi_match(ZextSrc, MRI, m_GTrunc(m_Reg(TruncSrc))) ||
+      !MRI.hasOneNonDBGUse(TruncSrc) ||
+      MRI.getVRegDef(TruncSrc)->getOpcode() != TargetOpcode::G_ICMP)
+    return false;
+
+  ICmpDef = TruncSrc;
+  return true;
+}
+
+void AMDGPURegBankCombinerHelper::applyMoveUniformICmpWOneUse(
+    MachineInstr &MI, Register &ICmpDef) {
+  MachineInstr &ICmp = *MRI.getVRegDef(ICmpDef);
+  B.setInstrAndDebugLoc(MI);
+  B.buildICmp((CmpInst::Predicate)ICmp.getOperand(1).getPredicate(),
+              ICmp.getOperand(0), ICmp.getOperand(2), ICmp.getOperand(3));
+  Observer.changingInstr(MI);
+  getConditionOp(MI).setReg(ICmpDef);
+  Observer.changedInstr(MI);
+  ICmp.eraseFromParent();
+}
+
+class AMDGPURegBankCombinerHelperState {
+protected:
+  CombinerHelper &Helper;
+  AMDGPURegBankCombinerHelper &RegBankHelper;
+
+public:
+  AMDGPURegBankCombinerHelperState(CombinerHelper &Helper,
+                                   AMDGPURegBankCombinerHelper &RegBankHelper)
+      : Helper(Helper), RegBankHelper(RegBankHelper) {}
+};
 
 #define AMDGPUREGBANKCOMBINERHELPER_GENCOMBINERHELPER_DEPS
 #include "AMDGPUGenRegBankGICombiner.inc"
 #undef AMDGPUREGBANKCOMBINERHELPER_GENCOMBINERHELPER_DEPS
 
 namespace {
 #define AMDGPUREGBANKCOMBINERHELPER_GENCOMBINERHELPER_H
 #include "AMDGPUGenRegBankGICombiner.inc"
 #undef AMDGPUREGBANKCOMBINERHELPER_GENCOMBINERHELPER_H
 
 class AMDGPURegBankCombinerInfo final : public CombinerInfo {
   GISelKnownBits *KB;
   MachineDominatorTree *MDT;
 
 public:
   AMDGPUGenRegBankCombinerHelperRuleConfig GeneratedRuleCfg;
 
   AMDGPURegBankCombinerInfo(bool EnableOpt, bool OptSize, bool MinSize,
                                   const AMDGPULegalizerInfo *LI,
                                   GISelKnownBits *KB, MachineDominatorTree *MDT)
       : CombinerInfo(/*AllowIllegalOps*/ false, /*ShouldLegalizeIllegal*/ true,
                      /*LegalizerInfo*/ LI, EnableOpt, OptSize, MinSize),
         KB(KB), MDT(MDT) {
     if (!GeneratedRuleCfg.parseCommandLineOption())
       report_fatal_error("Invalid rule identifier");
   }
 
   bool combine(GISelChangeObserver &Observer, MachineInstr &MI,
                MachineIRBuilder &B) const override;
 };
 
 bool AMDGPURegBankCombinerInfo::combine(GISelChangeObserver &Observer,
                                               MachineInstr &MI,
                                               MachineIRBuilder &B) const {
   CombinerHelper Helper(Observer, B, KB, MDT);
-  AMDGPUGenRegBankCombinerHelper Generated(GeneratedRuleCfg);
+  AMDGPURegBankCombinerHelper RegBankHelper(B, Helper, Observer);
+  AMDGPUGenRegBankCombinerHelper Generated(GeneratedRuleCfg, Helper,
+                                           RegBankHelper);
 
-  if (Generated.tryCombineAll(Observer, MI, B, Helper))
+  if (Generated.tryCombineAll(Observer, MI, B))
     return true;
 
   return false;
 }
 
 #define AMDGPUREGBANKCOMBINERHELPER_GENCOMBINERHELPER_CPP
 #include "AMDGPUGenRegBankGICombiner.inc"
 #undef AMDGPUREGBANKCOMBINERHELPER_GENCOMBINERHELPER_CPP

llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp

extern "C" LLVM_EXTERNAL_VISIBILITY void LLVMInitializeAMDGPUTarget() {
   initializeAMDGPUArgumentUsageInfoPass(*PR);
   initializeAMDGPUAtomicOptimizerPass(*PR);
   initializeAMDGPULowerKernelArgumentsPass(*PR);
   initializeAMDGPULowerKernelAttributesPass(*PR);
   initializeAMDGPULowerIntrinsicsPass(*PR);
   initializeAMDGPUOpenCLEnqueuedBlockLoweringPass(*PR);
   initializeAMDGPUPostLegalizerCombinerPass(*PR);
   initializeAMDGPUPreLegalizerCombinerPass(*PR);
+  initializeAMDGPURegBankCombinerPass(*PR);
   initializeAMDGPUPromoteAllocaPass(*PR);
   initializeAMDGPUPromoteAllocaToVectorPass(*PR);
   initializeAMDGPUCodeGenPreparePass(*PR);
   initializeAMDGPULateCodeGenPreparePass(*PR);
   initializeAMDGPUPropagateAttributesEarlyPass(*PR);
   initializeAMDGPUPropagateAttributesLatePass(*PR);
   initializeAMDGPURewriteOutArgumentsPass(*PR);
   initializeAMDGPUUnifyMetadataPass(*PR);
public:
   void addMachineSSAOptimization() override;
   bool addILPOpts() override;
   bool addInstSelector() override;
   bool addIRTranslator() override;
   void addPreLegalizeMachineIR() override;
   bool addLegalizeMachineIR() override;
   void addPreRegBankSelect() override;
   bool addRegBankSelect() override;
+  void addPreGlobalInstructionSelect() override;
   bool addGlobalInstructionSelect() override;
   void addFastRegAlloc() override;
   void addOptimizedRegAlloc() override;
   void addPreRegAlloc() override;
   bool addPreRewrite() override;
   void addPostRegAlloc() override;
   void addPreSched2() override;
   void addPreEmitPass() override;
void GCNPassConfig::addPreRegBankSelect() {
   addPass(createAMDGPUPostLegalizeCombiner(IsOptNone));
 }
 
 bool GCNPassConfig::addRegBankSelect() {
   addPass(new RegBankSelect());
   return false;
 }
 
+void GCNPassConfig::addPreGlobalInstructionSelect() {
+  bool IsOptNone = getOptLevel() == CodeGenOpt::None;
+  addPass(createAMDGPURegBankCombiner(IsOptNone));
+}
+
 bool GCNPassConfig::addGlobalInstructionSelect() {
   addPass(new InstructionSelect());
   // TODO: Fix instruction selection to do the right thing for image
   // instructions with tfe or lwe in the first place, instead of running a
   // separate pass to fix them up?
   addPass(createSIAddIMGInitPass());
   return false;
 }

llvm/test/CodeGen/AMDGPU/GlobalISel/combine-move-uniform-icmp-with-one-use.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -mtriple=amdgcn -mcpu=gfx1010 -run-pass=amdgpu-regbank-combiner -verify-machineinstrs %s -o - | FileCheck %s
+
+---
+name: uniform_icmp_select
+legalized: true
+regBankSelected: true
+tracksRegLiveness: true
+body: |
+  bb.0:
+    liveins: $sgpr0, $sgpr1, $sgpr2, $vgpr0_vgpr1
+
+    ; CHECK-LABEL: name: uniform_icmp_select
+    ; CHECK: liveins: $sgpr0, $sgpr1, $sgpr2, $vgpr0_vgpr1
+    ; CHECK: [[COPY:%[0-9]+]]:sgpr(s32) = COPY $sgpr0
+    ; CHECK: [[COPY1:%[0-9]+]]:sgpr(s32) = COPY $sgpr1
+    ; CHECK: [[COPY2:%[0-9]+]]:sgpr(s32) = COPY $sgpr2
+    ; CHECK: [[COPY3:%[0-9]+]]:vgpr(p1) = COPY $vgpr0_vgpr1
+    ; CHECK: [[C:%[0-9]+]]:sgpr(s32) = G_CONSTANT i32 -1
+    ; CHECK: [[C1:%[0-9]+]]:sgpr(s32) = G_CONSTANT i32 268435455
+    ; CHECK: [[AND:%[0-9]+]]:sgpr(s32) = G_AND [[COPY2]], [[C1]]
+    ; CHECK: [[ICMP:%[0-9]+]]:sgpr(s32) = G_ICMP intpred(sgt), [[COPY]](s32), [[C]]
+    ; CHECK: [[SELECT:%[0-9]+]]:sgpr(s32) = G_SELECT [[ICMP]](s32), [[COPY1]], [[AND]]
+    ; CHECK: [[COPY4:%[0-9]+]]:vgpr(s32) = COPY [[SELECT]](s32)
+    ; CHECK: G_STORE [[COPY4]](s32), [[COPY3]](p1) :: (store 4, addrspace 1)
+    ; CHECK: S_ENDPGM 0
+    %0:sgpr(s32) = COPY $sgpr0
+    %1:sgpr(s32) = COPY $sgpr1
+    %2:sgpr(s32) = COPY $sgpr2
+    %3:vgpr(p1) = COPY $vgpr0_vgpr1
+    %7:sgpr(s32) = G_CONSTANT i32 -1
+    %12:sgpr(s32) = G_ICMP intpred(sgt), %0(s32), %7
+    %8:sgpr(s1) = G_TRUNC %12(s32)
+    %9:sgpr(s32) = G_CONSTANT i32 268435455
+    %10:sgpr(s32) = G_AND %2, %9
+    %13:sgpr(s32) = G_ZEXT %8(s1)
+    %11:sgpr(s32) = G_SELECT %13(s32), %1, %10
+    %14:vgpr(s32) = COPY %11(s32)
+    G_STORE %14(s32), %3(p1) :: (store 4, addrspace 1)
+    S_ENDPGM 0
+
+...
+
+---
+name: uniform_icmp_select_not_single_use
+legalized: true
+regBankSelected: true
+tracksRegLiveness: true
+body: |
+  bb.0:
+    liveins: $sgpr0, $sgpr1, $sgpr2, $sgpr3, $vgpr0_vgpr1, $vgpr2_vgpr3
+
+    ; CHECK-LABEL: name: uniform_icmp_select_not_single_use
+    ; CHECK: liveins: $sgpr0, $sgpr1, $sgpr2, $sgpr3, $vgpr0_vgpr1, $vgpr2_vgpr3
+    ; CHECK: [[COPY:%[0-9]+]]:sgpr(s32) = COPY $sgpr0
+    ; CHECK: [[COPY1:%[0-9]+]]:sgpr(s32) = COPY $sgpr1
+    ; CHECK: [[COPY2:%[0-9]+]]:sgpr(s32) = COPY $sgpr2
+    ; CHECK: [[COPY3:%[0-9]+]]:sgpr(s32) = COPY $sgpr3
+    ; CHECK: [[COPY4:%[0-9]+]]:vgpr(p1) = COPY $vgpr0_vgpr1
+    ; CHECK: [[COPY5:%[0-9]+]]:vgpr(p1) = COPY $vgpr2_vgpr3
+    ; CHECK: [[C:%[0-9]+]]:sgpr(s32) = G_CONSTANT i32 -1
+    ; CHECK: [[ICMP:%[0-9]+]]:sgpr(s32) = G_ICMP intpred(sgt), [[COPY]](s32), [[C]]
+    ; CHECK: [[TRUNC:%[0-9]+]]:sgpr(s1) = G_TRUNC [[ICMP]](s32)
+    ; CHECK: [[C1:%[0-9]+]]:sgpr(s32) = G_CONSTANT i32 268435455
+    ; CHECK: [[AND:%[0-9]+]]:sgpr(s32) = G_AND [[COPY2]], [[C1]]
+    ; CHECK: [[ZEXT:%[0-9]+]]:sgpr(s32) = G_ZEXT [[TRUNC]](s1)
+    ; CHECK: [[SELECT:%[0-9]+]]:sgpr(s32) = G_SELECT [[ZEXT]](s32), [[COPY1]], [[AND]]
+    ; CHECK: [[COPY6:%[0-9]+]]:vgpr(s32) = COPY [[SELECT]](s32)
+    ; CHECK: G_STORE [[COPY6]](s32), [[COPY4]](p1) :: (store 4, addrspace 1)
+    ; CHECK: [[ZEXT1:%[0-9]+]]:sgpr(s32) = G_ZEXT [[TRUNC]](s1)
+    ; CHECK: [[SELECT1:%[0-9]+]]:sgpr(s32) = G_SELECT [[ZEXT1]](s32), [[AND]], [[COPY3]]
+    ; CHECK: [[COPY7:%[0-9]+]]:vgpr(s32) = COPY [[SELECT1]](s32)
+    ; CHECK: G_STORE [[COPY7]](s32), [[COPY5]](p1) :: (store 4, addrspace 1)
+    ; CHECK: S_ENDPGM 0
+    %0:sgpr(s32) = COPY $sgpr0
+    %1:sgpr(s32) = COPY $sgpr1
+    %2:sgpr(s32) = COPY $sgpr2
+    %3:sgpr(s32) = COPY $sgpr3
+    %4:vgpr(p1) = COPY $vgpr0_vgpr1
+    %5:vgpr(p1) = COPY $vgpr2_vgpr3
+    %11:sgpr(s32) = G_CONSTANT i32 -1
+    %17:sgpr(s32) = G_ICMP intpred(sgt), %0(s32), %11
+    %12:sgpr(s1) = G_TRUNC %17(s32)
+    %13:sgpr(s32) = G_CONSTANT i32 268435455
+    %14:sgpr(s32) = G_AND %2, %13
+    %18:sgpr(s32) = G_ZEXT %12(s1)
+    %15:sgpr(s32) = G_SELECT %18(s32), %1, %14
+    %19:vgpr(s32) = COPY %15(s32)
+    G_STORE %19(s32), %4(p1) :: (store 4, addrspace 1)
+    %20:sgpr(s32) = G_ZEXT %12(s1)
+    %16:sgpr(s32) = G_SELECT %20(s32), %14, %3
+    %21:vgpr(s32) = COPY %16(s32)
+    G_STORE %21(s32), %5(p1) :: (store 4, addrspace 1)
+    S_ENDPGM 0
+
+...
+
+---
+name: uniform_icmp_brcond
+legalized: true
+regBankSelected: true
+tracksRegLiveness: true
+body: |
+  ; CHECK-LABEL: name: uniform_icmp_brcond
+  ; CHECK: bb.0:
+  ; CHECK:   successors: %bb.1(0x50000000), %bb.2(0x30000000)
+  ; CHECK:   liveins: $sgpr0, $sgpr1, $vgpr0_vgpr1, $vgpr2_vgpr3
+  ; CHECK:   [[COPY:%[0-9]+]]:sgpr(s32) = COPY $sgpr0
+  ; CHECK:   [[COPY1:%[0-9]+]]:sgpr(s32) = COPY $sgpr1
+  ; CHECK:   [[COPY2:%[0-9]+]]:vgpr(p1) = COPY $vgpr0_vgpr1
+  ; CHECK:   [[COPY3:%[0-9]+]]:vgpr(p1) = COPY $vgpr2_vgpr3
+  ; CHECK:   [[C:%[0-9]+]]:sgpr(s32) = G_CONSTANT i32 -1
+  ; CHECK:   [[C1:%[0-9]+]]:sgpr(s32) = G_CONSTANT i32 268435455
+  ; CHECK:   [[AND:%[0-9]+]]:sgpr(s32) = G_AND [[COPY1]], [[C1]]
+  ; CHECK:   [[ICMP:%[0-9]+]]:sgpr(s32) = G_ICMP intpred(sle), [[COPY]](s32), [[C]]
+  ; CHECK:   G_BRCOND [[ICMP]](s32), %bb.1
+  ; CHECK:   G_BR %bb.2
+  ; CHECK: bb.1:
+  ; CHECK:   successors: %bb.2(0x80000000)
+  ; CHECK:   [[COPY4:%[0-9]+]]:vgpr(s32) = COPY [[AND]](s32)
+  ; CHECK:   G_STORE [[COPY4]](s32), [[COPY2]](p1) :: (store 4, addrspace 1)
+  ; CHECK: bb.2:
+  ; CHECK:   [[COPY5:%[0-9]+]]:vgpr(s32) = COPY [[AND]](s32)
+  ; CHECK:   G_STORE [[COPY5]](s32), [[COPY3]](p1) :: (store 4, addrspace 1)
+  ; CHECK:   S_ENDPGM 0
+  bb.0:
+    successors: %bb.1(0x50000000), %bb.2(0x30000000)
+    liveins: $sgpr0, $sgpr1, $vgpr0_vgpr1, $vgpr2_vgpr3
+
+    %0:sgpr(s32) = COPY $sgpr0
+    %1:sgpr(s32) = COPY $sgpr1
+    %2:vgpr(p1) = COPY $vgpr0_vgpr1
+    %3:vgpr(p1) = COPY $vgpr2_vgpr3
+    %9:sgpr(s32) = G_CONSTANT i32 -1
+    %15:sgpr(s32) = G_ICMP intpred(sle), %0(s32), %9
+    %10:sgpr(s1) = G_TRUNC %15(s32)
+    %11:sgpr(s32) = G_CONSTANT i32 268435455
+    %12:sgpr(s32) = G_AND %1, %11
+    %16:sgpr(s32) = G_ZEXT %10(s1)
+    G_BRCOND %16(s32), %bb.1
+    G_BR %bb.2
+
+  bb.1:
+    %17:vgpr(s32) = COPY %12(s32)
+    G_STORE %17(s32), %2(p1) :: (store 4, addrspace 1)
+
+  bb.2:
+    %18:vgpr(s32) = COPY %12(s32)
+    G_STORE %18(s32), %3(p1) :: (store 4, addrspace 1)
+    S_ENDPGM 0
+
+...
+
+---
+name: uniform_icmp_brcond_not_single_use
+legalized: true
+regBankSelected: true
+tracksRegLiveness: true
+body: |
+  ; CHECK-LABEL: name: uniform_icmp_brcond_not_single_use
+  ; CHECK: bb.0:
+  ; CHECK:   successors: %bb.1(0x50000000), %bb.2(0x30000000)
+  ; CHECK:   liveins: $sgpr0, $sgpr1, $sgpr2, $vgpr0_vgpr1, $vgpr2_vgpr3
+  ; CHECK:   [[COPY:%[0-9]+]]:sgpr(s32) = COPY $sgpr0
+  ; CHECK:   [[COPY1:%[0-9]+]]:sgpr(s32) = COPY $sgpr1
+  ; CHECK:   [[COPY2:%[0-9]+]]:sgpr(s32) = COPY $sgpr2
+  ; CHECK:   [[COPY3:%[0-9]+]]:vgpr(p1) = COPY $vgpr0_vgpr1
+  ; CHECK:   [[COPY4:%[0-9]+]]:vgpr(p1) = COPY $vgpr2_vgpr3
+  ; CHECK:   [[C:%[0-9]+]]:sgpr(s32) = G_CONSTANT i32 -1
+  ; CHECK:   [[ICMP:%[0-9]+]]:sgpr(s32) = G_ICMP intpred(sgt), [[COPY]](s32), [[C]]
+  ; CHECK:   [[TRUNC:%[0-9]+]]:sgpr(s1) = G_TRUNC [[ICMP]](s32)
+  ; CHECK:   [[C1:%[0-9]+]]:sgpr(s32) = G_CONSTANT i32 268435455
+  ; CHECK:   [[AND:%[0-9]+]]:sgpr(s32) = G_AND [[COPY2]], [[C1]]
+  ; CHECK:   [[C2:%[0-9]+]]:sgpr(s1) = G_CONSTANT i1 true
+  ; CHECK:   [[ANYEXT:%[0-9]+]]:sgpr(s32) = G_ANYEXT [[TRUNC]](s1)
+  ; CHECK:   [[ANYEXT1:%[0-9]+]]:sgpr(s32) = G_ANYEXT [[C2]](s1)
+  ; CHECK:   [[XOR:%[0-9]+]]:sgpr(s32) = G_XOR [[ANYEXT]], [[ANYEXT1]]
+  ; CHECK:   [[TRUNC1:%[0-9]+]]:sgpr(s1) = G_TRUNC [[XOR]](s32)
+  ; CHECK:   [[ZEXT:%[0-9]+]]:sgpr(s32) = G_ZEXT [[TRUNC1]](s1)
+  ; CHECK:   G_BRCOND [[ZEXT]](s32), %bb.2
+  ; CHECK:   G_BR %bb.1
+  ; CHECK: bb.1:
+  ; CHECK:   successors: %bb.2(0x80000000)
+  ; CHECK:   [[ZEXT1:%[0-9]+]]:sgpr(s32) = G_ZEXT [[TRUNC]](s1)
+  ; CHECK:   [[SELECT:%[0-9]+]]:sgpr(s32) = G_SELECT [[ZEXT1]](s32), [[COPY1]], [[AND]]
+  ; CHECK:   [[COPY5:%[0-9]+]]:vgpr(s32) = COPY [[SELECT]](s32)
+  ; CHECK:   G_STORE [[COPY5]](s32), [[COPY3]](p1) :: (store 4, addrspace 1)
+  ; CHECK: bb.2:
+  ; CHECK:   [[COPY6:%[0-9]+]]:vgpr(s32) = COPY [[AND]](s32)
+  ; CHECK:   G_STORE [[COPY6]](s32), [[COPY4]](p1) :: (store 4, addrspace 1)
+  ; CHECK:   S_ENDPGM 0
+  bb.0:
+    successors: %bb.1(0x50000000), %bb.2(0x30000000)
+    liveins: $sgpr0, $sgpr1, $sgpr2, $vgpr0_vgpr1, $vgpr2_vgpr3
+
+    %0:sgpr(s32) = COPY $sgpr0
+    %1:sgpr(s32) = COPY $sgpr1
+    %2:sgpr(s32) = COPY $sgpr2
+    %3:vgpr(p1) = COPY $vgpr0_vgpr1
+    %4:vgpr(p1) = COPY $vgpr2_vgpr3
+    %10:sgpr(s32) = G_CONSTANT i32 -1
+    %17:sgpr(s32) = G_ICMP intpred(sgt), %0(s32), %10
+    %11:sgpr(s1) = G_TRUNC %17(s32)
+    %12:sgpr(s32) = G_CONSTANT i32 268435455
+    %13:sgpr(s32) = G_AND %2, %12
+    %15:sgpr(s1) = G_CONSTANT i1 true
+    %18:sgpr(s32) = G_ANYEXT %11(s1)
+    %19:sgpr(s32) = G_ANYEXT %15(s1)
+    %20:sgpr(s32) = G_XOR %18, %19
+    %16:sgpr(s1) = G_TRUNC %20(s32)
+    %21:sgpr(s32) = G_ZEXT %16(s1)
+    G_BRCOND %21(s32), %bb.2
+    G_BR %bb.1
+
+  bb.1:
+    %22:sgpr(s32) = G_ZEXT %11(s1)
+    %14:sgpr(s32) = G_SELECT %22(s32), %1, %13
+    %23:vgpr(s32) = COPY %14(s32)
+    G_STORE %23(s32), %3(p1) :: (store 4, addrspace 1)
+
+  bb.2:
+    %24:vgpr(s32) = COPY %13(s32)
+    G_STORE %24(s32), %4(p1) :: (store 4, addrspace 1)
+    S_ENDPGM 0
+
+...

llvm/test/CodeGen/AMDGPU/GlobalISel/divergent-control-flow.ll

 ; CHECK-NEXT:    s_getpc_b64 s[4:5]
 ; CHECK-NEXT:    s_add_u32 s4, s4, external_constant@gotpcrel32@lo+4
 ; CHECK-NEXT:    s_addc_u32 s5, s5, external_constant@gotpcrel32@hi+12
 ; CHECK-NEXT:    s_load_dwordx2 s[4:5], s[4:5], 0x0
 ; CHECK-NEXT:    s_waitcnt lgkmcnt(0)
 ; CHECK-NEXT:    s_load_dword s4, s[4:5], 0x0
 ; CHECK-NEXT:    s_waitcnt lgkmcnt(0)
 ; CHECK-NEXT:    s_cmp_lg_u32 s4, 0
-; CHECK-NEXT:    s_cselect_b32 s4, 1, 0
-; CHECK-NEXT:    s_and_b32 s4, s4, 1
-; CHECK-NEXT:    s_cmp_lg_u32 s4, 0
 ; CHECK-NEXT:    s_cbranch_scc1 BB4_4
 ; CHECK-NEXT:  ; %bb.1: ; %bb2
 ; CHECK-NEXT:    s_getpc_b64 s[6:7]
 ; CHECK-NEXT:    s_add_u32 s6, s6, const.ptr@gotpcrel32@lo+4
 ; CHECK-NEXT:    s_addc_u32 s7, s7, const.ptr@gotpcrel32@hi+12
 ; CHECK-NEXT:    s_load_dwordx2 s[6:7], s[6:7], 0x0
 ; CHECK-NEXT:    v_mov_b32_e32 v0, 0
 ; CHECK-NEXT:    s_mov_b32 s4, -1
 ; CHECK-NEXT:    s_waitcnt lgkmcnt(0)
 ; CHECK-NEXT:    s_load_dwordx2 s[6:7], s[6:7], 0x0
 ; CHECK-NEXT:    s_waitcnt lgkmcnt(0)
 ; CHECK-NEXT:    global_load_dword v0, v0, s[6:7]
 ; CHECK-NEXT:    s_waitcnt vmcnt(0)
 ; CHECK-NEXT:    v_cmp_gt_f32_e32 vcc, 1.0, v0
 ; CHECK-NEXT:    s_cbranch_vccnz BB4_3
 ; CHECK-NEXT:  ; %bb.2: ; %bb7
 ; CHECK-NEXT:    s_mov_b32 s4, 0
 ; CHECK-NEXT:  BB4_3: ; %bb8
 ; CHECK-NEXT:    s_cmp_lg_u32 s4, 0
-; CHECK-NEXT:    s_cselect_b32 s4, 1, 0
-; CHECK-NEXT:    s_and_b32 s4, s4, 1
-; CHECK-NEXT:    s_cmp_lg_u32 s4, 0
 ; CHECK-NEXT:    s_cbranch_scc0 BB4_5
 ; CHECK-NEXT:  BB4_4: ; %bb12
 ; CHECK-NEXT:    s_setpc_b64 s[30:31]
 ; CHECK-NEXT:  BB4_5: ; %bb11
 ; CHECK-NEXT:    v_mov_b32_e32 v0, 4.0
 ; CHECK-NEXT:    buffer_store_dword v0, v0, s[0:3], 0 offen
 ; CHECK-NEXT:    s_waitcnt vmcnt(0)
 ; CHECK-NEXT:    s_setpc_b64 s[30:31]

llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.end.cf.i32.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -global-isel -mtriple=amdgcn--amdhsa -mcpu=gfx1010 -mattr=+wavefrontsize32,-wavefrontsize64 -verify-machineinstrs < %s | FileCheck -check-prefix=GCN %s
 
 define amdgpu_kernel void @test_wave32(i32 %arg0, [8 x i32], i32 %saved) {
 ; GCN-LABEL: test_wave32:
 ; GCN:       ; %bb.0: ; %entry
 ; GCN-NEXT:    s_clause 0x1
 ; GCN-NEXT:    s_load_dword s1, s[4:5], 0x0
 ; GCN-NEXT:    s_load_dword s0, s[4:5], 0x24
 ; GCN-NEXT:    s_waitcnt lgkmcnt(0)
 ; GCN-NEXT:    s_cmp_lg_u32 s1, 0
-; GCN-NEXT:    s_cselect_b32 s1, 1, 0
-; GCN-NEXT:    s_and_b32 s1, s1, 1
-; GCN-NEXT:    s_cmp_lg_u32 s1, 0
 ; GCN-NEXT:    s_cbranch_scc1 BB0_2
 ; GCN-NEXT:  ; %bb.1: ; %mid
 ; GCN-NEXT:    v_mov_b32_e32 v0, 0
 ; GCN-NEXT:    global_store_dword v[0:1], v0, off
 ; GCN-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GCN-NEXT:  BB0_2: ; %bb
 ; GCN-NEXT:    s_waitcnt_depctr 0xffe3
 ; GCN-NEXT:    s_or_b32 exec_lo, exec_lo, s0

llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.end.cf.i64.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -global-isel -mtriple=amdgcn--amdhsa -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -check-prefix=GCN %s
 
 define amdgpu_kernel void @test_wave64(i32 %arg0, i64 %saved) {
 ; GCN-LABEL: test_wave64:
 ; GCN:       ; %bb.0: ; %entry
 ; GCN-NEXT:    s_load_dword s2, s[4:5], 0x0
 ; GCN-NEXT:    s_load_dwordx2 s[0:1], s[4:5], 0x8
 ; GCN-NEXT:    s_waitcnt lgkmcnt(0)
 ; GCN-NEXT:    s_cmp_lg_u32 s2, 0
-; GCN-NEXT:    s_cselect_b32 s2, 1, 0
-; GCN-NEXT:    s_and_b32 s2, s2, 1
-; GCN-NEXT:    s_cmp_lg_u32 s2, 0
 ; GCN-NEXT:    s_cbranch_scc1 BB0_2
 ; GCN-NEXT:  ; %bb.1: ; %mid
 ; GCN-NEXT:    v_mov_b32_e32 v0, 0
 ; GCN-NEXT:    global_store_dword v[0:1], v0, off
 ; GCN-NEXT:    s_waitcnt vmcnt(0)
 ; GCN-NEXT:  BB0_2: ; %bb
 ; GCN-NEXT:    s_or_b64 exec, exec, s[0:1]
 ; GCN-NEXT:    v_mov_b32_e32 v0, 0

llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.is.private.ll

 define amdgpu_kernel void @is_private_sgpr(i8* %ptr) {
 ; CI-LABEL: is_private_sgpr:
 ; CI:       ; %bb.0:
 ; CI-NEXT:    s_load_dwordx2 s[0:1], s[6:7], 0x0
 ; CI-NEXT:    s_waitcnt lgkmcnt(0)
 ; CI-NEXT:    s_load_dword s0, s[4:5], 0x11
 ; CI-NEXT:    s_waitcnt lgkmcnt(0)
 ; CI-NEXT:    s_cmp_lg_u32 s1, s0
-; CI-NEXT:    s_cselect_b32 s0, 1, 0
-; CI-NEXT:    s_and_b32 s0, s0, 1
-; CI-NEXT:    s_cmp_lg_u32 s0, 0
 ; CI-NEXT:    s_cbranch_scc1 BB1_2
 ; CI-NEXT:  ; %bb.1: ; %bb0
 ; CI-NEXT:    v_mov_b32_e32 v0, 0
 ; CI-NEXT:    flat_store_dword v[0:1], v0
 ; CI-NEXT:    s_waitcnt vmcnt(0)
 ; CI-NEXT:  BB1_2: ; %bb1
 ; CI-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: is_private_sgpr:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[6:7], 0x0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_getreg_b32 s0, hwreg(HW_REG_SH_MEM_BASES, 0, 16)
 ; GFX9-NEXT:    s_lshl_b32 s0, s0, 16
 ; GFX9-NEXT:    s_cmp_lg_u32 s1, s0
-; GFX9-NEXT:    s_cselect_b32 s0, 1, 0
-; GFX9-NEXT:    s_and_b32 s0, s0, 1
-; GFX9-NEXT:    s_cmp_lg_u32 s0, 0
 ; GFX9-NEXT:    s_cbranch_scc1 BB1_2
 ; GFX9-NEXT:  ; %bb.1: ; %bb0
 ; GFX9-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX9-NEXT:    global_store_dword v[0:1], v0, off
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:  BB1_2: ; %bb1
 ; GFX9-NEXT:    s_endpgm
   %val = call i1 @llvm.amdgcn.is.private(i8* %ptr)

llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.is.shared.ll

 define amdgpu_kernel void @is_local_sgpr(i8* %ptr) {
 ; CI-LABEL: is_local_sgpr:
 ; CI:       ; %bb.0:
 ; CI-NEXT:    s_load_dwordx2 s[0:1], s[6:7], 0x0
 ; CI-NEXT:    s_waitcnt lgkmcnt(0)
 ; CI-NEXT:    s_load_dword s0, s[4:5], 0x10
 ; CI-NEXT:    s_waitcnt lgkmcnt(0)
 ; CI-NEXT:    s_cmp_lg_u32 s1, s0
-; CI-NEXT:    s_cselect_b32 s0, 1, 0
-; CI-NEXT:    s_and_b32 s0, s0, 1
-; CI-NEXT:    s_cmp_lg_u32 s0, 0
 ; CI-NEXT:    s_cbranch_scc1 BB1_2
 ; CI-NEXT:  ; %bb.1: ; %bb0
 ; CI-NEXT:    v_mov_b32_e32 v0, 0
 ; CI-NEXT:    flat_store_dword v[0:1], v0
 ; CI-NEXT:    s_waitcnt vmcnt(0)
 ; CI-NEXT:  BB1_2: ; %bb1
 ; CI-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: is_local_sgpr:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[6:7], 0x0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_getreg_b32 s0, hwreg(HW_REG_SH_MEM_BASES, 16, 16)
 ; GFX9-NEXT:    s_lshl_b32 s0, s0, 16
 ; GFX9-NEXT:    s_cmp_lg_u32 s1, s0
-; GFX9-NEXT:    s_cselect_b32 s0, 1, 0
-; GFX9-NEXT:    s_and_b32 s0, s0, 1
-; GFX9-NEXT:    s_cmp_lg_u32 s0, 0
 ; GFX9-NEXT:    s_cbranch_scc1 BB1_2
 ; GFX9-NEXT:  ; %bb.1: ; %bb0
 ; GFX9-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX9-NEXT:    global_store_dword v[0:1], v0, off
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:  BB1_2: ; %bb1
 ; GFX9-NEXT:    s_endpgm
   %val = call i1 @llvm.amdgcn.is.shared(i8* %ptr)

llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.set.inactive.ll

; GCN-NEXT:    s_endpgm
   %tmp = call i64 @llvm.amdgcn.set.inactive.i64(i64 %in, i64 0) #0
   store i64 %tmp, i64 addrspace(1)* %out
   ret void
 }
 
 define amdgpu_kernel void @set_inactive_scc(i32 addrspace(1)* %out, i32 %in, <4 x i32> inreg %desc) {
 ; GCN-LABEL: set_inactive_scc:
 ; GCN:       ; %bb.0:
+; GCN-NEXT:    s_load_dword s2, s[0:1], 0x2c
 ; GCN-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x34
 ; GCN-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NEXT:    s_buffer_load_dword s2, s[4:7], 0x0
-; GCN-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x24
-; GCN-NEXT:    s_load_dword s0, s[0:1], 0x2c
-; GCN-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NEXT:    s_cmp_lg_u32 s2, 56
-; GCN-NEXT:    v_mov_b32_e32 v0, s0
-; GCN-NEXT:    s_cselect_b32 s0, 1, 0
+; GCN-NEXT:    v_mov_b32_e32 v0, s2
+; GCN-NEXT:    s_buffer_load_dword s3, s[4:7], 0x0
 ; GCN-NEXT:    s_not_b64 exec, exec
 ; GCN-NEXT:    v_mov_b32_e32 v0, 42
 ; GCN-NEXT:    s_not_b64 exec, exec
-; GCN-NEXT:    s_and_b32 s0, s0, 1
-; GCN-NEXT:    s_cmp_lg_u32 s0, 0
+; GCN-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GCN-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN-NEXT:    s_cmp_lg_u32 s3, 56
 ; GCN-NEXT:    s_cbranch_scc0 BB2_2
 ; GCN-NEXT:  ; %bb.1: ; %.one
 ; GCN-NEXT:    v_add_u32_e32 v1, vcc, 1, v0
-; GCN-NEXT:    s_mov_b32 s6, -1
-; GCN-NEXT:    s_mov_b32 s7, 0xf000
-; GCN-NEXT:    s_mov_b32 s0, 0
-; GCN-NEXT:    buffer_store_dword v1, off, s[4:7], 0
+; GCN-NEXT:    s_mov_b32 s2, -1
+; GCN-NEXT:    s_mov_b32 s3, 0xf000
+; GCN-NEXT:    s_mov_b32 s4, 0
+; GCN-NEXT:    buffer_store_dword v1, off, s[0:3], 0
 ; GCN-NEXT:    s_branch BB2_3
 ; GCN-NEXT:  BB2_2:
-; GCN-NEXT:    s_mov_b32 s0, -1
+; GCN-NEXT:    s_mov_b32 s4, -1
 ; GCN-NEXT:  BB2_3: ; %Flow
-; GCN-NEXT:    s_xor_b32 s0, s0, -1
-; GCN-NEXT:    s_and_b32 s0, s0, 1
-; GCN-NEXT:    s_cmp_lg_u32 s0, 0
+; GCN-NEXT:    s_xor_b32 s2, s4, -1
+; GCN-NEXT:    s_and_b32 s2, s2, 1
+; GCN-NEXT:    s_cmp_lg_u32 s2, 0
 ; GCN-NEXT:    s_cbranch_scc1 BB2_5
 ; GCN-NEXT:  ; %bb.4: ; %.zero
-; GCN-NEXT:    s_mov_b32 s6, -1
-; GCN-NEXT:    s_mov_b32 s7, 0xf000
-; GCN-NEXT:    buffer_store_dword v0, off, s[4:7], 0
+; GCN-NEXT:    s_mov_b32 s2, -1
+; GCN-NEXT:    s_mov_b32 s3, 0xf000
+; GCN-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GCN-NEXT:  BB2_5: ; %.exit
 ; GCN-NEXT:    s_endpgm
   %val = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> %desc, i32 0, i32 0)
   %cmp = icmp eq i32 %val, 56
   %tmp = call i32 @llvm.amdgcn.set.inactive.i32(i32 %in, i32 42) #0
   br i1 %cmp, label %.zero, label %.one
 
 .zero:

llvm/test/CodeGen/AMDGPU/GlobalISel/move-uniform-icmp-with-one-use.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -global-isel -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs < %s | FileCheck --check-prefix=GFX10 %s
+
+define amdgpu_cs void @uniform_icmp_select(i32 inreg %cmp_val, i32 inreg %src1, i32 inreg %src2, i32 addrspace(1)* %out_ptr) {
+; GFX10-LABEL: uniform_icmp_select:
+; GFX10:       ; %bb.0:
+; GFX10-NEXT:    s_and_b32 s2, s2, 0xfffffff
+; GFX10-NEXT:    s_cmp_gt_i32 s0, -1
+; GFX10-NEXT:    s_cselect_b32 s0, s1, s2
+; GFX10-NEXT:    v_mov_b32_e32 v2, s0
+; GFX10-NEXT:    global_store_dword v[0:1], v2, off
+; GFX10-NEXT:    s_endpgm
+  %cmp = icmp sgt i32 %cmp_val, -1
+  %maskedsrc2 = and i32 %src2, 268435455
+  %result = select i1 %cmp, i32 %src1, i32 %maskedsrc2
+  store i32 %result, i32 addrspace(1)* %out_ptr
+  ret void
+}
+
+define amdgpu_cs void @uniform_icmp_select_not_single_use(i32 inreg %cmp_val, i32 inreg %src1, i32 inreg %src2, i32 inreg %src3, i32 addrspace(1)* %out_ptr1, i32 addrspace(1)* %out_ptr2) {
+; GFX10-LABEL: uniform_icmp_select_not_single_use:
+; GFX10:       ; %bb.0:
+; GFX10-NEXT:    s_cmp_gt_i32 s0, -1
+; GFX10-NEXT:    s_cselect_b32 s0, 1, 0
+; GFX10-NEXT:    s_and_b32 s2, s2, 0xfffffff
+; GFX10-NEXT:    s_and_b32 s0, s0, 1
+; GFX10-NEXT:    s_cmp_lg_u32 s0, 0
+; GFX10-NEXT:    s_cselect_b32 s1, s1, s2
+; GFX10-NEXT:    s_cselect_b32 s0, s2, s3
+; GFX10-NEXT:    v_mov_b32_e32 v7, s1
+; GFX10-NEXT:    v_mov_b32_e32 v5, s0
+; GFX10-NEXT:    global_store_dword v[0:1], v7, off
+; GFX10-NEXT:    global_store_dword v[2:3], v5, off
+; GFX10-NEXT:    s_endpgm
+  %cmp = icmp sgt i32 %cmp_val, -1
+  %maskedsrc2 = and i32 %src2, 268435455
+  %result1 = select i1 %cmp, i32 %src1, i32 %maskedsrc2
+  store i32 %result1, i32 addrspace(1)* %out_ptr1
+  %result2 = select i1 %cmp, i32 %maskedsrc2, i32 %src3
+  store i32 %result2, i32 addrspace(1)* %out_ptr2
+  ret void
+}
+
+define amdgpu_cs void @uniform_icmp_brcond(i32 inreg %cmp_val, i32 inreg %src, i32 addrspace(1)* %out_ptr1, i32 addrspace(1)* %out_ptr2) {
+; GFX10-LABEL: uniform_icmp_brcond:
+; GFX10:       ; %bb.0:
+; GFX10-NEXT:    s_and_b32 s1, s1, 0xfffffff
+; GFX10-NEXT:    s_cmp_le_i32 s0, -1
+; GFX10-NEXT:    s_cbranch_scc1 BB2_2
+; GFX10-NEXT:  ; %bb.1: ; %bb0
+; GFX10-NEXT:    v_mov_b32_e32 v7, s1
+; GFX10-NEXT:    global_store_dword v[0:1], v7, off
+; GFX10-NEXT:  BB2_2: ; %bb1
+; GFX10-NEXT:    v_mov_b32_e32 v0, s1
+; GFX10-NEXT:    global_store_dword v[2:3], v0, off
+; GFX10-NEXT:    s_endpgm
+  %cmp = icmp sgt i32 %cmp_val, -1
+  %maskedsrc = and i32 %src, 268435455
+  br i1 %cmp, label %bb0, label %bb1
+bb0:
+  store i32 %maskedsrc, i32 addrspace(1)* %out_ptr1
+  br label %bb1
+bb1:
+  store i32 %maskedsrc, i32 addrspace(1)* %out_ptr2
+  ret void
+}
+
+define amdgpu_cs void @uniform_icmp_brcond_not_single_use(i32 inreg %cmp_val, i32 inreg %src1, i32 inreg %src2, i32 addrspace(1)* %out_ptr1, i32 addrspace(1)* %out_ptr2) {
+; GFX10-LABEL: uniform_icmp_brcond_not_single_use:
+; GFX10:       ; %bb.0:
+; GFX10-NEXT:    s_cmp_gt_i32 s0, -1
+; GFX10-NEXT:    s_cselect_b32 s3, 1, 0
+; GFX10-NEXT:    s_xor_b32 s0, s3, -1
+; GFX10-NEXT:    s_and_b32 s4, s0, 1
+; GFX10-NEXT:    s_and_b32 s0, s2, 0xfffffff
+; GFX10-NEXT:    s_cmp_lg_u32 s4, 0
+; GFX10-NEXT:    s_cbranch_scc1 BB3_2
+; GFX10-NEXT:  ; %bb.1: ; %bb0
+; GFX10-NEXT:    s_and_b32 s2, s3, 1
+; GFX10-NEXT:    s_cmp_lg_u32 s2, 0
+; GFX10-NEXT:    s_cselect_b32 s1, s1, s0
+; GFX10-NEXT:    v_mov_b32_e32 v7, s1
+; GFX10-NEXT:    global_store_dword v[0:1], v7, off
+; GFX10-NEXT:  BB3_2: ; %bb1
+; GFX10-NEXT:    v_mov_b32_e32 v0, s0
+; GFX10-NEXT:    global_store_dword v[2:3], v0, off
+; GFX10-NEXT:    s_endpgm
+  %cmp = icmp sgt i32 %cmp_val, -1
+  %maskedsrc2 = and i32 %src2, 268435455
+  br i1 %cmp, label %bb0, label %bb1
+bb0:
+  %result1 = select i1 %cmp, i32 %src1, i32 %maskedsrc2
+  store i32 %result1, i32 addrspace(1)* %out_ptr1
+  br label %bb1
+bb1:
+  store i32 %maskedsrc2, i32 addrspace(1)* %out_ptr2
+  ret void
+}

llvm/test/CodeGen/AMDGPU/GlobalISel/non-entry-alloca.ll

 ; GCN-NEXT:    s_add_u32 flat_scratch_lo, s6, s9
 ; GCN-NEXT:    s_load_dword s6, s[4:5], 0x8
 ; GCN-NEXT:    s_addc_u32 flat_scratch_hi, s7, 0
 ; GCN-NEXT:    s_add_u32 s0, s0, s9
 ; GCN-NEXT:    s_addc_u32 s1, s1, 0
 ; GCN-NEXT:    s_movk_i32 s32, 0x400
 ; GCN-NEXT:    s_waitcnt lgkmcnt(0)
 ; GCN-NEXT:    s_cmp_lg_u32 s6, 0
-; GCN-NEXT:    s_cselect_b32 s6, 1, 0
-; GCN-NEXT:    s_and_b32 s6, s6, 1
-; GCN-NEXT:    s_cmp_lg_u32 s6, 0
 ; GCN-NEXT:    s_mov_b32 s33, 0
 ; GCN-NEXT:    s_cbranch_scc1 BB0_3
 ; GCN-NEXT:  ; %bb.1: ; %bb.0
 ; GCN-NEXT:    s_load_dword s6, s[4:5], 0xc
 ; GCN-NEXT:    s_waitcnt lgkmcnt(0)
 ; GCN-NEXT:    s_cmp_lg_u32 s6, 0
-; GCN-NEXT:    s_cselect_b32 s6, 1, 0
-; GCN-NEXT:    s_and_b32 s6, s6, 1
-; GCN-NEXT:    s_cmp_lg_u32 s6, 0
 ; GCN-NEXT:    s_cbranch_scc1 BB0_3
 ; GCN-NEXT:  ; %bb.2: ; %bb.1
 ; GCN-NEXT:    s_load_dwordx2 s[6:7], s[4:5], 0x0
 ; GCN-NEXT:    s_load_dword s8, s[4:5], 0x10
 ; GCN-NEXT:    s_add_u32 s4, s32, 0x1000
 ; GCN-NEXT:    s_add_u32 s5, s4, 4
 ; GCN-NEXT:    v_mov_b32_e32 v3, s5
 ; GCN-NEXT:    v_mov_b32_e32 v1, 0

 ; GCN-NEXT:    s_add_u32 flat_scratch_lo, s6, s9
 ; GCN-NEXT:    s_load_dword s6, s[4:5], 0x8
 ; GCN-NEXT:    s_addc_u32 flat_scratch_hi, s7, 0
 ; GCN-NEXT:    s_add_u32 s0, s0, s9
 ; GCN-NEXT:    s_addc_u32 s1, s1, 0
 ; GCN-NEXT:    s_movk_i32 s32, 0x1000
 ; GCN-NEXT:    s_waitcnt lgkmcnt(0)
 ; GCN-NEXT:    s_cmp_lg_u32 s6, 0
-; GCN-NEXT:    s_cselect_b32 s6, 1, 0
-; GCN-NEXT:    s_and_b32 s6, s6, 1
-; GCN-NEXT:    s_cmp_lg_u32 s6, 0
 ; GCN-NEXT:    s_mov_b32 s33, 0
 ; GCN-NEXT:    s_cbranch_scc1 BB1_2
 ; GCN-NEXT:  ; %bb.1: ; %bb.0
 ; GCN-NEXT:    s_load_dwordx2 s[6:7], s[4:5], 0x0
 ; GCN-NEXT:    s_load_dword s8, s[4:5], 0xc
 ; GCN-NEXT:    s_add_u32 s4, s32, 0x1000
 ; GCN-NEXT:    s_and_b32 s4, s4, 0xfffff000
 ; GCN-NEXT:    s_add_u32 s5, s4, 4

llvm/test/CodeGen/AMDGPU/GlobalISel/saddsat.ll

 ; GFX6-NEXT:    v_mov_b32_e32 v0, s2
 ; GFX6-NEXT:    s_addc_u32 s11, s3, s7
 ; GFX6-NEXT:    v_mov_b32_e32 v1, s3
 ; GFX6-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
 ; GFX6-NEXT:    v_cmp_lt_i64_e32 vcc, s[10:11], v[0:1]
 ; GFX6-NEXT:    v_cmp_lt_u64_e64 s[0:1], s[4:5], 0
 ; GFX6-NEXT:    v_cndmask_b32_e64 v3, 0, 1, vcc
 ; GFX6-NEXT:    v_cmp_eq_u64_e32 vcc, s[10:11], v[0:1]
-; GFX6-NEXT:    s_movk_i32 s2, 0x7f
+; GFX6-NEXT:    s_movk_i32 s12, 0x7f
 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, v3, v2, vcc
 ; GFX6-NEXT:    v_cndmask_b32_e64 v1, 0, 1, s[0:1]
 ; GFX6-NEXT:    v_cmp_lt_i64_e64 s[0:1], s[6:7], 0
 ; GFX6-NEXT:    v_cmp_eq_u64_e64 vcc, s[6:7], 0
-; GFX6-NEXT:    s_sub_i32 s6, s2, 64
-; GFX6-NEXT:    s_sub_i32 s4, 64, s2
-; GFX6-NEXT:    s_cmp_lt_u32 s2, 64
-; GFX6-NEXT:    s_cselect_b32 s12, 1, 0
-; GFX6-NEXT:    s_cmp_eq_u32 s2, 0
+; GFX6-NEXT:    s_sub_i32 s6, s12, 64
+; GFX6-NEXT:    s_sub_i32 s4, 64, s12
+; GFX6-NEXT:    s_cmp_lt_u32 s12, 64
 ; GFX6-NEXT:    s_cselect_b32 s13, 1, 0
-; GFX6-NEXT:    v_cndmask_b32_e64 v2, 0, 1, s[0:1]
-; GFX6-NEXT:    s_ashr_i64 s[0:1], s[10:11], s2
-; GFX6-NEXT:    s_lshr_b64 s[2:3], s[8:9], s2
+; GFX6-NEXT:    s_lshr_b64 s[2:3], s[8:9], s12
 ; GFX6-NEXT:    s_lshl_b64 s[4:5], s[10:11], s4
+; GFX6-NEXT:    v_cndmask_b32_e64 v2, 0, 1, s[0:1]
 ; GFX6-NEXT:    s_or_b64 s[2:3], s[2:3], s[4:5]
 ; GFX6-NEXT:    s_ashr_i32 s4, s11, 31
+; GFX6-NEXT:    s_ashr_i64 s[0:1], s[10:11], s12
 ; GFX6-NEXT:    s_ashr_i64 s[6:7], s[10:11], s6
-; GFX6-NEXT:    s_and_b32 s12, s12, 1
-; GFX6-NEXT:    s_cmp_lg_u32 s12, 0
+; GFX6-NEXT:    s_and_b32 s13, s13, 1
+; GFX6-NEXT:    s_cmp_lg_u32 s13, 0
 ; GFX6-NEXT:    s_cselect_b64 s[2:3], s[2:3], s[6:7]
-; GFX6-NEXT:    s_and_b32 s6, s13, 1
-; GFX6-NEXT:    s_cmp_lg_u32 s6, 0
+; GFX6-NEXT:    s_cmp_eq_u32 s12, 0
 ; GFX6-NEXT:    s_cselect_b64 s[2:3], s[8:9], s[2:3]
 ; GFX6-NEXT:    s_mov_b32 s5, s4
-; GFX6-NEXT:    s_cmp_lg_u32 s12, 0
+; GFX6-NEXT:    s_cmp_lg_u32 s13, 0
 ; GFX6-NEXT:    s_cselect_b64 s[0:1], s[0:1], s[4:5]
 ; GFX6-NEXT:    s_add_u32 s2, s2, 0
 ; GFX6-NEXT:    s_cselect_b32 s4, 1, 0
 ; GFX6-NEXT:    s_and_b32 s4, s4, 1
 ; GFX6-NEXT:    s_cmp_lg_u32 s4, 0
 ; GFX6-NEXT:    s_addc_u32 s3, s3, 0
 ; GFX6-NEXT:    s_cselect_b32 s4, 1, 0
 ; GFX6-NEXT:    s_and_b32 s4, s4, 1

 ; GFX8-NEXT:    s_and_b32 s0, 1, s2
 ; GFX8-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
 ; GFX8-NEXT:    v_cmp_ne_u32_e64 vcc, 0, s0
 ; GFX8-NEXT:    v_cmp_lt_u64_e64 s[0:1], s[4:5], 0
 ; GFX8-NEXT:    s_cmp_eq_u64 s[6:7], 0
 ; GFX8-NEXT:    v_cndmask_b32_e64 v1, 0, 1, s[0:1]
 ; GFX8-NEXT:    v_cmp_lt_i64_e64 s[0:1], s[6:7], 0
 ; GFX8-NEXT:    s_cselect_b32 s2, 1, 0
+; GFX8-NEXT:    s_movk_i32 s12, 0x7f
 ; GFX8-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; GFX8-NEXT:    v_cndmask_b32_e64 v2, 0, 1, s[0:1]
 ; GFX8-NEXT:    s_and_b32 s0, 1, s2
-; GFX8-NEXT:    s_movk_i32 s2, 0x7f
-; GFX8-NEXT:    s_sub_i32 s6, s2, 64
-; GFX8-NEXT:    s_sub_i32 s4, 64, s2
-; GFX8-NEXT:    s_cmp_lt_u32 s2, 64
-; GFX8-NEXT:    s_cselect_b32 s12, 1, 0
-; GFX8-NEXT:    s_cmp_eq_u32 s2, 0
+; GFX8-NEXT:    s_sub_i32 s6, s12, 64
+; GFX8-NEXT:    s_sub_i32 s4, 64, s12
+; GFX8-NEXT:    s_cmp_lt_u32 s12, 64
 ; GFX8-NEXT:    s_cselect_b32 s13, 1, 0
-; GFX8-NEXT:    v_cmp_ne_u32_e64 vcc, 0, s0
-; GFX8-NEXT:    s_ashr_i64 s[0:1], s[10:11], s2
-; GFX8-NEXT:    s_lshr_b64 s[2:3], s[8:9], s2
+; GFX8-NEXT:    s_lshr_b64 s[2:3], s[8:9], s12
 ; GFX8-NEXT:    s_lshl_b64 s[4:5], s[10:11], s4
+; GFX8-NEXT:    v_cmp_ne_u32_e64 vcc, 0, s0
 ; GFX8-NEXT:    s_or_b64 s[2:3], s[2:3], s[4:5]
 ; GFX8-NEXT:    s_ashr_i32 s4, s11, 31
+; GFX8-NEXT:    s_ashr_i64 s[0:1], s[10:11], s12
 ; GFX8-NEXT:    s_ashr_i64 s[6:7], s[10:11], s6
-; GFX8-NEXT:    s_and_b32 s12, s12, 1
-; GFX8-NEXT:    s_cmp_lg_u32 s12, 0
+; GFX8-NEXT:    s_and_b32 s13, s13, 1
+; GFX8-NEXT:    s_cmp_lg_u32 s13, 0
 ; GFX8-NEXT:    s_cselect_b64 s[2:3], s[2:3], s[6:7]
-; GFX8-NEXT:    s_and_b32 s6, s13, 1
-; GFX8-NEXT:    s_cmp_lg_u32 s6, 0
+; GFX8-NEXT:    s_cmp_eq_u32 s12, 0
 ; GFX8-NEXT:    s_cselect_b64 s[2:3], s[8:9], s[2:3]
 ; GFX8-NEXT:    s_mov_b32 s5, s4
-; GFX8-NEXT:    s_cmp_lg_u32 s12, 0
+; GFX8-NEXT:    s_cmp_lg_u32 s13, 0
 ; GFX8-NEXT:    s_cselect_b64 s[0:1], s[0:1], s[4:5]
 ; GFX8-NEXT:    s_add_u32 s2, s2, 0
 ; GFX8-NEXT:    s_cselect_b32 s4, 1, 0
 ; GFX8-NEXT:    s_and_b32 s4, s4, 1
 ; GFX8-NEXT:    s_cmp_lg_u32 s4, 0
 ; GFX8-NEXT:    s_addc_u32 s3, s3, 0
 ; GFX8-NEXT:    s_cselect_b32 s4, 1, 0
 ; GFX8-NEXT:    s_and_b32 s4, s4, 1

 ; GFX9-NEXT:    s_and_b32 s0, 1, s2
 ; GFX9-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
 ; GFX9-NEXT:    v_cmp_ne_u32_e64 vcc, 0, s0
 ; GFX9-NEXT:    v_cmp_lt_u64_e64 s[0:1], s[4:5], 0
 ; GFX9-NEXT:    s_cmp_eq_u64 s[6:7], 0
 ; GFX9-NEXT:    v_cndmask_b32_e64 v1, 0, 1, s[0:1]
 ; GFX9-NEXT:    v_cmp_lt_i64_e64 s[0:1], s[6:7], 0
 ; GFX9-NEXT:    s_cselect_b32 s2, 1, 0
+; GFX9-NEXT:    s_movk_i32 s12, 0x7f
 ; GFX9-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; GFX9-NEXT:    v_cndmask_b32_e64 v2, 0, 1, s[0:1]
 ; GFX9-NEXT:    s_and_b32 s0, 1, s2
-; GFX9-NEXT:    s_movk_i32 s2, 0x7f
-; GFX9-NEXT:    s_sub_i32 s6, s2, 64
-; GFX9-NEXT:    s_sub_i32 s4, 64, s2
-; GFX9-NEXT:    s_cmp_lt_u32 s2, 64
-; GFX9-NEXT:    s_cselect_b32 s12, 1, 0
-; GFX9-NEXT:    s_cmp_eq_u32 s2, 0
+; GFX9-NEXT:    s_sub_i32 s6, s12, 64
+; GFX9-NEXT:    s_sub_i32 s4, 64, s12
+; GFX9-NEXT:    s_cmp_lt_u32 s12, 64
 ; GFX9-NEXT:    s_cselect_b32 s13, 1, 0
-; GFX9-NEXT:    v_cmp_ne_u32_e64 vcc, 0, s0
-; GFX9-NEXT:    s_ashr_i64 s[0:1], s[10:11], s2
-; GFX9-NEXT:    s_lshr_b64 s[2:3], s[8:9], s2
+; GFX9-NEXT:    s_lshr_b64 s[2:3], s[8:9], s12
 ; GFX9-NEXT:    s_lshl_b64 s[4:5], s[10:11], s4
+; GFX9-NEXT:    v_cmp_ne_u32_e64 vcc, 0, s0
 ; GFX9-NEXT:    s_or_b64 s[2:3], s[2:3], s[4:5]
 ; GFX9-NEXT:    s_ashr_i32 s4, s11, 31
+; GFX9-NEXT:    s_ashr_i64 s[0:1], s[10:11], s12
 ; GFX9-NEXT:    s_ashr_i64 s[6:7], s[10:11], s6
-; GFX9-NEXT:    s_and_b32 s12, s12, 1
-; GFX9-NEXT:    s_cmp_lg_u32 s12, 0
+; GFX9-NEXT:    s_and_b32 s13, s13, 1
+; GFX9-NEXT:    s_cmp_lg_u32 s13, 0
 ; GFX9-NEXT:    s_cselect_b64 s[2:3], s[2:3], s[6:7]
-; GFX9-NEXT:    s_and_b32 s6, s13, 1
-; GFX9-NEXT:    s_cmp_lg_u32 s6, 0
+; GFX9-NEXT:    s_cmp_eq_u32 s12, 0
 ; GFX9-NEXT:    s_cselect_b64 s[2:3], s[8:9], s[2:3]
 ; GFX9-NEXT:    s_mov_b32 s5, s4
-; GFX9-NEXT:    s_cmp_lg_u32 s12, 0
+; GFX9-NEXT:    s_cmp_lg_u32 s13, 0
 ; GFX9-NEXT:    s_cselect_b64 s[0:1], s[0:1], s[4:5]
 ; GFX9-NEXT:    s_add_u32 s2, s2, 0
 ; GFX9-NEXT:    s_cselect_b32 s4, 1, 0
 ; GFX9-NEXT:    s_and_b32 s4, s4, 1
 ; GFX9-NEXT:    s_cmp_lg_u32 s4, 0
 ; GFX9-NEXT:    s_addc_u32 s3, s3, 0
 ; GFX9-NEXT:    s_cselect_b32 s4, 1, 0
 ; GFX9-NEXT:    s_and_b32 s4, s4, 1

 ; GFX10-NEXT:    s_cselect_b32 s0, 1, 0
 ; GFX10-NEXT:    v_mov_b32_e32 v4, s11
 ; GFX10-NEXT:    s_and_b32 s0, 1, s0
 ; GFX10-NEXT:    s_cmp_eq_u64 s[6:7], 0
 ; GFX10-NEXT:    v_cmp_ne_u32_e64 vcc_lo, 0, s0
 ; GFX10-NEXT:    v_cmp_lt_u64_e64 s0, s[4:5], 0
 ; GFX10-NEXT:    v_cndmask_b32_e64 v1, 0, 1, s1
 ; GFX10-NEXT:    s_cselect_b32 s1, 1, 0
-; GFX10-NEXT:    s_sub_i32 s13, s12, 64
-; GFX10-NEXT:    s_and_b32 s14, 1, s1
+; GFX10-NEXT:    s_sub_i32 s14, s12, 64
+; GFX10-NEXT:    s_and_b32 s13, 1, s1
 ; GFX10-NEXT:    s_sub_i32 s2, 64, s12
 ; GFX10-NEXT:    v_cndmask_b32_e32 v0, v1, v0, vcc_lo
 ; GFX10-NEXT:    v_cndmask_b32_e64 v1, 0, 1, s0
 ; GFX10-NEXT:    v_cmp_lt_i64_e64 s0, s[6:7], 0
 ; GFX10-NEXT:    s_cmp_lt_u32 s12, 64
-; GFX10-NEXT:    v_cmp_ne_u32_e64 vcc_lo, 0, s14
+; GFX10-NEXT:    v_cmp_ne_u32_e64 vcc_lo, 0, s13
 ; GFX10-NEXT:    s_cselect_b32 s15, 1, 0
-; GFX10-NEXT:    s_cmp_eq_u32 s12, 0
-; GFX10-NEXT:    s_cselect_b32 s16, 1, 0
+; GFX10-NEXT:    s_lshl_b64 s[2:3], s[10:11], s2
+; GFX10-NEXT:    s_and_b32 s15, s15, 1
 ; GFX10-NEXT:    v_cndmask_b32_e64 v2, 0, 1, s0
 ; GFX10-NEXT:    s_lshr_b64 s[0:1], s[8:9], s12
-; GFX10-NEXT:    s_lshl_b64 s[2:3], s[10:11], s2
 ; GFX10-NEXT:    s_ashr_i64 s[4:5], s[10:11], s12
-; GFX10-NEXT:    s_and_b32 s12, s15, 1
 ; GFX10-NEXT:    s_or_b64 s[0:1], s[0:1], s[2:3]
 ; GFX10-NEXT:    s_ashr_i32 s2, s11, 31
-; GFX10-NEXT:    s_ashr_i64 s[6:7], s[10:11], s13
-; GFX10-NEXT:    s_cmp_lg_u32 s12, 0
+; GFX10-NEXT:    s_ashr_i64 s[6:7], s[10:11], s14
+; GFX10-NEXT:    s_cmp_lg_u32 s15, 0
 ; GFX10-NEXT:    s_mov_b32 s3, s2
 ; GFX10-NEXT:    s_cselect_b64 s[0:1], s[0:1], s[6:7]
-; GFX10-NEXT:    s_and_b32 s6, s16, 1
+; GFX10-NEXT:    s_cmp_eq_u32 s12, 0
 ; GFX10-NEXT:    v_cndmask_b32_e32 v1, v2, v1, vcc_lo
-; GFX10-NEXT:    s_cmp_lg_u32 s6, 0
-; GFX10-NEXT:    v_mov_b32_e32 v2, s9
 ; GFX10-NEXT:    s_cselect_b64 s[0:1], s[8:9], s[0:1]
-; GFX10-NEXT:    s_cmp_lg_u32 s12, 0
-; GFX10-NEXT:    v_xor_b32_e32 v0, v1, v0
+; GFX10-NEXT:    s_cmp_lg_u32 s15, 0
+; GFX10-NEXT:    v_mov_b32_e32 v2, s9
 ; GFX10-NEXT:    s_cselect_b64 s[2:3], s[4:5], s[2:3]
 ; GFX10-NEXT:    s_add_u32 s0, s0, 0
 ; GFX10-NEXT:    s_cselect_b32 s4, 1, 0
-; GFX10-NEXT:    v_mov_b32_e32 v1, s8
+; GFX10-NEXT:    v_xor_b32_e32 v0, v1, v0
 ; GFX10-NEXT:    s_and_b32 s4, s4, 1
-; GFX10-NEXT:    v_and_b32_e32 v0, 1, v0
+; GFX10-NEXT:    v_mov_b32_e32 v1, s8
 ; GFX10-NEXT:    s_cmp_lg_u32 s4, 0
 ; GFX10-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX10-NEXT:    s_cselect_b32 s4, 1, 0
-; GFX10-NEXT:    v_cmp_ne_u32_e32 vcc_lo, 0, v0
+; GFX10-NEXT:    v_and_b32_e32 v0, 1, v0
 ; GFX10-NEXT:    s_and_b32 s4, s4, 1
 ; GFX10-NEXT:    s_cmp_lg_u32 s4, 0
 ; GFX10-NEXT:    s_addc_u32 s2, s2, 0
 ; GFX10-NEXT:    s_cselect_b32 s4, 1, 0
-; GFX10-NEXT:    v_cndmask_b32_e64 v0, v1, s0, vcc_lo
+; GFX10-NEXT:    v_cmp_ne_u32_e32 vcc_lo, 0, v0
 ; GFX10-NEXT:    s_and_b32 s4, s4, 1
-; GFX10-NEXT:    v_cndmask_b32_e64 v1, v2, s1, vcc_lo
 ; GFX10-NEXT:    s_cmp_lg_u32 s4, 0
-; GFX10-NEXT:    v_cndmask_b32_e64 v2, v3, s2, vcc_lo
 ; GFX10-NEXT:    s_addc_u32 s3, s3, 0x80000000
-; GFX10-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX10-NEXT:    v_cndmask_b32_e64 v0, v1, s0, vcc_lo
+; GFX10-NEXT:    v_cndmask_b32_e64 v1, v2, s1, vcc_lo
+; GFX10-NEXT:    v_cndmask_b32_e64 v2, v3, s2, vcc_lo
 ; GFX10-NEXT:    v_cndmask_b32_e64 v3, v4, s3, vcc_lo
+; GFX10-NEXT:    v_readfirstlane_b32 s0, v0
 ; GFX10-NEXT:    v_readfirstlane_b32 s1, v1
 ; GFX10-NEXT:    v_readfirstlane_b32 s2, v2
 ; GFX10-NEXT:    v_readfirstlane_b32 s3, v3
 ; GFX10-NEXT:    ; return to shader part epilog
   %result = call i128 @llvm.sadd.sat.i128(i128 %lhs, i128 %rhs)
   ret i128 %result
 }
 

 ; GFX6-NEXT:    s_cselect_b32 s19, 1, 0
 ; GFX6-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX6-NEXT:    s_and_b32 s19, s19, 1
 ; GFX6-NEXT:    v_mov_b32_e32 v2, s0
 ; GFX6-NEXT:    s_cmp_lg_u32 s19, 0
 ; GFX6-NEXT:    v_cmp_lt_u64_e32 vcc, s[16:17], v[2:3]
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s2
 ; GFX6-NEXT:    s_addc_u32 s19, s3, s11
-; GFX6-NEXT:    s_movk_i32 s20, 0x7f
 ; GFX6-NEXT:    v_mov_b32_e32 v1, s3
+; GFX6-NEXT:    s_movk_i32 s20, 0x7f
 ; GFX6-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
 ; GFX6-NEXT:    v_cmp_lt_i64_e32 vcc, s[18:19], v[0:1]
 ; GFX6-NEXT:    v_cmp_lt_u64_e64 s[0:1], s[8:9], 0
 ; GFX6-NEXT:    s_sub_i32 s21, s20, 64
 ; GFX6-NEXT:    s_sub_i32 s22, 64, s20
-; GFX6-NEXT:    s_cmp_lt_u32 s20, 64
 ; GFX6-NEXT:    v_cndmask_b32_e64 v3, 0, 1, vcc
 ; GFX6-NEXT:    v_cmp_eq_u64_e32 vcc, s[18:19], v[0:1]
 ; GFX6-NEXT:    v_cndmask_b32_e64 v1, 0, 1, s[0:1]
-; GFX6-NEXT:    s_cselect_b32 s23, 1, 0
-; GFX6-NEXT:    s_cmp_eq_u32 s20, 0
+; GFX6-NEXT:    s_cmp_lt_u32 s20, 64
 ; GFX6-NEXT:    v_cmp_lt_i64_e64 s[0:1], s[10:11], 0
-; GFX6-NEXT:    s_cselect_b32 s24, 1, 0
+; GFX6-NEXT:    s_cselect_b32 s23, 1, 0
 ; GFX6-NEXT:    s_lshr_b64 s[2:3], s[16:17], s20
 ; GFX6-NEXT:    s_lshl_b64 s[8:9], s[18:19], s22
 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, v3, v2, vcc
 ; GFX6-NEXT:    v_cndmask_b32_e64 v2, 0, 1, s[0:1]
 ; GFX6-NEXT:    v_cmp_eq_u64_e64 vcc, s[10:11], 0
 ; GFX6-NEXT:    s_or_b64 s[2:3], s[2:3], s[8:9]
 ; GFX6-NEXT:    s_ashr_i32 s8, s19, 31
 ; GFX6-NEXT:    s_ashr_i64 s[0:1], s[18:19], s20
 ; GFX6-NEXT:    s_ashr_i64 s[10:11], s[18:19], s21
 ; GFX6-NEXT:    s_and_b32 s23, s23, 1
 ; GFX6-NEXT:    s_cmp_lg_u32 s23, 0
 ; GFX6-NEXT:    s_cselect_b64 s[2:3], s[2:3], s[10:11]
-; GFX6-NEXT:    s_and_b32 s10, s24, 1
-; GFX6-NEXT:    s_cmp_lg_u32 s10, 0
+; GFX6-NEXT:    s_cmp_eq_u32 s20, 0
 ; GFX6-NEXT:    s_cselect_b64 s[2:3], s[16:17], s[2:3]
 ; GFX6-NEXT:    s_cmp_lg_u32 s23, 0
 ; GFX6-NEXT:    s_mov_b32 s9, s8
 ; GFX6-NEXT:    s_cselect_b64 s[0:1], s[0:1], s[8:9]
 ; GFX6-NEXT:    s_add_u32 s2, s2, 0
 ; GFX6-NEXT:    s_cselect_b32 s8, 1, 0
 ; GFX6-NEXT:    s_and_b32 s8, s8, 1
 ; GFX6-NEXT:    s_cmp_lg_u32 s8, 0

 ; GFX6-NEXT:    s_cmp_lg_u32 s3, 0
 ; GFX6-NEXT:    v_cmp_lt_u64_e32 vcc, s[0:1], v[2:3]
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s6
 ; GFX6-NEXT:    s_addc_u32 s3, s7, s15
 ; GFX6-NEXT:    v_mov_b32_e32 v1, s7
 ; GFX6-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
 ; GFX6-NEXT:    v_cmp_lt_i64_e32 vcc, s[2:3], v[0:1]
 ; GFX6-NEXT:    v_cmp_lt_u64_e64 s[4:5], s[12:13], 0
-; GFX6-NEXT:    s_cmp_lt_u32 s20, 64
 ; GFX6-NEXT:    v_cndmask_b32_e64 v3, 0, 1, vcc
 ; GFX6-NEXT:    v_cmp_eq_u64_e32 vcc, s[2:3], v[0:1]
 ; GFX6-NEXT:    v_cndmask_b32_e64 v1, 0, 1, s[4:5]
-; GFX6-NEXT:    s_cselect_b32 s12, 1, 0
-; GFX6-NEXT:    s_cmp_eq_u32 s20, 0
+; GFX6-NEXT:    s_cmp_lt_u32 s20, 64
 ; GFX6-NEXT:    v_cmp_lt_i64_e64 s[4:5], s[14:15], 0
-; GFX6-NEXT:    s_cselect_b32 s13, 1, 0
+; GFX6-NEXT:    s_cselect_b32 s12, 1, 0
 ; GFX6-NEXT:    s_lshr_b64 s[6:7], s[0:1], s20
 ; GFX6-NEXT:    s_lshl_b64 s[8:9], s[2:3], s22
 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, v3, v2, vcc
 ; GFX6-NEXT:    v_cndmask_b32_e64 v2, 0, 1, s[4:5]
 ; GFX6-NEXT:    s_or_b64 s[6:7], s[6:7], s[8:9]
 ; GFX6-NEXT:    s_ashr_i32 s8, s3, 31
 ; GFX6-NEXT:    s_ashr_i64 s[4:5], s[2:3], s20
 ; GFX6-NEXT:    s_ashr_i64 s[10:11], s[2:3], s21
 ; GFX6-NEXT:    s_and_b32 s12, s12, 1
 ; GFX6-NEXT:    s_cmp_lg_u32 s12, 0
 ; GFX6-NEXT:    s_cselect_b64 s[6:7], s[6:7], s[10:11]
-; GFX6-NEXT:    s_and_b32 s10, s13, 1
-; GFX6-NEXT:    s_cmp_lg_u32 s10, 0
+; GFX6-NEXT:    s_cmp_eq_u32 s20, 0
 ; GFX6-NEXT:    s_cselect_b64 s[6:7], s[0:1], s[6:7]
 ; GFX6-NEXT:    s_mov_b32 s9, s8
 ; GFX6-NEXT:    s_cmp_lg_u32 s12, 0
 ; GFX6-NEXT:    s_cselect_b64 s[4:5], s[4:5], s[8:9]
 ; GFX6-NEXT:    s_add_u32 s6, s6, 0
 ; GFX6-NEXT:    s_cselect_b32 s8, 1, 0
 ; GFX6-NEXT:    s_and_b32 s8, s8, 1
 ; GFX6-NEXT:    s_cmp_lg_u32 s8, 0

 ; GFX8-NEXT:    s_movk_i32 s20, 0x7f
 ; GFX8-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; GFX8-NEXT:    v_cndmask_b32_e64 v2, 0, 1, s[0:1]
 ; GFX8-NEXT:    s_and_b32 s0, 1, s2
 ; GFX8-NEXT:    s_sub_i32 s21, s20, 64
 ; GFX8-NEXT:    s_sub_i32 s22, 64, s20
 ; GFX8-NEXT:    s_cmp_lt_u32 s20, 64
 ; GFX8-NEXT:    s_cselect_b32 s23, 1, 0
-; GFX8-NEXT:    s_cmp_eq_u32 s20, 0
-; GFX8-NEXT:    s_cselect_b32 s24, 1, 0
 ; GFX8-NEXT:    s_lshr_b64 s[2:3], s[16:17], s20
 ; GFX8-NEXT:    s_lshl_b64 s[8:9], s[18:19], s22
 ; GFX8-NEXT:    v_cmp_ne_u32_e64 vcc, 0, s0
 ; GFX8-NEXT:    s_or_b64 s[2:3], s[2:3], s[8:9]
 ; GFX8-NEXT:    s_ashr_i32 s8, s19, 31
 ; GFX8-NEXT:    s_ashr_i64 s[0:1], s[18:19], s20
 ; GFX8-NEXT:    s_ashr_i64 s[10:11], s[18:19], s21
 ; GFX8-NEXT:    s_and_b32 s23, s23, 1
 ; GFX8-NEXT:    s_cmp_lg_u32 s23, 0
 ; GFX8-NEXT:    s_cselect_b64 s[2:3], s[2:3], s[10:11]
-; GFX8-NEXT:    s_and_b32 s10, s24, 1
-; GFX8-NEXT:    s_cmp_lg_u32 s10, 0
+; GFX8-NEXT:    s_cmp_eq_u32 s20, 0
 ; GFX8-NEXT:    s_cselect_b64 s[2:3], s[16:17], s[2:3]
 ; GFX8-NEXT:    s_cmp_lg_u32 s23, 0
 ; GFX8-NEXT:    s_mov_b32 s9, s8
 ; GFX8-NEXT:    s_cselect_b64 s[0:1], s[0:1], s[8:9]
 ; GFX8-NEXT:    s_add_u32 s2, s2, 0
 ; GFX8-NEXT:    s_cselect_b32 s8, 1, 0
 ; GFX8-NEXT:    s_and_b32 s8, s8, 1
 ; GFX8-NEXT:    s_cmp_lg_u32 s8, 0

 ; GFX8-NEXT:    v_cndmask_b32_e64 v1, 0, 1, s[4:5]
 ; GFX8-NEXT:    v_cmp_lt_i64_e64 s[4:5], s[14:15], 0
 ; GFX8-NEXT:    s_cselect_b32 s6, 1, 0
 ; GFX8-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; GFX8-NEXT:    v_cndmask_b32_e64 v2, 0, 1, s[4:5]
 ; GFX8-NEXT:    s_and_b32 s4, 1, s6
 ; GFX8-NEXT:    s_cmp_lt_u32 s20, 64
 ; GFX8-NEXT:    s_cselect_b32 s12, 1, 0
-; GFX8-NEXT:    s_cmp_eq_u32 s20, 0
-; GFX8-NEXT:    s_cselect_b32 s13, 1, 0
 ; GFX8-NEXT:    s_lshr_b64 s[6:7], s[0:1], s20
 ; GFX8-NEXT:    s_lshl_b64 s[8:9], s[2:3], s22
 ; GFX8-NEXT:    v_cmp_ne_u32_e64 vcc, 0, s4
 ; GFX8-NEXT:    s_or_b64 s[6:7], s[6:7], s[8:9]
 ; GFX8-NEXT:    s_ashr_i32 s8, s3, 31
 ; GFX8-NEXT:    s_ashr_i64 s[4:5], s[2:3], s20
 ; GFX8-NEXT:    s_ashr_i64 s[10:11], s[2:3], s21
 ; GFX8-NEXT:    s_and_b32 s12, s12, 1
 ; GFX8-NEXT:    s_cmp_lg_u32 s12, 0
 ; GFX8-NEXT:    s_cselect_b64 s[6:7], s[6:7], s[10:11]
-; GFX8-NEXT:    s_and_b32 s10, s13, 1
-; GFX8-NEXT:    s_cmp_lg_u32 s10, 0
+; GFX8-NEXT:    s_cmp_eq_u32 s20, 0
 ; GFX8-NEXT:    s_cselect_b64 s[6:7], s[0:1], s[6:7]
 ; GFX8-NEXT:    s_mov_b32 s9, s8
 ; GFX8-NEXT:    s_cmp_lg_u32 s12, 0
 ; GFX8-NEXT:    s_cselect_b64 s[4:5], s[4:5], s[8:9]
 ; GFX8-NEXT:    s_add_u32 s6, s6, 0
 ; GFX8-NEXT:    s_cselect_b32 s8, 1, 0
 ; GFX8-NEXT:    s_and_b32 s8, s8, 1
 ; GFX8-NEXT:    s_cmp_lg_u32 s8, 0

 ; GFX9-NEXT:    s_movk_i32 s20, 0x7f
 ; GFX9-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; GFX9-NEXT:    v_cndmask_b32_e64 v2, 0, 1, s[0:1]
 ; GFX9-NEXT:    s_and_b32 s0, 1, s2
 ; GFX9-NEXT:    s_sub_i32 s21, s20, 64
 ; GFX9-NEXT:    s_sub_i32 s22, 64, s20
 ; GFX9-NEXT:    s_cmp_lt_u32 s20, 64
 ; GFX9-NEXT:    s_cselect_b32 s23, 1, 0
-; GFX9-NEXT:    s_cmp_eq_u32 s20, 0
-; GFX9-NEXT:    s_cselect_b32 s24, 1, 0
 ; GFX9-NEXT:    s_lshr_b64 s[2:3], s[16:17], s20
 ; GFX9-NEXT:    s_lshl_b64 s[8:9], s[18:19], s22
 ; GFX9-NEXT:    v_cmp_ne_u32_e64 vcc, 0, s0
 ; GFX9-NEXT:    s_or_b64 s[2:3], s[2:3], s[8:9]
 ; GFX9-NEXT:    s_ashr_i32 s8, s19, 31
 ; GFX9-NEXT:    s_ashr_i64 s[0:1], s[18:19], s20
 ; GFX9-NEXT:    s_ashr_i64 s[10:11], s[18:19], s21
 ; GFX9-NEXT:    s_and_b32 s23, s23, 1
 ; GFX9-NEXT:    s_cmp_lg_u32 s23, 0
 ; GFX9-NEXT:    s_cselect_b64 s[2:3], s[2:3], s[10:11]
-; GFX9-NEXT:    s_and_b32 s10, s24, 1
-; GFX9-NEXT:    s_cmp_lg_u32 s10, 0
+; GFX9-NEXT:    s_cmp_eq_u32 s20, 0
 ; GFX9-NEXT:    s_cselect_b64 s[2:3], s[16:17], s[2:3]
 ; GFX9-NEXT:    s_cmp_lg_u32 s23, 0
 ; GFX9-NEXT:    s_mov_b32 s9, s8
 ; GFX9-NEXT:    s_cselect_b64 s[0:1], s[0:1], s[8:9]
 ; GFX9-NEXT:    s_add_u32 s2, s2, 0
 ; GFX9-NEXT:    s_cselect_b32 s8, 1, 0
 ; GFX9-NEXT:    s_and_b32 s8, s8, 1
 ; GFX9-NEXT:    s_cmp_lg_u32 s8, 0

 ; GFX9-NEXT:    v_cndmask_b32_e64 v1, 0, 1, s[4:5]
 ; GFX9-NEXT:    v_cmp_lt_i64_e64 s[4:5], s[14:15], 0
 ; GFX9-NEXT:    s_cselect_b32 s6, 1, 0
 ; GFX9-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; GFX9-NEXT:    v_cndmask_b32_e64 v2, 0, 1, s[4:5]
 ; GFX9-NEXT:    s_and_b32 s4, 1, s6
 ; GFX9-NEXT:    s_cmp_lt_u32 s20, 64
 ; GFX9-NEXT:    s_cselect_b32 s12, 1, 0
-; GFX9-NEXT:    s_cmp_eq_u32 s20, 0
-; GFX9-NEXT:    s_cselect_b32 s13, 1, 0
 ; GFX9-NEXT:    s_lshr_b64 s[6:7], s[0:1], s20
 ; GFX9-NEXT:    s_lshl_b64 s[8:9], s[2:3], s22
 ; GFX9-NEXT:    v_cmp_ne_u32_e64 vcc, 0, s4
 ; GFX9-NEXT:    s_or_b64 s[6:7], s[6:7], s[8:9]
 ; GFX9-NEXT:    s_ashr_i32 s8, s3, 31
 ; GFX9-NEXT:    s_ashr_i64 s[4:5], s[2:3], s20
 ; GFX9-NEXT:    s_ashr_i64 s[10:11], s[2:3], s21
 ; GFX9-NEXT:    s_and_b32 s12, s12, 1
 ; GFX9-NEXT:    s_cmp_lg_u32 s12, 0
 ; GFX9-NEXT:    s_cselect_b64 s[6:7], s[6:7], s[10:11]
-; GFX9-NEXT:    s_and_b32 s10, s13, 1
-; GFX9-NEXT:    s_cmp_lg_u32 s10, 0
+; GFX9-NEXT:    s_cmp_eq_u32 s20, 0
 ; GFX9-NEXT:    s_cselect_b64 s[6:7], s[0:1], s[6:7]
 ; GFX9-NEXT:    s_mov_b32 s9, s8
 ; GFX9-NEXT:    s_cmp_lg_u32 s12, 0
 ; GFX9-NEXT:    s_cselect_b64 s[4:5], s[4:5], s[8:9]
 ; GFX9-NEXT:    s_add_u32 s6, s6, 0
 ; GFX9-NEXT:    s_cselect_b32 s8, 1, 0
 ; GFX9-NEXT:    s_and_b32 s8, s8, 1
 ; GFX9-NEXT:    s_cmp_lg_u32 s8, 0

 ; GFX10-NEXT:    s_mov_b32 s46, s0
 ; GFX10-NEXT:    s_and_b32 s17, s17, 1
 ; GFX10-NEXT:    s_mov_b32 s47, s1
 ; GFX10-NEXT:    s_cmp_lg_u32 s17, 0
 ; GFX10-NEXT:    s_addc_u32 s29, s1, s9
 ; GFX10-NEXT:    s_cselect_b32 s18, 1, 0
 ; GFX10-NEXT:    v_cmp_lt_u64_e64 s0, s[28:29], s[46:47]
 ; GFX10-NEXT:    s_and_b32 s18, s18, 1
+; GFX10-NEXT:    v_cmp_lt_u64_e64 s1, s[8:9], 0
 ; GFX10-NEXT:    s_cmp_lg_u32 s18, 0
 ; GFX10-NEXT:    s_addc_u32 s30, s2, s10
 ; GFX10-NEXT:    s_cselect_b32 s19, 1, 0
 ; GFX10-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s0
 ; GFX10-NEXT:    s_and_b32 s19, s19, 1
 ; GFX10-NEXT:    s_cmp_lg_u32 s19, 0
 ; GFX10-NEXT:    s_addc_u32 s31, s3, s11
 ; GFX10-NEXT:    v_cmp_lt_i64_e64 s0, s[30:31], s[2:3]
 ; GFX10-NEXT:    s_cmp_eq_u64 s[30:31], s[2:3]
-; GFX10-NEXT:    v_cmp_lt_u64_e64 s2, s[8:9], 0
+; GFX10-NEXT:    v_mov_b32_e32 v3, s31
 ; GFX10-NEXT:    s_cselect_b32 s20, 1, 0
 ; GFX10-NEXT:    v_cndmask_b32_e64 v1, 0, 1, s0
 ; GFX10-NEXT:    s_and_b32 s0, 1, s20
 ; GFX10-NEXT:    s_cmp_eq_u64 s[10:11], 0
-; GFX10-NEXT:    s_movk_i32 s20, 0x7f
-; GFX10-NEXT:    s_cselect_b32 s1, 1, 0
-; GFX10-NEXT:    v_cndmask_b32_e64 v2, 0, 1, s2
-; GFX10-NEXT:    v_cmp_lt_i64_e64 s2, s[10:11], 0
 ; GFX10-NEXT:    v_cmp_ne_u32_e64 vcc_lo, 0, s0
-; GFX10-NEXT:    s_and_b32 s1, 1, s1
+; GFX10-NEXT:    s_cselect_b32 s0, 1, 0
+; GFX10-NEXT:    s_movk_i32 s20, 0x7f
+; GFX10-NEXT:    s_and_b32 s0, 1, s0
 ; GFX10-NEXT:    s_sub_i32 s21, s20, 64
+; GFX10-NEXT:    v_cndmask_b32_e32 v0, v1, v0, vcc_lo
+; GFX10-NEXT:    v_cndmask_b32_e64 v1, 0, 1, s1
+; GFX10-NEXT:    v_cmp_lt_i64_e64 s1, s[10:11], 0
 ; GFX10-NEXT:    s_sub_i32 s22, 64, s20
 ; GFX10-NEXT:    s_cmp_lt_u32 s20, 64
-; GFX10-NEXT:    v_cndmask_b32_e32 v0, v1, v0, vcc_lo
+; GFX10-NEXT:    v_cmp_ne_u32_e64 vcc_lo, 0, s0
 ; GFX10-NEXT:    s_cselect_b32 s10, 1, 0
-; GFX10-NEXT:    s_cmp_eq_u32 s20, 0
-; GFX10-NEXT:    v_cndmask_b32_e64 v3, 0, 1, s2
-; GFX10-NEXT:    s_cselect_b32 s23, 1, 0
-; GFX10-NEXT:    v_cmp_ne_u32_e64 vcc_lo, 0, s1
-; GFX10-NEXT:    s_lshr_b64 s[0:1], s[28:29], s20
 ; GFX10-NEXT:    s_lshl_b64 s[2:3], s[30:31], s22
-; GFX10-NEXT:    s_and_b32 s24, s10, 1
+; GFX10-NEXT:    v_cndmask_b32_e64 v2, 0, 1, s1
+; GFX10-NEXT:    s_lshr_b64 s[0:1], s[28:29], s20
+; GFX10-NEXT:    s_and_b32 s23, s10, 1
 ; GFX10-NEXT:    s_or_b64 s[0:1], s[0:1], s[2:3]
 ; GFX10-NEXT:    s_ashr_i32 s2, s31, 31
 ; GFX10-NEXT:    s_ashr_i64 s[8:9], s[30:31], s20
 ; GFX10-NEXT:    s_ashr_i64 s[10:11], s[30:31], s21
-; GFX10-NEXT:    s_cmp_lg_u32 s24, 0
+; GFX10-NEXT:    s_cmp_lg_u32 s23, 0
 ; GFX10-NEXT:    s_mov_b32 s3, s2
 ; GFX10-NEXT:    s_cselect_b64 s[0:1], s[0:1], s[10:11]
-; GFX10-NEXT:    s_and_b32 s10, s23, 1
-; GFX10-NEXT:    v_cndmask_b32_e32 v1, v3, v2, vcc_lo
-; GFX10-NEXT:    s_cmp_lg_u32 s10, 0
-; GFX10-NEXT:    v_mov_b32_e32 v2, s29
+; GFX10-NEXT:    s_cmp_eq_u32 s20, 0
+; GFX10-NEXT:    v_cndmask_b32_e32 v1, v2, v1, vcc_lo
 ; GFX10-NEXT:    s_cselect_b64 s[0:1], s[28:29], s[0:1]
-; GFX10-NEXT:    s_cmp_lg_u32 s24, 0
-; GFX10-NEXT:    v_xor_b32_e32 v0, v1, v0
+; GFX10-NEXT:    s_cmp_lg_u32 s23, 0
+; GFX10-NEXT:    s_brev_b32 s23, 1
 ; GFX10-NEXT:    s_cselect_b64 s[2:3], s[8:9], s[2:3]
 ; GFX10-NEXT:    s_add_u32 s0, s0, 0
 ; GFX10-NEXT:    s_cselect_b32 s8, 1, 0
-; GFX10-NEXT:    v_mov_b32_e32 v1, s28
+; GFX10-NEXT:    v_xor_b32_e32 v0, v1, v0
 ; GFX10-NEXT:    s_and_b32 s8, s8, 1
-; GFX10-NEXT:    v_and_b32_e32 v0, 1, v0
+; GFX10-NEXT:    v_mov_b32_e32 v1, s28
 ; GFX10-NEXT:    s_cmp_lg_u32 s8, 0
-; GFX10-NEXT:    s_brev_b32 s23, 1
+; GFX10-NEXT:    v_mov_b32_e32 v2, s29
 ; GFX10-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX10-NEXT:    s_cselect_b32 s8, 1, 0
-; GFX10-NEXT:    v_cmp_ne_u32_e32 vcc_lo, 0, v0
+; GFX10-NEXT:    v_and_b32_e32 v0, 1, v0
 ; GFX10-NEXT:    s_and_b32 s8, s8, 1
-; GFX10-NEXT:    v_mov_b32_e32 v3, s31
 ; GFX10-NEXT:    s_cmp_lg_u32 s8, 0
 ; GFX10-NEXT:    s_addc_u32 s2, s2, 0
 ; GFX10-NEXT:    s_cselect_b32 s8, 1, 0
-; GFX10-NEXT:    v_cndmask_b32_e64 v0, v1, s0, vcc_lo
+; GFX10-NEXT:    v_cmp_ne_u32_e32 vcc_lo, 0, v0
 ; GFX10-NEXT:    s_and_b32 s8, s8, 1
-; GFX10-NEXT:    v_cndmask_b32_e64 v1, v2, s1, vcc_lo
 ; GFX10-NEXT:    s_cmp_lg_u32 s8, 0
-; GFX10-NEXT:    v_mov_b32_e32 v2, s30
 ; GFX10-NEXT:    s_addc_u32 s3, s3, s23
+; GFX10-NEXT:    v_cndmask_b32_e64 v0, v1, s0, vcc_lo
 ; GFX10-NEXT:    s_add_u32 s0, s4, s12
-; GFX10-NEXT:    s_cselect_b32 s1, 1, 0
+; GFX10-NEXT:    s_cselect_b32 s8, 1, 0
+; GFX10-NEXT:    v_cndmask_b32_e64 v1, v2, s1, vcc_lo
+; GFX10-NEXT:    s_and_b32 s8, s8, 1
+; GFX10-NEXT:    v_mov_b32_e32 v2, s30
+; GFX10-NEXT:    s_cmp_lg_u32 s8, 0
 ; GFX10-NEXT:    v_cndmask_b32_e64 v3, v3, s3, vcc_lo
-; GFX10-NEXT:    s_and_b32 s1, s1, 1
-; GFX10-NEXT:    v_cndmask_b32_e64 v2, v2, s2, vcc_lo
-; GFX10-NEXT:    s_cmp_lg_u32 s1, 0
 ; GFX10-NEXT:    s_addc_u32 s1, s5, s13
 ; GFX10-NEXT:    s_cselect_b32 s8, 1, 0
-; GFX10-NEXT:    v_cmp_lt_u64_e64 s3, s[0:1], s[4:5]
+; GFX10-NEXT:    v_cmp_lt_u64_e64 s4, s[0:1], s[4:5]
 ; GFX10-NEXT:    s_and_b32 s8, s8, 1
+; GFX10-NEXT:    v_cndmask_b32_e64 v2, v2, s2, vcc_lo
 ; GFX10-NEXT:    s_cmp_lg_u32 s8, 0
+; GFX10-NEXT:    v_cmp_lt_u64_e64 s3, s[12:13], 0
 ; GFX10-NEXT:    s_addc_u32 s8, s6, s14
 ; GFX10-NEXT:    s_cselect_b32 s9, 1, 0
-; GFX10-NEXT:    v_cndmask_b32_e64 v4, 0, 1, s3
+; GFX10-NEXT:    v_cndmask_b32_e64 v4, 0, 1, s4
 ; GFX10-NEXT:    s_and_b32 s9, s9, 1
 ; GFX10-NEXT:    v_mov_b32_e32 v7, s8
 ; GFX10-NEXT:    s_cmp_lg_u32 s9, 0
 ; GFX10-NEXT:    s_addc_u32 s9, s7, s15
 ; GFX10-NEXT:    s_cmp_eq_u64 s[8:9], s[6:7]
-; GFX10-NEXT:    v_cmp_lt_i64_e64 s3, s[8:9], s[6:7]
+; GFX10-NEXT:    v_cmp_lt_i64_e64 s4, s[8:9], s[6:7]
 ; GFX10-NEXT:    s_cselect_b32 s2, 1, 0
 ; GFX10-NEXT:    v_mov_b32_e32 v8, s9
 ; GFX10-NEXT:    s_and_b32 s2, 1, s2
 ; GFX10-NEXT:    s_cmp_eq_u64 s[14:15], 0
 ; GFX10-NEXT:    v_cmp_ne_u32_e64 vcc_lo, 0, s2
-; GFX10-NEXT:    v_cmp_lt_u64_e64 s2, s[12:13], 0
-; GFX10-NEXT:    v_cndmask_b32_e64 v5, 0, 1, s3
-; GFX10-NEXT:    s_cselect_b32 s3, 1, 0
-; GFX10-NEXT:    s_and_b32 s16, 1, s3
-; GFX10-NEXT:    s_cmp_lt_u32 s20, 64
-; GFX10-NEXT:    v_cndmask_b32_e32 v4, v5, v4, vcc_lo
-; GFX10-NEXT:    v_cndmask_b32_e64 v5, 0, 1, s2
+; GFX10-NEXT:    s_cselect_b32 s2, 1, 0
+; GFX10-NEXT:    v_cndmask_b32_e64 v5, 0, 1, s4
+; GFX10-NEXT:    s_and_b32 s12, 1, s2
 ; GFX10-NEXT:    v_cmp_lt_i64_e64 s2, s[14:15], 0
+; GFX10-NEXT:    s_cmp_lt_u32 s20, 64
 ; GFX10-NEXT:    s_cselect_b32 s10, 1, 0
-; GFX10-NEXT:    s_cmp_eq_u32 s20, 0
-; GFX10-NEXT:    v_cmp_ne_u32_e64 vcc_lo, 0, s16
-; GFX10-NEXT:    s_cselect_b32 s12, 1, 0
-; GFX10-NEXT:    s_lshl_b64 s[4:5], s[8:9], s22
+; GFX10-NEXT:    v_cndmask_b32_e32 v4, v5, v4, vcc_lo
+; GFX10-NEXT:    v_cndmask_b32_e64 v5, 0, 1, s3
 ; GFX10-NEXT:    v_cndmask_b32_e64 v6, 0, 1, s2
 ; GFX10-NEXT:    s_lshr_b64 s[2:3], s[0:1], s20
+; GFX10-NEXT:    s_lshl_b64 s[4:5], s[8:9], s22
 ; GFX10-NEXT:    s_and_b32 s13, s10, 1
 ; GFX10-NEXT:    s_or_b64 s[2:3], s[2:3], s[4:5]
 ; GFX10-NEXT:    s_ashr_i32 s4, s9, 31
 ; GFX10-NEXT:    s_ashr_i64 s[6:7], s[8:9], s20
 ; GFX10-NEXT:    s_ashr_i64 s[10:11], s[8:9], s21
 ; GFX10-NEXT:    s_cmp_lg_u32 s13, 0
-; GFX10-NEXT:    s_mov_b32 s5, s4
+; GFX10-NEXT:    v_cmp_ne_u32_e64 vcc_lo, 0, s12
 ; GFX10-NEXT:    s_cselect_b64 s[2:3], s[2:3], s[10:11]
-; GFX10-NEXT:    s_and_b32 s10, s12, 1
-; GFX10-NEXT:    v_cndmask_b32_e32 v5, v6, v5, vcc_lo
-; GFX10-NEXT:    s_cmp_lg_u32 s10, 0
-; GFX10-NEXT:    v_mov_b32_e32 v6, s1
+; GFX10-NEXT:    s_cmp_eq_u32 s20, 0
+; GFX10-NEXT:    s_mov_b32 s5, s4
 ; GFX10-NEXT:    s_cselect_b64 s[2:3], s[0:1], s[2:3]
 ; GFX10-NEXT:    s_cmp_lg_u32 s13, 0
-; GFX10-NEXT:    v_xor_b32_e32 v4, v5, v4
+; GFX10-NEXT:    v_cndmask_b32_e32 v5, v6, v5, vcc_lo
 ; GFX10-NEXT:    s_cselect_b64 s[4:5], s[6:7], s[4:5]
 ; GFX10-NEXT:    s_add_u32 s2, s2, 0
 ; GFX10-NEXT:    s_cselect_b32 s6, 1, 0
-; GFX10-NEXT:    v_mov_b32_e32 v5, s0
+; GFX10-NEXT:    v_mov_b32_e32 v6, s1
 ; GFX10-NEXT:    s_and_b32 s6, s6, 1
-; GFX10-NEXT:    v_and_b32_e32 v4, 1, v4
+; GFX10-NEXT:    v_xor_b32_e32 v4, v5, v4
 ; GFX10-NEXT:    s_cmp_lg_u32 s6, 0
-; GFX10-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX10-NEXT:    v_mov_b32_e32 v5, s0
 ; GFX10-NEXT:    s_addc_u32 s3, s3, 0
 ; GFX10-NEXT:    s_cselect_b32 s6, 1, 0
-; GFX10-NEXT:    v_cmp_ne_u32_e32 vcc_lo, 0, v4
+; GFX10-NEXT:    v_and_b32_e32 v4, 1, v4
 ; GFX10-NEXT:    s_and_b32 s6, s6, 1
+; GFX10-NEXT:    v_readfirstlane_b32 s0, v0
 ; GFX10-NEXT:    s_cmp_lg_u32 s6, 0
 ; GFX10-NEXT:    s_addc_u32 s4, s4, 0
 ; GFX10-NEXT:    s_cselect_b32 s6, 1, 0
-; GFX10-NEXT:    v_cndmask_b32_e64 v4, v5, s2, vcc_lo
+; GFX10-NEXT:    v_cmp_ne_u32_e32 vcc_lo, 0, v4
 ; GFX10-NEXT:    s_and_b32 s6, s6, 1
-; GFX10-NEXT:    v_cndmask_b32_e64 v5, v6, s3, vcc_lo
 ; GFX10-NEXT:    s_cmp_lg_u32 s6, 0
-; GFX10-NEXT:    v_cndmask_b32_e64 v6, v7, s4, vcc_lo
 ; GFX10-NEXT:    s_addc_u32 s1, s5, s23
-; GFX10-NEXT:    v_readfirstlane_b32 s2, v2
+; GFX10-NEXT:    v_cndmask_b32_e64 v4, v5, s2, vcc_lo
+; GFX10-NEXT:    v_cndmask_b32_e64 v5, v6, s3, vcc_lo
+; GFX10-NEXT:    v_cndmask_b32_e64 v6, v7, s4, vcc_lo
 ; GFX10-NEXT:    v_cndmask_b32_e64 v7, v8, s1, vcc_lo
 ; GFX10-NEXT:    v_readfirstlane_b32 s1, v1
+; GFX10-NEXT:    v_readfirstlane_b32 s2, v2
 ; GFX10-NEXT:    v_readfirstlane_b32 s3, v3
 ; GFX10-NEXT:    v_readfirstlane_b32 s4, v4
 ; GFX10-NEXT:    v_readfirstlane_b32 s5, v5
 ; GFX10-NEXT:    v_readfirstlane_b32 s6, v6
 ; GFX10-NEXT:    v_readfirstlane_b32 s7, v7
 ; GFX10-NEXT:    ; return to shader part epilog
   %result = call <2 x i128> @llvm.sadd.sat.v2i128(<2 x i128> %lhs, <2 x i128> %rhs)
   ret <2 x i128> %result

llvm/test/CodeGen/AMDGPU/GlobalISel/ssubsat.ll

 ; GFX6-NEXT:    v_mov_b32_e32 v0, s2
 ; GFX6-NEXT:    s_subb_u32 s11, s3, s7
 ; GFX6-NEXT:    v_mov_b32_e32 v1, s3
 ; GFX6-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
 ; GFX6-NEXT:    v_cmp_lt_i64_e32 vcc, s[10:11], v[0:1]
 ; GFX6-NEXT:    v_cmp_gt_u64_e64 s[0:1], s[4:5], 0
 ; GFX6-NEXT:    v_cndmask_b32_e64 v3, 0, 1, vcc
 ; GFX6-NEXT:    v_cmp_eq_u64_e32 vcc, s[10:11], v[0:1]
-; GFX6-NEXT:    s_movk_i32 s2, 0x7f
+; GFX6-NEXT:    s_movk_i32 s12, 0x7f
 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, v3, v2, vcc
 ; GFX6-NEXT:    v_cndmask_b32_e64 v1, 0, 1, s[0:1]
 ; GFX6-NEXT:    v_cmp_gt_i64_e64 s[0:1], s[6:7], 0
 ; GFX6-NEXT:    v_cmp_eq_u64_e64 vcc, s[6:7], 0
-; GFX6-NEXT:    s_sub_i32 s6, s2, 64
-; GFX6-NEXT:    s_sub_i32 s4, 64, s2
-; GFX6-NEXT:    s_cmp_lt_u32 s2, 64
-; GFX6-NEXT:    s_cselect_b32 s12, 1, 0
-; GFX6-NEXT:    s_cmp_eq_u32 s2, 0
+; GFX6-NEXT:    s_sub_i32 s6, s12, 64
+; GFX6-NEXT:    s_sub_i32 s4, 64, s12
+; GFX6-NEXT:    s_cmp_lt_u32 s12, 64
 ; GFX6-NEXT:    s_cselect_b32 s13, 1, 0
-; GFX6-NEXT:    v_cndmask_b32_e64 v2, 0, 1, s[0:1]
-; GFX6-NEXT:    s_ashr_i64 s[0:1], s[10:11], s2
-; GFX6-NEXT:    s_lshr_b64 s[2:3], s[8:9], s2
+; GFX6-NEXT:    s_lshr_b64 s[2:3], s[8:9], s12
 ; GFX6-NEXT:    s_lshl_b64 s[4:5], s[10:11], s4
+; GFX6-NEXT:    v_cndmask_b32_e64 v2, 0, 1, s[0:1]
 ; GFX6-NEXT:    s_or_b64 s[2:3], s[2:3], s[4:5]
 ; GFX6-NEXT:    s_ashr_i32 s4, s11, 31
+; GFX6-NEXT:    s_ashr_i64 s[0:1], s[10:11], s12
 ; GFX6-NEXT:    s_ashr_i64 s[6:7], s[10:11], s6
-; GFX6-NEXT:    s_and_b32 s12, s12, 1
-; GFX6-NEXT:    s_cmp_lg_u32 s12, 0
+; GFX6-NEXT:    s_and_b32 s13, s13, 1
+; GFX6-NEXT:    s_cmp_lg_u32 s13, 0
 ; GFX6-NEXT:    s_cselect_b64 s[2:3], s[2:3], s[6:7]
-; GFX6-NEXT:    s_and_b32 s6, s13, 1
-; GFX6-NEXT:    s_cmp_lg_u32 s6, 0
+; GFX6-NEXT:    s_cmp_eq_u32 s12, 0
 ; GFX6-NEXT:    s_cselect_b64 s[2:3], s[8:9], s[2:3]
 ; GFX6-NEXT:    s_mov_b32 s5, s4
-; GFX6-NEXT:    s_cmp_lg_u32 s12, 0
+; GFX6-NEXT:    s_cmp_lg_u32 s13, 0
 ; GFX6-NEXT:    s_cselect_b64 s[0:1], s[0:1], s[4:5]
 ; GFX6-NEXT:    s_add_u32 s2, s2, 0
 ; GFX6-NEXT:    s_cselect_b32 s4, 1, 0
 ; GFX6-NEXT:    s_and_b32 s4, s4, 1
 ; GFX6-NEXT:    s_cmp_lg_u32 s4, 0
 ; GFX6-NEXT:    s_addc_u32 s3, s3, 0
 ; GFX6-NEXT:    s_cselect_b32 s4, 1, 0
 ; GFX6-NEXT:    s_and_b32 s4, s4, 1

 ; GFX8-NEXT:    s_and_b32 s0, 1, s2
 ; GFX8-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
 ; GFX8-NEXT:    v_cmp_ne_u32_e64 vcc, 0, s0
 ; GFX8-NEXT:    v_cmp_gt_u64_e64 s[0:1], s[4:5], 0
 ; GFX8-NEXT:    s_cmp_eq_u64 s[6:7], 0
 ; GFX8-NEXT:    v_cndmask_b32_e64 v1, 0, 1, s[0:1]
 ; GFX8-NEXT:    v_cmp_gt_i64_e64 s[0:1], s[6:7], 0
 ; GFX8-NEXT:    s_cselect_b32 s2, 1, 0
+; GFX8-NEXT:    s_movk_i32 s12, 0x7f
 ; GFX8-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; GFX8-NEXT:    v_cndmask_b32_e64 v2, 0, 1, s[0:1]
 ; GFX8-NEXT:    s_and_b32 s0, 1, s2
-; GFX8-NEXT:    s_movk_i32 s2, 0x7f
-; GFX8-NEXT:    s_sub_i32 s6, s2, 64
-; GFX8-NEXT:    s_sub_i32 s4, 64, s2
-; GFX8-NEXT:    s_cmp_lt_u32 s2, 64
-; GFX8-NEXT:    s_cselect_b32 s12, 1, 0
-; GFX8-NEXT:    s_cmp_eq_u32 s2, 0
+; GFX8-NEXT:    s_sub_i32 s6, s12, 64
+; GFX8-NEXT:    s_sub_i32 s4, 64, s12
+; GFX8-NEXT:    s_cmp_lt_u32 s12, 64
 ; GFX8-NEXT:    s_cselect_b32 s13, 1, 0
-; GFX8-NEXT:    v_cmp_ne_u32_e64 vcc, 0, s0
-; GFX8-NEXT:    s_ashr_i64 s[0:1], s[10:11], s2
-; GFX8-NEXT:    s_lshr_b64 s[2:3], s[8:9], s2
+; GFX8-NEXT:    s_lshr_b64 s[2:3], s[8:9], s12
 ; GFX8-NEXT:    s_lshl_b64 s[4:5], s[10:11], s4
+; GFX8-NEXT:    v_cmp_ne_u32_e64 vcc, 0, s0
 ; GFX8-NEXT:    s_or_b64 s[2:3], s[2:3], s[4:5]
 ; GFX8-NEXT:    s_ashr_i32 s4, s11, 31
+; GFX8-NEXT:    s_ashr_i64 s[0:1], s[10:11], s12
 ; GFX8-NEXT:    s_ashr_i64 s[6:7], s[10:11], s6
-; GFX8-NEXT:    s_and_b32 s12, s12, 1
-; GFX8-NEXT:    s_cmp_lg_u32 s12, 0
+; GFX8-NEXT:    s_and_b32 s13, s13, 1
+; GFX8-NEXT:    s_cmp_lg_u32 s13, 0
 ; GFX8-NEXT:    s_cselect_b64 s[2:3], s[2:3], s[6:7]
-; GFX8-NEXT:    s_and_b32 s6, s13, 1
-; GFX8-NEXT:    s_cmp_lg_u32 s6, 0
+; GFX8-NEXT:    s_cmp_eq_u32 s12, 0
 ; GFX8-NEXT:    s_cselect_b64 s[2:3], s[8:9], s[2:3]
 ; GFX8-NEXT:    s_mov_b32 s5, s4
-; GFX8-NEXT:    s_cmp_lg_u32 s12, 0
+; GFX8-NEXT:    s_cmp_lg_u32 s13, 0
 ; GFX8-NEXT:    s_cselect_b64 s[0:1], s[0:1], s[4:5]
 ; GFX8-NEXT:    s_add_u32 s2, s2, 0
 ; GFX8-NEXT:    s_cselect_b32 s4, 1, 0
 ; GFX8-NEXT:    s_and_b32 s4, s4, 1
 ; GFX8-NEXT:    s_cmp_lg_u32 s4, 0
 ; GFX8-NEXT:    s_addc_u32 s3, s3, 0
 ; GFX8-NEXT:    s_cselect_b32 s4, 1, 0
 ; GFX8-NEXT:    s_and_b32 s4, s4, 1

 ; GFX9-NEXT:    s_and_b32 s0, 1, s2
 ; GFX9-NEXT:    v_cndmask_b32_e64 v0, 0, 1, vcc
 ; GFX9-NEXT:    v_cmp_ne_u32_e64 vcc, 0, s0
 ; GFX9-NEXT:    v_cmp_gt_u64_e64 s[0:1], s[4:5], 0
 ; GFX9-NEXT:    s_cmp_eq_u64 s[6:7], 0
 ; GFX9-NEXT:    v_cndmask_b32_e64 v1, 0, 1, s[0:1]
 ; GFX9-NEXT:    v_cmp_gt_i64_e64 s[0:1], s[6:7], 0
 ; GFX9-NEXT:    s_cselect_b32 s2, 1, 0
+; GFX9-NEXT:    s_movk_i32 s12, 0x7f
 ; GFX9-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; GFX9-NEXT:    v_cndmask_b32_e64 v2, 0, 1, s[0:1]
 ; GFX9-NEXT:    s_and_b32 s0, 1, s2
-; GFX9-NEXT:    s_movk_i32 s2, 0x7f
-; GFX9-NEXT:    s_sub_i32 s6, s2, 64
-; GFX9-NEXT:    s_sub_i32 s4, 64, s2
-; GFX9-NEXT:    s_cmp_lt_u32 s2, 64
-; GFX9-NEXT:    s_cselect_b32 s12, 1, 0
-; GFX9-NEXT:    s_cmp_eq_u32 s2, 0
+; GFX9-NEXT:    s_sub_i32 s6, s12, 64
+; GFX9-NEXT:    s_sub_i32 s4, 64, s12
+; GFX9-NEXT:    s_cmp_lt_u32 s12, 64
 ; GFX9-NEXT:    s_cselect_b32 s13, 1, 0
-; GFX9-NEXT:    v_cmp_ne_u32_e64 vcc, 0, s0
-; GFX9-NEXT:    s_ashr_i64 s[0:1], s[10:11], s2
-; GFX9-NEXT:    s_lshr_b64 s[2:3], s[8:9], s2
+; GFX9-NEXT:    s_lshr_b64 s[2:3], s[8:9], s12
 ; GFX9-NEXT:    s_lshl_b64 s[4:5], s[10:11], s4
+; GFX9-NEXT:    v_cmp_ne_u32_e64 vcc, 0, s0
 ; GFX9-NEXT:    s_or_b64 s[2:3], s[2:3], s[4:5]
 ; GFX9-NEXT:    s_ashr_i32 s4, s11, 31
+; GFX9-NEXT:    s_ashr_i64 s[0:1], s[10:11], s12
 ; GFX9-NEXT:    s_ashr_i64 s[6:7], s[10:11], s6
-; GFX9-NEXT:    s_and_b32 s12, s12, 1
-; GFX9-NEXT:    s_cmp_lg_u32 s12, 0
+; GFX9-NEXT:    s_and_b32 s13, s13, 1
+; GFX9-NEXT:    s_cmp_lg_u32 s13, 0
 ; GFX9-NEXT:    s_cselect_b64 s[2:3], s[2:3], s[6:7]
-; GFX9-NEXT:    s_and_b32 s6, s13, 1
-; GFX9-NEXT:    s_cmp_lg_u32 s6, 0
+; GFX9-NEXT:    s_cmp_eq_u32 s12, 0
 ; GFX9-NEXT:    s_cselect_b64 s[2:3], s[8:9], s[2:3]
 ; GFX9-NEXT:    s_mov_b32 s5, s4
-; GFX9-NEXT:    s_cmp_lg_u32 s12, 0
+; GFX9-NEXT:    s_cmp_lg_u32 s13, 0
 ; GFX9-NEXT:    s_cselect_b64 s[0:1], s[0:1], s[4:5]
 ; GFX9-NEXT:    s_add_u32 s2, s2, 0
 ; GFX9-NEXT:    s_cselect_b32 s4, 1, 0
 ; GFX9-NEXT:    s_and_b32 s4, s4, 1
 ; GFX9-NEXT:    s_cmp_lg_u32 s4, 0
 ; GFX9-NEXT:    s_addc_u32 s3, s3, 0
 ; GFX9-NEXT:    s_cselect_b32 s4, 1, 0
 ; GFX9-NEXT:    s_and_b32 s4, s4, 1

 ; GFX10-NEXT:    s_cselect_b32 s0, 1, 0
 ; GFX10-NEXT:    v_mov_b32_e32 v4, s11
 ; GFX10-NEXT:    s_and_b32 s0, 1, s0
 ; GFX10-NEXT:    s_cmp_eq_u64 s[6:7], 0
 ; GFX10-NEXT:    v_cmp_ne_u32_e64 vcc_lo, 0, s0
 ; GFX10-NEXT:    v_cmp_gt_u64_e64 s0, s[4:5], 0
 ; GFX10-NEXT:    v_cndmask_b32_e64 v1, 0, 1, s1
 ; GFX10-NEXT:    s_cselect_b32 s1, 1, 0
-; GFX10-NEXT:    s_sub_i32 s13, s12, 64
-; GFX10-NEXT:    s_and_b32 s14, 1, s1
+; GFX10-NEXT:    s_sub_i32 s14, s12, 64
+; GFX10-NEXT:    s_and_b32 s13, 1, s1
 ; GFX10-NEXT:    s_sub_i32 s2, 64, s12
 ; GFX10-NEXT:    v_cndmask_b32_e32 v0, v1, v0, vcc_lo
 ; GFX10-NEXT:    v_cndmask_b32_e64 v1, 0, 1, s0
 ; GFX10-NEXT:    v_cmp_gt_i64_e64 s0, s[6:7], 0
 ; GFX10-NEXT:    s_cmp_lt_u32 s12, 64
-; GFX10-NEXT:    v_cmp_ne_u32_e64 vcc_lo, 0, s14
+; GFX10-NEXT:    v_cmp_ne_u32_e64 vcc_lo, 0, s13
 ; GFX10-NEXT:    s_cselect_b32 s15, 1, 0
-; GFX10-NEXT:    s_cmp_eq_u32 s12, 0
-; GFX10-NEXT:    s_cselect_b32 s16, 1, 0
+; GFX10-NEXT:    s_lshl_b64 s[2:3], s[10:11], s2
+; GFX10-NEXT:    s_and_b32 s15, s15, 1
 ; GFX10-NEXT:    v_cndmask_b32_e64 v2, 0, 1, s0
 ; GFX10-NEXT:    s_lshr_b64 s[0:1], s[8:9], s12
-; GFX10-NEXT:    s_lshl_b64 s[2:3], s[10:11], s2
 ; GFX10-NEXT:    s_ashr_i64 s[4:5], s[10:11], s12
-; GFX10-NEXT:    s_and_b32 s12, s15, 1
 ; GFX10-NEXT:    s_or_b64 s[0:1], s[0:1], s[2:3]
 ; GFX10-NEXT:    s_ashr_i32 s2, s11, 31
-; GFX10-NEXT:    s_ashr_i64 s[6:7], s[10:11], s13
-; GFX10-NEXT:    s_cmp_lg_u32 s12, 0
+; GFX10-NEXT:    s_ashr_i64 s[6:7], s[10:11], s14
+; GFX10-NEXT:    s_cmp_lg_u32 s15, 0
 ; GFX10-NEXT:    s_mov_b32 s3, s2
 ; GFX10-NEXT:    s_cselect_b64 s[0:1], s[0:1], s[6:7]
-; GFX10-NEXT:    s_and_b32 s6, s16, 1
+; GFX10-NEXT:    s_cmp_eq_u32 s12, 0
 ; GFX10-NEXT:    v_cndmask_b32_e32 v1, v2, v1, vcc_lo
-; GFX10-NEXT:    s_cmp_lg_u32 s6, 0
-; GFX10-NEXT:    v_mov_b32_e32 v2, s9
 ; GFX10-NEXT:    s_cselect_b64 s[0:1], s[8:9], s[0:1]
-; GFX10-NEXT:    s_cmp_lg_u32 s12, 0
-; GFX10-NEXT:    v_xor_b32_e32 v0, v1, v0
+; GFX10-NEXT:    s_cmp_lg_u32 s15, 0
+; GFX10-NEXT:    v_mov_b32_e32 v2, s9
 ; GFX10-NEXT:    s_cselect_b64 s[2:3], s[4:5], s[2:3]
 ; GFX10-NEXT:    s_add_u32 s0, s0, 0
 ; GFX10-NEXT:    s_cselect_b32 s4, 1, 0
-; GFX10-NEXT:    v_mov_b32_e32 v1, s8
+; GFX10-NEXT:    v_xor_b32_e32 v0, v1, v0
 ; GFX10-NEXT:    s_and_b32 s4, s4, 1
-; GFX10-NEXT:    v_and_b32_e32 v0, 1, v0
+; GFX10-NEXT:    v_mov_b32_e32 v1, s8
 ; GFX10-NEXT:    s_cmp_lg_u32 s4, 0
 ; GFX10-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX10-NEXT:    s_cselect_b32 s4, 1, 0
-; GFX10-NEXT:    v_cmp_ne_u32_e32 vcc_lo, 0, v0
+; GFX10-NEXT:    v_and_b32_e32 v0, 1, v0
 ; GFX10-NEXT:    s_and_b32 s4, s4, 1
 ; GFX10-NEXT:    s_cmp_lg_u32 s4, 0
 ; GFX10-NEXT:    s_addc_u32 s2, s2, 0
 ; GFX10-NEXT:    s_cselect_b32 s4, 1, 0
-; GFX10-NEXT:    v_cndmask_b32_e64 v0, v1, s0, vcc_lo
+; GFX10-NEXT:    v_cmp_ne_u32_e32 vcc_lo, 0, v0
 ; GFX10-NEXT:    s_and_b32 s4, s4, 1
-; GFX10-NEXT:    v_cndmask_b32_e64 v1, v2, s1, vcc_lo
 ; GFX10-NEXT:    s_cmp_lg_u32 s4, 0
-; GFX10-NEXT:    v_cndmask_b32_e64 v2, v3, s2, vcc_lo
 ; GFX10-NEXT:    s_addc_u32 s3, s3, 0x80000000
-; GFX10-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX10-NEXT:    v_cndmask_b32_e64 v0, v1, s0, vcc_lo
+; GFX10-NEXT:    v_cndmask_b32_e64 v1, v2, s1, vcc_lo
+; GFX10-NEXT:    v_cndmask_b32_e64 v2, v3, s2, vcc_lo
 ; GFX10-NEXT:    v_cndmask_b32_e64 v3, v4, s3, vcc_lo
+; GFX10-NEXT:    v_readfirstlane_b32 s0, v0
 ; GFX10-NEXT:    v_readfirstlane_b32 s1, v1
 ; GFX10-NEXT:    v_readfirstlane_b32 s2, v2
 ; GFX10-NEXT:    v_readfirstlane_b32 s3, v3
 ; GFX10-NEXT:    ; return to shader part epilog
   %result = call i128 @llvm.ssub.sat.i128(i128 %lhs, i128 %rhs)
   ret i128 %result
 }
 

 ; GFX6-NEXT:    s_cselect_b32 s19, 1, 0
 ; GFX6-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX6-NEXT:    s_and_b32 s19, s19, 1
 ; GFX6-NEXT:    v_mov_b32_e32 v2, s0
 ; GFX6-NEXT:    s_cmp_lg_u32 s19, 0
 ; GFX6-NEXT:    v_cmp_lt_u64_e32 vcc, s[16:17], v[2:3]
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s2
 ; GFX6-NEXT:    s_subb_u32 s19, s3, s11
-; GFX6-NEXT:    s_movk_i32 s20, 0x7f
 ; GFX6-NEXT:    v_mov_b32_e32 v1, s3
+; GFX6-NEXT:    s_movk_i32 s20, 0x7f
 ; GFX6-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
 ; GFX6-NEXT:    v_cmp_lt_i64_e32 vcc, s[18:19], v[0:1]
 ; GFX6-NEXT:    v_cmp_gt_u64_e64 s[0:1], s[8:9], 0
 ; GFX6-NEXT:    s_sub_i32 s21, s20, 64
 ; GFX6-NEXT:    s_sub_i32 s22, 64, s20
-; GFX6-NEXT:    s_cmp_lt_u32 s20, 64
 ; GFX6-NEXT:    v_cndmask_b32_e64 v3, 0, 1, vcc
 ; GFX6-NEXT:    v_cmp_eq_u64_e32 vcc, s[18:19], v[0:1]
 ; GFX6-NEXT:    v_cndmask_b32_e64 v1, 0, 1, s[0:1]
-; GFX6-NEXT:    s_cselect_b32 s23, 1, 0
-; GFX6-NEXT:    s_cmp_eq_u32 s20, 0
+; GFX6-NEXT:    s_cmp_lt_u32 s20, 64
 ; GFX6-NEXT:    v_cmp_gt_i64_e64 s[0:1], s[10:11], 0
-; GFX6-NEXT:    s_cselect_b32 s24, 1, 0
+; GFX6-NEXT:    s_cselect_b32 s23, 1, 0
 ; GFX6-NEXT:    s_lshr_b64 s[2:3], s[16:17], s20
 ; GFX6-NEXT:    s_lshl_b64 s[8:9], s[18:19], s22
 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, v3, v2, vcc
 ; GFX6-NEXT:    v_cndmask_b32_e64 v2, 0, 1, s[0:1]
 ; GFX6-NEXT:    v_cmp_eq_u64_e64 vcc, s[10:11], 0
 ; GFX6-NEXT:    s_or_b64 s[2:3], s[2:3], s[8:9]
 ; GFX6-NEXT:    s_ashr_i32 s8, s19, 31
 ; GFX6-NEXT:    s_ashr_i64 s[0:1], s[18:19], s20
 ; GFX6-NEXT:    s_ashr_i64 s[10:11], s[18:19], s21
 ; GFX6-NEXT:    s_and_b32 s23, s23, 1
 ; GFX6-NEXT:    s_cmp_lg_u32 s23, 0
 ; GFX6-NEXT:    s_cselect_b64 s[2:3], s[2:3], s[10:11]
-; GFX6-NEXT:    s_and_b32 s10, s24, 1
-; GFX6-NEXT:    s_cmp_lg_u32 s10, 0
+; GFX6-NEXT:    s_cmp_eq_u32 s20, 0
 ; GFX6-NEXT:    s_cselect_b64 s[2:3], s[16:17], s[2:3]
 ; GFX6-NEXT:    s_cmp_lg_u32 s23, 0
 ; GFX6-NEXT:    s_mov_b32 s9, s8
 ; GFX6-NEXT:    s_cselect_b64 s[0:1], s[0:1], s[8:9]
 ; GFX6-NEXT:    s_add_u32 s2, s2, 0
 ; GFX6-NEXT:    s_cselect_b32 s8, 1, 0
 ; GFX6-NEXT:    s_and_b32 s8, s8, 1
 ; GFX6-NEXT:    s_cmp_lg_u32 s8, 0

 ; GFX6-NEXT:    s_cmp_lg_u32 s3, 0
 ; GFX6-NEXT:    v_cmp_lt_u64_e32 vcc, s[0:1], v[2:3]
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s6
 ; GFX6-NEXT:    s_subb_u32 s3, s7, s15
 ; GFX6-NEXT:    v_mov_b32_e32 v1, s7
 ; GFX6-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
 ; GFX6-NEXT:    v_cmp_lt_i64_e32 vcc, s[2:3], v[0:1]
 ; GFX6-NEXT:    v_cmp_gt_u64_e64 s[4:5], s[12:13], 0
-; GFX6-NEXT:    s_cmp_lt_u32 s20, 64
 ; GFX6-NEXT:    v_cndmask_b32_e64 v3, 0, 1, vcc
 ; GFX6-NEXT:    v_cmp_eq_u64_e32 vcc, s[2:3], v[0:1]
 ; GFX6-NEXT:    v_cndmask_b32_e64 v1, 0, 1, s[4:5]
-; GFX6-NEXT:    s_cselect_b32 s12, 1, 0
-; GFX6-NEXT:    s_cmp_eq_u32 s20, 0
+; GFX6-NEXT:    s_cmp_lt_u32 s20, 64
 ; GFX6-NEXT:    v_cmp_gt_i64_e64 s[4:5], s[14:15], 0
-; GFX6-NEXT:    s_cselect_b32 s13, 1, 0
+; GFX6-NEXT:    s_cselect_b32 s12, 1, 0
 ; GFX6-NEXT:    s_lshr_b64 s[6:7], s[0:1], s20
 ; GFX6-NEXT:    s_lshl_b64 s[8:9], s[2:3], s22
 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, v3, v2, vcc
 ; GFX6-NEXT:    v_cndmask_b32_e64 v2, 0, 1, s[4:5]
 ; GFX6-NEXT:    s_or_b64 s[6:7], s[6:7], s[8:9]
 ; GFX6-NEXT:    s_ashr_i32 s8, s3, 31
 ; GFX6-NEXT:    s_ashr_i64 s[4:5], s[2:3], s20
 ; GFX6-NEXT:    s_ashr_i64 s[10:11], s[2:3], s21
 ; GFX6-NEXT:    s_and_b32 s12, s12, 1
 ; GFX6-NEXT:    s_cmp_lg_u32 s12, 0
 ; GFX6-NEXT:    s_cselect_b64 s[6:7], s[6:7], s[10:11]
-; GFX6-NEXT:    s_and_b32 s10, s13, 1
-; GFX6-NEXT:    s_cmp_lg_u32 s10, 0
+; GFX6-NEXT:    s_cmp_eq_u32 s20, 0
 ; GFX6-NEXT:    s_cselect_b64 s[6:7], s[0:1], s[6:7]
 ; GFX6-NEXT:    s_mov_b32 s9, s8
 ; GFX6-NEXT:    s_cmp_lg_u32 s12, 0
 ; GFX6-NEXT:    s_cselect_b64 s[4:5], s[4:5], s[8:9]
 ; GFX6-NEXT:    s_add_u32 s6, s6, 0
 ; GFX6-NEXT:    s_cselect_b32 s8, 1, 0
 ; GFX6-NEXT:    s_and_b32 s8, s8, 1
 ; GFX6-NEXT:    s_cmp_lg_u32 s8, 0

 ; GFX8-NEXT:    s_movk_i32 s20, 0x7f
 ; GFX8-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; GFX8-NEXT:    v_cndmask_b32_e64 v2, 0, 1, s[0:1]
 ; GFX8-NEXT:    s_and_b32 s0, 1, s2
 ; GFX8-NEXT:    s_sub_i32 s21, s20, 64
 ; GFX8-NEXT:    s_sub_i32 s22, 64, s20
 ; GFX8-NEXT:    s_cmp_lt_u32 s20, 64
 ; GFX8-NEXT:    s_cselect_b32 s23, 1, 0
-; GFX8-NEXT:    s_cmp_eq_u32 s20, 0
-; GFX8-NEXT:    s_cselect_b32 s24, 1, 0
 ; GFX8-NEXT:    s_lshr_b64 s[2:3], s[16:17], s20
 ; GFX8-NEXT:    s_lshl_b64 s[8:9], s[18:19], s22
 ; GFX8-NEXT:    v_cmp_ne_u32_e64 vcc, 0, s0
 ; GFX8-NEXT:    s_or_b64 s[2:3], s[2:3], s[8:9]
 ; GFX8-NEXT:    s_ashr_i32 s8, s19, 31
 ; GFX8-NEXT:    s_ashr_i64 s[0:1], s[18:19], s20
 ; GFX8-NEXT:    s_ashr_i64 s[10:11], s[18:19], s21
 ; GFX8-NEXT:    s_and_b32 s23, s23, 1
 ; GFX8-NEXT:    s_cmp_lg_u32 s23, 0
 ; GFX8-NEXT:    s_cselect_b64 s[2:3], s[2:3], s[10:11]
-; GFX8-NEXT:    s_and_b32 s10, s24, 1
-; GFX8-NEXT:    s_cmp_lg_u32 s10, 0
+; GFX8-NEXT:    s_cmp_eq_u32 s20, 0
 ; GFX8-NEXT:    s_cselect_b64 s[2:3], s[16:17], s[2:3]
 ; GFX8-NEXT:    s_cmp_lg_u32 s23, 0
 ; GFX8-NEXT:    s_mov_b32 s9, s8
 ; GFX8-NEXT:    s_cselect_b64 s[0:1], s[0:1], s[8:9]
 ; GFX8-NEXT:    s_add_u32 s2, s2, 0
 ; GFX8-NEXT:    s_cselect_b32 s8, 1, 0
 ; GFX8-NEXT:    s_and_b32 s8, s8, 1
 ; GFX8-NEXT:    s_cmp_lg_u32 s8, 0

 ; GFX8-NEXT:    v_cndmask_b32_e64 v1, 0, 1, s[4:5]
 ; GFX8-NEXT:    v_cmp_gt_i64_e64 s[4:5], s[14:15], 0
 ; GFX8-NEXT:    s_cselect_b32 s6, 1, 0
 ; GFX8-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; GFX8-NEXT:    v_cndmask_b32_e64 v2, 0, 1, s[4:5]
 ; GFX8-NEXT:    s_and_b32 s4, 1, s6
 ; GFX8-NEXT:    s_cmp_lt_u32 s20, 64
 ; GFX8-NEXT:    s_cselect_b32 s12, 1, 0
-; GFX8-NEXT:    s_cmp_eq_u32 s20, 0
-; GFX8-NEXT:    s_cselect_b32 s13, 1, 0
 ; GFX8-NEXT:    s_lshr_b64 s[6:7], s[0:1], s20
 ; GFX8-NEXT:    s_lshl_b64 s[8:9], s[2:3], s22
 ; GFX8-NEXT:    v_cmp_ne_u32_e64 vcc, 0, s4
 ; GFX8-NEXT:    s_or_b64 s[6:7], s[6:7], s[8:9]
 ; GFX8-NEXT:    s_ashr_i32 s8, s3, 31
 ; GFX8-NEXT:    s_ashr_i64 s[4:5], s[2:3], s20
 ; GFX8-NEXT:    s_ashr_i64 s[10:11], s[2:3], s21
 ; GFX8-NEXT:    s_and_b32 s12, s12, 1
 ; GFX8-NEXT:    s_cmp_lg_u32 s12, 0
 ; GFX8-NEXT:    s_cselect_b64 s[6:7], s[6:7], s[10:11]
-; GFX8-NEXT:    s_and_b32 s10, s13, 1
-; GFX8-NEXT:    s_cmp_lg_u32 s10, 0
+; GFX8-NEXT:    s_cmp_eq_u32 s20, 0
 ; GFX8-NEXT:    s_cselect_b64 s[6:7], s[0:1], s[6:7]
 ; GFX8-NEXT:    s_mov_b32 s9, s8
 ; GFX8-NEXT:    s_cmp_lg_u32 s12, 0
 ; GFX8-NEXT:    s_cselect_b64 s[4:5], s[4:5], s[8:9]
 ; GFX8-NEXT:    s_add_u32 s6, s6, 0
 ; GFX8-NEXT:    s_cselect_b32 s8, 1, 0
 ; GFX8-NEXT:    s_and_b32 s8, s8, 1
 ; GFX8-NEXT:    s_cmp_lg_u32 s8, 0

 ; GFX9-NEXT:    s_movk_i32 s20, 0x7f
 ; GFX9-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; GFX9-NEXT:    v_cndmask_b32_e64 v2, 0, 1, s[0:1]
 ; GFX9-NEXT:    s_and_b32 s0, 1, s2
 ; GFX9-NEXT:    s_sub_i32 s21, s20, 64
 ; GFX9-NEXT:    s_sub_i32 s22, 64, s20
 ; GFX9-NEXT:    s_cmp_lt_u32 s20, 64
 ; GFX9-NEXT:    s_cselect_b32 s23, 1, 0
-; GFX9-NEXT:    s_cmp_eq_u32 s20, 0
-; GFX9-NEXT:    s_cselect_b32 s24, 1, 0
 ; GFX9-NEXT:    s_lshr_b64 s[2:3], s[16:17], s20
 ; GFX9-NEXT:    s_lshl_b64 s[8:9], s[18:19], s22
 ; GFX9-NEXT:    v_cmp_ne_u32_e64 vcc, 0, s0
 ; GFX9-NEXT:    s_or_b64 s[2:3], s[2:3], s[8:9]
 ; GFX9-NEXT:    s_ashr_i32 s8, s19, 31
 ; GFX9-NEXT:    s_ashr_i64 s[0:1], s[18:19], s20
 ; GFX9-NEXT:    s_ashr_i64 s[10:11], s[18:19], s21
 ; GFX9-NEXT:    s_and_b32 s23, s23, 1
 ; GFX9-NEXT:    s_cmp_lg_u32 s23, 0
 ; GFX9-NEXT:    s_cselect_b64 s[2:3], s[2:3], s[10:11]
-; GFX9-NEXT:    s_and_b32 s10, s24, 1
-; GFX9-NEXT:    s_cmp_lg_u32 s10, 0
+; GFX9-NEXT:    s_cmp_eq_u32 s20, 0
 ; GFX9-NEXT:    s_cselect_b64 s[2:3], s[16:17], s[2:3]
 ; GFX9-NEXT:    s_cmp_lg_u32 s23, 0
 ; GFX9-NEXT:    s_mov_b32 s9, s8
 ; GFX9-NEXT:    s_cselect_b64 s[0:1], s[0:1], s[8:9]
 ; GFX9-NEXT:    s_add_u32 s2, s2, 0
 ; GFX9-NEXT:    s_cselect_b32 s8, 1, 0
 ; GFX9-NEXT:    s_and_b32 s8, s8, 1
 ; GFX9-NEXT:    s_cmp_lg_u32 s8, 0

 ; GFX9-NEXT:    v_cndmask_b32_e64 v1, 0, 1, s[4:5]
 ; GFX9-NEXT:    v_cmp_gt_i64_e64 s[4:5], s[14:15], 0
 ; GFX9-NEXT:    s_cselect_b32 s6, 1, 0
 ; GFX9-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; GFX9-NEXT:    v_cndmask_b32_e64 v2, 0, 1, s[4:5]
 ; GFX9-NEXT:    s_and_b32 s4, 1, s6
 ; GFX9-NEXT:    s_cmp_lt_u32 s20, 64
 ; GFX9-NEXT:    s_cselect_b32 s12, 1, 0
-; GFX9-NEXT:    s_cmp_eq_u32 s20, 0
-; GFX9-NEXT:    s_cselect_b32 s13, 1, 0
 ; GFX9-NEXT:    s_lshr_b64 s[6:7], s[0:1], s20
 ; GFX9-NEXT:    s_lshl_b64 s[8:9], s[2:3], s22
 ; GFX9-NEXT:    v_cmp_ne_u32_e64 vcc, 0, s4
 ; GFX9-NEXT:    s_or_b64 s[6:7], s[6:7], s[8:9]
 ; GFX9-NEXT:    s_ashr_i32 s8, s3, 31
 ; GFX9-NEXT:    s_ashr_i64 s[4:5], s[2:3], s20
 ; GFX9-NEXT:    s_ashr_i64 s[10:11], s[2:3], s21
 ; GFX9-NEXT:    s_and_b32 s12, s12, 1
 ; GFX9-NEXT:    s_cmp_lg_u32 s12, 0
 ; GFX9-NEXT:    s_cselect_b64 s[6:7], s[6:7], s[10:11]
-; GFX9-NEXT:    s_and_b32 s10, s13, 1
-; GFX9-NEXT:    s_cmp_lg_u32 s10, 0
+; GFX9-NEXT:    s_cmp_eq_u32 s20, 0
 ; GFX9-NEXT:    s_cselect_b64 s[6:7], s[0:1], s[6:7]
 ; GFX9-NEXT:    s_mov_b32 s9, s8
 ; GFX9-NEXT:    s_cmp_lg_u32 s12, 0
 ; GFX9-NEXT:    s_cselect_b64 s[4:5], s[4:5], s[8:9]
 ; GFX9-NEXT:    s_add_u32 s6, s6, 0
 ; GFX9-NEXT:    s_cselect_b32 s8, 1, 0
 ; GFX9-NEXT:    s_and_b32 s8, s8, 1
 ; GFX9-NEXT:    s_cmp_lg_u32 s8, 0

 ; GFX10-NEXT:    s_mov_b32 s46, s0
 ; GFX10-NEXT:    s_and_b32 s17, s17, 1
 ; GFX10-NEXT:    s_mov_b32 s47, s1
 ; GFX10-NEXT:    s_cmp_lg_u32 s17, 0
 ; GFX10-NEXT:    s_subb_u32 s29, s1, s9
 ; GFX10-NEXT:    s_cselect_b32 s18, 1, 0
 ; GFX10-NEXT:    v_cmp_lt_u64_e64 s0, s[28:29], s[46:47]
 ; GFX10-NEXT:    s_and_b32 s18, s18, 1
+; GFX10-NEXT:    v_cmp_gt_u64_e64 s1, s[8:9], 0
 ; GFX10-NEXT:    s_cmp_lg_u32 s18, 0
 ; GFX10-NEXT:    s_subb_u32 s30, s2, s10
 ; GFX10-NEXT:    s_cselect_b32 s19, 1, 0
 ; GFX10-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s0
 ; GFX10-NEXT:    s_and_b32 s19, s19, 1
 ; GFX10-NEXT:    s_cmp_lg_u32 s19, 0
 ; GFX10-NEXT:    s_subb_u32 s31, s3, s11
 ; GFX10-NEXT:    v_cmp_lt_i64_e64 s0, s[30:31], s[2:3]
 ; GFX10-NEXT:    s_cmp_eq_u64 s[30:31], s[2:3]
-; GFX10-NEXT:    v_cmp_gt_u64_e64 s2, s[8:9], 0
+; GFX10-NEXT:    v_mov_b32_e32 v3, s31
 ; GFX10-NEXT:    s_cselect_b32 s20, 1, 0
 ; GFX10-NEXT:    v_cndmask_b32_e64 v1, 0, 1, s0
 ; GFX10-NEXT:    s_and_b32 s0, 1, s20
 ; GFX10-NEXT:    s_cmp_eq_u64 s[10:11], 0
-; GFX10-NEXT:    s_movk_i32 s20, 0x7f
-; GFX10-NEXT:    s_cselect_b32 s1, 1, 0
-; GFX10-NEXT:    v_cndmask_b32_e64 v2, 0, 1, s2
-; GFX10-NEXT:    v_cmp_gt_i64_e64 s2, s[10:11], 0
 ; GFX10-NEXT:    v_cmp_ne_u32_e64 vcc_lo, 0, s0
-; GFX10-NEXT:    s_and_b32 s1, 1, s1
+; GFX10-NEXT:    s_cselect_b32 s0, 1, 0
+; GFX10-NEXT:    s_movk_i32 s20, 0x7f
+; GFX10-NEXT:    s_and_b32 s0, 1, s0
 ; GFX10-NEXT:    s_sub_i32 s21, s20, 64
+; GFX10-NEXT:    v_cndmask_b32_e32 v0, v1, v0, vcc_lo
+; GFX10-NEXT:    v_cndmask_b32_e64 v1, 0, 1, s1
+; GFX10-NEXT:    v_cmp_gt_i64_e64 s1, s[10:11], 0
 ; GFX10-NEXT:    s_sub_i32 s22, 64, s20
 ; GFX10-NEXT:    s_cmp_lt_u32 s20, 64
-; GFX10-NEXT:    v_cndmask_b32_e32 v0, v1, v0, vcc_lo
+; GFX10-NEXT:    v_cmp_ne_u32_e64 vcc_lo, 0, s0
 ; GFX10-NEXT:    s_cselect_b32 s10, 1, 0
-; GFX10-NEXT:    s_cmp_eq_u32 s20, 0
-; GFX10-NEXT:    v_cndmask_b32_e64 v3, 0, 1, s2
-; GFX10-NEXT:    s_cselect_b32 s23, 1, 0
-; GFX10-NEXT:    v_cmp_ne_u32_e64 vcc_lo, 0, s1
-; GFX10-NEXT:    s_lshr_b64 s[0:1], s[28:29], s20
 ; GFX10-NEXT:    s_lshl_b64 s[2:3], s[30:31], s22
-; GFX10-NEXT:    s_and_b32 s24, s10, 1
+; GFX10-NEXT:    v_cndmask_b32_e64 v2, 0, 1, s1
+; GFX10-NEXT:    s_lshr_b64 s[0:1], s[28:29], s20
+; GFX10-NEXT:    s_and_b32 s23, s10, 1
 ; GFX10-NEXT:    s_or_b64 s[0:1], s[0:1], s[2:3]
 ; GFX10-NEXT:    s_ashr_i32 s2, s31, 31
 ; GFX10-NEXT:    s_ashr_i64 s[8:9], s[30:31], s20
 ; GFX10-NEXT:    s_ashr_i64 s[10:11], s[30:31], s21
-; GFX10-NEXT:    s_cmp_lg_u32 s24, 0
+; GFX10-NEXT:    s_cmp_lg_u32 s23, 0
 ; GFX10-NEXT:    s_mov_b32 s3, s2
 ; GFX10-NEXT:    s_cselect_b64 s[0:1], s[0:1], s[10:11]
-; GFX10-NEXT:    s_and_b32 s10, s23, 1
-; GFX10-NEXT:    v_cndmask_b32_e32 v1, v3, v2, vcc_lo
-; GFX10-NEXT:    s_cmp_lg_u32 s10, 0
-; GFX10-NEXT:    v_mov_b32_e32 v2, s29
+; GFX10-NEXT:    s_cmp_eq_u32 s20, 0
+; GFX10-NEXT:    v_cndmask_b32_e32 v1, v2, v1, vcc_lo
 ; GFX10-NEXT:    s_cselect_b64 s[0:1], s[28:29], s[0:1]
-; GFX10-NEXT:    s_cmp_lg_u32 s24, 0
-; GFX10-NEXT:    v_xor_b32_e32 v0, v1, v0
+; GFX10-NEXT:    s_cmp_lg_u32 s23, 0
+; GFX10-NEXT:    s_brev_b32 s23, 1
 ; GFX10-NEXT:    s_cselect_b64 s[2:3], s[8:9], s[2:3]
 ; GFX10-NEXT:    s_add_u32 s0, s0, 0
 ; GFX10-NEXT:    s_cselect_b32 s8, 1, 0
-; GFX10-NEXT:    v_mov_b32_e32 v1, s28
+; GFX10-NEXT:    v_xor_b32_e32 v0, v1, v0
 ; GFX10-NEXT:    s_and_b32 s8, s8, 1
-; GFX10-NEXT:    v_and_b32_e32 v0, 1, v0
+; GFX10-NEXT:    v_mov_b32_e32 v1, s28
 ; GFX10-NEXT:    s_cmp_lg_u32 s8, 0
-; GFX10-NEXT:    s_brev_b32 s23, 1
+; GFX10-NEXT:    v_mov_b32_e32 v2, s29
 ; GFX10-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX10-NEXT:    s_cselect_b32 s8, 1, 0
-; GFX10-NEXT:    v_cmp_ne_u32_e32 vcc_lo, 0, v0
+; GFX10-NEXT:    v_and_b32_e32 v0, 1, v0
 ; GFX10-NEXT:    s_and_b32 s8, s8, 1
-; GFX10-NEXT:    v_mov_b32_e32 v3, s31
 ; GFX10-NEXT:    s_cmp_lg_u32 s8, 0
 ; GFX10-NEXT:    s_addc_u32 s2, s2, 0
 ; GFX10-NEXT:    s_cselect_b32 s8, 1, 0
-; GFX10-NEXT:    v_cndmask_b32_e64 v0, v1, s0, vcc_lo
+; GFX10-NEXT:    v_cmp_ne_u32_e32 vcc_lo, 0, v0
 ; GFX10-NEXT:    s_and_b32 s8, s8, 1
-; GFX10-NEXT:    v_cndmask_b32_e64 v1, v2, s1, vcc_lo
 ; GFX10-NEXT:    s_cmp_lg_u32 s8, 0
-; GFX10-NEXT:    v_mov_b32_e32 v2, s30
 ; GFX10-NEXT:    s_addc_u32 s3, s3, s23
+; GFX10-NEXT:    v_cndmask_b32_e64 v0, v1, s0, vcc_lo
 ; GFX10-NEXT:    s_sub_u32 s0, s4, s12
-; GFX10-NEXT:    s_cselect_b32 s1, 1, 0
+; GFX10-NEXT:    s_cselect_b32 s8, 1, 0
+; GFX10-NEXT:    v_cndmask_b32_e64 v1, v2, s1, vcc_lo
+; GFX10-NEXT:    s_and_b32 s8, s8, 1
+; GFX10-NEXT:    v_mov_b32_e32 v2, s30
+; GFX10-NEXT:    s_cmp_lg_u32 s8, 0
 ; GFX10-NEXT:    v_cndmask_b32_e64 v3, v3, s3, vcc_lo
-; GFX10-NEXT:    s_and_b32 s1, s1, 1
-; GFX10-NEXT:    v_cndmask_b32_e64 v2, v2, s2, vcc_lo
-; GFX10-NEXT:    s_cmp_lg_u32 s1, 0
 ; GFX10-NEXT:    s_subb_u32 s1, s5, s13
 ; GFX10-NEXT:    s_cselect_b32 s8, 1, 0
-; GFX10-NEXT:    v_cmp_lt_u64_e64 s3, s[0:1], s[4:5]
+; GFX10-NEXT:    v_cmp_lt_u64_e64 s4, s[0:1], s[4:5]
 ; GFX10-NEXT:    s_and_b32 s8, s8, 1
+; GFX10-NEXT:    v_cndmask_b32_e64 v2, v2, s2, vcc_lo
 ; GFX10-NEXT:    s_cmp_lg_u32 s8, 0
+; GFX10-NEXT:    v_cmp_gt_u64_e64 s3, s[12:13], 0
 ; GFX10-NEXT:    s_subb_u32 s8, s6, s14
 ; GFX10-NEXT:    s_cselect_b32 s9, 1, 0
-; GFX10-NEXT:    v_cndmask_b32_e64 v4, 0, 1, s3
+; GFX10-NEXT:    v_cndmask_b32_e64 v4, 0, 1, s4
 ; GFX10-NEXT:    s_and_b32 s9, s9, 1
 ; GFX10-NEXT:    v_mov_b32_e32 v7, s8
 ; GFX10-NEXT:    s_cmp_lg_u32 s9, 0
 ; GFX10-NEXT:    s_subb_u32 s9, s7, s15
 ; GFX10-NEXT:    s_cmp_eq_u64 s[8:9], s[6:7]
-; GFX10-NEXT:    v_cmp_lt_i64_e64 s3, s[8:9], s[6:7]
+; GFX10-NEXT:    v_cmp_lt_i64_e64 s4, s[8:9], s[6:7]
 ; GFX10-NEXT:    s_cselect_b32 s2, 1, 0
 ; GFX10-NEXT:    v_mov_b32_e32 v8, s9
 ; GFX10-NEXT:    s_and_b32 s2, 1, s2
 ; GFX10-NEXT:    s_cmp_eq_u64 s[14:15], 0
 ; GFX10-NEXT:    v_cmp_ne_u32_e64 vcc_lo, 0, s2
-; GFX10-NEXT:    v_cmp_gt_u64_e64 s2, s[12:13], 0
-; GFX10-NEXT:    v_cndmask_b32_e64 v5, 0, 1, s3
-; GFX10-NEXT:    s_cselect_b32 s3, 1, 0
-; GFX10-NEXT:    s_and_b32 s16, 1, s3
-; GFX10-NEXT:    s_cmp_lt_u32 s20, 64
-; GFX10-NEXT:    v_cndmask_b32_e32 v4, v5, v4, vcc_lo
-; GFX10-NEXT:    v_cndmask_b32_e64 v5, 0, 1, s2
+; GFX10-NEXT:    s_cselect_b32 s2, 1, 0
+; GFX10-NEXT:    v_cndmask_b32_e64 v5, 0, 1, s4
+; GFX10-NEXT:    s_and_b32 s12, 1, s2
 ; GFX10-NEXT:    v_cmp_gt_i64_e64 s2, s[14:15], 0
+; GFX10-NEXT:    s_cmp_lt_u32 s20, 64
 ; GFX10-NEXT:    s_cselect_b32 s10, 1, 0
-; GFX10-NEXT:    s_cmp_eq_u32 s20, 0
-; GFX10-NEXT:    v_cmp_ne_u32_e64 vcc_lo, 0, s16
-; GFX10-NEXT:    s_cselect_b32 s12, 1, 0
-; GFX10-NEXT:    s_lshl_b64 s[4:5], s[8:9], s22
+; GFX10-NEXT:    v_cndmask_b32_e32 v4, v5, v4, vcc_lo
+; GFX10-NEXT:    v_cndmask_b32_e64 v5, 0, 1, s3
 ; GFX10-NEXT:    v_cndmask_b32_e64 v6, 0, 1, s2
 ; GFX10-NEXT:    s_lshr_b64 s[2:3], s[0:1], s20
+; GFX10-NEXT:    s_lshl_b64 s[4:5], s[8:9], s22
 ; GFX10-NEXT:    s_and_b32 s13, s10, 1
 ; GFX10-NEXT:    s_or_b64 s[2:3], s[2:3], s[4:5]
 ; GFX10-NEXT:    s_ashr_i32 s4, s9, 31
 ; GFX10-NEXT:    s_ashr_i64 s[6:7], s[8:9], s20
 ; GFX10-NEXT:    s_ashr_i64 s[10:11], s[8:9], s21
 ; GFX10-NEXT:    s_cmp_lg_u32 s13, 0
-; GFX10-NEXT:    s_mov_b32 s5, s4
+; GFX10-NEXT:    v_cmp_ne_u32_e64 vcc_lo, 0, s12
 ; GFX10-NEXT:    s_cselect_b64 s[2:3], s[2:3], s[10:11]
-; GFX10-NEXT:    s_and_b32 s10, s12, 1
-; GFX10-NEXT:    v_cndmask_b32_e32 v5, v6, v5, vcc_lo
-; GFX10-NEXT:    s_cmp_lg_u32 s10, 0
-; GFX10-NEXT:    v_mov_b32_e32 v6, s1
+; GFX10-NEXT:    s_cmp_eq_u32 s20, 0
+; GFX10-NEXT:    s_mov_b32 s5, s4
 ; GFX10-NEXT:    s_cselect_b64 s[2:3], s[0:1], s[2:3]
 ; GFX10-NEXT:    s_cmp_lg_u32 s13, 0
-; GFX10-NEXT:    v_xor_b32_e32 v4, v5, v4
+; GFX10-NEXT:    v_cndmask_b32_e32 v5, v6, v5, vcc_lo
 ; GFX10-NEXT:    s_cselect_b64 s[4:5], s[6:7], s[4:5]
 ; GFX10-NEXT:    s_add_u32 s2, s2, 0
 ; GFX10-NEXT:    s_cselect_b32 s6, 1, 0
-; GFX10-NEXT:    v_mov_b32_e32 v5, s0
+; GFX10-NEXT:    v_mov_b32_e32 v6, s1
 ; GFX10-NEXT:    s_and_b32 s6, s6, 1
-; GFX10-NEXT:    v_and_b32_e32 v4, 1, v4
+; GFX10-NEXT:    v_xor_b32_e32 v4, v5, v4
 ; GFX10-NEXT:    s_cmp_lg_u32 s6, 0
-; GFX10-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX10-NEXT:    v_mov_b32_e32 v5, s0
 ; GFX10-NEXT:    s_addc_u32 s3, s3, 0
 ; GFX10-NEXT:    s_cselect_b32 s6, 1, 0
-; GFX10-NEXT:    v_cmp_ne_u32_e32 vcc_lo, 0, v4
+; GFX10-NEXT:    v_and_b32_e32 v4, 1, v4
 ; GFX10-NEXT:    s_and_b32 s6, s6, 1
+; GFX10-NEXT:    v_readfirstlane_b32 s0, v0
 ; GFX10-NEXT:    s_cmp_lg_u32 s6, 0
 ; GFX10-NEXT:    s_addc_u32 s4, s4, 0
 ; GFX10-NEXT:    s_cselect_b32 s6, 1, 0
-; GFX10-NEXT:    v_cndmask_b32_e64 v4, v5, s2, vcc_lo
+; GFX10-NEXT:    v_cmp_ne_u32_e32 vcc_lo, 0, v4
 ; GFX10-NEXT:    s_and_b32 s6, s6, 1
-; GFX10-NEXT:    v_cndmask_b32_e64 v5, v6, s3, vcc_lo
 ; GFX10-NEXT:    s_cmp_lg_u32 s6, 0
-; GFX10-NEXT:    v_cndmask_b32_e64 v6, v7, s4, vcc_lo
 ; GFX10-NEXT:    s_addc_u32 s1, s5, s23
-; GFX10-NEXT:    v_readfirstlane_b32 s2, v2
+; GFX10-NEXT:    v_cndmask_b32_e64 v4, v5, s2, vcc_lo
+; GFX10-NEXT:    v_cndmask_b32_e64 v5, v6, s3, vcc_lo
+; GFX10-NEXT:    v_cndmask_b32_e64 v6, v7, s4, vcc_lo
 ; GFX10-NEXT:    v_cndmask_b32_e64 v7, v8, s1, vcc_lo
 ; GFX10-NEXT:    v_readfirstlane_b32 s1, v1
+; GFX10-NEXT:    v_readfirstlane_b32 s2, v2
 ; GFX10-NEXT:    v_readfirstlane_b32 s3, v3
 ; GFX10-NEXT:    v_readfirstlane_b32 s4, v4
 ; GFX10-NEXT:    v_readfirstlane_b32 s5, v5
 ; GFX10-NEXT:    v_readfirstlane_b32 s6, v6
 ; GFX10-NEXT:    v_readfirstlane_b32 s7, v7
 ; GFX10-NEXT:    ; return to shader part epilog
   %result = call <2 x i128> @llvm.ssub.sat.v2i128(<2 x i128> %lhs, <2 x i128> %rhs)
   ret <2 x i128> %result