[GISel] Add new combines for G_ADD

[mkitzan]

Summary

Patch adds new GICombineRules for G_ADD:

• G_ADD(x, G_SUB(y, x)) -> y
• G_ADD(G_SUB(y, x), x) -> y

Patch additionally adds new combine tests for AArch64 target for these new rules.

[tschuett]

Out of curiosity, is the first rule a win or canonicalisation?

[arsenm]

llvm/include/llvm/CodeGen/GlobalISel/CombinerHelper.h

322

APInt to allow it to work for wide integers?

llvm/test/CodeGen/AArch64/GlobalISel/combine-add.mir

86

Add an s128 case? This currently won't work for vectors, but it would be nice to handle a few vector tests too

llvm/test/CodeGen/AMDGPU/GlobalISel/extractelement.i128.ll

23–25

I don't understand where this regression came from (but isn't particularly important)

54–63

This is a much more interesting regression

[mkitzan]

In D87936#2282772, @tschuett wrote:

Out of curiosity, is the first rule a win or canonicalisation?

Primarily canonicalization, however depending on the target there can be perf wins. By eliminating the signed constant, there's a better chance the constant can be folded into the opcode as an immediate operand rather than being loaded into a register before being used (if the target supports immediate operands, otherwise nothing lost, eh?).

Assume hypothetical target only supports 2b immediate operands to arithmetic ops and registers are 8b. Here's an example where the combine is a perf win.

mov r1, -1
add r0, r1
---
sub r0, 1

llvm/include/llvm/CodeGen/GlobalISel/CombinerHelper.h

322

Make sense, will add

llvm/test/CodeGen/AArch64/GlobalISel/combine-add.mir

86

Will add an s128 and a vector test case for each of these

llvm/test/CodeGen/AMDGPU/GlobalISel/extractelement.i128.ll

54–63

I noticed regressions in this test file too. The combine causing them is the: G_ADD(x, y) -> G_OR(x, y) (iff x and y share no common bits). That combine rule is the cause of nearly all the AMDGPU test changes (with the exception of three or four tests).

[arsenm]

llvm/test/CodeGen/AMDGPU/GlobalISel/extractelement.i128.ll

54–63

So we need the equivalent utility for SelectionDAG::isBaseWithConstantOffset(

[mkitzan]

llvm/test/CodeGen/AMDGPU/GlobalISel/extractelement.i128.ll

54–63

Not quite sure where that fits into the implementation of the combine. The added GlobalISel/Utils.cpp:haveNoCommonBitsSet acts the same as SelectionDAG.pp:haveNoCommonBitsSet which is used to gate the DAG combine version (x + y) -> (x | y) iff x and y share no bits.

[mkitzan]

• Changed G_ADD(x, -cst) -> G_SUB(x, cst) to use APInt
• Added vector and s128 test cases (though some combines don't fire with these input types)

[arsenm]

llvm/test/CodeGen/AMDGPU/GlobalISel/extractelement.i128.ll

54–63

It's not for the combine itself, it's for all of the other code to deal with the side effect of it. In general this is a terrible combine because nobody expects adds to turn into ors. Without a utility function you can easily use to match adds that were turned into ors, I think it's generally worse to do this combine. I think having this utility function is a prerequisite for this

[mkitzan]

I can remove the G_ADD(x, y)->G_OR(x, y) combine if that's a blocker. Let me know if there's anything else specifically blocking the approval of this patch.

[mkitzan]

Updates:

• Rebased patch
• Removed G_ADD(x, y) -> G_OR(x, y) combine
• Exchanged my custom applyAddSubSameReg function for replaceSingleDefInstWithReg
• Add @aemerson and @foad to reviewers list

[arsenm]

I do think the add -> or has some value, but it should be separate with the helper function

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

2793–2798

Probably should use m_match here (maybe even have a negative constant matcher)

[mkitzan]

Good point about using mi_match, updated the diff.

Can work on the add -> or combine + helper function in a future patch. Better to not block committing the simpler combines.

[foad]

llvm/include/llvm/CodeGen/GlobalISel/CombinerHelper.h

435

Does SelectionDAG do this? I suppose it can be useful to canonicalize, but my gut feeling is that it would be better to do the converse, i.e. canonicalize G_SUB(x, cst) -> G_ADD(x, -cst). That way you are canonicalizing on using G_ADD instead of G_SUB wherever possible, which feels more useful than using positive constants instead of negative constants wherever possible. E.g. if you're writing a pattern to match a "reg + simm" addressing mode you would only need to match G_ADD.

[mkitzan]

llvm/include/llvm/CodeGen/GlobalISel/CombinerHelper.h

435

No, I can't seem to find it in DAGCombiner. Would you rather it be removed from the patch?

[arsenm]

I think addressing modes are less of a factor given that g_ptr_add is separate

[arsenm]

In D87936#3525578, @arsenm wrote:

I think addressing modes are less of a factor given that g_ptr_add is separate

I guess this breaks down as soon as you have a 2 component mode

[mkitzan]

I'll remove that combine as well. If there's more discussion and it's found to be beneficial then I can open another revision for it.

[mkitzan]

Updates:

• Removed G_ADD(x, -cst) -> G_SUB(x, cst)
• Very minor NFC addressing https://reviews.llvm.org/D125819#inline-1207969

[paquette]

I think this looks reasonable now? I basically only have one nit.

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

2799

FYI, in Register.h:

/// Wrapper class representing virtual and physical registers. Should be passed
/// by value.
class Register {
  unsigned Reg;

Since neither MaybeSub nor MaybeSameReg is being mutated, they might as well be passed by value.

[mkitzan]

Thanks for the review @paquette. I'll update the diff addressing your nit when I push the patch

llvm/include/llvm/CodeGen/GlobalISel/CombinerHelper.h

public:
 
   /// Transform trunc (shl x, K) to shl (trunc x),
   /// K => K < VT.getScalarSizeInBits().
   bool matchCombineTruncOfShl(MachineInstr &MI,
                               std::pair<Register, Register> &MatchInfo);
   bool applyCombineTruncOfShl(MachineInstr &MI,
                               std::pair<Register, Register> &MatchInfo);
 
+  /// Transform G_ADD(x, -cst) to G_SUB(x, cst).
+  bool matchAddNegConstant(MachineInstr &MI, APInt &Cst);
+  bool applyAddNegConstant(MachineInstr &MI, APInt &Cst);
+
+  /// Transform G_ADD(x, G_SUB(y, x)) to y.
+  /// Transform G_ADD(G_SUB(y, x), x) to y.
+  bool matchAddSubSameReg(MachineInstr &MI, Register &Src);
+  bool applyAddSubSameReg(MachineInstr &MI, Register &Src);
+
+  /// Transform G_ADD(x, y) to G_OR(x, y) iff x and y share no common bits.
+  bool matchAddToOr(MachineInstr &MI);
+  bool applyAddToOr(MachineInstr &MI);
+
   /// Transform G_MUL(x, -1) to G_SUB(0, x)
   bool applyCombineMulByNegativeOne(MachineInstr &MI);
 
   /// Return true if any explicit use operand on \p MI is defined by a
   /// G_IMPLICIT_DEF.
   bool matchAnyExplicitUseIsUndef(MachineInstr &MI);
 
   /// Return true if all register explicit use operands on \p MI are defined by
public:
   /// Combine inverting a result of a compare into the opposite cond code.
   bool matchNotCmp(MachineInstr &MI, SmallVectorImpl<Register> &RegsToNegate);
   bool applyNotCmp(MachineInstr &MI, SmallVectorImpl<Register> &RegsToNegate);
 
   /// Try to transform \p MI by using all of the above
   /// combine functions. Returns true if changed.
   bool tryCombine(MachineInstr &MI);
 
 private:
   // Memcpy family optimization helpers.
   bool optimizeMemcpy(MachineInstr &MI, Register Dst, Register Src,
                       unsigned KnownLen, Align DstAlign, Align SrcAlign,
                       bool IsVolatile);
   bool optimizeMemmove(MachineInstr &MI, Register Dst, Register Src,
                        unsigned KnownLen, Align DstAlign, Align SrcAlign,
                        bool IsVolatile);
   bool optimizeMemset(MachineInstr &MI, Register Dst, Register Val,

llvm/include/llvm/CodeGen/GlobalISel/Utils.h

 #include "llvm/CodeGen/Register.h"
 #include "llvm/Support/Alignment.h"
 #include "llvm/Support/LowLevelTypeImpl.h"
 #include <cstdint>
 
 namespace llvm {
 
 class AnalysisUsage;
+class GISelKnownBits;
 class MachineFunction;
 class MachineInstr;
 class MachineOperand;
 class MachineOptimizationRemarkEmitter;
 class MachineOptimizationRemarkMissed;
 struct MachinePointerInfo;
 class MachineRegisterInfo;
 class MCInstrDesc;

 /// Returns true if given the TargetLowering's boolean contents information,
 /// the value \p Val contains a true value.
 bool isConstTrueVal(const TargetLowering &TLI, int64_t Val, bool IsVector,
                     bool IsFP);
 
 /// Returns an integer representing true, as defined by the
 /// TargetBooleanContents.
 int64_t getICmpTrueVal(const TargetLowering &TLI, bool IsVector, bool IsFP);
+
+/// Returns true if \p A and \p B are known to have no common bits sets.
+bool haveNoCommonBitsSet(const Register A, const Register B, GISelKnownBits &KB,
+                         MachineRegisterInfo &MRI);
+
 } // End namespace llvm.
 #endif

llvm/include/llvm/Target/GlobalISel/Combine.td

 // Transform (mul x, -1) -> (sub 0, x)
 def mul_by_neg_one: GICombineRule <
   (defs root:$root),
   (match (wip_match_opcode G_MUL):$root,
          [{ return Helper.matchConstantOp(${root}->getOperand(2), -1); }]),
   (apply [{ return Helper.applyCombineMulByNegativeOne(*${root}); }])
 >;
 
+// Transform (add x, -cst) -> (sub x, cst)
+def add_neg_constant_matchinfo : GIDefMatchData<"APInt">;
+def add_neg_constant: GICombineRule<
+  (defs root:$root, add_neg_constant_matchinfo:$matchinfo),
+  (match (wip_match_opcode G_ADD):$root,
+         [{ return Helper.matchAddNegConstant(*${root}, ${matchinfo}); }]),
+  (apply [{ return Helper.applyAddNegConstant(*${root}, ${matchinfo}); }])
+>;
+
+// Transform (add x, (sub y, x)) -> y
+// Transform (add (sub y, x), x) -> y
+def add_sub_reg_matchinfo : GIDefMatchData<"Register">;
+def add_sub_reg: GICombineRule <
+  (defs root:$root, add_sub_reg_matchinfo:$matchinfo),
+  (match (wip_match_opcode G_ADD):$root,
+         [{ return Helper.matchAddSubSameReg(*${root}, ${matchinfo}); }]),
+  (apply [{ return Helper.applyAddSubSameReg(*${root}, ${matchinfo}); }])
+>;
+
+// Transform (add x, y) -> or(x, y) iff x and y share no common bits
+def add_to_or: GICombineRule <
+  (defs root:$root),
+  (match (wip_match_opcode G_ADD):$root,
+         [{ return Helper.matchAddToOr(*${root}); }]),
+  (apply [{ return Helper.applyAddToOr(*${root}); }])
+>;
+
 // FIXME: These should use the custom predicate feature once it lands.
 def undef_combines : GICombineGroup<[undef_to_fp_zero, undef_to_int_zero,
                                      undef_to_negative_one,
                                      binop_left_undef_to_zero,
                                      propagate_undef_any_op,
                                      propagate_undef_all_ops,
                                      propagate_undef_shuffle_mask,
                                      erase_undef_store]>;
 
 def identity_combines : GICombineGroup<[select_same_val, right_identity_zero,
                                         binop_same_val, binop_left_to_zero,
                                         binop_right_to_zero, p2i_to_i2p,
                                         i2p_to_p2i, anyext_trunc_fold,
                                         fneg_fneg_fold, right_identity_one]>;
 
 def known_bits_simplifications : GICombineGroup<[
-  and_trivial_mask, redundant_sext_inreg]>;
+  and_trivial_mask, redundant_sext_inreg, add_to_or]>;
 
 def width_reduction_combines : GICombineGroup<[reduce_shl_of_extend]>;
 
 def select_combines : GICombineGroup<[select_undef_cmp, select_constant_cmp]>;
 
 def trivial_combines : GICombineGroup<[copy_prop, mul_to_shl, add_p2i_to_ptradd,
-                                       mul_by_neg_one]>;
+                                       mul_by_neg_one, add_neg_constant]>;
 
 def all_combines : GICombineGroup<[trivial_combines, ptr_add_immed_chain,
     combines_for_extload, combine_indexed_load_store, undef_combines,
     identity_combines, simplify_add_to_sub,
     hoist_logic_op_with_same_opcode_hands,
     shl_ashr_to_sext_inreg, sext_inreg_of_load,
     width_reduction_combines, select_combines,
     known_bits_simplifications, ext_ext_fold,
     not_cmp_fold, opt_brcond_by_inverting_cond,
     unmerge_merge, fabs_fabs_fold, unmerge_cst, unmerge_dead_to_trunc,
     unmerge_zext_to_zext, trunc_ext_fold, trunc_shl,
-    constant_fp_op]>;
+    constant_fp_op, add_sub_reg]>;

llvm/lib/CodeGen/GlobalISel/CombinerHelper.cpp

bool CombinerHelper::applyCombineTruncOfShl(
   Builder.setInstrAndDebugLoc(MI);
   auto TruncShiftSrc = Builder.buildTrunc(DstTy, ShiftSrc);
   auto TruncShiftAmt = Builder.buildTrunc(DstTy, ShiftAmt);
   Builder.buildShl(DstReg, TruncShiftSrc, TruncShiftAmt, SrcMI->getFlags());
   MI.eraseFromParent();
   return true;
 }
 
+bool CombinerHelper::matchAddNegConstant(MachineInstr &MI, APInt &Cst) {
+  assert(MI.getOpcode() == TargetOpcode::G_ADD && "Expected a G_ADD");
+  Register CstSrc = MI.getOperand(2).getReg();
+  MachineInstr *MaybeCst = MRI.getVRegDef(CstSrc);
+
+  if (MaybeCst->getOpcode() != TargetOpcode::G_CONSTANT)
+    return false;
+  Cst = MaybeCst->getOperand(1).getCImm()->getValue();
+  return Cst.isNegative();
+}
+
+bool CombinerHelper::applyAddNegConstant(MachineInstr &MI, APInt &Cst) {
+  assert(MI.getOpcode() == TargetOpcode::G_ADD && Cst.isNegative() &&
+         "Expected a G_ADD and negative constant");
+  Register Dst = MI.getOperand(0).getReg();
+  Register Reg = MI.getOperand(1).getReg();
+
+  Builder.setInstrAndDebugLoc(MI);
+  Builder.buildSub(Dst, Reg, Builder.buildConstant(MRI.getType(Reg), -Cst),
+                   MI.getFlags());
+  MI.eraseFromParent();
+  return true;
+}
+
+bool CombinerHelper::matchAddSubSameReg(MachineInstr &MI, Register &Src) {
+  assert(MI.getOpcode() == TargetOpcode::G_ADD && "Expected a G_ADD");
+  Register LHS = MI.getOperand(1).getReg();
+  Register RHS = MI.getOperand(2).getReg();
+
+  // Helper lambda to check for opportunities for
+  // A + (B - A) -> B
+  // (B - A) + A -> B
+  auto CheckFold = [&](Register &MaybeSub, Register &MaybeSameReg) {
+    Register Reg;
+    return mi_match(MaybeSub, MRI, m_GSub(m_Reg(Src), m_Reg(Reg))) &&
+           Reg == MaybeSameReg;
+  };
+  return CheckFold(LHS, RHS) || CheckFold(RHS, LHS);
+}
+
+bool CombinerHelper::applyAddSubSameReg(MachineInstr &MI, Register &Src) {
+  assert(MI.getOpcode() == TargetOpcode::G_ADD && "Expected a G_ADD");
+  Register Dst = MI.getOperand(0).getReg();
+  MI.eraseFromParent();
+  replaceRegWith(MRI, Dst, Src);
+  return true;
+}
+
+bool CombinerHelper::matchAddToOr(MachineInstr &MI) {
+  assert(MI.getOpcode() == TargetOpcode::G_ADD && "Expected a G_ADD");
+  Register LHS = MI.getOperand(1).getReg();
+  Register RHS = MI.getOperand(2).getReg();
+  return haveNoCommonBitsSet(LHS, RHS, *KB, MRI);
+}
+
+bool CombinerHelper::applyAddToOr(MachineInstr &MI) {
+  assert(MI.getOpcode() == TargetOpcode::G_ADD && "Expected a G_ADD");
+  Register Dst = MI.getOperand(0).getReg();
+  Register LHS = MI.getOperand(1).getReg();
+  Register RHS = MI.getOperand(2).getReg();
+
+  Builder.setInstrAndDebugLoc(MI);
+  Builder.buildOr(Dst, LHS, RHS);
+  MI.eraseFromParent();
+  return true;
+}
+
 bool CombinerHelper::matchAnyExplicitUseIsUndef(MachineInstr &MI) {
   return any_of(MI.explicit_uses(), [this](const MachineOperand &MO) {
     return MO.isReg() &&
            getOpcodeDef(TargetOpcode::G_IMPLICIT_DEF, MO.getReg(), MRI);
   });
 }
 
 bool CombinerHelper::matchAllExplicitUsesAreUndef(MachineInstr &MI) {
bool CombinerHelper::applyNotCmp(MachineInstr &MI,
 
   replaceRegWith(MRI, MI.getOperand(0).getReg(), MI.getOperand(1).getReg());
   MI.eraseFromParent();
   return true;
 }
 
 bool CombinerHelper::tryCombine(MachineInstr &MI) {
   if (tryCombineCopy(MI))
     return true;
   if (tryCombineExtendingLoads(MI))
     return true;
   if (tryCombineIndexedLoadStore(MI))
     return true;
   return false;
 }

llvm/lib/CodeGen/GlobalISel/Utils.cpp

 //===- llvm/CodeGen/GlobalISel/Utils.cpp -------------------------*- C++ -*-==//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 /// \file This file implements the utility functions used by the GlobalISel
 /// pipeline.
 //===----------------------------------------------------------------------===//
 
 #include "llvm/CodeGen/GlobalISel/Utils.h"
 #include "llvm/ADT/APFloat.h"
 #include "llvm/ADT/APInt.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/CodeGen/GlobalISel/GISelChangeObserver.h"
+#include "llvm/CodeGen/GlobalISel/GISelKnownBits.h"
 #include "llvm/CodeGen/GlobalISel/MIPatternMatch.h"
 #include "llvm/CodeGen/GlobalISel/RegisterBankInfo.h"
 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/CodeGen/MachineInstrBuilder.h"
 #include "llvm/CodeGen/MachineOptimizationRemarkEmitter.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/CodeGen/StackProtector.h"
 #include "llvm/CodeGen/TargetInstrInfo.h"
int64_t llvm::getICmpTrueVal(const TargetLowering &TLI, bool IsVector,
   case TargetLowering::UndefinedBooleanContent:
   case TargetLowering::ZeroOrOneBooleanContent:
     return 1;
   case TargetLowering::ZeroOrNegativeOneBooleanContent:
     return -1;
   }
   llvm_unreachable("Invalid boolean contents");
 }
+
+bool llvm::haveNoCommonBitsSet(const Register A, const Register B,
+                               GISelKnownBits &KB, MachineRegisterInfo &MRI) {
+  assert(MRI.getType(B) == MRI.getType(A) && "Invalid operands");
+  KnownBits AKnown = KB.getKnownBits(A);
+  KnownBits BKnown = KB.getKnownBits(B);
+  return (AKnown.Zero | BKnown.Zero).isAllOnesValue();
+}

llvm/test/CodeGen/AArch64/GlobalISel/combine-add.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -run-pass=aarch64-prelegalizer-combiner -verify-machineinstrs -mtriple aarch64-unknown-unknown %s -o - | FileCheck %s
+
+---
+name:            add_neg_constant
+alignment:       4
+tracksRegLiveness: true
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $w0
+    ; CHECK-LABEL: name: add_neg_constant
+    ; CHECK: liveins: $w0
+    ; CHECK: [[COPY:%[0-9]+]]:_(s32) = COPY $w0
+    ; CHECK: [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 1
+    ; CHECK: [[SUB:%[0-9]+]]:_(s32) = G_SUB [[COPY]], [[C]]
+    ; CHECK: $w0 = COPY [[SUB]](s32)
+    %0:_(s32) = COPY $w0
+    %1:_(s32) = G_CONSTANT i32 -1
+    %2:_(s32) = G_ADD %0, %1
+    $w0 = COPY %2
+...
+---
+name:            add_neg_constant_wide
+alignment:       4
+tracksRegLiveness: true
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $q0
+    ; CHECK-LABEL: name: add_neg_constant_wide
+    ; CHECK: liveins: $q0
+    ; CHECK: [[COPY:%[0-9]+]]:_(s128) = COPY $q0
+    ; CHECK: [[C:%[0-9]+]]:_(s128) = G_CONSTANT i128 1
+    ; CHECK: [[SUB:%[0-9]+]]:_(s128) = G_SUB [[COPY]], [[C]]
+    ; CHECK: $q0 = COPY [[SUB]](s128)
+    %0:_(s128) = COPY $q0
+    %1:_(s128) = G_CONSTANT i128 -1
+    %2:_(s128) = G_ADD %0, %1
+    $q0 = COPY %2
+...
+---
+name:            add_neg_constant_vec
+alignment:       4
+tracksRegLiveness: true
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $x0
+    ; CHECK-LABEL: name: add_neg_constant_vec
+    ; CHECK: liveins: $x0
+    ; CHECK: [[COPY:%[0-9]+]]:_(<4 x s16>) = COPY $x0
+    ; CHECK: [[C:%[0-9]+]]:_(s16) = G_CONSTANT i16 -1
+    ; CHECK: [[BUILD_VECTOR:%[0-9]+]]:_(<4 x s16>) = G_BUILD_VECTOR [[C]](s16), [[C]](s16), [[C]](s16), [[C]](s16)
+    ; CHECK: [[ADD:%[0-9]+]]:_(<4 x s16>) = G_ADD [[COPY]], [[BUILD_VECTOR]]
+    ; CHECK: $x0 = COPY [[ADD]](<4 x s16>)
+    %0:_(<4 x s16>) = COPY $x0
+    %1:_(s16) = G_CONSTANT i16 -1
+    %2:_(<4 x s16>) = G_BUILD_VECTOR %1, %1, %1, %1
+    %3:_(<4 x s16>) = G_ADD %0, %2
+    $x0 = COPY %3(<4 x s16>)
+...
+---
+name:            add_lhs_sub_reg
+alignment:       4
+tracksRegLiveness: true
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $w0, $w1
+    ; CHECK-LABEL: name: add_lhs_sub_reg
+    ; CHECK: liveins: $w0, $w1
+    ; CHECK: [[COPY:%[0-9]+]]:_(s32) = COPY $w0
+    ; CHECK: $w0 = COPY [[COPY]](s32)
+    %0:_(s32) = COPY $w0
+    %1:_(s32) = COPY $w1
+    %2:_(s32) = G_SUB %0, %1
+    %3:_(s32) = G_ADD %2, %1
+    $w0 = COPY %3
+...
+---
+name:            add_lhs_sub_reg_wide
+alignment:       4
+tracksRegLiveness: true
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $q0, $q1
+    ; CHECK-LABEL: name: add_lhs_sub_reg_wide
+    ; CHECK: liveins: $q0, $q1
+    ; CHECK: [[COPY:%[0-9]+]]:_(s128) = COPY $q0
+    ; CHECK: $q0 = COPY [[COPY]](s128)
+    %0:_(s128) = COPY $q0
+    %1:_(s128) = COPY $q1
+    %2:_(s128) = G_SUB %0, %1
+    %3:_(s128) = G_ADD %2, %1
+    $q0 = COPY %3
+...
+---
+name:            add_lhs_sub_reg_vec
+alignment:       4
+tracksRegLiveness: true
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $x0, $x1
+    ; CHECK-LABEL: name: add_lhs_sub_reg_vec
+    ; CHECK: liveins: $x0, $x1
+    ; CHECK: [[COPY:%[0-9]+]]:_(<4 x s16>) = COPY $x0
+    ; CHECK: $x0 = COPY [[COPY]](<4 x s16>)
+    %0:_(<4 x s16>) = COPY $x0
+    %1:_(<4 x s16>) = COPY $x1
+    %2:_(<4 x s16>) = G_SUB %0, %1
+    %3:_(<4 x s16>) = G_ADD %2, %1
+    $x0 = COPY %3
+...
+---
+name:            add_rhs_sub_reg
+alignment:       4
+tracksRegLiveness: true
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $w0, $w1
+    ; CHECK-LABEL: name: add_rhs_sub_reg
+    ; CHECK: liveins: $w0, $w1
+    ; CHECK: [[COPY:%[0-9]+]]:_(s32) = COPY $w0
+    ; CHECK: $w0 = COPY [[COPY]](s32)
+    %0:_(s32) = COPY $w0
+    %1:_(s32) = COPY $w1
+    %2:_(s32) = G_SUB %0, %1
+    %3:_(s32) = G_ADD %1, %2
+    $w0 = COPY %3
+...
+---
+name:            add_rhs_sub_reg_wide
+alignment:       4
+tracksRegLiveness: true
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $q0, $q1
+    ; CHECK-LABEL: name: add_rhs_sub_reg_wide
+    ; CHECK: liveins: $q0, $q1
+    ; CHECK: [[COPY:%[0-9]+]]:_(s128) = COPY $q0
+    ; CHECK: $q0 = COPY [[COPY]](s128)
+    %0:_(s128) = COPY $q0
+    %1:_(s128) = COPY $q1
+    %2:_(s128) = G_SUB %0, %1
+    %3:_(s128) = G_ADD %1, %2
+    $q0 = COPY %3
+...
+---
+name:            add_rhs_sub_reg_vec
+alignment:       4
+tracksRegLiveness: true
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $x0, $x1
+    ; CHECK-LABEL: name: add_rhs_sub_reg_vec
+    ; CHECK: liveins: $x0, $x1
+    ; CHECK: [[COPY:%[0-9]+]]:_(<4 x s16>) = COPY $x0
+    ; CHECK: $x0 = COPY [[COPY]](<4 x s16>)
+    %0:_(<4 x s16>) = COPY $x0
+    %1:_(<4 x s16>) = COPY $x1
+    %2:_(<4 x s16>) = G_SUB %0, %1
+    %3:_(<4 x s16>) = G_ADD %1, %2
+    $x0 = COPY %3
+...
+---
+name:            add_to_or
+body:             |
+  bb.0:
+    liveins: $w0, $w1
+    ; CHECK-LABEL: name: add_to_or
+    ; CHECK: [[COPY:%[0-9]+]]:_(s32) = COPY $w0
+    ; CHECK: [[COPY1:%[0-9]+]]:_(s32) = COPY $w1
+    ; CHECK: [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 16
+    ; CHECK: [[SHL:%[0-9]+]]:_(s32) = G_SHL [[COPY]], [[C]](s32)
+    ; CHECK: [[LSHR:%[0-9]+]]:_(s32) = G_LSHR [[COPY1]], [[C]](s32)
+    ; CHECK: [[OR:%[0-9]+]]:_(s32) = G_OR [[SHL]], [[LSHR]]
+    ; CHECK: $w0 = COPY [[OR]](s32)
+    %0:_(s32) = COPY $w0
+    %1:_(s32) = COPY $w1
+    %2:_(s32) = G_CONSTANT i32 16
+    %3:_(s32) = G_SHL %0, %2
+    %4:_(s32) = G_LSHR %1, %2
+    %5:_(s32) = G_ADD %3, %4
+    $w0 = COPY %5
+...
+---
+name:            add_to_or_wide
+body:             |
+  bb.0:
+    liveins: $q0, $q1
+    ; CHECK-LABEL: name: add_to_or_wide
+    ; CHECK: [[COPY:%[0-9]+]]:_(s128) = COPY $q0
+    ; CHECK: [[COPY1:%[0-9]+]]:_(s128) = COPY $q1
+    ; CHECK: [[C:%[0-9]+]]:_(s128) = G_CONSTANT i128 64
+    ; CHECK: [[SHL:%[0-9]+]]:_(s128) = G_SHL [[COPY]], [[C]](s128)
+    ; CHECK: [[LSHR:%[0-9]+]]:_(s128) = G_LSHR [[COPY1]], [[C]](s128)
+    ; CHECK: [[ADD:%[0-9]+]]:_(s128) = G_ADD [[SHL]], [[LSHR]]
+    ; CHECK: $q0 = COPY [[ADD]](s128)
+    %0:_(s128) = COPY $q0
+    %1:_(s128) = COPY $q1
+    %2:_(s128) = G_CONSTANT i128 64
+    %3:_(s128) = G_SHL %0, %2
+    %4:_(s128) = G_LSHR %1, %2
+    %5:_(s128) = G_ADD %3, %4
+    $q0 = COPY %5
+...
+---
+name:            add_to_or_vec
+body:             |
+  bb.0:
+    liveins: $x0, $x1
+    ; CHECK-LABEL: name: add_to_or_vec
+    ; CHECK: [[COPY:%[0-9]+]]:_(<4 x s16>) = COPY $x0
+    ; CHECK: [[COPY1:%[0-9]+]]:_(<4 x s16>) = COPY $x1
+    ; CHECK: [[C:%[0-9]+]]:_(s16) = G_CONSTANT i16 8
+    ; CHECK: [[BUILD_VECTOR:%[0-9]+]]:_(<4 x s16>) = G_BUILD_VECTOR [[C]](s16), [[C]](s16), [[C]](s16), [[C]](s16)
+    ; CHECK: [[SHL:%[0-9]+]]:_(<4 x s16>) = G_SHL [[COPY]], [[BUILD_VECTOR]](<4 x s16>)
+    ; CHECK: [[LSHR:%[0-9]+]]:_(<4 x s16>) = G_LSHR [[COPY1]], [[BUILD_VECTOR]](<4 x s16>)
+    ; CHECK: [[ADD:%[0-9]+]]:_(<4 x s16>) = G_ADD [[SHL]], [[LSHR]]
+    ; CHECK: $x0 = COPY [[ADD]](<4 x s16>)
+    %0:_(<4 x s16>) = COPY $x0
+    %1:_(<4 x s16>) = COPY $x1
+    %2:_(s16) = G_CONSTANT i16 8
+    %3:_(<4 x s16>) = G_BUILD_VECTOR %2, %2, %2, %2
+    %4:_(<4 x s16>) = G_SHL %0, %3
+    %5:_(<4 x s16>) = G_LSHR %1, %3
+    %6:_(<4 x s16>) = G_ADD %4, %5
+    $x0 = COPY %6
+...
+

llvm/test/CodeGen/AMDGPU/GlobalISel/add.v2i16.ll

 ; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX9-NEXT:    v_mov_b32_e32 v1, 0xffc0ffc0
 ; GFX9-NEXT:    v_pk_add_u16 v0, v0, v1
 ; GFX9-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX8-LABEL: v_add_v2i16_neg_inline_imm_splat:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX8-NEXT:    s_movk_i32 s4, 0xffc0
-; GFX8-NEXT:    v_mov_b32_e32 v2, s4
-; GFX8-NEXT:    v_add_u16_e32 v1, s4, v0
-; GFX8-NEXT:    v_add_u16_sdwa v0, v0, v2 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX8-NEXT:    v_mov_b32_e32 v2, 64
+; GFX8-NEXT:    v_subrev_u16_e32 v1, 64, v0
+; GFX8-NEXT:    v_sub_u16_sdwa v0, v0, v2 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
 ; GFX8-NEXT:    v_or_b32_e32 v0, v1, v0
 ; GFX8-NEXT:    s_setpc_b64 s[30:31]
   %add = add <2 x i16> %a, <i16 -64, i16 -64>
   ret <2 x i16> %add
 }
 
 define <2 x i16> @v_add_v2i16_neg_inline_imm_lo(<2 x i16> %a) {
 ; GFX9-LABEL: v_add_v2i16_neg_inline_imm_lo:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX9-NEXT:    v_mov_b32_e32 v1, 0x4ffc0
 ; GFX9-NEXT:    v_pk_add_u16 v0, v0, v1
 ; GFX9-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX8-LABEL: v_add_v2i16_neg_inline_imm_lo:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX8-NEXT:    v_mov_b32_e32 v2, 4
-; GFX8-NEXT:    v_add_u16_e32 v1, 0xffc0, v0
+; GFX8-NEXT:    v_subrev_u16_e32 v1, 64, v0
 ; GFX8-NEXT:    v_add_u16_sdwa v0, v0, v2 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
 ; GFX8-NEXT:    v_or_b32_e32 v0, v1, v0
 ; GFX8-NEXT:    s_setpc_b64 s[30:31]
   %add = add <2 x i16> %a, <i16 -64, i16 4>
   ret <2 x i16> %add
 }
 
 define <2 x i16> @v_add_v2i16_neg_inline_imm_hi(<2 x i16> %a) {
 ; GFX9-LABEL: v_add_v2i16_neg_inline_imm_hi:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX9-NEXT:    v_mov_b32_e32 v1, 0xffc00004
 ; GFX9-NEXT:    v_pk_add_u16 v0, v0, v1
 ; GFX9-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX8-LABEL: v_add_v2i16_neg_inline_imm_hi:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX8-NEXT:    v_mov_b32_e32 v1, 0xffffffc0
-; GFX8-NEXT:    v_add_u16_e32 v2, 4, v0
-; GFX8-NEXT:    v_add_u16_sdwa v0, v0, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
-; GFX8-NEXT:    v_or_b32_e32 v0, v2, v0
+; GFX8-NEXT:    v_mov_b32_e32 v2, 64
+; GFX8-NEXT:    v_add_u16_e32 v1, 4, v0
+; GFX8-NEXT:    v_sub_u16_sdwa v0, v0, v2 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX8-NEXT:    v_or_b32_e32 v0, v1, v0
 ; GFX8-NEXT:    s_setpc_b64 s[30:31]
   %add = add <2 x i16> %a, <i16 4, i16 -64>
   ret <2 x i16> %add
 }
 
 define amdgpu_ps i32 @s_add_v2i16_neg_inline_imm_splat(<2 x i16> inreg %a) {
 ; GFX9-LABEL: s_add_v2i16_neg_inline_imm_splat:
 ; GFX9:       ; %bb.0:

llvm/test/CodeGen/AMDGPU/GlobalISel/extractelement.i128.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -global-isel -mtriple=amdgcn-mesa-mesa3d -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX9 %s
 ; RUN: llc -global-isel -mtriple=amdgcn-mesa-mesa3d -mcpu=fiji -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX8 %s
 ; RUN: llc -global-isel -mtriple=amdgcn-mesa-mesa3d -mcpu=hawaii -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN,GFX7 %s
 
 define amdgpu_ps i128 @extractelement_sgpr_v4i128_sgpr_idx(<4 x i128> addrspace(4)* inreg %ptr, i32 inreg %idx) {
 ; GFX9-LABEL: extractelement_sgpr_v4i128_sgpr_idx:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx16 s[8:23], s[2:3], 0x0
-; GFX9-NEXT:    s_lshl_b32 m0, s4, 1
+; GFX9-NEXT:    s_lshl_b32 s2, s4, 1
+; GFX9-NEXT:    s_mov_b32 m0, s2
 ; GFX9-NEXT:    s_nop 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_movrels_b64 s[0:1], s[8:9]
-; GFX9-NEXT:    s_movrels_b64 s[2:3], s[10:11]
+; GFX9-NEXT:    s_or_b32 m0, s2, 1
+; GFX9-NEXT:    s_nop 0
+; GFX9-NEXT:    s_movrels_b64 s[2:3], s[8:9]
 ; GFX9-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-LABEL: extractelement_sgpr_v4i128_sgpr_idx:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_load_dwordx16 s[8:23], s[2:3], 0x0
-; GFX8-NEXT:    s_lshl_b32 m0, s4, 1
+; GFX8-NEXT:    s_lshl_b32 s2, s4, 1
+; GFX8-NEXT:    s_mov_b32 m0, s2
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    s_movrels_b64 s[0:1], s[8:9]
-; GFX8-NEXT:    s_movrels_b64 s[2:3], s[10:11]
+; GFX8-NEXT:    s_or_b32 m0, s2, 1
+; GFX8-NEXT:    s_movrels_b64 s[2:3], s[8:9]
 ; GFX8-NEXT:    ; return to shader part epilog
 ;
 ; GFX7-LABEL: extractelement_sgpr_v4i128_sgpr_idx:
 ; GFX7:       ; %bb.0:
 ; GFX7-NEXT:    s_load_dwordx16 s[8:23], s[2:3], 0x0
-; GFX7-NEXT:    s_lshl_b32 m0, s4, 1
+; GFX7-NEXT:    s_lshl_b32 s2, s4, 1
+; GFX7-NEXT:    s_mov_b32 m0, s2
 ; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX7-NEXT:    s_movrels_b64 s[0:1], s[8:9]
-; GFX7-NEXT:    s_movrels_b64 s[2:3], s[10:11]
+; GFX7-NEXT:    s_or_b32 m0, s2, 1
+; GFX7-NEXT:    s_movrels_b64 s[2:3], s[8:9]
 ; GFX7-NEXT:    ; return to shader part epilog
   %vector = load <4 x i128>, <4 x i128> addrspace(4)* %ptr
   %element = extractelement <4 x i128> %vector, i32 %idx
   ret i128 %element
 }
 
 define amdgpu_ps i128 @extractelement_vgpr_v4i128_sgpr_idx(<4 x i128> addrspace(1)* %ptr, i32 inreg %idx) {
 ; GFX9-LABEL: extractelement_vgpr_v4i128_sgpr_idx:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    global_load_dwordx4 v[2:5], v[0:1], off
 ; GFX9-NEXT:    global_load_dwordx4 v[6:9], v[0:1], off offset:16
 ; GFX9-NEXT:    global_load_dwordx4 v[10:13], v[0:1], off offset:32
 ; GFX9-NEXT:    global_load_dwordx4 v[14:17], v[0:1], off offset:48
 ; GFX9-NEXT:    s_lshl_b32 s0, s2, 1
-; GFX9-NEXT:    s_lshl_b32 s0, s0, 1
-; GFX9-NEXT:    s_set_gpr_idx_on s0, gpr_idx(SRC0)
+; GFX9-NEXT:    s_lshl_b32 s1, s0, 1
+; GFX9-NEXT:    s_set_gpr_idx_on s1, gpr_idx(SRC0)
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    v_mov_b32_e32 v0, v2
 ; GFX9-NEXT:    v_mov_b32_e32 v1, v3
+; GFX9-NEXT:    s_set_gpr_idx_off
+; GFX9-NEXT:    s_or_b32 s0, s0, 1
+; GFX9-NEXT:    s_lshl_b32 s0, s0, 1
+; GFX9-NEXT:    s_set_gpr_idx_on s0, gpr_idx(SRC0)
 ; GFX9-NEXT:    v_mov_b32_e32 v18, v2
 ; GFX9-NEXT:    v_mov_b32_e32 v3, v3
 ; GFX9-NEXT:    s_set_gpr_idx_off
 ; GFX9-NEXT:    v_readfirstlane_b32 s0, v0
 ; GFX9-NEXT:    v_readfirstlane_b32 s1, v1
 ; GFX9-NEXT:    v_readfirstlane_b32 s2, v18
 ; GFX9-NEXT:    v_readfirstlane_b32 s3, v3
 ; GFX9-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-LABEL: extractelement_vgpr_v4i128_sgpr_idx:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    v_add_u32_e32 v6, vcc, 16, v0
 ; GFX8-NEXT:    v_addc_u32_e32 v7, vcc, 0, v1, vcc
 ; GFX8-NEXT:    v_add_u32_e32 v10, vcc, 32, v0
 ; GFX8-NEXT:    v_addc_u32_e32 v11, vcc, 0, v1, vcc
 ; GFX8-NEXT:    flat_load_dwordx4 v[2:5], v[0:1]
 ; GFX8-NEXT:    flat_load_dwordx4 v[6:9], v[6:7]
 ; GFX8-NEXT:    v_add_u32_e32 v0, vcc, 48, v0
 ; GFX8-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
 ; GFX8-NEXT:    flat_load_dwordx4 v[10:13], v[10:11]
 ; GFX8-NEXT:    flat_load_dwordx4 v[14:17], v[0:1]
 ; GFX8-NEXT:    s_lshl_b32 s0, s2, 1
 ; GFX8-NEXT:    s_lshl_b32 m0, s0, 1
+; GFX8-NEXT:    s_or_b32 s0, s0, 1
 ; GFX8-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX8-NEXT:    v_movrels_b32_e32 v1, v3
 ; GFX8-NEXT:    v_movrels_b32_e32 v0, v2
-; GFX8-NEXT:    v_mov_b32_e32 v3, v1
-; GFX8-NEXT:    v_mov_b32_e32 v2, v0
+; GFX8-NEXT:    s_lshl_b32 m0, s0, 1
+; GFX8-NEXT:    v_movrels_b32_e32 v18, v2
+; GFX8-NEXT:    v_movrels_b32_e32 v3, v3
 ; GFX8-NEXT:    v_readfirstlane_b32 s0, v0
 ; GFX8-NEXT:    v_readfirstlane_b32 s1, v1
-; GFX8-NEXT:    v_readfirstlane_b32 s2, v2
+; GFX8-NEXT:    v_readfirstlane_b32 s2, v18
 ; GFX8-NEXT:    v_readfirstlane_b32 s3, v3
 ; GFX8-NEXT:    ; return to shader part epilog
 ;
 ; GFX7-LABEL: extractelement_vgpr_v4i128_sgpr_idx:
 ; GFX7:       ; %bb.0:
 ; GFX7-NEXT:    s_mov_b32 s6, 0
 ; GFX7-NEXT:    s_mov_b32 s7, 0xf000
 ; GFX7-NEXT:    s_mov_b64 s[4:5], 0
 ; GFX7-NEXT:    buffer_load_dwordx4 v[2:5], v[0:1], s[4:7], 0 addr64
 ; GFX7-NEXT:    buffer_load_dwordx4 v[6:9], v[0:1], s[4:7], 0 addr64 offset:16
 ; GFX7-NEXT:    buffer_load_dwordx4 v[10:13], v[0:1], s[4:7], 0 addr64 offset:32
 ; GFX7-NEXT:    buffer_load_dwordx4 v[14:17], v[0:1], s[4:7], 0 addr64 offset:48
 ; GFX7-NEXT:    s_lshl_b32 s0, s2, 1
 ; GFX7-NEXT:    s_lshl_b32 m0, s0, 1
+; GFX7-NEXT:    s_or_b32 s0, s0, 1
 ; GFX7-NEXT:    s_waitcnt vmcnt(0)
 ; GFX7-NEXT:    v_movrels_b32_e32 v1, v3
 ; GFX7-NEXT:    v_movrels_b32_e32 v0, v2
-; GFX7-NEXT:    v_mov_b32_e32 v3, v1
-; GFX7-NEXT:    v_mov_b32_e32 v2, v0
+; GFX7-NEXT:    s_lshl_b32 m0, s0, 1
+; GFX7-NEXT:    v_movrels_b32_e32 v18, v2
+; GFX7-NEXT:    v_movrels_b32_e32 v3, v3
 ; GFX7-NEXT:    v_readfirstlane_b32 s0, v0
 ; GFX7-NEXT:    v_readfirstlane_b32 s1, v1
-; GFX7-NEXT:    v_readfirstlane_b32 s2, v2
+; GFX7-NEXT:    v_readfirstlane_b32 s2, v18
 ; GFX7-NEXT:    v_readfirstlane_b32 s3, v3
 ; GFX7-NEXT:    ; return to shader part epilog
   %vector = load <4 x i128>, <4 x i128> addrspace(1)* %ptr
   %element = extractelement <4 x i128> %vector, i32 %idx
   ret i128 %element
 }
 
 define i128 @extractelement_vgpr_v4i128_vgpr_idx(<4 x i128> addrspace(1)* %ptr, i32 %idx) {
 ; GFX9-LABEL: extractelement_vgpr_v4i128_vgpr_idx:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX9-NEXT:    v_lshlrev_b32_e32 v16, 1, v2
 ; GFX9-NEXT:    global_load_dwordx4 v[2:5], v[0:1], off
 ; GFX9-NEXT:    global_load_dwordx4 v[6:9], v[0:1], off offset:16
-; GFX9-NEXT:    v_add_u32_e32 v17, 1, v16
+; GFX9-NEXT:    v_or_b32_e32 v17, 1, v16
 ; GFX9-NEXT:    v_cmp_eq_u32_e64 s[4:5], 1, v16
 ; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v17
 ; GFX9-NEXT:    v_cmp_eq_u32_e64 s[6:7], 6, v16
 ; GFX9-NEXT:    v_cmp_eq_u32_e64 s[8:9], 7, v16
 ; GFX9-NEXT:    s_waitcnt vmcnt(1)
 ; GFX9-NEXT:    v_cndmask_b32_e64 v10, v2, v4, s[4:5]
 ; GFX9-NEXT:    v_cndmask_b32_e64 v11, v3, v5, s[4:5]
 ; GFX9-NEXT:    v_cndmask_b32_e32 v2, v2, v4, vcc

 ; GFX8-LABEL: extractelement_vgpr_v4i128_vgpr_idx:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX8-NEXT:    v_add_u32_e32 v3, vcc, 16, v0
 ; GFX8-NEXT:    v_addc_u32_e32 v4, vcc, 0, v1, vcc
 ; GFX8-NEXT:    flat_load_dwordx4 v[8:11], v[0:1]
 ; GFX8-NEXT:    flat_load_dwordx4 v[4:7], v[3:4]
 ; GFX8-NEXT:    v_lshlrev_b32_e32 v16, 1, v2
-; GFX8-NEXT:    v_add_u32_e32 v17, vcc, 1, v16
+; GFX8-NEXT:    v_or_b32_e32 v17, 1, v16
 ; GFX8-NEXT:    v_cmp_eq_u32_e64 s[4:5], 1, v16
 ; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v17
 ; GFX8-NEXT:    v_cmp_eq_u32_e64 s[6:7], 6, v16
 ; GFX8-NEXT:    v_cmp_eq_u32_e64 s[8:9], 7, v16
 ; GFX8-NEXT:    s_waitcnt vmcnt(1) lgkmcnt(1)
 ; GFX8-NEXT:    v_cndmask_b32_e64 v2, v8, v10, s[4:5]
 ; GFX8-NEXT:    v_cndmask_b32_e64 v3, v9, v11, s[4:5]
 ; GFX8-NEXT:    v_cndmask_b32_e32 v8, v8, v10, vcc

 ; GFX7:       ; %bb.0:
 ; GFX7-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX7-NEXT:    s_mov_b32 s10, 0
 ; GFX7-NEXT:    s_mov_b32 s11, 0xf000
 ; GFX7-NEXT:    s_mov_b64 s[8:9], 0
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v16, 1, v2
 ; GFX7-NEXT:    buffer_load_dwordx4 v[2:5], v[0:1], s[8:11], 0 addr64
 ; GFX7-NEXT:    buffer_load_dwordx4 v[6:9], v[0:1], s[8:11], 0 addr64 offset:16
-; GFX7-NEXT:    v_add_i32_e32 v17, vcc, 1, v16
+; GFX7-NEXT:    v_or_b32_e32 v17, 1, v16
 ; GFX7-NEXT:    v_cmp_eq_u32_e64 s[4:5], 1, v16
 ; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v17
 ; GFX7-NEXT:    v_cmp_eq_u32_e64 s[6:7], 6, v16
 ; GFX7-NEXT:    s_waitcnt vmcnt(1)
 ; GFX7-NEXT:    v_cndmask_b32_e64 v10, v2, v4, s[4:5]
 ; GFX7-NEXT:    v_cndmask_b32_e64 v11, v3, v5, s[4:5]
 ; GFX7-NEXT:    v_cndmask_b32_e32 v2, v2, v4, vcc
 ; GFX7-NEXT:    v_cndmask_b32_e32 v3, v3, v5, vcc

 ; GFX7-NEXT:    v_cndmask_b32_e64 v3, v3, v15, s[4:5]
 ; GFX7-NEXT:    s_setpc_b64 s[30:31]
   %vector = load <4 x i128>, <4 x i128> addrspace(1)* %ptr
   %element = extractelement <4 x i128> %vector, i32 %idx
   ret i128 %element
 }
 
 define amdgpu_ps i128 @extractelement_sgpr_v4i128_vgpr_idx(<4 x i128> addrspace(4)* inreg %ptr, i32 %idx) {
-; GFX9-LABEL: extractelement_sgpr_v4i128_vgpr_idx:
-; GFX9:       ; %bb.0:
-; GFX9-NEXT:    s_load_dwordx16 s[0:15], s[2:3], 0x0
-; GFX9-NEXT:    v_lshlrev_b32_e32 v2, 1, v0
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v2
-; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    v_mov_b32_e32 v3, s0
-; GFX9-NEXT:    v_mov_b32_e32 v1, s2
-; GFX9-NEXT:    v_mov_b32_e32 v0, s1
-; GFX9-NEXT:    v_mov_b32_e32 v4, s3
-; GFX9-NEXT:    v_mov_b32_e32 v5, s4
-; GFX9-NEXT:    v_mov_b32_e32 v6, s5
-; GFX9-NEXT:    v_cndmask_b32_e32 v1, v3, v1, vcc
-; GFX9-NEXT:    v_cndmask_b32_e32 v0, v0, v4, vcc
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 2, v2
-; GFX9-NEXT:    v_cndmask_b32_e32 v1, v1, v5, vcc
-; GFX9-NEXT:    v_cndmask_b32_e32 v0, v0, v6, vcc
-; GFX9-NEXT:    v_mov_b32_e32 v7, s6
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 3, v2
-; GFX9-NEXT:    v_mov_b32_e32 v8, s7
-; GFX9-NEXT:    v_cndmask_b32_e32 v1, v1, v7, vcc
-; GFX9-NEXT:    v_cndmask_b32_e32 v0, v0, v8, vcc
-; GFX9-NEXT:    v_mov_b32_e32 v9, s8
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 4, v2
-; GFX9-NEXT:    v_mov_b32_e32 v10, s9
-; GFX9-NEXT:    v_cndmask_b32_e32 v1, v1, v9, vcc
-; GFX9-NEXT:    v_cndmask_b32_e32 v0, v0, v10, vcc
-; GFX9-NEXT:    v_mov_b32_e32 v11, s10
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 5, v2
-; GFX9-NEXT:    v_mov_b32_e32 v12, s11
-; GFX9-NEXT:    v_cndmask_b32_e32 v1, v1, v11, vcc
-; GFX9-NEXT:    v_cndmask_b32_e32 v0, v0, v12, vcc
-; GFX9-NEXT:    v_mov_b32_e32 v13, s12
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 6, v2
-; GFX9-NEXT:    v_mov_b32_e32 v14, s13
-; GFX9-NEXT:    v_cndmask_b32_e32 v1, v1, v13, vcc
-; GFX9-NEXT:    v_cndmask_b32_e32 v4, v0, v14, vcc
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 7, v2
-; GFX9-NEXT:    v_mov_b32_e32 v15, s14
-; GFX9-NEXT:    v_mov_b32_e32 v16, s15
-; GFX9-NEXT:    v_add_u32_e32 v2, 1, v2
-; GFX9-NEXT:    v_cndmask_b32_e32 v0, v1, v15, vcc
-; GFX9-NEXT:    v_cndmask_b32_e32 v1, v4, v16, vcc
-; GFX9-NEXT:    v_mov_b32_e32 v4, s1
-; GFX9-NEXT:    v_mov_b32_e32 v5, s2
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v2
-; GFX9-NEXT:    v_mov_b32_e32 v6, s3
-; GFX9-NEXT:    v_cndmask_b32_e32 v3, v3, v5, vcc
-; GFX9-NEXT:    v_cndmask_b32_e32 v4, v4, v6, vcc
-; GFX9-NEXT:    v_mov_b32_e32 v7, s4
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 2, v2
-; GFX9-NEXT:    v_mov_b32_e32 v8, s5
-; GFX9-NEXT:    v_cndmask_b32_e32 v3, v3, v7, vcc
-; GFX9-NEXT:    v_cndmask_b32_e32 v4, v4, v8, vcc
-; GFX9-NEXT:    v_mov_b32_e32 v9, s6
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 3, v2
-; GFX9-NEXT:    v_mov_b32_e32 v10, s7
-; GFX9-NEXT:    v_cndmask_b32_e32 v3, v3, v9, vcc
-; GFX9-NEXT:    v_cndmask_b32_e32 v4, v4, v10, vcc
-; GFX9-NEXT:    v_mov_b32_e32 v11, s8
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 4, v2
-; GFX9-NEXT:    v_mov_b32_e32 v12, s9
-; GFX9-NEXT:    v_mov_b32_e32 v13, s10
-; GFX9-NEXT:    v_mov_b32_e32 v5, s11
-; GFX9-NEXT:    v_cndmask_b32_e32 v3, v3, v11, vcc
-; GFX9-NEXT:    v_cndmask_b32_e32 v4, v4, v12, vcc
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 5, v2
-; GFX9-NEXT:    v_mov_b32_e32 v6, s12
-; GFX9-NEXT:    v_mov_b32_e32 v7, s13
-; GFX9-NEXT:    v_cndmask_b32_e32 v3, v3, v13, vcc
-; GFX9-NEXT:    v_cndmask_b32_e32 v4, v4, v5, vcc
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 6, v2
-; GFX9-NEXT:    v_mov_b32_e32 v8, s14
-; GFX9-NEXT:    v_mov_b32_e32 v9, s15
-; GFX9-NEXT:    v_cndmask_b32_e32 v3, v3, v6, vcc
-; GFX9-NEXT:    v_cndmask_b32_e32 v4, v4, v7, vcc
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 7, v2
-; GFX9-NEXT:    v_cndmask_b32_e32 v2, v3, v8, vcc
-; GFX9-NEXT:    v_cndmask_b32_e32 v3, v4, v9, vcc
-; GFX9-NEXT:    v_readfirstlane_b32 s0, v0
-; GFX9-NEXT:    v_readfirstlane_b32 s1, v1
-; GFX9-NEXT:    v_readfirstlane_b32 s2, v2
-; GFX9-NEXT:    v_readfirstlane_b32 s3, v3
-; GFX9-NEXT:    ; return to shader part epilog
-;
-; GFX8-LABEL: extractelement_sgpr_v4i128_vgpr_idx:
-; GFX8:       ; %bb.0:
-; GFX8-NEXT:    s_load_dwordx16 s[0:15], s[2:3], 0x0
-; GFX8-NEXT:    v_lshlrev_b32_e32 v2, 1, v0
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v2
-; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    v_mov_b32_e32 v3, s0
-; GFX8-NEXT:    v_mov_b32_e32 v1, s2
-; GFX8-NEXT:    v_mov_b32_e32 v0, s1
-; GFX8-NEXT:    v_mov_b32_e32 v4, s3
-; GFX8-NEXT:    v_mov_b32_e32 v5, s4
-; GFX8-NEXT:    v_mov_b32_e32 v6, s5
-; GFX8-NEXT:    v_cndmask_b32_e32 v1, v3, v1, vcc
-; GFX8-NEXT:    v_cndmask_b32_e32 v0, v0, v4, vcc
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 2, v2
-; GFX8-NEXT:    v_cndmask_b32_e32 v1, v1, v5, vcc
-; GFX8-NEXT:    v_cndmask_b32_e32 v0, v0, v6, vcc
-; GFX8-NEXT:    v_mov_b32_e32 v7, s6
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 3, v2
-; GFX8-NEXT:    v_mov_b32_e32 v8, s7
-; GFX8-NEXT:    v_cndmask_b32_e32 v1, v1, v7, vcc
-; GFX8-NEXT:    v_cndmask_b32_e32 v0, v0, v8, vcc
-; GFX8-NEXT:    v_mov_b32_e32 v9, s8
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 4, v2
-; GFX8-NEXT:    v_mov_b32_e32 v10, s9
-; GFX8-NEXT:    v_cndmask_b32_e32 v1, v1, v9, vcc
-; GFX8-NEXT:    v_cndmask_b32_e32 v0, v0, v10, vcc
-; GFX8-NEXT:    v_mov_b32_e32 v11, s10
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 5, v2
-; GFX8-NEXT:    v_mov_b32_e32 v12, s11
-; GFX8-NEXT:    v_cndmask_b32_e32 v1, v1, v11, vcc
-; GFX8-NEXT:    v_cndmask_b32_e32 v0, v0, v12, vcc
-; GFX8-NEXT:    v_mov_b32_e32 v13, s12
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 6, v2
-; GFX8-NEXT:    v_mov_b32_e32 v14, s13
-; GFX8-NEXT:    v_cndmask_b32_e32 v1, v1, v13, vcc
-; GFX8-NEXT:    v_cndmask_b32_e32 v4, v0, v14, vcc
-; GFX8-NEXT:    v_mov_b32_e32 v15, s14
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 7, v2
-; GFX8-NEXT:    v_mov_b32_e32 v16, s15
-; GFX8-NEXT:    v_cndmask_b32_e32 v0, v1, v15, vcc
-; GFX8-NEXT:    v_cndmask_b32_e32 v1, v4, v16, vcc
-; GFX8-NEXT:    v_add_u32_e32 v2, vcc, 1, v2
-; GFX8-NEXT:    v_mov_b32_e32 v4, s1
-; GFX8-NEXT:    v_mov_b32_e32 v5, s2
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v2
-; GFX8-NEXT:    v_mov_b32_e32 v6, s3
-; GFX8-NEXT:    v_cndmask_b32_e32 v3, v3, v5, vcc
-; GFX8-NEXT:    v_cndmask_b32_e32 v4, v4, v6, vcc
-; GFX8-NEXT:    v_mov_b32_e32 v7, s4
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 2, v2
-; GFX8-NEXT:    v_mov_b32_e32 v8, s5
-; GFX8-NEXT:    v_cndmask_b32_e32 v3, v3, v7, vcc
-; GFX8-NEXT:    v_cndmask_b32_e32 v4, v4, v8, vcc
-; GFX8-NEXT:    v_mov_b32_e32 v9, s6
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 3, v2
-; GFX8-NEXT:    v_mov_b32_e32 v10, s7
-; GFX8-NEXT:    v_cndmask_b32_e32 v3, v3, v9, vcc
-; GFX8-NEXT:    v_cndmask_b32_e32 v4, v4, v10, vcc
-; GFX8-NEXT:    v_mov_b32_e32 v11, s8
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 4, v2
-; GFX8-NEXT:    v_mov_b32_e32 v12, s9
-; GFX8-NEXT:    v_mov_b32_e32 v13, s10
-; GFX8-NEXT:    v_mov_b32_e32 v5, s11
-; GFX8-NEXT:    v_cndmask_b32_e32 v3, v3, v11, vcc
-; GFX8-NEXT:    v_cndmask_b32_e32 v4, v4, v12, vcc
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 5, v2
-; GFX8-NEXT:    v_mov_b32_e32 v6, s12
-; GFX8-NEXT:    v_mov_b32_e32 v7, s13
-; GFX8-NEXT:    v_cndmask_b32_e32 v3, v3, v13, vcc
-; GFX8-NEXT:    v_cndmask_b32_e32 v4, v4, v5, vcc
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 6, v2
-; GFX8-NEXT:    v_mov_b32_e32 v8, s14
-; GFX8-NEXT:    v_mov_b32_e32 v9, s15
-; GFX8-NEXT:    v_cndmask_b32_e32 v3, v3, v6, vcc
-; GFX8-NEXT:    v_cndmask_b32_e32 v4, v4, v7, vcc
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 7, v2
-; GFX8-NEXT:    v_cndmask_b32_e32 v2, v3, v8, vcc
-; GFX8-NEXT:    v_cndmask_b32_e32 v3, v4, v9, vcc
-; GFX8-NEXT:    v_readfirstlane_b32 s0, v0
-; GFX8-NEXT:    v_readfirstlane_b32 s1, v1
-; GFX8-NEXT:    v_readfirstlane_b32 s2, v2
-; GFX8-NEXT:    v_readfirstlane_b32 s3, v3
-; GFX8-NEXT:    ; return to shader part epilog
-;
-; GFX7-LABEL: extractelement_sgpr_v4i128_vgpr_idx:
-; GFX7:       ; %bb.0:
-; GFX7-NEXT:    s_load_dwordx16 s[0:15], s[2:3], 0x0
-; GFX7-NEXT:    v_lshlrev_b32_e32 v2, 1, v0
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v2
-; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX7-NEXT:    v_mov_b32_e32 v3, s0
-; GFX7-NEXT:    v_mov_b32_e32 v1, s2
-; GFX7-NEXT:    v_mov_b32_e32 v0, s1
-; GFX7-NEXT:    v_mov_b32_e32 v4, s3
-; GFX7-NEXT:    v_mov_b32_e32 v5, s4
-; GFX7-NEXT:    v_mov_b32_e32 v6, s5
-; GFX7-NEXT:    v_cndmask_b32_e32 v1, v3, v1, vcc
-; GFX7-NEXT:    v_cndmask_b32_e32 v0, v0, v4, vcc
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 2, v2
-; GFX7-NEXT:    v_cndmask_b32_e32 v1, v1, v5, vcc
-; GFX7-NEXT:    v_cndmask_b32_e32 v0, v0, v6, vcc
-; GFX7-NEXT:    v_mov_b32_e32 v7, s6
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 3, v2
-; GFX7-NEXT:    v_mov_b32_e32 v8, s7
-; GFX7-NEXT:    v_cndmask_b32_e32 v1, v1, v7, vcc
-; GFX7-NEXT:    v_cndmask_b32_e32 v0, v0, v8, vcc
-; GFX7-NEXT:    v_mov_b32_e32 v9, s8
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 4, v2
-; GFX7-NEXT:    v_mov_b32_e32 v10, s9
-; GFX7-NEXT:    v_cndmask_b32_e32 v1, v1, v9, vcc
-; GFX7-NEXT:    v_cndmask_b32_e32 v0, v0, v10, vcc
-; GFX7-NEXT:    v_mov_b32_e32 v11, s10
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 5, v2
-; GFX7-NEXT:    v_mov_b32_e32 v12, s11
-; GFX7-NEXT:    v_cndmask_b32_e32 v1, v1, v11, vcc
-; GFX7-NEXT:    v_cndmask_b32_e32 v0, v0, v12, vcc
-; GFX7-NEXT:    v_mov_b32_e32 v13, s12
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 6, v2
-; GFX7-NEXT:    v_mov_b32_e32 v14, s13
-; GFX7-NEXT:    v_cndmask_b32_e32 v1, v1, v13, vcc
-; GFX7-NEXT:    v_cndmask_b32_e32 v4, v0, v14, vcc
-; GFX7-NEXT:    v_mov_b32_e32 v15, s14
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 7, v2
-; GFX7-NEXT:    v_mov_b32_e32 v16, s15
-; GFX7-NEXT:    v_cndmask_b32_e32 v0, v1, v15, vcc
-; GFX7-NEXT:    v_cndmask_b32_e32 v1, v4, v16, vcc
-; GFX7-NEXT:    v_add_i32_e32 v2, vcc, 1, v2
-; GFX7-NEXT:    v_mov_b32_e32 v4, s1
-; GFX7-NEXT:    v_mov_b32_e32 v5, s2
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v2
-; GFX7-NEXT:    v_mov_b32_e32 v6, s3
-; GFX7-NEXT:    v_cndmask_b32_e32 v3, v3, v5, vcc
-; GFX7-NEXT:    v_cndmask_b32_e32 v4, v4, v6, vcc
-; GFX7-NEXT:    v_mov_b32_e32 v7, s4
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 2, v2
-; GFX7-NEXT:    v_mov_b32_e32 v8, s5
-; GFX7-NEXT:    v_cndmask_b32_e32 v3, v3, v7, vcc
-; GFX7-NEXT:    v_cndmask_b32_e32 v4, v4, v8, vcc
-; GFX7-NEXT:    v_mov_b32_e32 v9, s6
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 3, v2
-; GFX7-NEXT:    v_mov_b32_e32 v10, s7
-; GFX7-NEXT:    v_cndmask_b32_e32 v3, v3, v9, vcc
-; GFX7-NEXT:    v_cndmask_b32_e32 v4, v4, v10, vcc
-; GFX7-NEXT:    v_mov_b32_e32 v11, s8
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 4, v2
-; GFX7-NEXT:    v_mov_b32_e32 v12, s9
-; GFX7-NEXT:    v_mov_b32_e32 v13, s10
-; GFX7-NEXT:    v_mov_b32_e32 v5, s11
-; GFX7-NEXT:    v_cndmask_b32_e32 v3, v3, v11, vcc
-; GFX7-NEXT:    v_cndmask_b32_e32 v4, v4, v12, vcc
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 5, v2
-; GFX7-NEXT:    v_mov_b32_e32 v6, s12
-; GFX7-NEXT:    v_mov_b32_e32 v7, s13
-; GFX7-NEXT:    v_cndmask_b32_e32 v3, v3, v13, vcc
-; GFX7-NEXT:    v_cndmask_b32_e32 v4, v4, v5, vcc
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 6, v2
-; GFX7-NEXT:    v_mov_b32_e32 v8, s14
-; GFX7-NEXT:    v_mov_b32_e32 v9, s15
-; GFX7-NEXT:    v_cndmask_b32_e32 v3, v3, v6, vcc
-; GFX7-NEXT:    v_cndmask_b32_e32 v4, v4, v7, vcc
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 7, v2
-; GFX7-NEXT:    v_cndmask_b32_e32 v2, v3, v8, vcc
-; GFX7-NEXT:    v_cndmask_b32_e32 v3, v4, v9, vcc
-; GFX7-NEXT:    v_readfirstlane_b32 s0, v0
-; GFX7-NEXT:    v_readfirstlane_b32 s1, v1
-; GFX7-NEXT:    v_readfirstlane_b32 s2, v2
-; GFX7-NEXT:    v_readfirstlane_b32 s3, v3
-; GFX7-NEXT:    ; return to shader part epilog
+; GCN-LABEL: extractelement_sgpr_v4i128_vgpr_idx:
+; GCN:       ; %bb.0:
+; GCN-NEXT:    s_load_dwordx16 s[0:15], s[2:3], 0x0
+; GCN-NEXT:    v_lshlrev_b32_e32 v2, 1, v0
+; GCN-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v2
+; GCN-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN-NEXT:    v_mov_b32_e32 v3, s0
+; GCN-NEXT:    v_mov_b32_e32 v1, s2
+; GCN-NEXT:    v_mov_b32_e32 v0, s1
+; GCN-NEXT:    v_mov_b32_e32 v4, s3
+; GCN-NEXT:    v_mov_b32_e32 v5, s4
+; GCN-NEXT:    v_mov_b32_e32 v6, s5
+; GCN-NEXT:    v_cndmask_b32_e32 v1, v3, v1, vcc
+; GCN-NEXT:    v_cndmask_b32_e32 v0, v0, v4, vcc
+; GCN-NEXT:    v_cmp_eq_u32_e32 vcc, 2, v2
+; GCN-NEXT:    v_cndmask_b32_e32 v1, v1, v5, vcc
+; GCN-NEXT:    v_cndmask_b32_e32 v0, v0, v6, vcc
+; GCN-NEXT:    v_mov_b32_e32 v7, s6
+; GCN-NEXT:    v_cmp_eq_u32_e32 vcc, 3, v2
+; GCN-NEXT:    v_mov_b32_e32 v8, s7
+; GCN-NEXT:    v_cndmask_b32_e32 v1, v1, v7, vcc
+; GCN-NEXT:    v_cndmask_b32_e32 v0, v0, v8, vcc
+; GCN-NEXT:    v_mov_b32_e32 v9, s8
+; GCN-NEXT:    v_cmp_eq_u32_e32 vcc, 4, v2
+; GCN-NEXT:    v_mov_b32_e32 v10, s9
+; GCN-NEXT:    v_cndmask_b32_e32 v1, v1, v9, vcc
+; GCN-NEXT:    v_cndmask_b32_e32 v0, v0, v10, vcc
+; GCN-NEXT:    v_mov_b32_e32 v11, s10
+; GCN-NEXT:    v_cmp_eq_u32_e32 vcc, 5, v2
+; GCN-NEXT:    v_mov_b32_e32 v12, s11
+; GCN-NEXT:    v_cndmask_b32_e32 v1, v1, v11, vcc
+; GCN-NEXT:    v_cndmask_b32_e32 v0, v0, v12, vcc
+; GCN-NEXT:    v_mov_b32_e32 v13, s12
+; GCN-NEXT:    v_cmp_eq_u32_e32 vcc, 6, v2
+; GCN-NEXT:    v_mov_b32_e32 v14, s13
+; GCN-NEXT:    v_cndmask_b32_e32 v1, v1, v13, vcc
+; GCN-NEXT:    v_cndmask_b32_e32 v4, v0, v14, vcc
+; GCN-NEXT:    v_cmp_eq_u32_e32 vcc, 7, v2
+; GCN-NEXT:    v_mov_b32_e32 v15, s14
+; GCN-NEXT:    v_mov_b32_e32 v16, s15
+; GCN-NEXT:    v_or_b32_e32 v2, 1, v2
+; GCN-NEXT:    v_cndmask_b32_e32 v0, v1, v15, vcc
+; GCN-NEXT:    v_cndmask_b32_e32 v1, v4, v16, vcc
+; GCN-NEXT:    v_mov_b32_e32 v4, s1
+; GCN-NEXT:    v_mov_b32_e32 v5, s2
+; GCN-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v2
+; GCN-NEXT:    v_mov_b32_e32 v6, s3
+; GCN-NEXT:    v_cndmask_b32_e32 v3, v3, v5, vcc
+; GCN-NEXT:    v_cndmask_b32_e32 v4, v4, v6, vcc
+; GCN-NEXT:    v_mov_b32_e32 v7, s4
+; GCN-NEXT:    v_cmp_eq_u32_e32 vcc, 2, v2
+; GCN-NEXT:    v_mov_b32_e32 v8, s5
+; GCN-NEXT:    v_cndmask_b32_e32 v3, v3, v7, vcc
+; GCN-NEXT:    v_cndmask_b32_e32 v4, v4, v8, vcc
+; GCN-NEXT:    v_mov_b32_e32 v9, s6
+; GCN-NEXT:    v_cmp_eq_u32_e32 vcc, 3, v2
+; GCN-NEXT:    v_mov_b32_e32 v10, s7
+; GCN-NEXT:    v_cndmask_b32_e32 v3, v3, v9, vcc
+; GCN-NEXT:    v_cndmask_b32_e32 v4, v4, v10, vcc
+; GCN-NEXT:    v_mov_b32_e32 v11, s8
+; GCN-NEXT:    v_cmp_eq_u32_e32 vcc, 4, v2
+; GCN-NEXT:    v_mov_b32_e32 v12, s9
+; GCN-NEXT:    v_mov_b32_e32 v13, s10
+; GCN-NEXT:    v_mov_b32_e32 v5, s11
+; GCN-NEXT:    v_cndmask_b32_e32 v3, v3, v11, vcc
+; GCN-NEXT:    v_cndmask_b32_e32 v4, v4, v12, vcc
+; GCN-NEXT:    v_cmp_eq_u32_e32 vcc, 5, v2
+; GCN-NEXT:    v_mov_b32_e32 v6, s12
+; GCN-NEXT:    v_mov_b32_e32 v7, s13
+; GCN-NEXT:    v_cndmask_b32_e32 v3, v3, v13, vcc
+; GCN-NEXT:    v_cndmask_b32_e32 v4, v4, v5, vcc
+; GCN-NEXT:    v_cmp_eq_u32_e32 vcc, 6, v2
+; GCN-NEXT:    v_mov_b32_e32 v8, s14
+; GCN-NEXT:    v_mov_b32_e32 v9, s15
+; GCN-NEXT:    v_cndmask_b32_e32 v3, v3, v6, vcc
+; GCN-NEXT:    v_cndmask_b32_e32 v4, v4, v7, vcc
+; GCN-NEXT:    v_cmp_eq_u32_e32 vcc, 7, v2
+; GCN-NEXT:    v_cndmask_b32_e32 v2, v3, v8, vcc
+; GCN-NEXT:    v_cndmask_b32_e32 v3, v4, v9, vcc
+; GCN-NEXT:    v_readfirstlane_b32 s0, v0
+; GCN-NEXT:    v_readfirstlane_b32 s1, v1
+; GCN-NEXT:    v_readfirstlane_b32 s2, v2
+; GCN-NEXT:    v_readfirstlane_b32 s3, v3
+; GCN-NEXT:    ; return to shader part epilog
   %vector = load <4 x i128>, <4 x i128> addrspace(4)* %ptr
   %element = extractelement <4 x i128> %vector, i32 %idx
   ret i128 %element
 }
 
 define amdgpu_ps i128 @extractelement_sgpr_v4i128_idx0(<4 x i128> addrspace(4)* inreg %ptr) {
 ; GCN-LABEL: extractelement_sgpr_v4i128_idx0:
 ; GCN:       ; %bb.0:

llvm/test/CodeGen/AMDGPU/GlobalISel/extractelement.ll

entry:
   ret double %ext
 }
 
 define amdgpu_ps double @dyn_extract_v8f64_s_s_offsetm1(<8 x double> inreg %vec, i32 inreg %sel) {
 ; GCN-LABEL: dyn_extract_v8f64_s_s_offsetm1:
 ; GCN:       ; %bb.0: ; %entry
 ; GCN-NEXT:    s_mov_b32 s0, s2
 ; GCN-NEXT:    s_mov_b32 s1, s3
-; GCN-NEXT:    s_add_i32 m0, s18, -1
+; GCN-NEXT:    s_sub_i32 m0, s18, 1
 ; GCN-NEXT:    s_mov_b32 s2, s4
 ; GCN-NEXT:    s_mov_b32 s3, s5
 ; GCN-NEXT:    s_mov_b32 s4, s6
 ; GCN-NEXT:    s_mov_b32 s5, s7
 ; GCN-NEXT:    s_mov_b32 s6, s8
 ; GCN-NEXT:    s_mov_b32 s7, s9
 ; GCN-NEXT:    s_mov_b32 s8, s10
 ; GCN-NEXT:    s_mov_b32 s9, s11

llvm/test/CodeGen/AMDGPU/GlobalISel/saddsat.ll

 ; GFX8-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v0, v1
 ; GFX8-NEXT:    v_bfe_i32 v3, v2, 0, 24
 ; GFX8-NEXT:    v_bfe_i32 v0, v0, 0, 24
 ; GFX8-NEXT:    v_cmp_lt_i32_e64 s[4:5], v3, v0
 ; GFX8-NEXT:    v_bfe_i32 v0, v1, 0, 24
 ; GFX8-NEXT:    v_cmp_gt_i32_e64 s[6:7], 0, v0
 ; GFX8-NEXT:    v_ashrrev_i32_e32 v0, 23, v3
-; GFX8-NEXT:    v_add_u32_e32 v0, vcc, 0xff800000, v0
+; GFX8-NEXT:    v_subrev_u32_e32 v0, vcc, 0x800000, v0
 ; GFX8-NEXT:    s_xor_b64 vcc, s[6:7], s[4:5]
 ; GFX8-NEXT:    v_cndmask_b32_e32 v0, v2, v0, vcc
 ; GFX8-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX9-LABEL: v_saddsat_i24:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 8, v0

 ; GFX8-NEXT:    s_bfe_i32 s0, s0, 0x180000
 ; GFX8-NEXT:    s_cmp_lt_i32 s3, s0
 ; GFX8-NEXT:    s_cselect_b32 s0, 1, 0
 ; GFX8-NEXT:    s_bfe_i32 s1, s1, 0x180000
 ; GFX8-NEXT:    s_cmp_lt_i32 s1, 0
 ; GFX8-NEXT:    s_cselect_b32 s1, 1, 0
 ; GFX8-NEXT:    s_xor_b32 s0, s1, s0
 ; GFX8-NEXT:    s_ashr_i32 s1, s3, 23
-; GFX8-NEXT:    s_add_i32 s1, s1, 0xff800000
+; GFX8-NEXT:    s_sub_i32 s1, s1, 0x800000
 ; GFX8-NEXT:    s_and_b32 s0, s0, 1
 ; GFX8-NEXT:    s_cmp_lg_u32 s0, 0
 ; GFX8-NEXT:    s_cselect_b32 s0, s1, s2
 ; GFX8-NEXT:    ; return to shader part epilog
 ;
 ; GFX9-LABEL: s_saddsat_i24:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_lshl_b32 s1, s1, 8

llvm/test/CodeGen/AMDGPU/GlobalISel/sdiv.i32.ll

 ; CGP-NEXT:    v_cvt_f32_u32_e32 v2, v1
 ; CGP-NEXT:    v_sub_i32_e32 v3, vcc, 0, v1
 ; CGP-NEXT:    v_rcp_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_f32_e32 v2, 0x4f7ffffe, v2
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_lo_u32 v3, v3, v2
 ; CGP-NEXT:    v_mul_lo_u32 v5, 0, v3
 ; CGP-NEXT:    v_mul_hi_u32 v3, v2, v3
-; CGP-NEXT:    v_add_i32_e32 v3, vcc, v5, v3
+; CGP-NEXT:    v_or_b32_e32 v3, v5, v3
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; CGP-NEXT:    v_mul_lo_u32 v3, 0, v2
 ; CGP-NEXT:    v_mul_hi_u32 v2, v0, v2
-; CGP-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
+; CGP-NEXT:    v_or_b32_e32 v2, v3, v2
 ; CGP-NEXT:    v_mul_lo_u32 v3, v2, v1
 ; CGP-NEXT:    v_add_i32_e32 v5, vcc, 1, v2
 ; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v3
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v1
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v2, v5, vcc
 ; CGP-NEXT:    v_sub_i32_e64 v3, s[4:5], v0, v1
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v3, vcc
 ; CGP-NEXT:    v_add_i32_e32 v3, vcc, 1, v2

 ; CGP-NEXT:    v_cvt_f32_u32_e32 v0, s2
 ; CGP-NEXT:    s_sub_i32 s1, 0, s2
 ; CGP-NEXT:    v_rcp_f32_e32 v0, v0
 ; CGP-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; CGP-NEXT:    v_mul_lo_u32 v1, s1, v0
 ; CGP-NEXT:    v_mul_lo_u32 v2, 0, v1
 ; CGP-NEXT:    v_mul_hi_u32 v1, v0, v1
-; CGP-NEXT:    v_add_i32_e32 v1, vcc, v2, v1
+; CGP-NEXT:    v_or_b32_e32 v1, v2, v1
 ; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; CGP-NEXT:    v_mul_lo_u32 v1, 0, v0
 ; CGP-NEXT:    v_mul_hi_u32 v0, s0, v0
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
+; CGP-NEXT:    v_or_b32_e32 v0, v1, v0
 ; CGP-NEXT:    v_mul_lo_u32 v1, v0, s2
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, 1, v0
 ; CGP-NEXT:    v_sub_i32_e32 v1, vcc, s0, v1
 ; CGP-NEXT:    v_cmp_le_u32_e32 vcc, s2, v1
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; CGP-NEXT:    v_subrev_i32_e64 v2, s[0:1], s2, v1
 ; CGP-NEXT:    v_cndmask_b32_e32 v1, v1, v2, vcc
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, 1, v0

 ; CGP-NEXT:    v_cvt_u32_f32_e32 v4, v4
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v6, v6
 ; CGP-NEXT:    v_mul_lo_u32 v5, v5, v4
 ; CGP-NEXT:    v_mul_lo_u32 v7, v7, v6
 ; CGP-NEXT:    v_mul_lo_u32 v10, 0, v5
 ; CGP-NEXT:    v_mul_hi_u32 v5, v4, v5
 ; CGP-NEXT:    v_mul_lo_u32 v11, 0, v7
 ; CGP-NEXT:    v_mul_hi_u32 v7, v6, v7
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v10, v5
-; CGP-NEXT:    v_add_i32_e32 v7, vcc, v11, v7
+; CGP-NEXT:    v_or_b32_e32 v5, v10, v5
+; CGP-NEXT:    v_or_b32_e32 v7, v11, v7
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v5
 ; CGP-NEXT:    v_add_i32_e32 v5, vcc, v6, v7
 ; CGP-NEXT:    v_mul_lo_u32 v6, 0, v4
 ; CGP-NEXT:    v_mul_hi_u32 v4, v0, v4
 ; CGP-NEXT:    v_mul_lo_u32 v7, 0, v5
 ; CGP-NEXT:    v_mul_hi_u32 v5, v1, v5
-; CGP-NEXT:    v_add_i32_e32 v4, vcc, v6, v4
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
+; CGP-NEXT:    v_or_b32_e32 v4, v6, v4
+; CGP-NEXT:    v_or_b32_e32 v5, v7, v5
 ; CGP-NEXT:    v_mul_lo_u32 v6, v4, v2
 ; CGP-NEXT:    v_add_i32_e32 v7, vcc, 1, v4
 ; CGP-NEXT:    v_mul_lo_u32 v10, v5, v3
 ; CGP-NEXT:    v_add_i32_e32 v11, vcc, 1, v5
 ; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v6
 ; CGP-NEXT:    v_sub_i32_e32 v1, vcc, v1, v10
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v2
 ; CGP-NEXT:    v_cndmask_b32_e32 v4, v4, v7, vcc
; CHECK-NEXT:    s_setpc_b64 s[30:31]
   ret i32 %result
 }
 
 define <2 x i32> @v_sdiv_v2i32_pow2k_denom(<2 x i32> %num) {
 ; GISEL-LABEL: v_sdiv_v2i32_pow2k_denom:
 ; GISEL:       ; %bb.0:
 ; GISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GISEL-NEXT:    v_ashrrev_i32_e32 v2, 31, v0
-; GISEL-NEXT:    s_add_i32 s8, 0x1000, 0
+; GISEL-NEXT:    s_movk_i32 s8, 0x1000
 ; GISEL-NEXT:    v_ashrrev_i32_e32 v3, 31, v1
 ; GISEL-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GISEL-NEXT:    v_cvt_f32_u32_e32 v4, s8
 ; GISEL-NEXT:    s_sub_i32 s4, 0, s8
 ; GISEL-NEXT:    v_add_i32_e32 v1, vcc, v1, v3
 ; GISEL-NEXT:    v_xor_b32_e32 v0, v0, v2
 ; GISEL-NEXT:    v_rcp_iflag_f32_e32 v4, v4
 ; GISEL-NEXT:    v_xor_b32_e32 v1, v1, v3
; CHECK-NEXT:    s_setpc_b64 s[30:31]
   ret i32 %result
 }
 
 define <2 x i32> @v_sdiv_v2i32_oddk_denom(<2 x i32> %num) {
 ; GISEL-LABEL: v_sdiv_v2i32_oddk_denom:
 ; GISEL:       ; %bb.0:
 ; GISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GISEL-NEXT:    v_ashrrev_i32_e32 v2, 31, v0
-; GISEL-NEXT:    s_add_i32 s8, 0x12d8fb, 0
+; GISEL-NEXT:    s_mov_b32 s8, 0x12d8fb
 ; GISEL-NEXT:    v_ashrrev_i32_e32 v3, 31, v1
 ; GISEL-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GISEL-NEXT:    v_cvt_f32_u32_e32 v4, s8
 ; GISEL-NEXT:    s_sub_i32 s4, 0, s8
 ; GISEL-NEXT:    v_add_i32_e32 v1, vcc, v1, v3
 ; GISEL-NEXT:    v_xor_b32_e32 v0, v0, v2
 ; GISEL-NEXT:    v_rcp_iflag_f32_e32 v4, v4
 ; GISEL-NEXT:    v_xor_b32_e32 v1, v1, v3

 ; CGP-NEXT:    v_cvt_u32_f32_e32 v6, v6
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v8, v8
 ; CGP-NEXT:    v_mul_lo_u32 v7, v7, v6
 ; CGP-NEXT:    v_mul_lo_u32 v9, v9, v8
 ; CGP-NEXT:    v_mul_lo_u32 v10, 0, v7
 ; CGP-NEXT:    v_mul_hi_u32 v7, v6, v7
 ; CGP-NEXT:    v_mul_lo_u32 v11, 0, v9
 ; CGP-NEXT:    v_mul_hi_u32 v9, v8, v9
-; CGP-NEXT:    v_add_i32_e32 v7, vcc, v10, v7
-; CGP-NEXT:    v_add_i32_e32 v9, vcc, v11, v9
+; CGP-NEXT:    v_or_b32_e32 v7, v10, v7
+; CGP-NEXT:    v_or_b32_e32 v9, v11, v9
 ; CGP-NEXT:    v_add_i32_e32 v6, vcc, v6, v7
 ; CGP-NEXT:    v_add_i32_e32 v7, vcc, v8, v9
 ; CGP-NEXT:    v_mul_lo_u32 v8, 0, v6
 ; CGP-NEXT:    v_mul_hi_u32 v6, v0, v6
 ; CGP-NEXT:    v_mul_lo_u32 v9, 0, v7
 ; CGP-NEXT:    v_mul_hi_u32 v7, v1, v7
-; CGP-NEXT:    v_add_i32_e32 v6, vcc, v8, v6
-; CGP-NEXT:    v_add_i32_e32 v7, vcc, v9, v7
+; CGP-NEXT:    v_or_b32_e32 v6, v8, v6
+; CGP-NEXT:    v_or_b32_e32 v7, v9, v7
 ; CGP-NEXT:    v_mul_lo_u32 v8, v6, v2
 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, 1, v6
 ; CGP-NEXT:    v_mul_lo_u32 v10, v7, v3
 ; CGP-NEXT:    v_add_i32_e32 v11, vcc, 1, v7
 ; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v8
 ; CGP-NEXT:    v_sub_i32_e32 v1, vcc, v1, v10
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v2
 ; CGP-NEXT:    v_cndmask_b32_e32 v6, v6, v9, vcc

 ; GISEL-LABEL: v_sdiv_i32_24bit:
 ; GISEL:       ; %bb.0:
 ; GISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GISEL-NEXT:    s_mov_b32 s4, 0xffffff
 ; GISEL-NEXT:    v_and_b32_e32 v0, s4, v0
 ; GISEL-NEXT:    v_and_b32_e32 v1, s4, v1
 ; GISEL-NEXT:    v_ashrrev_i32_e32 v2, 31, v0
 ; GISEL-NEXT:    v_ashrrev_i32_e32 v3, 31, v1
-; GISEL-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
-; GISEL-NEXT:    v_add_i32_e32 v1, vcc, v1, v3
+; GISEL-NEXT:    v_or_b32_e32 v0, v0, v2
+; GISEL-NEXT:    v_or_b32_e32 v1, v1, v3
 ; GISEL-NEXT:    v_xor_b32_e32 v0, v0, v2
 ; GISEL-NEXT:    v_xor_b32_e32 v1, v1, v3
 ; GISEL-NEXT:    v_cvt_f32_u32_e32 v4, v1
 ; GISEL-NEXT:    v_sub_i32_e32 v5, vcc, 0, v1
 ; GISEL-NEXT:    v_rcp_iflag_f32_e32 v4, v4
 ; GISEL-NEXT:    v_mul_f32_e32 v4, 0x4f7ffffe, v4
 ; GISEL-NEXT:    v_cvt_u32_f32_e32 v4, v4
 ; GISEL-NEXT:    v_mul_lo_u32 v5, v5, v4

 ; CGP-NEXT:    v_cvt_f32_u32_e32 v2, v1
 ; CGP-NEXT:    v_sub_i32_e32 v3, vcc, 0, v1
 ; CGP-NEXT:    v_rcp_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_f32_e32 v2, 0x4f7ffffe, v2
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_lo_u32 v3, v3, v2
 ; CGP-NEXT:    v_mul_lo_u32 v4, 0, v3
 ; CGP-NEXT:    v_mul_hi_u32 v3, v2, v3
-; CGP-NEXT:    v_add_i32_e32 v3, vcc, v4, v3
+; CGP-NEXT:    v_or_b32_e32 v3, v4, v3
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; CGP-NEXT:    v_mul_lo_u32 v3, 0, v2
 ; CGP-NEXT:    v_mul_hi_u32 v2, v0, v2
-; CGP-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
+; CGP-NEXT:    v_or_b32_e32 v2, v3, v2
 ; CGP-NEXT:    v_mul_lo_u32 v3, v2, v1
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, 1, v2
 ; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v3
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v1
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v2, v4, vcc
 ; CGP-NEXT:    v_sub_i32_e64 v3, s[4:5], v0, v1
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v3, vcc
 ; CGP-NEXT:    v_add_i32_e32 v3, vcc, 1, v2

 ; CGP-NEXT:    v_cvt_u32_f32_e32 v4, v4
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v6, v6
 ; CGP-NEXT:    v_mul_lo_u32 v5, v5, v4
 ; CGP-NEXT:    v_mul_lo_u32 v7, v7, v6
 ; CGP-NEXT:    v_mul_lo_u32 v8, 0, v5
 ; CGP-NEXT:    v_mul_hi_u32 v5, v4, v5
 ; CGP-NEXT:    v_mul_lo_u32 v9, 0, v7
 ; CGP-NEXT:    v_mul_hi_u32 v7, v6, v7
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v8, v5
-; CGP-NEXT:    v_add_i32_e32 v7, vcc, v9, v7
+; CGP-NEXT:    v_or_b32_e32 v5, v8, v5
+; CGP-NEXT:    v_or_b32_e32 v7, v9, v7
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v5
 ; CGP-NEXT:    v_add_i32_e32 v5, vcc, v6, v7
 ; CGP-NEXT:    v_mul_lo_u32 v6, 0, v4
 ; CGP-NEXT:    v_mul_hi_u32 v4, v0, v4
 ; CGP-NEXT:    v_mul_lo_u32 v7, 0, v5
 ; CGP-NEXT:    v_mul_hi_u32 v5, v1, v5
-; CGP-NEXT:    v_add_i32_e32 v4, vcc, v6, v4
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
+; CGP-NEXT:    v_or_b32_e32 v4, v6, v4
+; CGP-NEXT:    v_or_b32_e32 v5, v7, v5
 ; CGP-NEXT:    v_mul_lo_u32 v6, v4, v2
 ; CGP-NEXT:    v_add_i32_e32 v7, vcc, 1, v4
 ; CGP-NEXT:    v_mul_lo_u32 v8, v5, v3
 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, 1, v5
 ; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v6
 ; CGP-NEXT:    v_sub_i32_e32 v1, vcc, v1, v8
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v2
 ; CGP-NEXT:    v_cndmask_b32_e32 v4, v4, v7, vcc

llvm/test/CodeGen/AMDGPU/GlobalISel/sdiv.i64.ll

 ; CHECK-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v5
 ; CHECK-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v5, vcc, v6, v5
 ; CHECK-NEXT:    v_add_i32_e32 v4, vcc, v4, v5
 ; CHECK-NEXT:    v_mul_lo_u32 v5, 0, v2
 ; CHECK-NEXT:    v_mul_lo_u32 v6, s6, v4
-; CHECK-NEXT:    v_mul_hi_u32 v8, s6, v2
-; CHECK-NEXT:    v_mul_lo_u32 v7, s6, v2
-; CHECK-NEXT:    v_add_i32_e32 v5, vcc, v5, v6
-; CHECK-NEXT:    v_add_i32_e32 v5, vcc, v5, v8
-; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v0, v7
+; CHECK-NEXT:    v_mul_hi_u32 v7, s6, v2
+; CHECK-NEXT:    v_mul_lo_u32 v8, s6, v2
+; CHECK-NEXT:    v_or_b32_e32 v5, v5, v6
+; CHECK-NEXT:    v_add_i32_e32 v5, vcc, v5, v7
+; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v0, v8
 ; CHECK-NEXT:    v_subb_u32_e64 v6, s[4:5], v1, v5, vcc
 ; CHECK-NEXT:    v_sub_i32_e64 v1, s[4:5], v1, v5
 ; CHECK-NEXT:    v_cmp_le_u32_e64 s[4:5], 0, v6
 ; CHECK-NEXT:    v_subbrev_u32_e32 v1, vcc, 0, v1, vcc
 ; CHECK-NEXT:    v_cndmask_b32_e64 v5, 0, -1, s[4:5]
 ; CHECK-NEXT:    v_cmp_le_u32_e64 s[4:5], s6, v0
 ; CHECK-NEXT:    v_subrev_i32_e32 v0, vcc, s6, v0
 ; CHECK-NEXT:    v_subbrev_u32_e32 v1, vcc, 0, v1, vcc

 ; CGP-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v10, vcc, v11, v10
 ; CGP-NEXT:    v_add_i32_e32 v7, vcc, v7, v9
 ; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, v10, v9
 ; CGP-NEXT:    v_add_i32_e32 v8, vcc, v8, v9
 ; CGP-NEXT:    v_mul_lo_u32 v9, 0, v7
 ; CGP-NEXT:    v_mul_lo_u32 v10, s7, v8
-; CGP-NEXT:    v_mul_hi_u32 v12, s7, v7
-; CGP-NEXT:    v_mul_lo_u32 v11, s7, v7
-; CGP-NEXT:    v_add_i32_e32 v9, vcc, v9, v10
-; CGP-NEXT:    v_add_i32_e32 v9, vcc, v9, v12
-; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v11
+; CGP-NEXT:    v_mul_hi_u32 v11, s7, v7
+; CGP-NEXT:    v_mul_lo_u32 v12, s7, v7
+; CGP-NEXT:    v_or_b32_e32 v9, v9, v10
+; CGP-NEXT:    v_add_i32_e32 v9, vcc, v9, v11
+; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v12
 ; CGP-NEXT:    v_subb_u32_e64 v10, s[4:5], v1, v9, vcc
 ; CGP-NEXT:    v_sub_i32_e64 v1, s[4:5], v1, v9
 ; CGP-NEXT:    v_cmp_le_u32_e64 s[4:5], 0, v10
 ; CGP-NEXT:    v_subbrev_u32_e32 v1, vcc, 0, v1, vcc
 ; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, -1, s[4:5]
 ; CGP-NEXT:    v_cmp_le_u32_e64 s[4:5], s7, v0
 ; CGP-NEXT:    v_subrev_i32_e32 v0, vcc, s7, v0
 ; CGP-NEXT:    v_subbrev_u32_e32 v1, vcc, 0, v1, vcc

 ; CGP-NEXT:    v_mul_lo_u32 v10, s6, v4
 ; CGP-NEXT:    v_addc_u32_e32 v3, vcc, v3, v6, vcc
 ; CGP-NEXT:    v_add_i32_e32 v8, vcc, v8, v9
 ; CGP-NEXT:    v_add_i32_e32 v8, vcc, v8, v11
 ; CGP-NEXT:    v_mul_lo_u32 v9, v7, v10
 ; CGP-NEXT:    v_mul_lo_u32 v11, v4, v8
 ; CGP-NEXT:    v_mul_hi_u32 v12, v4, v10
 ; CGP-NEXT:    v_mul_hi_u32 v10, v7, v10
-; CGP-NEXT:    v_xor_b32_e32 v0, v0, v5
+; CGP-NEXT:    v_xor_b32_e32 v2, v2, v6
 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, v9, v11
 ; CGP-NEXT:    v_cndmask_b32_e64 v11, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, v9, v12
 ; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
 ; CGP-NEXT:    v_mul_lo_u32 v12, v7, v8
 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, v11, v9
 ; CGP-NEXT:    v_mul_hi_u32 v11, v4, v8
 ; CGP-NEXT:    v_mul_hi_u32 v8, v7, v8

 ; CGP-NEXT:    v_mul_lo_u32 v12, s6, v4
 ; CGP-NEXT:    v_add_i32_e64 v7, s[4:5], v7, v8
 ; CGP-NEXT:    v_add_i32_e64 v10, s[4:5], v10, v11
 ; CGP-NEXT:    v_add_i32_e64 v10, s[4:5], v10, v13
 ; CGP-NEXT:    v_mul_lo_u32 v11, v9, v12
 ; CGP-NEXT:    v_mul_lo_u32 v13, v4, v10
 ; CGP-NEXT:    v_mul_hi_u32 v8, v4, v12
 ; CGP-NEXT:    v_mul_hi_u32 v12, v9, v12
-; CGP-NEXT:    v_xor_b32_e32 v2, v2, v6
+; CGP-NEXT:    v_xor_b32_e32 v3, v3, v6
 ; CGP-NEXT:    v_add_i32_e64 v11, s[4:5], v11, v13
 ; CGP-NEXT:    v_cndmask_b32_e64 v13, 0, 1, s[4:5]
 ; CGP-NEXT:    v_add_i32_e64 v8, s[4:5], v11, v8
 ; CGP-NEXT:    v_cndmask_b32_e64 v8, 0, 1, s[4:5]
 ; CGP-NEXT:    v_mul_lo_u32 v11, v9, v10
 ; CGP-NEXT:    v_add_i32_e64 v8, s[4:5], v13, v8
 ; CGP-NEXT:    v_mul_hi_u32 v13, v4, v10
 ; CGP-NEXT:    v_mul_hi_u32 v9, v9, v10
 ; CGP-NEXT:    v_add_i32_e64 v11, s[4:5], v11, v12
 ; CGP-NEXT:    v_cndmask_b32_e64 v12, 0, 1, s[4:5]
 ; CGP-NEXT:    v_add_i32_e64 v11, s[4:5], v11, v13
 ; CGP-NEXT:    v_cndmask_b32_e64 v13, 0, 1, s[4:5]
 ; CGP-NEXT:    v_add_i32_e64 v12, s[4:5], v12, v13
 ; CGP-NEXT:    v_add_i32_e64 v8, s[4:5], v11, v8
 ; CGP-NEXT:    v_cndmask_b32_e64 v11, 0, 1, s[4:5]
 ; CGP-NEXT:    v_add_i32_e64 v10, s[4:5], v12, v11
 ; CGP-NEXT:    v_add_i32_e64 v9, s[4:5], v9, v10
 ; CGP-NEXT:    v_addc_u32_e32 v7, vcc, v7, v9, vcc
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v8
 ; CGP-NEXT:    v_addc_u32_e32 v7, vcc, 0, v7, vcc
-; CGP-NEXT:    v_xor_b32_e32 v3, v3, v6
-; CGP-NEXT:    v_xor_b32_e32 v1, v1, v5
-; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v5
 ; CGP-NEXT:    v_mul_lo_u32 v8, v3, v4
 ; CGP-NEXT:    v_mul_lo_u32 v9, v2, v7
-; CGP-NEXT:    v_subb_u32_e32 v1, vcc, v1, v5, vcc
-; CGP-NEXT:    v_mul_hi_u32 v5, v2, v4
+; CGP-NEXT:    v_mul_hi_u32 v10, v2, v4
 ; CGP-NEXT:    v_mul_hi_u32 v4, v3, v4
+; CGP-NEXT:    v_xor_b32_e32 v0, v0, v5
 ; CGP-NEXT:    v_add_i32_e32 v8, vcc, v8, v9
 ; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v8, v5
-; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
-; CGP-NEXT:    v_mul_lo_u32 v8, v3, v7
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v9, v5
+; CGP-NEXT:    v_add_i32_e32 v8, vcc, v8, v10
+; CGP-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
+; CGP-NEXT:    v_mul_lo_u32 v10, v3, v7
+; CGP-NEXT:    v_add_i32_e32 v8, vcc, v9, v8
 ; CGP-NEXT:    v_mul_hi_u32 v9, v2, v7
 ; CGP-NEXT:    v_mul_hi_u32 v7, v3, v7
-; CGP-NEXT:    v_add_i32_e32 v4, vcc, v8, v4
-; CGP-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v4, vcc, v10, v4
+; CGP-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v9
 ; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v8, vcc, v8, v9
-; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v5
-; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v8, v5
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
-; CGP-NEXT:    v_mul_lo_u32 v7, 0, v4
-; CGP-NEXT:    v_mul_lo_u32 v8, s7, v5
-; CGP-NEXT:    v_mul_hi_u32 v10, s7, v4
-; CGP-NEXT:    v_mul_lo_u32 v9, s7, v4
+; CGP-NEXT:    v_add_i32_e32 v9, vcc, v10, v9
+; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v8
+; CGP-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v8, vcc, v9, v8
 ; CGP-NEXT:    v_add_i32_e32 v7, vcc, v7, v8
-; CGP-NEXT:    v_add_i32_e32 v7, vcc, v7, v10
-; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v2, v9
-; CGP-NEXT:    v_subb_u32_e64 v8, s[4:5], v3, v7, vcc
-; CGP-NEXT:    v_sub_i32_e64 v3, s[4:5], v3, v7
+; CGP-NEXT:    v_xor_b32_e32 v1, v1, v5
+; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v5
+; CGP-NEXT:    v_mul_lo_u32 v8, 0, v4
+; CGP-NEXT:    v_mul_lo_u32 v9, s7, v7
+; CGP-NEXT:    v_subb_u32_e32 v1, vcc, v1, v5, vcc
+; CGP-NEXT:    v_mul_hi_u32 v5, s7, v4
+; CGP-NEXT:    v_mul_lo_u32 v10, s7, v4
+; CGP-NEXT:    v_or_b32_e32 v8, v8, v9
+; CGP-NEXT:    v_add_i32_e32 v5, vcc, v8, v5
+; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v2, v10
+; CGP-NEXT:    v_subb_u32_e64 v8, s[4:5], v3, v5, vcc
+; CGP-NEXT:    v_sub_i32_e64 v3, s[4:5], v3, v5
 ; CGP-NEXT:    v_cmp_le_u32_e64 s[4:5], 0, v8
 ; CGP-NEXT:    v_subbrev_u32_e32 v3, vcc, 0, v3, vcc
-; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, -1, s[4:5]
+; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, -1, s[4:5]
 ; CGP-NEXT:    v_cmp_le_u32_e64 s[4:5], s7, v2
 ; CGP-NEXT:    v_subrev_i32_e32 v2, vcc, s7, v2
 ; CGP-NEXT:    v_subbrev_u32_e32 v3, vcc, 0, v3, vcc
 ; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, -1, s[4:5]
 ; CGP-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v8
 ; CGP-NEXT:    v_add_i32_e32 v8, vcc, 1, v4
-; CGP-NEXT:    v_cndmask_b32_e64 v7, v7, v9, s[4:5]
-; CGP-NEXT:    v_addc_u32_e32 v9, vcc, 0, v5, vcc
+; CGP-NEXT:    v_cndmask_b32_e64 v5, v5, v9, s[4:5]
+; CGP-NEXT:    v_addc_u32_e32 v9, vcc, 0, v7, vcc
 ; CGP-NEXT:    v_cmp_le_u32_e32 vcc, 0, v3
 ; CGP-NEXT:    v_cndmask_b32_e64 v10, 0, -1, vcc
 ; CGP-NEXT:    v_cmp_le_u32_e32 vcc, s7, v2
 ; CGP-NEXT:    v_cndmask_b32_e64 v2, 0, -1, vcc
 ; CGP-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v3
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v10, v2, vcc
 ; CGP-NEXT:    v_add_i32_e32 v3, vcc, 1, v8
 ; CGP-NEXT:    v_addc_u32_e32 v10, vcc, 0, v9, vcc
 ; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v2
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v8, v3, vcc
 ; CGP-NEXT:    v_cndmask_b32_e32 v3, v9, v10, vcc
-; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v7
+; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v5
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v4, v2, vcc
-; CGP-NEXT:    v_cndmask_b32_e32 v3, v5, v3, vcc
+; CGP-NEXT:    v_cndmask_b32_e32 v3, v7, v3, vcc
 ; CGP-NEXT:    v_xor_b32_e32 v2, v2, v6
 ; CGP-NEXT:    v_xor_b32_e32 v3, v3, v6
 ; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v2, v6
 ; CGP-NEXT:    v_subb_u32_e32 v3, vcc, v3, v6, vcc
 ; CGP-NEXT:    s_setpc_b64 s[30:31]
   %result = sdiv <2 x i64> %num, <i64 4096, i64 4096>
   ret <2 x i64> %result
 }

 ; CHECK-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v5
 ; CHECK-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v5, vcc, v6, v5
 ; CHECK-NEXT:    v_add_i32_e32 v4, vcc, v4, v5
 ; CHECK-NEXT:    v_mul_lo_u32 v5, 0, v2
 ; CHECK-NEXT:    v_mul_lo_u32 v6, s6, v4
-; CHECK-NEXT:    v_mul_hi_u32 v8, s6, v2
-; CHECK-NEXT:    v_mul_lo_u32 v7, s6, v2
-; CHECK-NEXT:    v_add_i32_e32 v5, vcc, v5, v6
-; CHECK-NEXT:    v_add_i32_e32 v5, vcc, v5, v8
-; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v0, v7
+; CHECK-NEXT:    v_mul_hi_u32 v7, s6, v2
+; CHECK-NEXT:    v_mul_lo_u32 v8, s6, v2
+; CHECK-NEXT:    v_or_b32_e32 v5, v5, v6
+; CHECK-NEXT:    v_add_i32_e32 v5, vcc, v5, v7
+; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v0, v8
 ; CHECK-NEXT:    v_subb_u32_e64 v6, s[4:5], v1, v5, vcc
 ; CHECK-NEXT:    v_sub_i32_e64 v1, s[4:5], v1, v5
 ; CHECK-NEXT:    v_cmp_le_u32_e64 s[4:5], 0, v6
 ; CHECK-NEXT:    v_subbrev_u32_e32 v1, vcc, 0, v1, vcc
 ; CHECK-NEXT:    v_cndmask_b32_e64 v5, 0, -1, s[4:5]
 ; CHECK-NEXT:    v_cmp_le_u32_e64 s[4:5], s6, v0
 ; CHECK-NEXT:    v_subrev_i32_e32 v0, vcc, s6, v0
 ; CHECK-NEXT:    v_subbrev_u32_e32 v1, vcc, 0, v1, vcc

 ; CGP-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v10, vcc, v11, v10
 ; CGP-NEXT:    v_add_i32_e32 v7, vcc, v7, v9
 ; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, v10, v9
 ; CGP-NEXT:    v_add_i32_e32 v8, vcc, v8, v9
 ; CGP-NEXT:    v_mul_lo_u32 v9, 0, v7
 ; CGP-NEXT:    v_mul_lo_u32 v10, s7, v8
-; CGP-NEXT:    v_mul_hi_u32 v12, s7, v7
-; CGP-NEXT:    v_mul_lo_u32 v11, s7, v7
-; CGP-NEXT:    v_add_i32_e32 v9, vcc, v9, v10
-; CGP-NEXT:    v_add_i32_e32 v9, vcc, v9, v12
-; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v11
+; CGP-NEXT:    v_mul_hi_u32 v11, s7, v7
+; CGP-NEXT:    v_mul_lo_u32 v12, s7, v7
+; CGP-NEXT:    v_or_b32_e32 v9, v9, v10
+; CGP-NEXT:    v_add_i32_e32 v9, vcc, v9, v11
+; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v12
 ; CGP-NEXT:    v_subb_u32_e64 v10, s[4:5], v1, v9, vcc
 ; CGP-NEXT:    v_sub_i32_e64 v1, s[4:5], v1, v9
 ; CGP-NEXT:    v_cmp_le_u32_e64 s[4:5], 0, v10
 ; CGP-NEXT:    v_subbrev_u32_e32 v1, vcc, 0, v1, vcc
 ; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, -1, s[4:5]
 ; CGP-NEXT:    v_cmp_le_u32_e64 s[4:5], s7, v0
 ; CGP-NEXT:    v_subrev_i32_e32 v0, vcc, s7, v0
 ; CGP-NEXT:    v_subbrev_u32_e32 v1, vcc, 0, v1, vcc

 ; CGP-NEXT:    v_mul_lo_u32 v10, s6, v4
 ; CGP-NEXT:    v_addc_u32_e32 v3, vcc, v3, v6, vcc
 ; CGP-NEXT:    v_add_i32_e32 v8, vcc, v8, v9
 ; CGP-NEXT:    v_add_i32_e32 v8, vcc, v8, v11
 ; CGP-NEXT:    v_mul_lo_u32 v9, v7, v10
 ; CGP-NEXT:    v_mul_lo_u32 v11, v4, v8
 ; CGP-NEXT:    v_mul_hi_u32 v12, v4, v10
 ; CGP-NEXT:    v_mul_hi_u32 v10, v7, v10
-; CGP-NEXT:    v_xor_b32_e32 v0, v0, v5
+; CGP-NEXT:    v_xor_b32_e32 v2, v2, v6
 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, v9, v11
 ; CGP-NEXT:    v_cndmask_b32_e64 v11, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, v9, v12
 ; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
 ; CGP-NEXT:    v_mul_lo_u32 v12, v7, v8
 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, v11, v9
 ; CGP-NEXT:    v_mul_hi_u32 v11, v4, v8
 ; CGP-NEXT:    v_mul_hi_u32 v8, v7, v8

 ; CGP-NEXT:    v_mul_lo_u32 v12, s6, v4
 ; CGP-NEXT:    v_add_i32_e64 v7, s[4:5], v7, v8
 ; CGP-NEXT:    v_add_i32_e64 v10, s[4:5], v10, v11
 ; CGP-NEXT:    v_add_i32_e64 v10, s[4:5], v10, v13
 ; CGP-NEXT:    v_mul_lo_u32 v11, v9, v12
 ; CGP-NEXT:    v_mul_lo_u32 v13, v4, v10
 ; CGP-NEXT:    v_mul_hi_u32 v8, v4, v12
 ; CGP-NEXT:    v_mul_hi_u32 v12, v9, v12
-; CGP-NEXT:    v_xor_b32_e32 v2, v2, v6
+; CGP-NEXT:    v_xor_b32_e32 v3, v3, v6
 ; CGP-NEXT:    v_add_i32_e64 v11, s[4:5], v11, v13
 ; CGP-NEXT:    v_cndmask_b32_e64 v13, 0, 1, s[4:5]
 ; CGP-NEXT:    v_add_i32_e64 v8, s[4:5], v11, v8
 ; CGP-NEXT:    v_cndmask_b32_e64 v8, 0, 1, s[4:5]
 ; CGP-NEXT:    v_mul_lo_u32 v11, v9, v10
 ; CGP-NEXT:    v_add_i32_e64 v8, s[4:5], v13, v8
 ; CGP-NEXT:    v_mul_hi_u32 v13, v4, v10
 ; CGP-NEXT:    v_mul_hi_u32 v9, v9, v10
 ; CGP-NEXT:    v_add_i32_e64 v11, s[4:5], v11, v12
 ; CGP-NEXT:    v_cndmask_b32_e64 v12, 0, 1, s[4:5]
 ; CGP-NEXT:    v_add_i32_e64 v11, s[4:5], v11, v13
 ; CGP-NEXT:    v_cndmask_b32_e64 v13, 0, 1, s[4:5]
 ; CGP-NEXT:    v_add_i32_e64 v12, s[4:5], v12, v13
 ; CGP-NEXT:    v_add_i32_e64 v8, s[4:5], v11, v8
 ; CGP-NEXT:    v_cndmask_b32_e64 v11, 0, 1, s[4:5]
 ; CGP-NEXT:    v_add_i32_e64 v10, s[4:5], v12, v11
 ; CGP-NEXT:    v_add_i32_e64 v9, s[4:5], v9, v10
 ; CGP-NEXT:    v_addc_u32_e32 v7, vcc, v7, v9, vcc
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v8
 ; CGP-NEXT:    v_addc_u32_e32 v7, vcc, 0, v7, vcc
-; CGP-NEXT:    v_xor_b32_e32 v3, v3, v6
-; CGP-NEXT:    v_xor_b32_e32 v1, v1, v5
-; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v5
 ; CGP-NEXT:    v_mul_lo_u32 v8, v3, v4
 ; CGP-NEXT:    v_mul_lo_u32 v9, v2, v7
-; CGP-NEXT:    v_subb_u32_e32 v1, vcc, v1, v5, vcc
-; CGP-NEXT:    v_mul_hi_u32 v5, v2, v4
+; CGP-NEXT:    v_mul_hi_u32 v10, v2, v4
 ; CGP-NEXT:    v_mul_hi_u32 v4, v3, v4
+; CGP-NEXT:    v_xor_b32_e32 v0, v0, v5
 ; CGP-NEXT:    v_add_i32_e32 v8, vcc, v8, v9
 ; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v8, v5
-; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
-; CGP-NEXT:    v_mul_lo_u32 v8, v3, v7
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v9, v5
+; CGP-NEXT:    v_add_i32_e32 v8, vcc, v8, v10
+; CGP-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
+; CGP-NEXT:    v_mul_lo_u32 v10, v3, v7
+; CGP-NEXT:    v_add_i32_e32 v8, vcc, v9, v8
 ; CGP-NEXT:    v_mul_hi_u32 v9, v2, v7
 ; CGP-NEXT:    v_mul_hi_u32 v7, v3, v7
-; CGP-NEXT:    v_add_i32_e32 v4, vcc, v8, v4
-; CGP-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v4, vcc, v10, v4
+; CGP-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v9
 ; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v8, vcc, v8, v9
-; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v5
-; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v8, v5
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
-; CGP-NEXT:    v_mul_lo_u32 v7, 0, v4
-; CGP-NEXT:    v_mul_lo_u32 v8, s7, v5
-; CGP-NEXT:    v_mul_hi_u32 v10, s7, v4
-; CGP-NEXT:    v_mul_lo_u32 v9, s7, v4
+; CGP-NEXT:    v_add_i32_e32 v9, vcc, v10, v9
+; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v8
+; CGP-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v8, vcc, v9, v8
 ; CGP-NEXT:    v_add_i32_e32 v7, vcc, v7, v8
-; CGP-NEXT:    v_add_i32_e32 v7, vcc, v7, v10
-; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v2, v9
-; CGP-NEXT:    v_subb_u32_e64 v8, s[4:5], v3, v7, vcc
-; CGP-NEXT:    v_sub_i32_e64 v3, s[4:5], v3, v7
+; CGP-NEXT:    v_xor_b32_e32 v1, v1, v5
+; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v5
+; CGP-NEXT:    v_mul_lo_u32 v8, 0, v4
+; CGP-NEXT:    v_mul_lo_u32 v9, s7, v7
+; CGP-NEXT:    v_subb_u32_e32 v1, vcc, v1, v5, vcc
+; CGP-NEXT:    v_mul_hi_u32 v5, s7, v4
+; CGP-NEXT:    v_mul_lo_u32 v10, s7, v4
+; CGP-NEXT:    v_or_b32_e32 v8, v8, v9
+; CGP-NEXT:    v_add_i32_e32 v5, vcc, v8, v5
+; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v2, v10
+; CGP-NEXT:    v_subb_u32_e64 v8, s[4:5], v3, v5, vcc
+; CGP-NEXT:    v_sub_i32_e64 v3, s[4:5], v3, v5
 ; CGP-NEXT:    v_cmp_le_u32_e64 s[4:5], 0, v8
 ; CGP-NEXT:    v_subbrev_u32_e32 v3, vcc, 0, v3, vcc
-; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, -1, s[4:5]
+; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, -1, s[4:5]
 ; CGP-NEXT:    v_cmp_le_u32_e64 s[4:5], s7, v2
 ; CGP-NEXT:    v_subrev_i32_e32 v2, vcc, s7, v2
 ; CGP-NEXT:    v_subbrev_u32_e32 v3, vcc, 0, v3, vcc
 ; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, -1, s[4:5]
 ; CGP-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v8
 ; CGP-NEXT:    v_add_i32_e32 v8, vcc, 1, v4
-; CGP-NEXT:    v_cndmask_b32_e64 v7, v7, v9, s[4:5]
-; CGP-NEXT:    v_addc_u32_e32 v9, vcc, 0, v5, vcc
+; CGP-NEXT:    v_cndmask_b32_e64 v5, v5, v9, s[4:5]
+; CGP-NEXT:    v_addc_u32_e32 v9, vcc, 0, v7, vcc
 ; CGP-NEXT:    v_cmp_le_u32_e32 vcc, 0, v3
 ; CGP-NEXT:    v_cndmask_b32_e64 v10, 0, -1, vcc
 ; CGP-NEXT:    v_cmp_le_u32_e32 vcc, s7, v2
 ; CGP-NEXT:    v_cndmask_b32_e64 v2, 0, -1, vcc
 ; CGP-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v3
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v10, v2, vcc
 ; CGP-NEXT:    v_add_i32_e32 v3, vcc, 1, v8
 ; CGP-NEXT:    v_addc_u32_e32 v10, vcc, 0, v9, vcc
 ; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v2
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v8, v3, vcc
 ; CGP-NEXT:    v_cndmask_b32_e32 v3, v9, v10, vcc
-; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v7
+; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v5
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v4, v2, vcc
-; CGP-NEXT:    v_cndmask_b32_e32 v3, v5, v3, vcc
+; CGP-NEXT:    v_cndmask_b32_e32 v3, v7, v3, vcc
 ; CGP-NEXT:    v_xor_b32_e32 v2, v2, v6
 ; CGP-NEXT:    v_xor_b32_e32 v3, v3, v6
 ; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v2, v6
 ; CGP-NEXT:    v_subb_u32_e32 v3, vcc, v3, v6, vcc
 ; CGP-NEXT:    s_setpc_b64 s[30:31]
   %result = sdiv <2 x i64> %num, <i64 1235195, i64 1235195>
   ret <2 x i64> %result
 }

llvm/test/CodeGen/AMDGPU/GlobalISel/srem.i32.ll

 ; CGP-NEXT:    v_cvt_f32_u32_e32 v3, v1
 ; CGP-NEXT:    v_sub_i32_e32 v4, vcc, 0, v1
 ; CGP-NEXT:    v_rcp_f32_e32 v3, v3
 ; CGP-NEXT:    v_mul_f32_e32 v3, 0x4f7ffffe, v3
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v3, v3
 ; CGP-NEXT:    v_mul_lo_u32 v4, v4, v3
 ; CGP-NEXT:    v_mul_lo_u32 v5, 0, v4
 ; CGP-NEXT:    v_mul_hi_u32 v4, v3, v4
-; CGP-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
+; CGP-NEXT:    v_or_b32_e32 v4, v5, v4
 ; CGP-NEXT:    v_add_i32_e32 v3, vcc, v3, v4
 ; CGP-NEXT:    v_mul_lo_u32 v4, 0, v3
 ; CGP-NEXT:    v_mul_hi_u32 v3, v0, v3
-; CGP-NEXT:    v_add_i32_e32 v3, vcc, v4, v3
+; CGP-NEXT:    v_or_b32_e32 v3, v4, v3
 ; CGP-NEXT:    v_mul_lo_u32 v3, v3, v1
 ; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v3
 ; CGP-NEXT:    v_sub_i32_e32 v3, vcc, v0, v1
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v1
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v3, vcc
 ; CGP-NEXT:    v_sub_i32_e32 v3, vcc, v0, v1
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v1
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v3, vcc

 ; CGP-NEXT:    v_cvt_f32_u32_e32 v0, s1
 ; CGP-NEXT:    s_sub_i32 s3, 0, s1
 ; CGP-NEXT:    v_rcp_f32_e32 v0, v0
 ; CGP-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; CGP-NEXT:    v_mul_lo_u32 v1, s3, v0
 ; CGP-NEXT:    v_mul_lo_u32 v2, 0, v1
 ; CGP-NEXT:    v_mul_hi_u32 v1, v0, v1
-; CGP-NEXT:    v_add_i32_e32 v1, vcc, v2, v1
+; CGP-NEXT:    v_or_b32_e32 v1, v2, v1
 ; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; CGP-NEXT:    v_mul_lo_u32 v1, 0, v0
 ; CGP-NEXT:    v_mul_hi_u32 v0, s0, v0
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
+; CGP-NEXT:    v_or_b32_e32 v0, v1, v0
 ; CGP-NEXT:    v_mul_lo_u32 v0, v0, s1
 ; CGP-NEXT:    v_sub_i32_e32 v0, vcc, s0, v0
 ; CGP-NEXT:    v_subrev_i32_e32 v1, vcc, s1, v0
 ; CGP-NEXT:    v_cmp_le_u32_e32 vcc, s1, v0
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
 ; CGP-NEXT:    v_subrev_i32_e32 v1, vcc, s1, v0
 ; CGP-NEXT:    v_cmp_le_u32_e32 vcc, s1, v0
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc

 ; CGP-NEXT:    v_cvt_u32_f32_e32 v5, v5
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v8, v8
 ; CGP-NEXT:    v_mul_lo_u32 v7, v7, v5
 ; CGP-NEXT:    v_mul_lo_u32 v9, v9, v8
 ; CGP-NEXT:    v_mul_lo_u32 v10, 0, v7
 ; CGP-NEXT:    v_mul_hi_u32 v7, v5, v7
 ; CGP-NEXT:    v_mul_lo_u32 v11, 0, v9
 ; CGP-NEXT:    v_mul_hi_u32 v9, v8, v9
-; CGP-NEXT:    v_add_i32_e32 v7, vcc, v10, v7
-; CGP-NEXT:    v_add_i32_e32 v9, vcc, v11, v9
+; CGP-NEXT:    v_or_b32_e32 v7, v10, v7
+; CGP-NEXT:    v_or_b32_e32 v9, v11, v9
 ; CGP-NEXT:    v_add_i32_e32 v5, vcc, v5, v7
 ; CGP-NEXT:    v_add_i32_e32 v7, vcc, v8, v9
 ; CGP-NEXT:    v_mul_lo_u32 v8, 0, v5
 ; CGP-NEXT:    v_mul_hi_u32 v5, v0, v5
 ; CGP-NEXT:    v_mul_lo_u32 v9, 0, v7
 ; CGP-NEXT:    v_mul_hi_u32 v7, v1, v7
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v8, v5
-; CGP-NEXT:    v_add_i32_e32 v7, vcc, v9, v7
+; CGP-NEXT:    v_or_b32_e32 v5, v8, v5
+; CGP-NEXT:    v_or_b32_e32 v7, v9, v7
 ; CGP-NEXT:    v_mul_lo_u32 v5, v5, v2
 ; CGP-NEXT:    v_mul_lo_u32 v7, v7, v3
 ; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v5
 ; CGP-NEXT:    v_sub_i32_e32 v1, vcc, v1, v7
 ; CGP-NEXT:    v_sub_i32_e32 v5, vcc, v0, v2
 ; CGP-NEXT:    v_sub_i32_e32 v7, vcc, v1, v3
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v2
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v5, vcc
; CHECK-NEXT:    s_setpc_b64 s[30:31]
   ret i32 %result
 }
 
 define <2 x i32> @v_srem_v2i32_pow2k_denom(<2 x i32> %num) {
 ; GISEL-LABEL: v_srem_v2i32_pow2k_denom:
 ; GISEL:       ; %bb.0:
 ; GISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GISEL-NEXT:    v_ashrrev_i32_e32 v2, 31, v0
-; GISEL-NEXT:    s_add_i32 s4, 0x1000, 0
+; GISEL-NEXT:    s_movk_i32 s4, 0x1000
 ; GISEL-NEXT:    v_ashrrev_i32_e32 v3, 31, v1
 ; GISEL-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GISEL-NEXT:    v_cvt_f32_u32_e32 v4, s4
 ; GISEL-NEXT:    s_sub_i32 s5, 0, s4
 ; GISEL-NEXT:    v_add_i32_e32 v1, vcc, v1, v3
 ; GISEL-NEXT:    v_xor_b32_e32 v0, v0, v2
 ; GISEL-NEXT:    v_rcp_iflag_f32_e32 v4, v4
 ; GISEL-NEXT:    v_xor_b32_e32 v1, v1, v3
; CHECK-NEXT:    s_setpc_b64 s[30:31]
   ret i32 %result
 }
 
 define <2 x i32> @v_srem_v2i32_oddk_denom(<2 x i32> %num) {
 ; GISEL-LABEL: v_srem_v2i32_oddk_denom:
 ; GISEL:       ; %bb.0:
 ; GISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GISEL-NEXT:    v_ashrrev_i32_e32 v2, 31, v0
-; GISEL-NEXT:    s_add_i32 s4, 0x12d8fb, 0
+; GISEL-NEXT:    s_mov_b32 s4, 0x12d8fb
 ; GISEL-NEXT:    v_ashrrev_i32_e32 v3, 31, v1
 ; GISEL-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GISEL-NEXT:    v_cvt_f32_u32_e32 v4, s4
 ; GISEL-NEXT:    s_sub_i32 s5, 0, s4
 ; GISEL-NEXT:    v_add_i32_e32 v1, vcc, v1, v3
 ; GISEL-NEXT:    v_xor_b32_e32 v0, v0, v2
 ; GISEL-NEXT:    v_rcp_iflag_f32_e32 v4, v4
 ; GISEL-NEXT:    v_xor_b32_e32 v1, v1, v3

 ; CGP-NEXT:    v_cvt_u32_f32_e32 v6, v6
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v8, v8
 ; CGP-NEXT:    v_mul_lo_u32 v7, v7, v6
 ; CGP-NEXT:    v_mul_lo_u32 v9, v9, v8
 ; CGP-NEXT:    v_mul_lo_u32 v10, 0, v7
 ; CGP-NEXT:    v_mul_hi_u32 v7, v6, v7
 ; CGP-NEXT:    v_mul_lo_u32 v11, 0, v9
 ; CGP-NEXT:    v_mul_hi_u32 v9, v8, v9
-; CGP-NEXT:    v_add_i32_e32 v7, vcc, v10, v7
-; CGP-NEXT:    v_add_i32_e32 v9, vcc, v11, v9
+; CGP-NEXT:    v_or_b32_e32 v7, v10, v7
+; CGP-NEXT:    v_or_b32_e32 v9, v11, v9
 ; CGP-NEXT:    v_add_i32_e32 v6, vcc, v6, v7
 ; CGP-NEXT:    v_add_i32_e32 v7, vcc, v8, v9
 ; CGP-NEXT:    v_mul_lo_u32 v8, 0, v6
 ; CGP-NEXT:    v_mul_hi_u32 v6, v0, v6
 ; CGP-NEXT:    v_mul_lo_u32 v9, 0, v7
 ; CGP-NEXT:    v_mul_hi_u32 v7, v1, v7
-; CGP-NEXT:    v_add_i32_e32 v6, vcc, v8, v6
-; CGP-NEXT:    v_add_i32_e32 v7, vcc, v9, v7
+; CGP-NEXT:    v_or_b32_e32 v6, v8, v6
+; CGP-NEXT:    v_or_b32_e32 v7, v9, v7
 ; CGP-NEXT:    v_mul_lo_u32 v6, v6, v2
 ; CGP-NEXT:    v_mul_lo_u32 v7, v7, v3
 ; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v6
 ; CGP-NEXT:    v_sub_i32_e32 v1, vcc, v1, v7
 ; CGP-NEXT:    v_sub_i32_e32 v6, vcc, v0, v2
 ; CGP-NEXT:    v_sub_i32_e32 v7, vcc, v1, v3
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v2
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v6, vcc

 ; GISEL-LABEL: v_srem_i32_24bit:
 ; GISEL:       ; %bb.0:
 ; GISEL-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GISEL-NEXT:    s_mov_b32 s4, 0xffffff
 ; GISEL-NEXT:    v_and_b32_e32 v0, s4, v0
 ; GISEL-NEXT:    v_and_b32_e32 v1, s4, v1
 ; GISEL-NEXT:    v_ashrrev_i32_e32 v2, 31, v0
 ; GISEL-NEXT:    v_ashrrev_i32_e32 v3, 31, v1
-; GISEL-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
-; GISEL-NEXT:    v_add_i32_e32 v1, vcc, v1, v3
+; GISEL-NEXT:    v_or_b32_e32 v0, v0, v2
+; GISEL-NEXT:    v_or_b32_e32 v1, v1, v3
 ; GISEL-NEXT:    v_xor_b32_e32 v0, v0, v2
 ; GISEL-NEXT:    v_xor_b32_e32 v1, v1, v3
 ; GISEL-NEXT:    v_cvt_f32_u32_e32 v3, v1
 ; GISEL-NEXT:    v_sub_i32_e32 v4, vcc, 0, v1
 ; GISEL-NEXT:    v_rcp_iflag_f32_e32 v3, v3
 ; GISEL-NEXT:    v_mul_f32_e32 v3, 0x4f7ffffe, v3
 ; GISEL-NEXT:    v_cvt_u32_f32_e32 v3, v3
 ; GISEL-NEXT:    v_mul_lo_u32 v4, v4, v3

 ; CGP-NEXT:    v_cvt_f32_u32_e32 v2, v1
 ; CGP-NEXT:    v_sub_i32_e32 v3, vcc, 0, v1
 ; CGP-NEXT:    v_rcp_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_f32_e32 v2, 0x4f7ffffe, v2
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_lo_u32 v3, v3, v2
 ; CGP-NEXT:    v_mul_lo_u32 v4, 0, v3
 ; CGP-NEXT:    v_mul_hi_u32 v3, v2, v3
-; CGP-NEXT:    v_add_i32_e32 v3, vcc, v4, v3
+; CGP-NEXT:    v_or_b32_e32 v3, v4, v3
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; CGP-NEXT:    v_mul_lo_u32 v3, 0, v2
 ; CGP-NEXT:    v_mul_hi_u32 v2, v0, v2
-; CGP-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
+; CGP-NEXT:    v_or_b32_e32 v2, v3, v2
 ; CGP-NEXT:    v_mul_lo_u32 v2, v2, v1
 ; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v2
 ; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v0, v1
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v1
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v0, v1
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v1
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc

 ; CGP-NEXT:    v_cvt_u32_f32_e32 v4, v4
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v6, v6
 ; CGP-NEXT:    v_mul_lo_u32 v5, v5, v4
 ; CGP-NEXT:    v_mul_lo_u32 v7, v7, v6
 ; CGP-NEXT:    v_mul_lo_u32 v8, 0, v5
 ; CGP-NEXT:    v_mul_hi_u32 v5, v4, v5
 ; CGP-NEXT:    v_mul_lo_u32 v9, 0, v7
 ; CGP-NEXT:    v_mul_hi_u32 v7, v6, v7
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v8, v5
-; CGP-NEXT:    v_add_i32_e32 v7, vcc, v9, v7
+; CGP-NEXT:    v_or_b32_e32 v5, v8, v5
+; CGP-NEXT:    v_or_b32_e32 v7, v9, v7
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v5
 ; CGP-NEXT:    v_add_i32_e32 v5, vcc, v6, v7
 ; CGP-NEXT:    v_mul_lo_u32 v6, 0, v4
 ; CGP-NEXT:    v_mul_hi_u32 v4, v0, v4
 ; CGP-NEXT:    v_mul_lo_u32 v7, 0, v5
 ; CGP-NEXT:    v_mul_hi_u32 v5, v1, v5
-; CGP-NEXT:    v_add_i32_e32 v4, vcc, v6, v4
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
+; CGP-NEXT:    v_or_b32_e32 v4, v6, v4
+; CGP-NEXT:    v_or_b32_e32 v5, v7, v5
 ; CGP-NEXT:    v_mul_lo_u32 v4, v4, v2
 ; CGP-NEXT:    v_mul_lo_u32 v5, v5, v3
 ; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v4
 ; CGP-NEXT:    v_sub_i32_e32 v1, vcc, v1, v5
 ; CGP-NEXT:    v_sub_i32_e32 v4, vcc, v0, v2
 ; CGP-NEXT:    v_sub_i32_e32 v5, vcc, v1, v3
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v2
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v4, vcc

llvm/test/CodeGen/AMDGPU/GlobalISel/srem.i64.ll

 ; CHECK-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v5
 ; CHECK-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v5, vcc, v6, v5
 ; CHECK-NEXT:    v_add_i32_e32 v4, vcc, v4, v5
 ; CHECK-NEXT:    v_mul_lo_u32 v5, 0, v2
 ; CHECK-NEXT:    v_mul_lo_u32 v4, s6, v4
-; CHECK-NEXT:    v_mul_lo_u32 v6, s6, v2
-; CHECK-NEXT:    v_mul_hi_u32 v2, s6, v2
-; CHECK-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
-; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v4, v2
-; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v0, v6
-; CHECK-NEXT:    v_subb_u32_e64 v4, s[4:5], v1, v2, vcc
-; CHECK-NEXT:    v_sub_i32_e64 v1, s[4:5], v1, v2
-; CHECK-NEXT:    v_cmp_le_u32_e64 s[4:5], 0, v4
-; CHECK-NEXT:    v_cndmask_b32_e64 v2, 0, -1, s[4:5]
+; CHECK-NEXT:    v_mul_hi_u32 v6, s6, v2
+; CHECK-NEXT:    v_mul_lo_u32 v2, s6, v2
+; CHECK-NEXT:    v_or_b32_e32 v4, v5, v4
+; CHECK-NEXT:    v_add_i32_e32 v4, vcc, v4, v6
+; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v0, v2
+; CHECK-NEXT:    v_subb_u32_e64 v2, s[4:5], v1, v4, vcc
+; CHECK-NEXT:    v_sub_i32_e64 v1, s[4:5], v1, v4
+; CHECK-NEXT:    v_cmp_le_u32_e64 s[4:5], 0, v2
+; CHECK-NEXT:    v_cndmask_b32_e64 v4, 0, -1, s[4:5]
 ; CHECK-NEXT:    v_cmp_le_u32_e64 s[4:5], s6, v0
 ; CHECK-NEXT:    v_cndmask_b32_e64 v5, 0, -1, s[4:5]
-; CHECK-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v4
+; CHECK-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v2
 ; CHECK-NEXT:    v_subbrev_u32_e32 v1, vcc, 0, v1, vcc
-; CHECK-NEXT:    v_cndmask_b32_e64 v2, v2, v5, s[4:5]
+; CHECK-NEXT:    v_cndmask_b32_e64 v4, v4, v5, s[4:5]
 ; CHECK-NEXT:    v_subrev_i32_e32 v5, vcc, s6, v0
 ; CHECK-NEXT:    v_subbrev_u32_e32 v1, vcc, 0, v1, vcc
 ; CHECK-NEXT:    v_cmp_le_u32_e32 vcc, 0, v1
 ; CHECK-NEXT:    v_cndmask_b32_e64 v6, 0, -1, vcc
 ; CHECK-NEXT:    v_cmp_le_u32_e32 vcc, s6, v5
 ; CHECK-NEXT:    v_cndmask_b32_e64 v7, 0, -1, vcc
 ; CHECK-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v1
 ; CHECK-NEXT:    v_cndmask_b32_e32 v6, v6, v7, vcc
 ; CHECK-NEXT:    v_subrev_i32_e32 v7, vcc, s6, v5
 ; CHECK-NEXT:    v_subbrev_u32_e32 v8, vcc, 0, v1, vcc
 ; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v6
 ; CHECK-NEXT:    v_cndmask_b32_e32 v5, v5, v7, vcc
 ; CHECK-NEXT:    v_cndmask_b32_e32 v1, v1, v8, vcc
-; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v2
+; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v4
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v5, vcc
-; CHECK-NEXT:    v_cndmask_b32_e32 v1, v4, v1, vcc
+; CHECK-NEXT:    v_cndmask_b32_e32 v1, v2, v1, vcc
 ; CHECK-NEXT:    v_xor_b32_e32 v0, v0, v3
 ; CHECK-NEXT:    v_xor_b32_e32 v1, v1, v3
 ; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v0, v3
 ; CHECK-NEXT:    v_subb_u32_e32 v1, vcc, v1, v3, vcc
 ; CHECK-NEXT:    s_setpc_b64 s[30:31]
   %result = srem i64 %num, 4096
   ret i64 %result
 }

 ; CGP-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v10, vcc, v11, v10
 ; CGP-NEXT:    v_add_i32_e32 v7, vcc, v7, v9
 ; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, v10, v9
 ; CGP-NEXT:    v_add_i32_e32 v8, vcc, v8, v9
 ; CGP-NEXT:    v_mul_lo_u32 v9, 0, v7
 ; CGP-NEXT:    v_mul_lo_u32 v8, s7, v8
-; CGP-NEXT:    v_mul_lo_u32 v10, s7, v7
-; CGP-NEXT:    v_mul_hi_u32 v7, s7, v7
-; CGP-NEXT:    v_add_i32_e32 v8, vcc, v9, v8
-; CGP-NEXT:    v_add_i32_e32 v7, vcc, v8, v7
-; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v10
-; CGP-NEXT:    v_subb_u32_e64 v8, s[4:5], v1, v7, vcc
-; CGP-NEXT:    v_sub_i32_e64 v1, s[4:5], v1, v7
-; CGP-NEXT:    v_cmp_le_u32_e64 s[4:5], 0, v8
-; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, -1, s[4:5]
+; CGP-NEXT:    v_mul_hi_u32 v10, s7, v7
+; CGP-NEXT:    v_mul_lo_u32 v7, s7, v7
+; CGP-NEXT:    v_or_b32_e32 v8, v9, v8
+; CGP-NEXT:    v_add_i32_e32 v8, vcc, v8, v10
+; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v7
+; CGP-NEXT:    v_subb_u32_e64 v7, s[4:5], v1, v8, vcc
+; CGP-NEXT:    v_sub_i32_e64 v1, s[4:5], v1, v8
+; CGP-NEXT:    v_cmp_le_u32_e64 s[4:5], 0, v7
+; CGP-NEXT:    v_cndmask_b32_e64 v8, 0, -1, s[4:5]
 ; CGP-NEXT:    v_cmp_le_u32_e64 s[4:5], s7, v0
 ; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, -1, s[4:5]
-; CGP-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v8
+; CGP-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v7
 ; CGP-NEXT:    v_subbrev_u32_e32 v1, vcc, 0, v1, vcc
-; CGP-NEXT:    v_cndmask_b32_e64 v7, v7, v9, s[4:5]
+; CGP-NEXT:    v_cndmask_b32_e64 v8, v8, v9, s[4:5]
 ; CGP-NEXT:    v_subrev_i32_e32 v9, vcc, s7, v0
 ; CGP-NEXT:    v_subbrev_u32_e32 v1, vcc, 0, v1, vcc
 ; CGP-NEXT:    v_cmp_le_u32_e32 vcc, 0, v1
 ; CGP-NEXT:    v_cndmask_b32_e64 v10, 0, -1, vcc
 ; CGP-NEXT:    v_cmp_le_u32_e32 vcc, s7, v9
 ; CGP-NEXT:    v_cndmask_b32_e64 v11, 0, -1, vcc
 ; CGP-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v1
 ; CGP-NEXT:    v_cndmask_b32_e32 v10, v10, v11, vcc
 ; CGP-NEXT:    v_subrev_i32_e32 v11, vcc, s7, v9
 ; CGP-NEXT:    v_subbrev_u32_e32 v12, vcc, 0, v1, vcc
 ; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v10
 ; CGP-NEXT:    v_cndmask_b32_e32 v9, v9, v11, vcc
 ; CGP-NEXT:    v_cndmask_b32_e32 v1, v1, v12, vcc
-; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v7
+; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v8
+; CGP-NEXT:    v_cndmask_b32_e32 v1, v7, v1, vcc
 ; CGP-NEXT:    v_mul_f32_e32 v7, 0x2f800000, v4
 ; CGP-NEXT:    v_trunc_f32_e32 v7, v7
 ; CGP-NEXT:    v_mac_f32_e32 v4, 0xcf800000, v7
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v4, v4
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v7, v7
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v9, vcc
-; CGP-NEXT:    v_cndmask_b32_e32 v1, v8, v1, vcc
+; CGP-NEXT:    v_add_i32_e32 v2, vcc, v2, v6
 ; CGP-NEXT:    v_mul_lo_u32 v8, -1, v4
 ; CGP-NEXT:    v_mul_lo_u32 v9, s6, v7
 ; CGP-NEXT:    v_mul_hi_u32 v11, s6, v4
 ; CGP-NEXT:    v_mul_lo_u32 v10, s6, v4
-; CGP-NEXT:    v_add_i32_e32 v2, vcc, v2, v6
 ; CGP-NEXT:    v_addc_u32_e32 v3, vcc, v3, v6, vcc
 ; CGP-NEXT:    v_add_i32_e32 v8, vcc, v8, v9
 ; CGP-NEXT:    v_add_i32_e32 v8, vcc, v8, v11
 ; CGP-NEXT:    v_mul_lo_u32 v9, v7, v10
 ; CGP-NEXT:    v_mul_lo_u32 v11, v4, v8
 ; CGP-NEXT:    v_mul_hi_u32 v12, v4, v10
 ; CGP-NEXT:    v_mul_hi_u32 v10, v7, v10
-; CGP-NEXT:    v_xor_b32_e32 v0, v0, v5
+; CGP-NEXT:    v_xor_b32_e32 v2, v2, v6
 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, v9, v11
 ; CGP-NEXT:    v_cndmask_b32_e64 v11, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, v9, v12
 ; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
 ; CGP-NEXT:    v_mul_lo_u32 v12, v7, v8
 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, v11, v9
 ; CGP-NEXT:    v_mul_hi_u32 v11, v4, v8
 ; CGP-NEXT:    v_mul_hi_u32 v8, v7, v8

 ; CGP-NEXT:    v_mul_lo_u32 v12, s6, v4
 ; CGP-NEXT:    v_add_i32_e64 v7, s[4:5], v7, v8
 ; CGP-NEXT:    v_add_i32_e64 v10, s[4:5], v10, v11
 ; CGP-NEXT:    v_add_i32_e64 v10, s[4:5], v10, v13
 ; CGP-NEXT:    v_mul_lo_u32 v11, v9, v12
 ; CGP-NEXT:    v_mul_lo_u32 v13, v4, v10
 ; CGP-NEXT:    v_mul_hi_u32 v8, v4, v12
 ; CGP-NEXT:    v_mul_hi_u32 v12, v9, v12
-; CGP-NEXT:    v_xor_b32_e32 v2, v2, v6
+; CGP-NEXT:    v_xor_b32_e32 v3, v3, v6
 ; CGP-NEXT:    v_add_i32_e64 v11, s[4:5], v11, v13
 ; CGP-NEXT:    v_cndmask_b32_e64 v13, 0, 1, s[4:5]
 ; CGP-NEXT:    v_add_i32_e64 v8, s[4:5], v11, v8
 ; CGP-NEXT:    v_cndmask_b32_e64 v8, 0, 1, s[4:5]
 ; CGP-NEXT:    v_mul_lo_u32 v11, v9, v10
 ; CGP-NEXT:    v_add_i32_e64 v8, s[4:5], v13, v8
 ; CGP-NEXT:    v_mul_hi_u32 v13, v4, v10
 ; CGP-NEXT:    v_mul_hi_u32 v9, v9, v10
 ; CGP-NEXT:    v_add_i32_e64 v11, s[4:5], v11, v12
 ; CGP-NEXT:    v_cndmask_b32_e64 v12, 0, 1, s[4:5]
 ; CGP-NEXT:    v_add_i32_e64 v11, s[4:5], v11, v13
 ; CGP-NEXT:    v_cndmask_b32_e64 v13, 0, 1, s[4:5]
 ; CGP-NEXT:    v_add_i32_e64 v12, s[4:5], v12, v13
 ; CGP-NEXT:    v_add_i32_e64 v8, s[4:5], v11, v8
 ; CGP-NEXT:    v_cndmask_b32_e64 v11, 0, 1, s[4:5]
 ; CGP-NEXT:    v_add_i32_e64 v10, s[4:5], v12, v11
 ; CGP-NEXT:    v_add_i32_e64 v9, s[4:5], v9, v10
 ; CGP-NEXT:    v_addc_u32_e32 v7, vcc, v7, v9, vcc
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v8
 ; CGP-NEXT:    v_addc_u32_e32 v7, vcc, 0, v7, vcc
-; CGP-NEXT:    v_xor_b32_e32 v3, v3, v6
-; CGP-NEXT:    v_xor_b32_e32 v1, v1, v5
-; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v5
 ; CGP-NEXT:    v_mul_lo_u32 v8, v3, v4
 ; CGP-NEXT:    v_mul_lo_u32 v9, v2, v7
-; CGP-NEXT:    v_subb_u32_e32 v1, vcc, v1, v5, vcc
-; CGP-NEXT:    v_mul_hi_u32 v5, v2, v4
+; CGP-NEXT:    v_mul_hi_u32 v10, v2, v4
 ; CGP-NEXT:    v_mul_hi_u32 v4, v3, v4
+; CGP-NEXT:    v_xor_b32_e32 v0, v0, v5
 ; CGP-NEXT:    v_add_i32_e32 v8, vcc, v8, v9
 ; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v8, v5
-; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
-; CGP-NEXT:    v_mul_lo_u32 v8, v3, v7
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v9, v5
+; CGP-NEXT:    v_add_i32_e32 v8, vcc, v8, v10
+; CGP-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
+; CGP-NEXT:    v_mul_lo_u32 v10, v3, v7
+; CGP-NEXT:    v_add_i32_e32 v8, vcc, v9, v8
 ; CGP-NEXT:    v_mul_hi_u32 v9, v2, v7
 ; CGP-NEXT:    v_mul_hi_u32 v7, v3, v7
-; CGP-NEXT:    v_add_i32_e32 v4, vcc, v8, v4
-; CGP-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v4, vcc, v10, v4
+; CGP-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v9
 ; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v8, vcc, v8, v9
-; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v5
-; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v8, v5
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
-; CGP-NEXT:    v_mul_lo_u32 v7, 0, v4
-; CGP-NEXT:    v_mul_lo_u32 v5, s7, v5
-; CGP-NEXT:    v_mul_lo_u32 v8, s7, v4
-; CGP-NEXT:    v_mul_hi_u32 v4, s7, v4
+; CGP-NEXT:    v_add_i32_e32 v9, vcc, v10, v9
+; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v8
+; CGP-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v8, vcc, v9, v8
+; CGP-NEXT:    v_add_i32_e32 v7, vcc, v7, v8
+; CGP-NEXT:    v_xor_b32_e32 v1, v1, v5
+; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v5
+; CGP-NEXT:    v_mul_lo_u32 v8, 0, v4
+; CGP-NEXT:    v_mul_lo_u32 v7, s7, v7
+; CGP-NEXT:    v_subb_u32_e32 v1, vcc, v1, v5, vcc
+; CGP-NEXT:    v_mul_hi_u32 v5, s7, v4
+; CGP-NEXT:    v_mul_lo_u32 v4, s7, v4
+; CGP-NEXT:    v_or_b32_e32 v7, v8, v7
 ; CGP-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
-; CGP-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
-; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v2, v8
-; CGP-NEXT:    v_subb_u32_e64 v5, s[4:5], v3, v4, vcc
-; CGP-NEXT:    v_sub_i32_e64 v3, s[4:5], v3, v4
-; CGP-NEXT:    v_cmp_le_u32_e64 s[4:5], 0, v5
-; CGP-NEXT:    v_cndmask_b32_e64 v4, 0, -1, s[4:5]
+; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v2, v4
+; CGP-NEXT:    v_subb_u32_e64 v4, s[4:5], v3, v5, vcc
+; CGP-NEXT:    v_sub_i32_e64 v3, s[4:5], v3, v5
+; CGP-NEXT:    v_cmp_le_u32_e64 s[4:5], 0, v4
+; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, -1, s[4:5]
 ; CGP-NEXT:    v_cmp_le_u32_e64 s[4:5], s7, v2
 ; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, -1, s[4:5]
-; CGP-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v5
+; CGP-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v4
 ; CGP-NEXT:    v_subbrev_u32_e32 v3, vcc, 0, v3, vcc
-; CGP-NEXT:    v_cndmask_b32_e64 v4, v4, v7, s[4:5]
+; CGP-NEXT:    v_cndmask_b32_e64 v5, v5, v7, s[4:5]
 ; CGP-NEXT:    v_subrev_i32_e32 v7, vcc, s7, v2
 ; CGP-NEXT:    v_subbrev_u32_e32 v3, vcc, 0, v3, vcc
 ; CGP-NEXT:    v_cmp_le_u32_e32 vcc, 0, v3
 ; CGP-NEXT:    v_cndmask_b32_e64 v8, 0, -1, vcc
 ; CGP-NEXT:    v_cmp_le_u32_e32 vcc, s7, v7
 ; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, -1, vcc
 ; CGP-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v3
 ; CGP-NEXT:    v_cndmask_b32_e32 v8, v8, v9, vcc
 ; CGP-NEXT:    v_subrev_i32_e32 v9, vcc, s7, v7
 ; CGP-NEXT:    v_subbrev_u32_e32 v10, vcc, 0, v3, vcc
 ; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v8
 ; CGP-NEXT:    v_cndmask_b32_e32 v7, v7, v9, vcc
 ; CGP-NEXT:    v_cndmask_b32_e32 v3, v3, v10, vcc
-; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v4
+; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v5
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v2, v7, vcc
-; CGP-NEXT:    v_cndmask_b32_e32 v3, v5, v3, vcc
+; CGP-NEXT:    v_cndmask_b32_e32 v3, v4, v3, vcc
 ; CGP-NEXT:    v_xor_b32_e32 v2, v2, v6
 ; CGP-NEXT:    v_xor_b32_e32 v3, v3, v6
 ; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v2, v6
 ; CGP-NEXT:    v_subb_u32_e32 v3, vcc, v3, v6, vcc
 ; CGP-NEXT:    s_setpc_b64 s[30:31]
   %result = srem <2 x i64> %num, <i64 4096, i64 4096>
   ret <2 x i64> %result
 }

 ; CHECK-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v5
 ; CHECK-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v5, vcc, v6, v5
 ; CHECK-NEXT:    v_add_i32_e32 v4, vcc, v4, v5
 ; CHECK-NEXT:    v_mul_lo_u32 v5, 0, v2
 ; CHECK-NEXT:    v_mul_lo_u32 v4, s6, v4
-; CHECK-NEXT:    v_mul_lo_u32 v6, s6, v2
-; CHECK-NEXT:    v_mul_hi_u32 v2, s6, v2
-; CHECK-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
-; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v4, v2
-; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v0, v6
-; CHECK-NEXT:    v_subb_u32_e64 v4, s[4:5], v1, v2, vcc
-; CHECK-NEXT:    v_sub_i32_e64 v1, s[4:5], v1, v2
-; CHECK-NEXT:    v_cmp_le_u32_e64 s[4:5], 0, v4
-; CHECK-NEXT:    v_cndmask_b32_e64 v2, 0, -1, s[4:5]
+; CHECK-NEXT:    v_mul_hi_u32 v6, s6, v2
+; CHECK-NEXT:    v_mul_lo_u32 v2, s6, v2
+; CHECK-NEXT:    v_or_b32_e32 v4, v5, v4
+; CHECK-NEXT:    v_add_i32_e32 v4, vcc, v4, v6
+; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v0, v2
+; CHECK-NEXT:    v_subb_u32_e64 v2, s[4:5], v1, v4, vcc
+; CHECK-NEXT:    v_sub_i32_e64 v1, s[4:5], v1, v4
+; CHECK-NEXT:    v_cmp_le_u32_e64 s[4:5], 0, v2
+; CHECK-NEXT:    v_cndmask_b32_e64 v4, 0, -1, s[4:5]
 ; CHECK-NEXT:    v_cmp_le_u32_e64 s[4:5], s6, v0
 ; CHECK-NEXT:    v_cndmask_b32_e64 v5, 0, -1, s[4:5]
-; CHECK-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v4
+; CHECK-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v2
 ; CHECK-NEXT:    v_subbrev_u32_e32 v1, vcc, 0, v1, vcc
-; CHECK-NEXT:    v_cndmask_b32_e64 v2, v2, v5, s[4:5]
+; CHECK-NEXT:    v_cndmask_b32_e64 v4, v4, v5, s[4:5]
 ; CHECK-NEXT:    v_subrev_i32_e32 v5, vcc, s6, v0
 ; CHECK-NEXT:    v_subbrev_u32_e32 v1, vcc, 0, v1, vcc
 ; CHECK-NEXT:    v_cmp_le_u32_e32 vcc, 0, v1
 ; CHECK-NEXT:    v_cndmask_b32_e64 v6, 0, -1, vcc
 ; CHECK-NEXT:    v_cmp_le_u32_e32 vcc, s6, v5
 ; CHECK-NEXT:    v_cndmask_b32_e64 v7, 0, -1, vcc
 ; CHECK-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v1
 ; CHECK-NEXT:    v_cndmask_b32_e32 v6, v6, v7, vcc
 ; CHECK-NEXT:    v_subrev_i32_e32 v7, vcc, s6, v5
 ; CHECK-NEXT:    v_subbrev_u32_e32 v8, vcc, 0, v1, vcc
 ; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v6
 ; CHECK-NEXT:    v_cndmask_b32_e32 v5, v5, v7, vcc
 ; CHECK-NEXT:    v_cndmask_b32_e32 v1, v1, v8, vcc
-; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v2
+; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v4
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v5, vcc
-; CHECK-NEXT:    v_cndmask_b32_e32 v1, v4, v1, vcc
+; CHECK-NEXT:    v_cndmask_b32_e32 v1, v2, v1, vcc
 ; CHECK-NEXT:    v_xor_b32_e32 v0, v0, v3
 ; CHECK-NEXT:    v_xor_b32_e32 v1, v1, v3
 ; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v0, v3
 ; CHECK-NEXT:    v_subb_u32_e32 v1, vcc, v1, v3, vcc
 ; CHECK-NEXT:    s_setpc_b64 s[30:31]
   %result = srem i64 %num, 1235195
   ret i64 %result
 }

 ; CGP-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v10, vcc, v11, v10
 ; CGP-NEXT:    v_add_i32_e32 v7, vcc, v7, v9
 ; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, v10, v9
 ; CGP-NEXT:    v_add_i32_e32 v8, vcc, v8, v9
 ; CGP-NEXT:    v_mul_lo_u32 v9, 0, v7
 ; CGP-NEXT:    v_mul_lo_u32 v8, s7, v8
-; CGP-NEXT:    v_mul_lo_u32 v10, s7, v7
-; CGP-NEXT:    v_mul_hi_u32 v7, s7, v7
-; CGP-NEXT:    v_add_i32_e32 v8, vcc, v9, v8
-; CGP-NEXT:    v_add_i32_e32 v7, vcc, v8, v7
-; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v10
-; CGP-NEXT:    v_subb_u32_e64 v8, s[4:5], v1, v7, vcc
-; CGP-NEXT:    v_sub_i32_e64 v1, s[4:5], v1, v7
-; CGP-NEXT:    v_cmp_le_u32_e64 s[4:5], 0, v8
-; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, -1, s[4:5]
+; CGP-NEXT:    v_mul_hi_u32 v10, s7, v7
+; CGP-NEXT:    v_mul_lo_u32 v7, s7, v7
+; CGP-NEXT:    v_or_b32_e32 v8, v9, v8
+; CGP-NEXT:    v_add_i32_e32 v8, vcc, v8, v10
+; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v7
+; CGP-NEXT:    v_subb_u32_e64 v7, s[4:5], v1, v8, vcc
+; CGP-NEXT:    v_sub_i32_e64 v1, s[4:5], v1, v8
+; CGP-NEXT:    v_cmp_le_u32_e64 s[4:5], 0, v7
+; CGP-NEXT:    v_cndmask_b32_e64 v8, 0, -1, s[4:5]
 ; CGP-NEXT:    v_cmp_le_u32_e64 s[4:5], s7, v0
 ; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, -1, s[4:5]
-; CGP-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v8
+; CGP-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v7
 ; CGP-NEXT:    v_subbrev_u32_e32 v1, vcc, 0, v1, vcc
-; CGP-NEXT:    v_cndmask_b32_e64 v7, v7, v9, s[4:5]
+; CGP-NEXT:    v_cndmask_b32_e64 v8, v8, v9, s[4:5]
 ; CGP-NEXT:    v_subrev_i32_e32 v9, vcc, s7, v0
 ; CGP-NEXT:    v_subbrev_u32_e32 v1, vcc, 0, v1, vcc
 ; CGP-NEXT:    v_cmp_le_u32_e32 vcc, 0, v1
 ; CGP-NEXT:    v_cndmask_b32_e64 v10, 0, -1, vcc
 ; CGP-NEXT:    v_cmp_le_u32_e32 vcc, s7, v9
 ; CGP-NEXT:    v_cndmask_b32_e64 v11, 0, -1, vcc
 ; CGP-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v1
 ; CGP-NEXT:    v_cndmask_b32_e32 v10, v10, v11, vcc
 ; CGP-NEXT:    v_subrev_i32_e32 v11, vcc, s7, v9
 ; CGP-NEXT:    v_subbrev_u32_e32 v12, vcc, 0, v1, vcc
 ; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v10
 ; CGP-NEXT:    v_cndmask_b32_e32 v9, v9, v11, vcc
 ; CGP-NEXT:    v_cndmask_b32_e32 v1, v1, v12, vcc
-; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v7
+; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v8
+; CGP-NEXT:    v_cndmask_b32_e32 v1, v7, v1, vcc
 ; CGP-NEXT:    v_mul_f32_e32 v7, 0x2f800000, v4
 ; CGP-NEXT:    v_trunc_f32_e32 v7, v7
 ; CGP-NEXT:    v_mac_f32_e32 v4, 0xcf800000, v7
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v4, v4
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v7, v7
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v9, vcc
-; CGP-NEXT:    v_cndmask_b32_e32 v1, v8, v1, vcc
+; CGP-NEXT:    v_add_i32_e32 v2, vcc, v2, v6
 ; CGP-NEXT:    v_mul_lo_u32 v8, -1, v4
 ; CGP-NEXT:    v_mul_lo_u32 v9, s6, v7
 ; CGP-NEXT:    v_mul_hi_u32 v11, s6, v4
 ; CGP-NEXT:    v_mul_lo_u32 v10, s6, v4
-; CGP-NEXT:    v_add_i32_e32 v2, vcc, v2, v6
 ; CGP-NEXT:    v_addc_u32_e32 v3, vcc, v3, v6, vcc
 ; CGP-NEXT:    v_add_i32_e32 v8, vcc, v8, v9
 ; CGP-NEXT:    v_add_i32_e32 v8, vcc, v8, v11
 ; CGP-NEXT:    v_mul_lo_u32 v9, v7, v10
 ; CGP-NEXT:    v_mul_lo_u32 v11, v4, v8
 ; CGP-NEXT:    v_mul_hi_u32 v12, v4, v10
 ; CGP-NEXT:    v_mul_hi_u32 v10, v7, v10
-; CGP-NEXT:    v_xor_b32_e32 v0, v0, v5
+; CGP-NEXT:    v_xor_b32_e32 v2, v2, v6
 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, v9, v11
 ; CGP-NEXT:    v_cndmask_b32_e64 v11, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, v9, v12
 ; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
 ; CGP-NEXT:    v_mul_lo_u32 v12, v7, v8
 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, v11, v9
 ; CGP-NEXT:    v_mul_hi_u32 v11, v4, v8
 ; CGP-NEXT:    v_mul_hi_u32 v8, v7, v8

 ; CGP-NEXT:    v_mul_lo_u32 v12, s6, v4
 ; CGP-NEXT:    v_add_i32_e64 v7, s[4:5], v7, v8
 ; CGP-NEXT:    v_add_i32_e64 v10, s[4:5], v10, v11
 ; CGP-NEXT:    v_add_i32_e64 v10, s[4:5], v10, v13
 ; CGP-NEXT:    v_mul_lo_u32 v11, v9, v12
 ; CGP-NEXT:    v_mul_lo_u32 v13, v4, v10
 ; CGP-NEXT:    v_mul_hi_u32 v8, v4, v12
 ; CGP-NEXT:    v_mul_hi_u32 v12, v9, v12
-; CGP-NEXT:    v_xor_b32_e32 v2, v2, v6
+; CGP-NEXT:    v_xor_b32_e32 v3, v3, v6
 ; CGP-NEXT:    v_add_i32_e64 v11, s[4:5], v11, v13
 ; CGP-NEXT:    v_cndmask_b32_e64 v13, 0, 1, s[4:5]
 ; CGP-NEXT:    v_add_i32_e64 v8, s[4:5], v11, v8
 ; CGP-NEXT:    v_cndmask_b32_e64 v8, 0, 1, s[4:5]
 ; CGP-NEXT:    v_mul_lo_u32 v11, v9, v10
 ; CGP-NEXT:    v_add_i32_e64 v8, s[4:5], v13, v8
 ; CGP-NEXT:    v_mul_hi_u32 v13, v4, v10
 ; CGP-NEXT:    v_mul_hi_u32 v9, v9, v10
 ; CGP-NEXT:    v_add_i32_e64 v11, s[4:5], v11, v12
 ; CGP-NEXT:    v_cndmask_b32_e64 v12, 0, 1, s[4:5]
 ; CGP-NEXT:    v_add_i32_e64 v11, s[4:5], v11, v13
 ; CGP-NEXT:    v_cndmask_b32_e64 v13, 0, 1, s[4:5]
 ; CGP-NEXT:    v_add_i32_e64 v12, s[4:5], v12, v13
 ; CGP-NEXT:    v_add_i32_e64 v8, s[4:5], v11, v8
 ; CGP-NEXT:    v_cndmask_b32_e64 v11, 0, 1, s[4:5]
 ; CGP-NEXT:    v_add_i32_e64 v10, s[4:5], v12, v11
 ; CGP-NEXT:    v_add_i32_e64 v9, s[4:5], v9, v10
 ; CGP-NEXT:    v_addc_u32_e32 v7, vcc, v7, v9, vcc
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v8
 ; CGP-NEXT:    v_addc_u32_e32 v7, vcc, 0, v7, vcc
-; CGP-NEXT:    v_xor_b32_e32 v3, v3, v6
-; CGP-NEXT:    v_xor_b32_e32 v1, v1, v5
-; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v5
 ; CGP-NEXT:    v_mul_lo_u32 v8, v3, v4
 ; CGP-NEXT:    v_mul_lo_u32 v9, v2, v7
-; CGP-NEXT:    v_subb_u32_e32 v1, vcc, v1, v5, vcc
-; CGP-NEXT:    v_mul_hi_u32 v5, v2, v4
+; CGP-NEXT:    v_mul_hi_u32 v10, v2, v4
 ; CGP-NEXT:    v_mul_hi_u32 v4, v3, v4
+; CGP-NEXT:    v_xor_b32_e32 v0, v0, v5
 ; CGP-NEXT:    v_add_i32_e32 v8, vcc, v8, v9
 ; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v8, v5
-; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
-; CGP-NEXT:    v_mul_lo_u32 v8, v3, v7
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v9, v5
+; CGP-NEXT:    v_add_i32_e32 v8, vcc, v8, v10
+; CGP-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
+; CGP-NEXT:    v_mul_lo_u32 v10, v3, v7
+; CGP-NEXT:    v_add_i32_e32 v8, vcc, v9, v8
 ; CGP-NEXT:    v_mul_hi_u32 v9, v2, v7
 ; CGP-NEXT:    v_mul_hi_u32 v7, v3, v7
-; CGP-NEXT:    v_add_i32_e32 v4, vcc, v8, v4
-; CGP-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v4, vcc, v10, v4
+; CGP-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v9
 ; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v8, vcc, v8, v9
-; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v5
-; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v8, v5
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
-; CGP-NEXT:    v_mul_lo_u32 v7, 0, v4
-; CGP-NEXT:    v_mul_lo_u32 v5, s7, v5
-; CGP-NEXT:    v_mul_lo_u32 v8, s7, v4
-; CGP-NEXT:    v_mul_hi_u32 v4, s7, v4
+; CGP-NEXT:    v_add_i32_e32 v9, vcc, v10, v9
+; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v8
+; CGP-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
+; CGP-NEXT:    v_add_i32_e32 v8, vcc, v9, v8
+; CGP-NEXT:    v_add_i32_e32 v7, vcc, v7, v8
+; CGP-NEXT:    v_xor_b32_e32 v1, v1, v5
+; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v5
+; CGP-NEXT:    v_mul_lo_u32 v8, 0, v4
+; CGP-NEXT:    v_mul_lo_u32 v7, s7, v7
+; CGP-NEXT:    v_subb_u32_e32 v1, vcc, v1, v5, vcc
+; CGP-NEXT:    v_mul_hi_u32 v5, s7, v4
+; CGP-NEXT:    v_mul_lo_u32 v4, s7, v4
+; CGP-NEXT:    v_or_b32_e32 v7, v8, v7
 ; CGP-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
-; CGP-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
-; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v2, v8
-; CGP-NEXT:    v_subb_u32_e64 v5, s[4:5], v3, v4, vcc
-; CGP-NEXT:    v_sub_i32_e64 v3, s[4:5], v3, v4
-; CGP-NEXT:    v_cmp_le_u32_e64 s[4:5], 0, v5
-; CGP-NEXT:    v_cndmask_b32_e64 v4, 0, -1, s[4:5]
+; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v2, v4
+; CGP-NEXT:    v_subb_u32_e64 v4, s[4:5], v3, v5, vcc
+; CGP-NEXT:    v_sub_i32_e64 v3, s[4:5], v3, v5
+; CGP-NEXT:    v_cmp_le_u32_e64 s[4:5], 0, v4
+; CGP-NEXT:    v_cndmask_b32_e64 v5, 0, -1, s[4:5]
 ; CGP-NEXT:    v_cmp_le_u32_e64 s[4:5], s7, v2
 ; CGP-NEXT:    v_cndmask_b32_e64 v7, 0, -1, s[4:5]
-; CGP-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v5
+; CGP-NEXT:    v_cmp_eq_u32_e64 s[4:5], 0, v4
 ; CGP-NEXT:    v_subbrev_u32_e32 v3, vcc, 0, v3, vcc
-; CGP-NEXT:    v_cndmask_b32_e64 v4, v4, v7, s[4:5]
+; CGP-NEXT:    v_cndmask_b32_e64 v5, v5, v7, s[4:5]
 ; CGP-NEXT:    v_subrev_i32_e32 v7, vcc, s7, v2
 ; CGP-NEXT:    v_subbrev_u32_e32 v3, vcc, 0, v3, vcc
 ; CGP-NEXT:    v_cmp_le_u32_e32 vcc, 0, v3
 ; CGP-NEXT:    v_cndmask_b32_e64 v8, 0, -1, vcc
 ; CGP-NEXT:    v_cmp_le_u32_e32 vcc, s7, v7
 ; CGP-NEXT:    v_cndmask_b32_e64 v9, 0, -1, vcc
 ; CGP-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v3
 ; CGP-NEXT:    v_cndmask_b32_e32 v8, v8, v9, vcc
 ; CGP-NEXT:    v_subrev_i32_e32 v9, vcc, s7, v7
 ; CGP-NEXT:    v_subbrev_u32_e32 v10, vcc, 0, v3, vcc
 ; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v8
 ; CGP-NEXT:    v_cndmask_b32_e32 v7, v7, v9, vcc
 ; CGP-NEXT:    v_cndmask_b32_e32 v3, v3, v10, vcc
-; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v4
+; CGP-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v5
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v2, v7, vcc
-; CGP-NEXT:    v_cndmask_b32_e32 v3, v5, v3, vcc
+; CGP-NEXT:    v_cndmask_b32_e32 v3, v4, v3, vcc
 ; CGP-NEXT:    v_xor_b32_e32 v2, v2, v6
 ; CGP-NEXT:    v_xor_b32_e32 v3, v3, v6
 ; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v2, v6
 ; CGP-NEXT:    v_subb_u32_e32 v3, vcc, v3, v6, vcc
 ; CGP-NEXT:    s_setpc_b64 s[30:31]
   %result = srem <2 x i64> %num, <i64 1235195, i64 1235195>
   ret <2 x i64> %result
 }

llvm/test/CodeGen/AMDGPU/GlobalISel/ssubsat.ll

 ; GFX8-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX8-NEXT:    v_sub_u32_e32 v2, vcc, v0, v1
 ; GFX8-NEXT:    v_bfe_i32 v3, v2, 0, 24
 ; GFX8-NEXT:    v_bfe_i32 v0, v0, 0, 24
 ; GFX8-NEXT:    v_cmp_lt_i32_e64 s[4:5], v3, v0
 ; GFX8-NEXT:    v_bfe_i32 v0, v1, 0, 24
 ; GFX8-NEXT:    v_cmp_lt_i32_e64 s[6:7], 0, v0
 ; GFX8-NEXT:    v_ashrrev_i32_e32 v0, 23, v3
-; GFX8-NEXT:    v_add_u32_e32 v0, vcc, 0xff800000, v0
+; GFX8-NEXT:    v_subrev_u32_e32 v0, vcc, 0x800000, v0
 ; GFX8-NEXT:    s_xor_b64 vcc, s[6:7], s[4:5]
 ; GFX8-NEXT:    v_cndmask_b32_e32 v0, v2, v0, vcc
 ; GFX8-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX9-LABEL: v_ssubsat_i24:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 8, v0

 ; GFX8-NEXT:    s_bfe_i32 s0, s0, 0x180000
 ; GFX8-NEXT:    s_cmp_lt_i32 s3, s0
 ; GFX8-NEXT:    s_cselect_b32 s0, 1, 0
 ; GFX8-NEXT:    s_bfe_i32 s1, s1, 0x180000
 ; GFX8-NEXT:    s_cmp_gt_i32 s1, 0
 ; GFX8-NEXT:    s_cselect_b32 s1, 1, 0
 ; GFX8-NEXT:    s_xor_b32 s0, s1, s0
 ; GFX8-NEXT:    s_ashr_i32 s1, s3, 23
-; GFX8-NEXT:    s_add_i32 s1, s1, 0xff800000
+; GFX8-NEXT:    s_sub_i32 s1, s1, 0x800000
 ; GFX8-NEXT:    s_and_b32 s0, s0, 1
 ; GFX8-NEXT:    s_cmp_lg_u32 s0, 0
 ; GFX8-NEXT:    s_cselect_b32 s0, s1, s2
 ; GFX8-NEXT:    ; return to shader part epilog
 ;
 ; GFX9-LABEL: s_ssubsat_i24:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_lshl_b32 s1, s1, 8

llvm/test/CodeGen/AMDGPU/GlobalISel/udiv.i32.ll

 ; CGP-NEXT:    v_cvt_f32_u32_e32 v2, v1
 ; CGP-NEXT:    v_sub_i32_e32 v3, vcc, 0, v1
 ; CGP-NEXT:    v_rcp_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_f32_e32 v2, 0x4f7ffffe, v2
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_lo_u32 v3, v3, v2
 ; CGP-NEXT:    v_mul_lo_u32 v4, 0, v3
 ; CGP-NEXT:    v_mul_hi_u32 v3, v2, v3
-; CGP-NEXT:    v_add_i32_e32 v3, vcc, v4, v3
+; CGP-NEXT:    v_or_b32_e32 v3, v4, v3
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; CGP-NEXT:    v_mul_lo_u32 v3, 0, v2
 ; CGP-NEXT:    v_mul_hi_u32 v2, v0, v2
-; CGP-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
+; CGP-NEXT:    v_or_b32_e32 v2, v3, v2
 ; CGP-NEXT:    v_mul_lo_u32 v3, v2, v1
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, 1, v2
 ; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v3
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v1
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v2, v4, vcc
 ; CGP-NEXT:    v_sub_i32_e64 v3, s[4:5], v0, v1
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v3, vcc
 ; CGP-NEXT:    v_add_i32_e32 v3, vcc, 1, v2

 ; CGP-NEXT:    v_cvt_f32_u32_e32 v0, s1
 ; CGP-NEXT:    s_sub_i32 s2, 0, s1
 ; CGP-NEXT:    v_rcp_f32_e32 v0, v0
 ; CGP-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; CGP-NEXT:    v_mul_lo_u32 v1, s2, v0
 ; CGP-NEXT:    v_mul_lo_u32 v2, 0, v1
 ; CGP-NEXT:    v_mul_hi_u32 v1, v0, v1
-; CGP-NEXT:    v_add_i32_e32 v1, vcc, v2, v1
+; CGP-NEXT:    v_or_b32_e32 v1, v2, v1
 ; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; CGP-NEXT:    v_mul_lo_u32 v1, 0, v0
 ; CGP-NEXT:    v_mul_hi_u32 v0, s0, v0
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
+; CGP-NEXT:    v_or_b32_e32 v0, v1, v0
 ; CGP-NEXT:    v_mul_lo_u32 v1, v0, s1
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, 1, v0
 ; CGP-NEXT:    v_sub_i32_e32 v1, vcc, s0, v1
 ; CGP-NEXT:    v_cmp_le_u32_e32 vcc, s1, v1
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; CGP-NEXT:    v_subrev_i32_e64 v2, s[2:3], s1, v1
 ; CGP-NEXT:    v_cndmask_b32_e32 v1, v1, v2, vcc
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, 1, v0

 ; CGP-NEXT:    v_cvt_u32_f32_e32 v4, v4
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v6, v6
 ; CGP-NEXT:    v_mul_lo_u32 v5, v5, v4
 ; CGP-NEXT:    v_mul_lo_u32 v7, v7, v6
 ; CGP-NEXT:    v_mul_lo_u32 v8, 0, v5
 ; CGP-NEXT:    v_mul_hi_u32 v5, v4, v5
 ; CGP-NEXT:    v_mul_lo_u32 v9, 0, v7
 ; CGP-NEXT:    v_mul_hi_u32 v7, v6, v7
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v8, v5
-; CGP-NEXT:    v_add_i32_e32 v7, vcc, v9, v7
+; CGP-NEXT:    v_or_b32_e32 v5, v8, v5
+; CGP-NEXT:    v_or_b32_e32 v7, v9, v7
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v5
 ; CGP-NEXT:    v_add_i32_e32 v5, vcc, v6, v7
 ; CGP-NEXT:    v_mul_lo_u32 v6, 0, v4
 ; CGP-NEXT:    v_mul_hi_u32 v4, v0, v4
 ; CGP-NEXT:    v_mul_lo_u32 v7, 0, v5
 ; CGP-NEXT:    v_mul_hi_u32 v5, v1, v5
-; CGP-NEXT:    v_add_i32_e32 v4, vcc, v6, v4
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
+; CGP-NEXT:    v_or_b32_e32 v4, v6, v4
+; CGP-NEXT:    v_or_b32_e32 v5, v7, v5
 ; CGP-NEXT:    v_mul_lo_u32 v6, v4, v2
 ; CGP-NEXT:    v_add_i32_e32 v7, vcc, 1, v4
 ; CGP-NEXT:    v_mul_lo_u32 v8, v5, v3
 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, 1, v5
 ; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v6
 ; CGP-NEXT:    v_sub_i32_e32 v1, vcc, v1, v8
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v2
 ; CGP-NEXT:    v_cndmask_b32_e32 v4, v4, v7, vcc

 ; CGP-NEXT:    v_cvt_u32_f32_e32 v4, v4
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v6, v6
 ; CGP-NEXT:    v_mul_lo_u32 v5, v5, v4
 ; CGP-NEXT:    v_mul_lo_u32 v7, v7, v6
 ; CGP-NEXT:    v_mul_lo_u32 v8, 0, v5
 ; CGP-NEXT:    v_mul_hi_u32 v5, v4, v5
 ; CGP-NEXT:    v_mul_lo_u32 v9, 0, v7
 ; CGP-NEXT:    v_mul_hi_u32 v7, v6, v7
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v8, v5
-; CGP-NEXT:    v_add_i32_e32 v7, vcc, v9, v7
+; CGP-NEXT:    v_or_b32_e32 v5, v8, v5
+; CGP-NEXT:    v_or_b32_e32 v7, v9, v7
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v5
 ; CGP-NEXT:    v_add_i32_e32 v5, vcc, v6, v7
 ; CGP-NEXT:    v_mul_lo_u32 v6, 0, v4
 ; CGP-NEXT:    v_mul_hi_u32 v4, v0, v4
 ; CGP-NEXT:    v_mul_lo_u32 v7, 0, v5
 ; CGP-NEXT:    v_mul_hi_u32 v5, v1, v5
-; CGP-NEXT:    v_add_i32_e32 v4, vcc, v6, v4
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
+; CGP-NEXT:    v_or_b32_e32 v4, v6, v4
+; CGP-NEXT:    v_or_b32_e32 v5, v7, v5
 ; CGP-NEXT:    v_mul_lo_u32 v6, v4, v2
 ; CGP-NEXT:    v_add_i32_e32 v7, vcc, 1, v4
 ; CGP-NEXT:    v_mul_lo_u32 v8, v5, v3
 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, 1, v5
 ; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v6
 ; CGP-NEXT:    v_sub_i32_e32 v1, vcc, v1, v8
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v2
 ; CGP-NEXT:    v_cndmask_b32_e32 v4, v4, v7, vcc

 ; CGP-NEXT:    v_cvt_f32_u32_e32 v2, v1
 ; CGP-NEXT:    v_sub_i32_e32 v3, vcc, 0, v1
 ; CGP-NEXT:    v_rcp_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_f32_e32 v2, 0x4f7ffffe, v2
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_lo_u32 v3, v3, v2
 ; CGP-NEXT:    v_mul_lo_u32 v4, 0, v3
 ; CGP-NEXT:    v_mul_hi_u32 v3, v2, v3
-; CGP-NEXT:    v_add_i32_e32 v3, vcc, v4, v3
+; CGP-NEXT:    v_or_b32_e32 v3, v4, v3
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; CGP-NEXT:    v_mul_lo_u32 v3, 0, v2
 ; CGP-NEXT:    v_mul_hi_u32 v2, v0, v2
-; CGP-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
+; CGP-NEXT:    v_or_b32_e32 v2, v3, v2
 ; CGP-NEXT:    v_mul_lo_u32 v3, v2, v1
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, 1, v2
 ; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v3
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v1
 ; CGP-NEXT:    v_cndmask_b32_e32 v2, v2, v4, vcc
 ; CGP-NEXT:    v_sub_i32_e64 v3, s[4:5], v0, v1
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v3, vcc
 ; CGP-NEXT:    v_add_i32_e32 v3, vcc, 1, v2

 ; CGP-NEXT:    v_cvt_u32_f32_e32 v4, v4
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v6, v6
 ; CGP-NEXT:    v_mul_lo_u32 v5, v5, v4
 ; CGP-NEXT:    v_mul_lo_u32 v7, v7, v6
 ; CGP-NEXT:    v_mul_lo_u32 v8, 0, v5
 ; CGP-NEXT:    v_mul_hi_u32 v5, v4, v5
 ; CGP-NEXT:    v_mul_lo_u32 v9, 0, v7
 ; CGP-NEXT:    v_mul_hi_u32 v7, v6, v7
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v8, v5
-; CGP-NEXT:    v_add_i32_e32 v7, vcc, v9, v7
+; CGP-NEXT:    v_or_b32_e32 v5, v8, v5
+; CGP-NEXT:    v_or_b32_e32 v7, v9, v7
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v5
 ; CGP-NEXT:    v_add_i32_e32 v5, vcc, v6, v7
 ; CGP-NEXT:    v_mul_lo_u32 v6, 0, v4
 ; CGP-NEXT:    v_mul_hi_u32 v4, v0, v4
 ; CGP-NEXT:    v_mul_lo_u32 v7, 0, v5
 ; CGP-NEXT:    v_mul_hi_u32 v5, v1, v5
-; CGP-NEXT:    v_add_i32_e32 v4, vcc, v6, v4
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
+; CGP-NEXT:    v_or_b32_e32 v4, v6, v4
+; CGP-NEXT:    v_or_b32_e32 v5, v7, v5
 ; CGP-NEXT:    v_mul_lo_u32 v6, v4, v2
 ; CGP-NEXT:    v_add_i32_e32 v7, vcc, 1, v4
 ; CGP-NEXT:    v_mul_lo_u32 v8, v5, v3
 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, 1, v5
 ; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v6
 ; CGP-NEXT:    v_sub_i32_e32 v1, vcc, v1, v8
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v2
 ; CGP-NEXT:    v_cndmask_b32_e32 v4, v4, v7, vcc

llvm/test/CodeGen/AMDGPU/GlobalISel/udiv.i64.ll

 ; CHECK-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
 ; CHECK-NEXT:    v_mul_lo_u32 v5, s7, v2
 ; CHECK-NEXT:    v_mul_lo_u32 v6, 0, v2
 ; CHECK-NEXT:    v_mul_hi_u32 v7, s7, v2
 ; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v3, v4
 ; CHECK-NEXT:    v_mul_lo_u32 v4, s7, v3
 ; CHECK-NEXT:    v_add_i32_e32 v8, vcc, 1, v2
 ; CHECK-NEXT:    v_addc_u32_e32 v9, vcc, 0, v3, vcc
-; CHECK-NEXT:    v_add_i32_e32 v4, vcc, v6, v4
+; CHECK-NEXT:    v_or_b32_e32 v4, v6, v4
 ; CHECK-NEXT:    v_add_i32_e32 v6, vcc, 1, v8
 ; CHECK-NEXT:    v_addc_u32_e32 v10, vcc, 0, v9, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v4, vcc, v4, v7
 ; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v0, v5
 ; CHECK-NEXT:    v_subb_u32_e64 v5, s[4:5], v1, v4, vcc
 ; CHECK-NEXT:    v_sub_i32_e64 v1, s[4:5], v1, v4
 ; CHECK-NEXT:    v_cmp_le_u32_e64 s[4:5], s7, v0
 ; CHECK-NEXT:    v_cndmask_b32_e64 v4, 0, -1, s[4:5]

 ; GISEL-NEXT:    v_add_i32_e32 v6, vcc, v6, v8
 ; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v9
 ; GISEL-NEXT:    v_mul_lo_u32 v8, s10, v6
 ; GISEL-NEXT:    v_add_i32_e32 v9, vcc, 1, v4
 ; GISEL-NEXT:    v_addc_u32_e32 v16, vcc, 0, v6, vcc
 ; GISEL-NEXT:    v_mul_lo_u32 v17, s10, v7
 ; GISEL-NEXT:    v_add_i32_e32 v18, vcc, 1, v5
 ; GISEL-NEXT:    v_addc_u32_e32 v19, vcc, 0, v7, vcc
-; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v12, v8
-; GISEL-NEXT:    v_add_i32_e32 v12, vcc, v14, v17
+; GISEL-NEXT:    v_or_b32_e32 v8, v12, v8
+; GISEL-NEXT:    v_or_b32_e32 v12, v14, v17
 ; GISEL-NEXT:    v_add_i32_e32 v14, vcc, 1, v9
 ; GISEL-NEXT:    v_addc_u32_e32 v17, vcc, 0, v16, vcc
 ; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v8, v13
 ; GISEL-NEXT:    v_add_i32_e32 v12, vcc, v12, v15
 ; GISEL-NEXT:    v_add_i32_e32 v13, vcc, 1, v18
 ; GISEL-NEXT:    v_addc_u32_e32 v15, vcc, 0, v19, vcc
 ; GISEL-NEXT:    v_sub_i32_e32 v2, vcc, v2, v10
 ; GISEL-NEXT:    v_subb_u32_e64 v10, s[4:5], v3, v8, vcc

 ; CGP-NEXT:    v_add_i32_e32 v6, vcc, v6, v8
 ; CGP-NEXT:    v_add_i32_e32 v7, vcc, v7, v9
 ; CGP-NEXT:    v_mul_lo_u32 v8, s10, v6
 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, 1, v4
 ; CGP-NEXT:    v_addc_u32_e32 v16, vcc, 0, v6, vcc
 ; CGP-NEXT:    v_mul_lo_u32 v17, s10, v7
 ; CGP-NEXT:    v_add_i32_e32 v18, vcc, 1, v5
 ; CGP-NEXT:    v_addc_u32_e32 v19, vcc, 0, v7, vcc
-; CGP-NEXT:    v_add_i32_e32 v8, vcc, v12, v8
-; CGP-NEXT:    v_add_i32_e32 v12, vcc, v14, v17
+; CGP-NEXT:    v_or_b32_e32 v8, v12, v8
+; CGP-NEXT:    v_or_b32_e32 v12, v14, v17
 ; CGP-NEXT:    v_add_i32_e32 v14, vcc, 1, v9
 ; CGP-NEXT:    v_addc_u32_e32 v17, vcc, 0, v16, vcc
 ; CGP-NEXT:    v_add_i32_e32 v8, vcc, v8, v13
 ; CGP-NEXT:    v_add_i32_e32 v12, vcc, v12, v15
 ; CGP-NEXT:    v_add_i32_e32 v13, vcc, 1, v18
 ; CGP-NEXT:    v_addc_u32_e32 v15, vcc, 0, v19, vcc
 ; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v2, v10
 ; CGP-NEXT:    v_subb_u32_e64 v10, s[4:5], v3, v8, vcc

 ; CHECK-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
 ; CHECK-NEXT:    v_mul_lo_u32 v5, s7, v2
 ; CHECK-NEXT:    v_mul_lo_u32 v6, 0, v2
 ; CHECK-NEXT:    v_mul_hi_u32 v7, s7, v2
 ; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v3, v4
 ; CHECK-NEXT:    v_mul_lo_u32 v4, s7, v3
 ; CHECK-NEXT:    v_add_i32_e32 v8, vcc, 1, v2
 ; CHECK-NEXT:    v_addc_u32_e32 v9, vcc, 0, v3, vcc
-; CHECK-NEXT:    v_add_i32_e32 v4, vcc, v6, v4
+; CHECK-NEXT:    v_or_b32_e32 v4, v6, v4
 ; CHECK-NEXT:    v_add_i32_e32 v6, vcc, 1, v8
 ; CHECK-NEXT:    v_addc_u32_e32 v10, vcc, 0, v9, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v4, vcc, v4, v7
 ; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v0, v5
 ; CHECK-NEXT:    v_subb_u32_e64 v5, s[4:5], v1, v4, vcc
 ; CHECK-NEXT:    v_sub_i32_e64 v1, s[4:5], v1, v4
 ; CHECK-NEXT:    v_cmp_le_u32_e64 s[4:5], s7, v0
 ; CHECK-NEXT:    v_cndmask_b32_e64 v4, 0, -1, s[4:5]

 ; GISEL-NEXT:    v_add_i32_e32 v6, vcc, v6, v8
 ; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v9
 ; GISEL-NEXT:    v_mul_lo_u32 v8, s10, v6
 ; GISEL-NEXT:    v_add_i32_e32 v9, vcc, 1, v4
 ; GISEL-NEXT:    v_addc_u32_e32 v16, vcc, 0, v6, vcc
 ; GISEL-NEXT:    v_mul_lo_u32 v17, s10, v7
 ; GISEL-NEXT:    v_add_i32_e32 v18, vcc, 1, v5
 ; GISEL-NEXT:    v_addc_u32_e32 v19, vcc, 0, v7, vcc
-; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v12, v8
-; GISEL-NEXT:    v_add_i32_e32 v12, vcc, v14, v17
+; GISEL-NEXT:    v_or_b32_e32 v8, v12, v8
+; GISEL-NEXT:    v_or_b32_e32 v12, v14, v17
 ; GISEL-NEXT:    v_add_i32_e32 v14, vcc, 1, v9
 ; GISEL-NEXT:    v_addc_u32_e32 v17, vcc, 0, v16, vcc
 ; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v8, v13
 ; GISEL-NEXT:    v_add_i32_e32 v12, vcc, v12, v15
 ; GISEL-NEXT:    v_add_i32_e32 v13, vcc, 1, v18
 ; GISEL-NEXT:    v_addc_u32_e32 v15, vcc, 0, v19, vcc
 ; GISEL-NEXT:    v_sub_i32_e32 v2, vcc, v2, v10
 ; GISEL-NEXT:    v_subb_u32_e64 v10, s[4:5], v3, v8, vcc

 ; CGP-NEXT:    v_add_i32_e32 v6, vcc, v6, v8
 ; CGP-NEXT:    v_add_i32_e32 v7, vcc, v7, v9
 ; CGP-NEXT:    v_mul_lo_u32 v8, s10, v6
 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, 1, v4
 ; CGP-NEXT:    v_addc_u32_e32 v16, vcc, 0, v6, vcc
 ; CGP-NEXT:    v_mul_lo_u32 v17, s10, v7
 ; CGP-NEXT:    v_add_i32_e32 v18, vcc, 1, v5
 ; CGP-NEXT:    v_addc_u32_e32 v19, vcc, 0, v7, vcc
-; CGP-NEXT:    v_add_i32_e32 v8, vcc, v12, v8
-; CGP-NEXT:    v_add_i32_e32 v12, vcc, v14, v17
+; CGP-NEXT:    v_or_b32_e32 v8, v12, v8
+; CGP-NEXT:    v_or_b32_e32 v12, v14, v17
 ; CGP-NEXT:    v_add_i32_e32 v14, vcc, 1, v9
 ; CGP-NEXT:    v_addc_u32_e32 v17, vcc, 0, v16, vcc
 ; CGP-NEXT:    v_add_i32_e32 v8, vcc, v8, v13
 ; CGP-NEXT:    v_add_i32_e32 v12, vcc, v12, v15
 ; CGP-NEXT:    v_add_i32_e32 v13, vcc, 1, v18
 ; CGP-NEXT:    v_addc_u32_e32 v15, vcc, 0, v19, vcc
 ; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v2, v10
 ; CGP-NEXT:    v_subb_u32_e64 v10, s[4:5], v3, v8, vcc

 ; GISEL-NEXT:    v_add_i32_e32 v6, vcc, v6, v7
 ; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v8, v9
 ; GISEL-NEXT:    v_mul_lo_u32 v8, v3, v6
 ; GISEL-NEXT:    v_add_i32_e32 v9, vcc, 1, v1
 ; GISEL-NEXT:    v_addc_u32_e32 v16, vcc, 0, v6, vcc
 ; GISEL-NEXT:    v_mul_lo_u32 v17, v4, v7
 ; GISEL-NEXT:    v_add_i32_e32 v18, vcc, 1, v5
 ; GISEL-NEXT:    v_addc_u32_e32 v19, vcc, 0, v7, vcc
-; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v12, v8
-; GISEL-NEXT:    v_add_i32_e32 v12, vcc, v14, v17
+; GISEL-NEXT:    v_or_b32_e32 v8, v12, v8
+; GISEL-NEXT:    v_or_b32_e32 v12, v14, v17
 ; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v8, v13
 ; GISEL-NEXT:    v_add_i32_e32 v12, vcc, v12, v15
 ; GISEL-NEXT:    v_sub_i32_e32 v0, vcc, v0, v10
 ; GISEL-NEXT:    v_subb_u32_e64 v10, s[4:5], 0, v8, vcc
 ; GISEL-NEXT:    v_cmp_ge_u32_e64 s[4:5], v0, v3
 ; GISEL-NEXT:    v_cndmask_b32_e64 v13, 0, -1, s[4:5]
 ; GISEL-NEXT:    v_sub_i32_e64 v2, s[4:5], v2, v11
 ; GISEL-NEXT:    v_subb_u32_e64 v11, s[6:7], 0, v12, s[4:5]

llvm/test/CodeGen/AMDGPU/GlobalISel/urem.i32.ll

 ; CGP-NEXT:    v_cvt_f32_u32_e32 v2, v1
 ; CGP-NEXT:    v_sub_i32_e32 v3, vcc, 0, v1
 ; CGP-NEXT:    v_rcp_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_f32_e32 v2, 0x4f7ffffe, v2
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_lo_u32 v3, v3, v2
 ; CGP-NEXT:    v_mul_lo_u32 v4, 0, v3
 ; CGP-NEXT:    v_mul_hi_u32 v3, v2, v3
-; CGP-NEXT:    v_add_i32_e32 v3, vcc, v4, v3
+; CGP-NEXT:    v_or_b32_e32 v3, v4, v3
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; CGP-NEXT:    v_mul_lo_u32 v3, 0, v2
 ; CGP-NEXT:    v_mul_hi_u32 v2, v0, v2
-; CGP-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
+; CGP-NEXT:    v_or_b32_e32 v2, v3, v2
 ; CGP-NEXT:    v_mul_lo_u32 v2, v2, v1
 ; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v2
 ; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v0, v1
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v1
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v0, v1
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v1
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc

 ; CGP-NEXT:    v_cvt_f32_u32_e32 v0, s1
 ; CGP-NEXT:    s_sub_i32 s2, 0, s1
 ; CGP-NEXT:    v_rcp_f32_e32 v0, v0
 ; CGP-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; CGP-NEXT:    v_mul_lo_u32 v1, s2, v0
 ; CGP-NEXT:    v_mul_lo_u32 v2, 0, v1
 ; CGP-NEXT:    v_mul_hi_u32 v1, v0, v1
-; CGP-NEXT:    v_add_i32_e32 v1, vcc, v2, v1
+; CGP-NEXT:    v_or_b32_e32 v1, v2, v1
 ; CGP-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; CGP-NEXT:    v_mul_lo_u32 v1, 0, v0
 ; CGP-NEXT:    v_mul_hi_u32 v0, s0, v0
-; CGP-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
+; CGP-NEXT:    v_or_b32_e32 v0, v1, v0
 ; CGP-NEXT:    v_mul_lo_u32 v0, v0, s1
 ; CGP-NEXT:    v_sub_i32_e32 v0, vcc, s0, v0
 ; CGP-NEXT:    v_subrev_i32_e32 v1, vcc, s1, v0
 ; CGP-NEXT:    v_cmp_le_u32_e32 vcc, s1, v0
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
 ; CGP-NEXT:    v_subrev_i32_e32 v1, vcc, s1, v0
 ; CGP-NEXT:    v_cmp_le_u32_e32 vcc, s1, v0
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc

 ; CGP-NEXT:    v_cvt_u32_f32_e32 v4, v4
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v6, v6
 ; CGP-NEXT:    v_mul_lo_u32 v5, v5, v4
 ; CGP-NEXT:    v_mul_lo_u32 v7, v7, v6
 ; CGP-NEXT:    v_mul_lo_u32 v8, 0, v5
 ; CGP-NEXT:    v_mul_hi_u32 v5, v4, v5
 ; CGP-NEXT:    v_mul_lo_u32 v9, 0, v7
 ; CGP-NEXT:    v_mul_hi_u32 v7, v6, v7
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v8, v5
-; CGP-NEXT:    v_add_i32_e32 v7, vcc, v9, v7
+; CGP-NEXT:    v_or_b32_e32 v5, v8, v5
+; CGP-NEXT:    v_or_b32_e32 v7, v9, v7
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v5
 ; CGP-NEXT:    v_add_i32_e32 v5, vcc, v6, v7
 ; CGP-NEXT:    v_mul_lo_u32 v6, 0, v4
 ; CGP-NEXT:    v_mul_hi_u32 v4, v0, v4
 ; CGP-NEXT:    v_mul_lo_u32 v7, 0, v5
 ; CGP-NEXT:    v_mul_hi_u32 v5, v1, v5
-; CGP-NEXT:    v_add_i32_e32 v4, vcc, v6, v4
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
+; CGP-NEXT:    v_or_b32_e32 v4, v6, v4
+; CGP-NEXT:    v_or_b32_e32 v5, v7, v5
 ; CGP-NEXT:    v_mul_lo_u32 v4, v4, v2
 ; CGP-NEXT:    v_mul_lo_u32 v5, v5, v3
 ; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v4
 ; CGP-NEXT:    v_sub_i32_e32 v1, vcc, v1, v5
 ; CGP-NEXT:    v_sub_i32_e32 v4, vcc, v0, v2
 ; CGP-NEXT:    v_sub_i32_e32 v5, vcc, v1, v3
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v2
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v4, vcc

 ; CGP-NEXT:    v_cvt_u32_f32_e32 v4, v4
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v6, v6
 ; CGP-NEXT:    v_mul_lo_u32 v5, v5, v4
 ; CGP-NEXT:    v_mul_lo_u32 v7, v7, v6
 ; CGP-NEXT:    v_mul_lo_u32 v8, 0, v5
 ; CGP-NEXT:    v_mul_hi_u32 v5, v4, v5
 ; CGP-NEXT:    v_mul_lo_u32 v9, 0, v7
 ; CGP-NEXT:    v_mul_hi_u32 v7, v6, v7
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v8, v5
-; CGP-NEXT:    v_add_i32_e32 v7, vcc, v9, v7
+; CGP-NEXT:    v_or_b32_e32 v5, v8, v5
+; CGP-NEXT:    v_or_b32_e32 v7, v9, v7
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v5
 ; CGP-NEXT:    v_add_i32_e32 v5, vcc, v6, v7
 ; CGP-NEXT:    v_mul_lo_u32 v6, 0, v4
 ; CGP-NEXT:    v_mul_hi_u32 v4, v0, v4
 ; CGP-NEXT:    v_mul_lo_u32 v7, 0, v5
 ; CGP-NEXT:    v_mul_hi_u32 v5, v1, v5
-; CGP-NEXT:    v_add_i32_e32 v4, vcc, v6, v4
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
+; CGP-NEXT:    v_or_b32_e32 v4, v6, v4
+; CGP-NEXT:    v_or_b32_e32 v5, v7, v5
 ; CGP-NEXT:    v_mul_lo_u32 v4, v4, v2
 ; CGP-NEXT:    v_mul_lo_u32 v5, v5, v3
 ; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v4
 ; CGP-NEXT:    v_sub_i32_e32 v1, vcc, v1, v5
 ; CGP-NEXT:    v_sub_i32_e32 v4, vcc, v0, v2
 ; CGP-NEXT:    v_sub_i32_e32 v5, vcc, v1, v3
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v2
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v4, vcc

 ; CGP-NEXT:    v_cvt_f32_u32_e32 v2, v1
 ; CGP-NEXT:    v_sub_i32_e32 v3, vcc, 0, v1
 ; CGP-NEXT:    v_rcp_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_f32_e32 v2, 0x4f7ffffe, v2
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v2, v2
 ; CGP-NEXT:    v_mul_lo_u32 v3, v3, v2
 ; CGP-NEXT:    v_mul_lo_u32 v4, 0, v3
 ; CGP-NEXT:    v_mul_hi_u32 v3, v2, v3
-; CGP-NEXT:    v_add_i32_e32 v3, vcc, v4, v3
+; CGP-NEXT:    v_or_b32_e32 v3, v4, v3
 ; CGP-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; CGP-NEXT:    v_mul_lo_u32 v3, 0, v2
 ; CGP-NEXT:    v_mul_hi_u32 v2, v0, v2
-; CGP-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
+; CGP-NEXT:    v_or_b32_e32 v2, v3, v2
 ; CGP-NEXT:    v_mul_lo_u32 v2, v2, v1
 ; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v2
 ; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v0, v1
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v1
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v0, v1
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v1
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc

 ; CGP-NEXT:    v_cvt_u32_f32_e32 v4, v4
 ; CGP-NEXT:    v_cvt_u32_f32_e32 v6, v6
 ; CGP-NEXT:    v_mul_lo_u32 v5, v5, v4
 ; CGP-NEXT:    v_mul_lo_u32 v7, v7, v6
 ; CGP-NEXT:    v_mul_lo_u32 v8, 0, v5
 ; CGP-NEXT:    v_mul_hi_u32 v5, v4, v5
 ; CGP-NEXT:    v_mul_lo_u32 v9, 0, v7
 ; CGP-NEXT:    v_mul_hi_u32 v7, v6, v7
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v8, v5
-; CGP-NEXT:    v_add_i32_e32 v7, vcc, v9, v7
+; CGP-NEXT:    v_or_b32_e32 v5, v8, v5
+; CGP-NEXT:    v_or_b32_e32 v7, v9, v7
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, v4, v5
 ; CGP-NEXT:    v_add_i32_e32 v5, vcc, v6, v7
 ; CGP-NEXT:    v_mul_lo_u32 v6, 0, v4
 ; CGP-NEXT:    v_mul_hi_u32 v4, v0, v4
 ; CGP-NEXT:    v_mul_lo_u32 v7, 0, v5
 ; CGP-NEXT:    v_mul_hi_u32 v5, v1, v5
-; CGP-NEXT:    v_add_i32_e32 v4, vcc, v6, v4
-; CGP-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
+; CGP-NEXT:    v_or_b32_e32 v4, v6, v4
+; CGP-NEXT:    v_or_b32_e32 v5, v7, v5
 ; CGP-NEXT:    v_mul_lo_u32 v4, v4, v2
 ; CGP-NEXT:    v_mul_lo_u32 v5, v5, v3
 ; CGP-NEXT:    v_sub_i32_e32 v0, vcc, v0, v4
 ; CGP-NEXT:    v_sub_i32_e32 v1, vcc, v1, v5
 ; CGP-NEXT:    v_sub_i32_e32 v4, vcc, v0, v2
 ; CGP-NEXT:    v_sub_i32_e32 v5, vcc, v1, v3
 ; CGP-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v2
 ; CGP-NEXT:    v_cndmask_b32_e32 v0, v0, v4, vcc

llvm/test/CodeGen/AMDGPU/GlobalISel/urem.i64.ll

 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
 ; CHECK-NEXT:    v_cndmask_b32_e64 v4, 0, 1, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
 ; CHECK-NEXT:    v_mul_lo_u32 v5, s7, v2
 ; CHECK-NEXT:    v_mul_lo_u32 v6, 0, v2
 ; CHECK-NEXT:    v_mul_hi_u32 v2, s7, v2
 ; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v3, v4
 ; CHECK-NEXT:    v_mul_lo_u32 v3, s7, v3
-; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v6, v3
+; CHECK-NEXT:    v_or_b32_e32 v3, v6, v3
 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
 ; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v0, v5
 ; CHECK-NEXT:    v_subb_u32_e64 v3, s[4:5], v1, v2, vcc
 ; CHECK-NEXT:    v_sub_i32_e64 v1, s[4:5], v1, v2
 ; CHECK-NEXT:    v_cmp_le_u32_e64 s[4:5], s7, v0
 ; CHECK-NEXT:    v_cndmask_b32_e64 v2, 0, -1, s[4:5]
 ; CHECK-NEXT:    v_cmp_le_u32_e64 s[4:5], 0, v3
 ; CHECK-NEXT:    v_cndmask_b32_e64 v4, 0, -1, s[4:5]

 ; GISEL-NEXT:    v_add_i32_e32 v9, vcc, v11, v9
 ; GISEL-NEXT:    v_mul_lo_u32 v11, s10, v5
 ; GISEL-NEXT:    v_mul_lo_u32 v13, 0, v5
 ; GISEL-NEXT:    v_mul_hi_u32 v5, s10, v5
 ; GISEL-NEXT:    v_add_i32_e32 v6, vcc, v6, v8
 ; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v9
 ; GISEL-NEXT:    v_mul_lo_u32 v6, s10, v6
 ; GISEL-NEXT:    v_mul_lo_u32 v7, s10, v7
-; GISEL-NEXT:    v_add_i32_e32 v6, vcc, v12, v6
-; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v13, v7
+; GISEL-NEXT:    v_or_b32_e32 v6, v12, v6
+; GISEL-NEXT:    v_or_b32_e32 v7, v13, v7
 ; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v6, v4
 ; GISEL-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
 ; GISEL-NEXT:    v_sub_i32_e32 v2, vcc, v2, v10
 ; GISEL-NEXT:    v_subb_u32_e64 v6, s[4:5], v3, v4, vcc
 ; GISEL-NEXT:    v_sub_i32_e64 v3, s[4:5], v3, v4
 ; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s10, v2
 ; GISEL-NEXT:    v_cndmask_b32_e64 v4, 0, -1, s[4:5]
 ; GISEL-NEXT:    v_sub_i32_e64 v0, s[4:5], v0, v11

 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, v11, v9
 ; CGP-NEXT:    v_mul_lo_u32 v11, s10, v5
 ; CGP-NEXT:    v_mul_lo_u32 v13, 0, v5
 ; CGP-NEXT:    v_mul_hi_u32 v5, s10, v5
 ; CGP-NEXT:    v_add_i32_e32 v6, vcc, v6, v8
 ; CGP-NEXT:    v_add_i32_e32 v7, vcc, v7, v9
 ; CGP-NEXT:    v_mul_lo_u32 v6, s10, v6
 ; CGP-NEXT:    v_mul_lo_u32 v7, s10, v7
-; CGP-NEXT:    v_add_i32_e32 v6, vcc, v12, v6
-; CGP-NEXT:    v_add_i32_e32 v7, vcc, v13, v7
+; CGP-NEXT:    v_or_b32_e32 v6, v12, v6
+; CGP-NEXT:    v_or_b32_e32 v7, v13, v7
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, v6, v4
 ; CGP-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
 ; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v2, v10
 ; CGP-NEXT:    v_subb_u32_e64 v6, s[4:5], v3, v4, vcc
 ; CGP-NEXT:    v_sub_i32_e64 v3, s[4:5], v3, v4
 ; CGP-NEXT:    v_cmp_le_u32_e64 s[4:5], s10, v2
 ; CGP-NEXT:    v_cndmask_b32_e64 v4, 0, -1, s[4:5]
 ; CGP-NEXT:    v_sub_i32_e64 v0, s[4:5], v0, v11

 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
 ; CHECK-NEXT:    v_cndmask_b32_e64 v4, 0, 1, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
 ; CHECK-NEXT:    v_mul_lo_u32 v5, s7, v2
 ; CHECK-NEXT:    v_mul_lo_u32 v6, 0, v2
 ; CHECK-NEXT:    v_mul_hi_u32 v2, s7, v2
 ; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v3, v4
 ; CHECK-NEXT:    v_mul_lo_u32 v3, s7, v3
-; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v6, v3
+; CHECK-NEXT:    v_or_b32_e32 v3, v6, v3
 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
 ; CHECK-NEXT:    v_sub_i32_e32 v0, vcc, v0, v5
 ; CHECK-NEXT:    v_subb_u32_e64 v3, s[4:5], v1, v2, vcc
 ; CHECK-NEXT:    v_sub_i32_e64 v1, s[4:5], v1, v2
 ; CHECK-NEXT:    v_cmp_le_u32_e64 s[4:5], s7, v0
 ; CHECK-NEXT:    v_cndmask_b32_e64 v2, 0, -1, s[4:5]
 ; CHECK-NEXT:    v_cmp_le_u32_e64 s[4:5], 0, v3
 ; CHECK-NEXT:    v_cndmask_b32_e64 v4, 0, -1, s[4:5]

 ; GISEL-NEXT:    v_add_i32_e32 v9, vcc, v11, v9
 ; GISEL-NEXT:    v_mul_lo_u32 v11, s10, v5
 ; GISEL-NEXT:    v_mul_lo_u32 v13, 0, v5
 ; GISEL-NEXT:    v_mul_hi_u32 v5, s10, v5
 ; GISEL-NEXT:    v_add_i32_e32 v6, vcc, v6, v8
 ; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v9
 ; GISEL-NEXT:    v_mul_lo_u32 v6, s10, v6
 ; GISEL-NEXT:    v_mul_lo_u32 v7, s10, v7
-; GISEL-NEXT:    v_add_i32_e32 v6, vcc, v12, v6
-; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v13, v7
+; GISEL-NEXT:    v_or_b32_e32 v6, v12, v6
+; GISEL-NEXT:    v_or_b32_e32 v7, v13, v7
 ; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v6, v4
 ; GISEL-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
 ; GISEL-NEXT:    v_sub_i32_e32 v2, vcc, v2, v10
 ; GISEL-NEXT:    v_subb_u32_e64 v6, s[4:5], v3, v4, vcc
 ; GISEL-NEXT:    v_sub_i32_e64 v3, s[4:5], v3, v4
 ; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s10, v2
 ; GISEL-NEXT:    v_cndmask_b32_e64 v4, 0, -1, s[4:5]
 ; GISEL-NEXT:    v_sub_i32_e64 v0, s[4:5], v0, v11

 ; CGP-NEXT:    v_add_i32_e32 v9, vcc, v11, v9
 ; CGP-NEXT:    v_mul_lo_u32 v11, s10, v5
 ; CGP-NEXT:    v_mul_lo_u32 v13, 0, v5
 ; CGP-NEXT:    v_mul_hi_u32 v5, s10, v5
 ; CGP-NEXT:    v_add_i32_e32 v6, vcc, v6, v8
 ; CGP-NEXT:    v_add_i32_e32 v7, vcc, v7, v9
 ; CGP-NEXT:    v_mul_lo_u32 v6, s10, v6
 ; CGP-NEXT:    v_mul_lo_u32 v7, s10, v7
-; CGP-NEXT:    v_add_i32_e32 v6, vcc, v12, v6
-; CGP-NEXT:    v_add_i32_e32 v7, vcc, v13, v7
+; CGP-NEXT:    v_or_b32_e32 v6, v12, v6
+; CGP-NEXT:    v_or_b32_e32 v7, v13, v7
 ; CGP-NEXT:    v_add_i32_e32 v4, vcc, v6, v4
 ; CGP-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
 ; CGP-NEXT:    v_sub_i32_e32 v2, vcc, v2, v10
 ; CGP-NEXT:    v_subb_u32_e64 v6, s[4:5], v3, v4, vcc
 ; CGP-NEXT:    v_sub_i32_e64 v3, s[4:5], v3, v4
 ; CGP-NEXT:    v_cmp_le_u32_e64 s[4:5], s10, v2
 ; CGP-NEXT:    v_cndmask_b32_e64 v4, 0, -1, s[4:5]
 ; CGP-NEXT:    v_sub_i32_e64 v0, s[4:5], v0, v11

 ; GISEL-NEXT:    v_add_i32_e32 v9, vcc, v11, v9
 ; GISEL-NEXT:    v_mul_lo_u32 v11, v4, v5
 ; GISEL-NEXT:    v_mul_lo_u32 v13, 0, v5
 ; GISEL-NEXT:    v_mul_hi_u32 v5, v4, v5
 ; GISEL-NEXT:    v_add_i32_e32 v6, vcc, v6, v7
 ; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v8, v9
 ; GISEL-NEXT:    v_mul_lo_u32 v6, v3, v6
 ; GISEL-NEXT:    v_mul_lo_u32 v7, v4, v7
-; GISEL-NEXT:    v_add_i32_e32 v6, vcc, v12, v6
-; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v13, v7
+; GISEL-NEXT:    v_or_b32_e32 v6, v12, v6
+; GISEL-NEXT:    v_or_b32_e32 v7, v13, v7
 ; GISEL-NEXT:    v_add_i32_e32 v1, vcc, v6, v1
 ; GISEL-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
 ; GISEL-NEXT:    v_sub_i32_e32 v0, vcc, v0, v10
 ; GISEL-NEXT:    v_subb_u32_e64 v6, s[4:5], 0, v1, vcc
 ; GISEL-NEXT:    v_sub_i32_e64 v1, s[4:5], 0, v1
 ; GISEL-NEXT:    v_cmp_ge_u32_e64 s[4:5], v0, v3
 ; GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, -1, s[4:5]
 ; GISEL-NEXT:    v_sub_i32_e64 v2, s[4:5], v2, v11