[AMDGPU] Introduce SIInstrWorklist to process instructions in moveToVALU

[skc7]

Summary

This patch introduces SIInstrWorklist to process a worklist of MI in moveToVALU. SIInstrWorklist maintains a vector of MachineInstr for worklist and another vector of MachineInstr for deferred list. Deferred list is used to delay the legalization of instruction.

Currently MBUF instructions are tracked in deferred list and will be processed after all the instructions in Initial worklist are done.

This patch is prerequisite for D141030

[skc7]

Fix mul.ll test failure.

[arsenm]

I guess this is one way to do it

llvm/lib/Target/AMDGPU/SIFixSGPRCopies.cpp

872

Should this avoid being reconstructed for each handled case?

llvm/lib/Target/AMDGPU/SIInstrInfo.cpp

6154–6155

Why did you revert from using a SetVector for this case?

6193–6195

Capitalize Impl. You can also do the refactor to use the implicit function with a different typedef for the wordlist in a precommit

6341

whitespace error

6488–6489

Don't understand why this null check was introduced

llvm/test/CodeGen/AMDGPU/control-flow-fastregalloc.ll

13–23

We don't want checks for all this old style metadata. If you're going to generate the checks, use a triple with a newer default code object. Also convert tests to generated checks in a pre-commit

[skc7]

Changes done as per @arsenm review.

[skc7]

Rebase.

[skc7]

Rebase. Update test control-flow-fastregalloc.ll

[skc7]

llvm/lib/Target/AMDGPU/SIFixSGPRCopies.cpp

872

SIInstrWorklist is needed for every moveToVALU call. In lowerVGPR2SGPRCopies method, all the copies are being pushed to worklist and one call to moveToVALU would be made. But in other methods, a new worklist object needs to be created and passed to moveToVALU.

llvm/lib/Target/AMDGPU/SIInstrInfo.cpp

6154–6155

The previous patch was a WIP, I was trying to experiment other things. Now, moved worklists to setVectors

6193–6195

Changed to moveToVALUImpl. This method is tied to this current implementation, where it uses newly created SIInstrWorklist . Not sure, how can it be made into precommit.

6341

Ran clang-format before submitting the patch. Should I move the break below the curly braces?

6488–6489

It was part of WIP patch. Was trying to fix few tests. Removed it.

llvm/test/CodeGen/AMDGPU/control-flow-fastregalloc.ll

13–23

Now updated the check lines which were failing. Reverted from generating automatic check-lines with script.

[arsenm]

llvm/lib/Target/AMDGPU/SIInstrInfo.cpp

20

Leftover new include?

6169

Can just return contains();

6186–6191

Add a comment explaining what deferred instructions are

6341

The break should be before the curly brace but clang-format probably doesn't handle moving across a brace

[skc7]

Changes done as per review by @arsenm

[arsenm]

LGTM with nit

llvm/lib/Target/AMDGPU/SIInstrInfo.cpp

6186–6192

Doesn't explain why these are deferred

[foad]

llvm/test/CodeGen/AMDGPU/sub.ll

2

Please use more descriptive prefixes: GFX6, GFX8, GFX9.

223–224

Should keep this explanatory comment (unless it is no longer relevant).

[skc7]

@foad As per your feedback, changes to sub.ll test have been made and merged upstream : LINK

[foad]

In D147168#4262072, @skc7 wrote:

@foad As per your feedback, changes to sub.ll test have been made and merged upstream : LINK

Thank you.

llvm/lib/Target/AMDGPU/SIFixSGPRCopies.cpp

if (DstReg == AMDGPU::M0 &&
             DefMI->eraseFromParent();
           MI.eraseFromParent();
         }
       }
     }
     return true;
   }
   if (!SrcReg.isVirtual() || TRI->isAGPR(*MRI, SrcReg)) {
-    TII->moveToVALU(MI, MDT);
+    SIInstrWorklist worklist;
+    worklist.insert(&MI);
+    TII->moveToVALU(worklist, MDT);
     return true;
   }
 
   unsigned SMovOp;
   int64_t Imm;
   // If we are just copying an immediate, we can replace the copy with
   // s_mov_b32.
   if (isSafeToFoldImmIntoCopy(&MI, MRI->getVRegDef(SrcReg), TII, SMovOp, Imm)) {

 void SIFixSGPRCopies::lowerVGPR2SGPRCopies(MachineFunction &MF) {
 
   SmallVector<unsigned, 8> LoweringWorklist;
   for (auto &C : V2SCopies) {
     if (needToBeConvertedToVALU(&C.second))
       LoweringWorklist.push_back(C.second.ID);
   }
 
+  // Store all the V2S copy instructions that need to be moved to VALU
+  // in the Copies worklist.
+  SIInstrWorklist Copies;
+
   while (!LoweringWorklist.empty()) {
     unsigned CurID = LoweringWorklist.pop_back_val();
     auto CurInfoIt = V2SCopies.find(CurID);
     if (CurInfoIt != V2SCopies.end()) {
       V2SCopyInfo C = CurInfoIt->getSecond();
       LLVM_DEBUG(dbgs() << "Processing ...\n"; C.dump());
       for (auto S : C.Siblings) {
         auto SibInfoIt = V2SCopies.find(S);
         if (SibInfoIt != V2SCopies.end()) {
           V2SCopyInfo &SI = SibInfoIt->getSecond();
           LLVM_DEBUG(dbgs() << "Sibling:\n"; SI.dump());
           if (!SI.NeedToBeConvertedToVALU) {
             set_subtract(SI.SChain, C.SChain);
             if (needToBeConvertedToVALU(&SI))
               LoweringWorklist.push_back(SI.ID);
           }
           SI.Siblings.remove_if([&](unsigned ID) { return ID == C.ID; });
         }
       }
       LLVM_DEBUG(dbgs() << "V2S copy " << *C.Copy
                         << " is being turned to VALU\n");
       V2SCopies.erase(C.ID);
-      TII->moveToVALU(*C.Copy, MDT);
+      Copies.insert(C.Copy);
     }
   }
 
+  TII->moveToVALU(Copies, MDT);
+  Copies.clear();
+
   // Now do actual lowering
   for (auto C : V2SCopies) {
     MachineInstr *MI = C.second.Copy;
     MachineBasicBlock *MBB = MI->getParent();
     // We decide to turn V2S copy to v_readfirstlane_b32
     // remove it from the V2SCopies and remove it from all its siblings
     LLVM_DEBUG(dbgs() << "V2S copy " << *MI
                       << " is being turned to v_readfirstlane_b32"

llvm/lib/Target/AMDGPU/SIInstrInfo.h

 class TargetRegisterClass;
 class ScheduleHazardRecognizer;
 
 /// Mark the MMO of a uniform load if there are no potentially clobbering stores
 /// on any path from the start of an entry function to this load.
 static const MachineMemOperand::Flags MONoClobber =
     MachineMemOperand::MOTargetFlag1;
 
+// Utility to store machine instructions worklist in order
+// as defined by comparator Cmp.
+struct SIInstrWorklist {
+  SIInstrWorklist() : InstrList() {}
+
+  void insert(MachineInstr *MI);
+
+  MachineInstr *top() const {
+    auto iter = InstrList.begin();
+    return *iter;
+  }
+
+  void erase_top() {
+    auto iter = InstrList.begin();
+    InstrList.erase(iter);
+  }
+
+  bool empty() const { return InstrList.empty(); }
+
+  void clear() {
+    InstrList.clear();
+    DeferredList.clear();
+  }
+
+  bool isDeferred(MachineInstr *MI);
+
+  std::vector<MachineInstr *> &getDeferredList() { return DeferredList; }
+
+private:
+  std::vector<MachineInstr *> InstrList;
+  std::vector<MachineInstr *> DeferredList;
+};
+
 class SIInstrInfo final : public AMDGPUGenInstrInfo {
 private:
   const SIRegisterInfo RI;
   const GCNSubtarget &ST;
   TargetSchedModel SchedModel;
   mutable std::unique_ptr<AMDGPUMIRFormatter> Formatter;
 
   // The inverse predicate should have the negative value.
MachineOperand buildExtractSubRegOrImm(MachineBasicBlock::iterator MI,
                                          MachineOperand &SuperReg,
                                          const TargetRegisterClass *SuperRC,
                                          unsigned SubIdx,
                                          const TargetRegisterClass *SubRC) const;
 private:
   void swapOperands(MachineInstr &Inst) const;
 
   std::pair<bool, MachineBasicBlock *>
-  moveScalarAddSub(SetVectorType &Worklist, MachineInstr &Inst,
+  moveScalarAddSub(SIInstrWorklist &Worklist, MachineInstr &Inst,
                    MachineDominatorTree *MDT = nullptr) const;
 
-  void lowerSelect(SetVectorType &Worklist, MachineInstr &Inst,
+  void lowerSelect(SIInstrWorklist &Worklist, MachineInstr &Inst,
                    MachineDominatorTree *MDT = nullptr) const;
 
-  void lowerScalarAbs(SetVectorType &Worklist,
-                      MachineInstr &Inst) const;
+  void lowerScalarAbs(SIInstrWorklist &Worklist, MachineInstr &Inst) const;
 
-  void lowerScalarXnor(SetVectorType &Worklist,
-                       MachineInstr &Inst) const;
+  void lowerScalarXnor(SIInstrWorklist &Worklist, MachineInstr &Inst) const;
 
-  void splitScalarNotBinop(SetVectorType &Worklist,
-                           MachineInstr &Inst,
+  void splitScalarNotBinop(SIInstrWorklist &Worklist, MachineInstr &Inst,
                            unsigned Opcode) const;
 
-  void splitScalarBinOpN2(SetVectorType &Worklist,
-                          MachineInstr &Inst,
+  void splitScalarBinOpN2(SIInstrWorklist &Worklist, MachineInstr &Inst,
                           unsigned Opcode) const;
 
-  void splitScalar64BitUnaryOp(SetVectorType &Worklist,
-                               MachineInstr &Inst, unsigned Opcode,
-                               bool Swap = false) const;
+  void splitScalar64BitUnaryOp(SIInstrWorklist &Worklist, MachineInstr &Inst,
+                               unsigned Opcode, bool Swap = false) const;
 
-  void splitScalar64BitAddSub(SetVectorType &Worklist, MachineInstr &Inst,
+  void splitScalar64BitAddSub(SIInstrWorklist &Worklist, MachineInstr &Inst,
                               MachineDominatorTree *MDT = nullptr) const;
 
-  void splitScalar64BitBinaryOp(SetVectorType &Worklist, MachineInstr &Inst,
+  void splitScalar64BitBinaryOp(SIInstrWorklist &Worklist, MachineInstr &Inst,
                                 unsigned Opcode,
                                 MachineDominatorTree *MDT = nullptr) const;
 
-  void splitScalar64BitXnor(SetVectorType &Worklist, MachineInstr &Inst,
-                                MachineDominatorTree *MDT = nullptr) const;
+  void splitScalar64BitXnor(SIInstrWorklist &Worklist, MachineInstr &Inst,
 
-  void splitScalar64BitBCNT(SetVectorType &Worklist,
+  void splitScalar64BitBCNT(SIInstrWorklist &Worklist,
                             MachineInstr &Inst) const;
-  void splitScalar64BitBFE(SetVectorType &Worklist,
-                           MachineInstr &Inst) const;
-  void movePackToVALU(SetVectorType &Worklist,
-                      MachineRegisterInfo &MRI,
+  void splitScalar64BitBFE(SIInstrWorklist &Worklist, MachineInstr &Inst) const;
+  void movePackToVALU(SIInstrWorklist &Worklist, MachineRegisterInfo &MRI,
                       MachineInstr &Inst) const;
 
   void addUsersToMoveToVALUWorklist(Register Reg, MachineRegisterInfo &MRI,
-                                    SetVectorType &Worklist) const;
+                                    SIInstrWorklist &Worklist) const;
 
   void addSCCDefUsersToVALUWorklist(MachineOperand &Op,
                                     MachineInstr &SCCDefInst,
-                                    SetVectorType &Worklist,
+                                    SIInstrWorklist &Worklist,
                                     Register NewCond = Register()) const;
   void addSCCDefsToVALUWorklist(MachineInstr *SCCUseInst,
-                                SetVectorType &Worklist) const;
+                                SIInstrWorklist &Worklist) const;
 
   const TargetRegisterClass *
   getDestEquivalentVGPRClass(const MachineInstr &Inst) const;
 
   bool checkInstOffsetsDoNotOverlap(const MachineInstr &MIa,
                                     const MachineInstr &MIb) const;
 
   Register findUsedSGPR(const MachineInstr &MI, int OpIndices[3]) const;
public:
   /// \returns A new basic block that contains \p MI if new blocks were created.
   MachineBasicBlock *
   legalizeOperands(MachineInstr &MI, MachineDominatorTree *MDT = nullptr) const;
 
   /// Change SADDR form of a FLAT \p Inst to its VADDR form if saddr operand
   /// was moved to VGPR. \returns true if succeeded.
   bool moveFlatAddrToVGPR(MachineInstr &Inst) const;
 
-  /// Replace this instruction's opcode with the equivalent VALU
-  /// opcode.  This function will also move the users of \p MI to the
-  /// VALU if necessary. If present, \p MDT is updated.
-  MachineBasicBlock *moveToVALU(MachineInstr &MI,
-                                MachineDominatorTree *MDT = nullptr) const;
+  /// Replace the instructions opcode with the equivalent VALU
+  /// opcode.  This function will also move the users of MachineInstruntions
+  /// in the \p WorkList to the VALU if necessary. If present, \p MDT is
+  /// updated.
+  void moveToVALU(SIInstrWorklist &Worklist, MachineDominatorTree *MDT) const;
+
+  void moveToVALUimpl(SIInstrWorklist &Worklist, MachineDominatorTree *MDT,
+                      MachineInstr &Inst) const;
 
   void insertNoop(MachineBasicBlock &MBB,
                   MachineBasicBlock::iterator MI) const override;
 
   void insertNoops(MachineBasicBlock &MBB, MachineBasicBlock::iterator MI,
                    unsigned Quantity) const override;
 
   void insertReturn(MachineBasicBlock &MBB) const;

llvm/lib/Target/AMDGPU/SIInstrInfo.cpp

 //===----------------------------------------------------------------------===//
 
 #include "SIInstrInfo.h"
 #include "AMDGPU.h"
 #include "AMDGPUInstrInfo.h"
 #include "GCNHazardRecognizer.h"
 #include "GCNSubtarget.h"
 #include "SIMachineFunctionInfo.h"
+#include "llvm/ADT/PostOrderIterator.h"
 #include "llvm/Analysis/ValueTracking.h"
 #include "llvm/CodeGen/LiveIntervals.h"
 #include "llvm/CodeGen/LiveVariables.h"
 #include "llvm/CodeGen/MachineDominators.h"
 #include "llvm/CodeGen/MachineFrameInfo.h"
 #include "llvm/CodeGen/MachineScheduler.h"
 #include "llvm/CodeGen/RegisterScavenging.h"
 #include "llvm/CodeGen/ScheduleDAG.h"
if (VAddr && AMDGPU::getIfAddr64Inst(MI.getOpcode()) != -1) {
       // to SGPRs.
       CreatedBB = loadSRsrcFromVGPR(*this, MI, *Rsrc, MDT);
       return CreatedBB;
     }
   }
   return CreatedBB;
 }
 
-MachineBasicBlock *SIInstrInfo::moveToVALU(MachineInstr &TopInst,
-                                           MachineDominatorTree *MDT) const {
-  SetVectorType Worklist;
-  Worklist.insert(&TopInst);
-  MachineBasicBlock *CreatedBB = nullptr;
-  MachineBasicBlock *CreatedBBTmp = nullptr;
+void SIInstrWorklist::insert(MachineInstr *MI) {
+  auto itr1 = std::find(InstrList.begin(), InstrList.end(), MI);
+  if (itr1 == InstrList.end())
+    InstrList.push_back(MI);
+
+  // Add MBUF instructiosn to deferred list.
+  int RsrcIdx =
+      AMDGPU::getNamedOperandIdx(MI->getOpcode(), AMDGPU::OpName::srsrc);
+  if (RsrcIdx != -1) {
+    auto itr2 = std::find(DeferredList.begin(), DeferredList.end(), MI);
+    if (!isDeferred(MI))
+      DeferredList.push_back(MI);
+  }
+}
+
+bool SIInstrWorklist::isDeferred(MachineInstr *MI) {
+  auto it = std::find(DeferredList.begin(), DeferredList.end(), MI);
+  if (it != DeferredList.end())
+    return true;
+  return false;
+}
+
+void SIInstrInfo::moveToVALU(SIInstrWorklist &Worklist,
 
   while (!Worklist.empty()) {
-    MachineInstr &Inst = *Worklist.pop_back_val();
-    MachineBasicBlock *MBB = Inst.getParent();
-    MachineRegisterInfo &MRI = MBB->getParent()->getRegInfo();
-
-    unsigned Opcode = Inst.getOpcode();
-    unsigned NewOpcode = getVALUOp(Inst);
-
-    // Handle some special cases
-    switch (Opcode) {
-    default:
-      break;
-    case AMDGPU::S_ADD_U64_PSEUDO:
-    case AMDGPU::S_SUB_U64_PSEUDO:
-      splitScalar64BitAddSub(Worklist, Inst, MDT);
-      Inst.eraseFromParent();
-      continue;
-    case AMDGPU::S_ADD_I32:
-    case AMDGPU::S_SUB_I32: {
-      // FIXME: The u32 versions currently selected use the carry.
-      bool Changed;
-      std::tie(Changed, CreatedBBTmp) = moveScalarAddSub(Worklist, Inst, MDT);
-      if (CreatedBBTmp && TopInst.getParent() == CreatedBBTmp)
-        CreatedBB = CreatedBBTmp;
-      if (Changed)
-        continue;
+    MachineInstr &Inst = *Worklist.top();
+    Worklist.erase_top();
+    if (Worklist.isDeferred(&Inst))
+      continue;
+    moveToVALUimpl(Worklist, MDT, Inst);
+  }
+
+  for (MachineInstr *Inst : Worklist.getDeferredList()) {
+    moveToVALUimpl(Worklist, MDT, *Inst);
+    assert(Worklist.empty() &&
+           "Deferred MachineInstr are not supposed to re-populate worklist");
+  }
+}
+
+void SIInstrInfo::moveToVALUimpl(SIInstrWorklist &Worklist,
+                                 MachineDominatorTree *MDT,
+                                 MachineInstr &Inst) const {
+
+  if (!MBB)
+    return;
+    return;
+    MachineBasicBlock *CreatedBBTmp = nullptr;
+      return;
 
-      // Default handling
-      break;
-    }
-    case AMDGPU::S_AND_B64:
-      splitScalar64BitBinaryOp(Worklist, Inst, AMDGPU::S_AND_B32, MDT);
-      Inst.eraseFromParent();
-      continue;
+    return;
 
-    case AMDGPU::S_OR_B64:
-      splitScalar64BitBinaryOp(Worklist, Inst, AMDGPU::S_OR_B32, MDT);
-      Inst.eraseFromParent();
-      continue;
+    return;
 
-    case AMDGPU::S_XOR_B64:
-      splitScalar64BitBinaryOp(Worklist, Inst, AMDGPU::S_XOR_B32, MDT);
-      Inst.eraseFromParent();
-      continue;
+    return;
 
-    case AMDGPU::S_NAND_B64:
-      splitScalar64BitBinaryOp(Worklist, Inst, AMDGPU::S_NAND_B32, MDT);
-      Inst.eraseFromParent();
-      continue;
+    return;
 
-    case AMDGPU::S_NOR_B64:
-      splitScalar64BitBinaryOp(Worklist, Inst, AMDGPU::S_NOR_B32, MDT);
-      Inst.eraseFromParent();
-      continue;
+    return;
 
-    case AMDGPU::S_XNOR_B64:
-      if (ST.hasDLInsts())
-        splitScalar64BitBinaryOp(Worklist, Inst, AMDGPU::S_XNOR_B32, MDT);
-      else
-        splitScalar64BitXnor(Worklist, Inst, MDT);
-      Inst.eraseFromParent();
-      continue;
+    return;
 
-    case AMDGPU::S_ANDN2_B64:
-      splitScalar64BitBinaryOp(Worklist, Inst, AMDGPU::S_ANDN2_B32, MDT);
-      Inst.eraseFromParent();
-      continue;
+    return;
 
-    case AMDGPU::S_ORN2_B64:
-      splitScalar64BitBinaryOp(Worklist, Inst, AMDGPU::S_ORN2_B32, MDT);
-      Inst.eraseFromParent();
-      continue;
+    return;
 
-    case AMDGPU::S_BREV_B64:
-      splitScalar64BitUnaryOp(Worklist, Inst, AMDGPU::S_BREV_B32, true);
-      Inst.eraseFromParent();
-      continue;
+    return;
 
-    case AMDGPU::S_NOT_B64:
-      splitScalar64BitUnaryOp(Worklist, Inst, AMDGPU::S_NOT_B32);
-      Inst.eraseFromParent();
-      continue;
+    return;
 
-    case AMDGPU::S_BCNT1_I32_B64:
-      splitScalar64BitBCNT(Worklist, Inst);
-      Inst.eraseFromParent();
-      continue;
+    return;
 
-    case AMDGPU::S_BFE_I64:
-      splitScalar64BitBFE(Worklist, Inst);
-      Inst.eraseFromParent();
-      continue;
+    return;
 
-    case AMDGPU::S_LSHL_B32:
-      if (ST.hasOnlyRevVALUShifts()) {
-        NewOpcode = AMDGPU::V_LSHLREV_B32_e64;
-        swapOperands(Inst);
-      }
-      break;
-    case AMDGPU::S_ASHR_I32:
-      if (ST.hasOnlyRevVALUShifts()) {
-        NewOpcode = AMDGPU::V_ASHRREV_I32_e64;
-        swapOperands(Inst);
-      }
-      break;
-    case AMDGPU::S_LSHR_B32:
-      if (ST.hasOnlyRevVALUShifts()) {
-        NewOpcode = AMDGPU::V_LSHRREV_B32_e64;
-        swapOperands(Inst);
-      }
-      break;
-    case AMDGPU::S_LSHL_B64:
-      if (ST.hasOnlyRevVALUShifts()) {
-        NewOpcode = AMDGPU::V_LSHLREV_B64_e64;
-        swapOperands(Inst);
-      }
-      break;
-    case AMDGPU::S_ASHR_I64:
-      if (ST.hasOnlyRevVALUShifts()) {
-        NewOpcode = AMDGPU::V_ASHRREV_I64_e64;
-        swapOperands(Inst);
-      }
-      break;
-    case AMDGPU::S_LSHR_B64:
-      if (ST.hasOnlyRevVALUShifts()) {
-        NewOpcode = AMDGPU::V_LSHRREV_B64_e64;
-        swapOperands(Inst);
-      }
-      break;
 
-    case AMDGPU::S_ABS_I32:
-      lowerScalarAbs(Worklist, Inst);
-      Inst.eraseFromParent();
-      continue;
+    return;
 
-    case AMDGPU::S_CBRANCH_SCC0:
-    case AMDGPU::S_CBRANCH_SCC1: {
-        // Clear unused bits of vcc
-        Register CondReg = Inst.getOperand(1).getReg();
-        bool IsSCC = CondReg == AMDGPU::SCC;
-        Register VCC = RI.getVCC();
-        Register EXEC = ST.isWave32() ? AMDGPU::EXEC_LO : AMDGPU::EXEC;
-        unsigned Opc = ST.isWave32() ? AMDGPU::S_AND_B32 : AMDGPU::S_AND_B64;
-        BuildMI(*MBB, Inst, Inst.getDebugLoc(), get(Opc), VCC)
-            .addReg(EXEC)
-            .addReg(IsSCC ? VCC : CondReg);
-        Inst.removeOperand(1);
-      }
-      break;
+  } break;
 
-    case AMDGPU::S_BFE_U64:
-    case AMDGPU::S_BFM_B64:
-      llvm_unreachable("Moving this op to VALU not implemented");
 
-    case AMDGPU::S_PACK_LL_B32_B16:
-    case AMDGPU::S_PACK_LH_B32_B16:
-    case AMDGPU::S_PACK_HL_B32_B16:
-    case AMDGPU::S_PACK_HH_B32_B16:
-      movePackToVALU(Worklist, MRI, Inst);
-      Inst.eraseFromParent();
-      continue;
+    return;
 
-    case AMDGPU::S_XNOR_B32:
-      lowerScalarXnor(Worklist, Inst);
-      Inst.eraseFromParent();
-      continue;
+    return;
 
-    case AMDGPU::S_NAND_B32:
-      splitScalarNotBinop(Worklist, Inst, AMDGPU::S_AND_B32);
-      Inst.eraseFromParent();
-      continue;
+    return;
 
-    case AMDGPU::S_NOR_B32:
-      splitScalarNotBinop(Worklist, Inst, AMDGPU::S_OR_B32);
-      Inst.eraseFromParent();
-      continue;
+    return;
 
-    case AMDGPU::S_ANDN2_B32:
-      splitScalarBinOpN2(Worklist, Inst, AMDGPU::S_AND_B32);
-      Inst.eraseFromParent();
-      continue;
+    return;
 
-    case AMDGPU::S_ORN2_B32:
-      splitScalarBinOpN2(Worklist, Inst, AMDGPU::S_OR_B32);
-      Inst.eraseFromParent();
-      continue;
+    return;
 
-    // TODO: remove as soon as everything is ready
-    // to replace VGPR to SGPR copy with V_READFIRSTLANEs.
-    // S_ADD/SUB_CO_PSEUDO as well as S_UADDO/USUBO_PSEUDO
-    // can only be selected from the uniform SDNode.
-    case AMDGPU::S_ADD_CO_PSEUDO:
-    case AMDGPU::S_SUB_CO_PSEUDO: {
-      unsigned Opc = (Inst.getOpcode() == AMDGPU::S_ADD_CO_PSEUDO)
-                         ? AMDGPU::V_ADDC_U32_e64
-                         : AMDGPU::V_SUBB_U32_e64;
-      const auto *CarryRC = RI.getRegClass(AMDGPU::SReg_1_XEXECRegClassID);
 
-      Register CarryInReg = Inst.getOperand(4).getReg();
-      if (!MRI.constrainRegClass(CarryInReg, CarryRC)) {
-        Register NewCarryReg = MRI.createVirtualRegister(CarryRC);
-        BuildMI(*MBB, &Inst, Inst.getDebugLoc(), get(AMDGPU::COPY), NewCarryReg)
-            .addReg(CarryInReg);
-      }
+      BuildMI(*MBB, Inst, Inst.getDebugLoc(), get(AMDGPU::COPY), NewCarryReg)
 
-      Register CarryOutReg = Inst.getOperand(1).getReg();
 
-      Register DestReg = MRI.createVirtualRegister(RI.getEquivalentVGPRClass(
-          MRI.getRegClass(Inst.getOperand(0).getReg())));
-      MachineInstr *CarryOp =
-          BuildMI(*MBB, &Inst, Inst.getDebugLoc(), get(Opc), DestReg)
-              .addReg(CarryOutReg, RegState::Define)
-              .add(Inst.getOperand(2))
-              .add(Inst.getOperand(3))
-              .addReg(CarryInReg)
-              .addImm(0);
-      CreatedBBTmp = legalizeOperands(*CarryOp);
-      if (CreatedBBTmp && TopInst.getParent() == CreatedBBTmp)
-        CreatedBB = CreatedBBTmp;
-      MRI.replaceRegWith(Inst.getOperand(0).getReg(), DestReg);
-      addUsersToMoveToVALUWorklist(DestReg, MRI, Worklist);
-      Inst.eraseFromParent();
-    }
-      continue;
-    case AMDGPU::S_UADDO_PSEUDO:
-    case AMDGPU::S_USUBO_PSEUDO: {
-      const DebugLoc &DL = Inst.getDebugLoc();
-      MachineOperand &Dest0 = Inst.getOperand(0);
-      MachineOperand &Dest1 = Inst.getOperand(1);
-      MachineOperand &Src0 = Inst.getOperand(2);
-      MachineOperand &Src1 = Inst.getOperand(3);
+    legalizeOperands(*CarryOp);
+    return;
 
-      unsigned Opc = (Inst.getOpcode() == AMDGPU::S_UADDO_PSEUDO)
-                         ? AMDGPU::V_ADD_CO_U32_e64
-                         : AMDGPU::V_SUB_CO_U32_e64;
-      const TargetRegisterClass *NewRC =
-          RI.getEquivalentVGPRClass(MRI.getRegClass(Dest0.getReg()));
-      Register DestReg = MRI.createVirtualRegister(NewRC);
-      MachineInstr *NewInstr = BuildMI(*MBB, &Inst, DL, get(Opc), DestReg)
-                                   .addReg(Dest1.getReg(), RegState::Define)
-                                   .add(Src0)
-                                   .add(Src1)
-                                   .addImm(0); // clamp bit
 
-      CreatedBBTmp = legalizeOperands(*NewInstr, MDT);
-      if (CreatedBBTmp && TopInst.getParent() == CreatedBBTmp)
-        CreatedBB = CreatedBBTmp;
-
-      MRI.replaceRegWith(Dest0.getReg(), DestReg);
-      addUsersToMoveToVALUWorklist(NewInstr->getOperand(0).getReg(), MRI,
-                                   Worklist);
-      Inst.eraseFromParent();
-    }
-      continue;
+    legalizeOperands(*NewInstr, MDT);
+    return;
 
-    case AMDGPU::S_CSELECT_B32:
-    case AMDGPU::S_CSELECT_B64:
-      lowerSelect(Worklist, Inst, MDT);
-      Inst.eraseFromParent();
-      continue;
-    case AMDGPU::S_CMP_EQ_I32:
-    case AMDGPU::S_CMP_LG_I32:
-    case AMDGPU::S_CMP_GT_I32:
-    case AMDGPU::S_CMP_GE_I32:
-    case AMDGPU::S_CMP_LT_I32:
-    case AMDGPU::S_CMP_LE_I32:
-    case AMDGPU::S_CMP_EQ_U32:
-    case AMDGPU::S_CMP_LG_U32:
-    case AMDGPU::S_CMP_GT_U32:
-    case AMDGPU::S_CMP_GE_U32:
-    case AMDGPU::S_CMP_LT_U32:
-    case AMDGPU::S_CMP_LE_U32:
-    case AMDGPU::S_CMP_EQ_U64:
-    case AMDGPU::S_CMP_LG_U64: {
-        const MCInstrDesc &NewDesc = get(NewOpcode);
-        Register CondReg = MRI.createVirtualRegister(RI.getWaveMaskRegClass());
-        MachineInstr *NewInstr =
-            BuildMI(*MBB, Inst, Inst.getDebugLoc(), NewDesc, CondReg)
-                .add(Inst.getOperand(0))
-                .add(Inst.getOperand(1));
-        legalizeOperands(*NewInstr, MDT);
-        int SCCIdx = Inst.findRegisterDefOperandIdx(AMDGPU::SCC);
-        MachineOperand SCCOp = Inst.getOperand(SCCIdx);
-        addSCCDefUsersToVALUWorklist(SCCOp, Inst, Worklist, CondReg);
-        Inst.eraseFromParent();
-      }
-      continue;
-    }
+    return;
+    return;
 
-    if (NewOpcode == AMDGPU::INSTRUCTION_LIST_END) {
-      // We cannot move this instruction to the VALU, so we should try to
-      // legalize its operands instead.
-      CreatedBBTmp = legalizeOperands(Inst, MDT);
-      if (CreatedBBTmp && TopInst.getParent() == CreatedBBTmp)
-        CreatedBB = CreatedBBTmp;
-      continue;
-    }
-
-    // Handle converting generic instructions like COPY-to-SGPR into
-    // COPY-to-VGPR.
-    if (NewOpcode == Opcode) {
-      Register DstReg = Inst.getOperand(0).getReg();
-      const TargetRegisterClass *NewDstRC = getDestEquivalentVGPRClass(Inst);
+    legalizeOperands(Inst, MDT);
+    return;
+    if (!NewDstRC)
+      return;
 
-      if (Inst.isCopy() && Inst.getOperand(1).getReg().isVirtual() &&
-          NewDstRC == RI.getRegClassForReg(MRI, Inst.getOperand(1).getReg())) {
-        // Instead of creating a copy where src and dst are the same register
-        // class, we just replace all uses of dst with src.  These kinds of
-        // copies interfere with the heuristics MachineSink uses to decide
-        // whether or not to split a critical edge.  Since the pass assumes
-        // that copies will end up as machine instructions and not be
-        // eliminated.
-        addUsersToMoveToVALUWorklist(DstReg, MRI, Worklist);
-        MRI.replaceRegWith(DstReg, Inst.getOperand(1).getReg());
-        MRI.clearKillFlags(Inst.getOperand(1).getReg());
-        Inst.getOperand(0).setReg(DstReg);
-
-        // Make sure we don't leave around a dead VGPR->SGPR copy. Normally
-        // these are deleted later, but at -O0 it would leave a suspicious
-        // looking illegal copy of an undef register.
-        for (unsigned I = Inst.getNumOperands() - 1; I != 0; --I)
-          Inst.removeOperand(I);
-        Inst.setDesc(get(AMDGPU::IMPLICIT_DEF));
-        continue;
-      }
-
-      Register NewDstReg = MRI.createVirtualRegister(NewDstRC);
-      MRI.replaceRegWith(DstReg, NewDstReg);
-      legalizeOperands(Inst, MDT);
-      addUsersToMoveToVALUWorklist(NewDstReg, MRI, Worklist);
-      continue;
-    }
+      return;
+    return;
 
-    // Use the new VALU Opcode.
-    auto NewInstr = BuildMI(*MBB, Inst, Inst.getDebugLoc(), get(NewOpcode))
-                        .setMIFlags(Inst.getFlags());
-    for (const MachineOperand &Op : Inst.explicit_operands())
-      NewInstr->addOperand(Op);
-
-    // Remove any references to SCC. Vector instructions can't read from it, and
-    // We're just about to add the implicit use / defs of VCC, and we don't want
-    // both.
-    for (MachineOperand &Op : Inst.implicit_operands()) {
-      if (Op.getReg() == AMDGPU::SCC) {
-        // Only propagate through live-def of SCC.
-        if (Op.isDef() && !Op.isDead())
-          addSCCDefUsersToVALUWorklist(Op, Inst, Worklist);
-        if (Op.isUse())
-          addSCCDefsToVALUWorklist(NewInstr, Worklist);
-      }
-    }
-
-    Inst.eraseFromParent();
-
-    Register NewDstReg;
-    if (NewInstr->getOperand(0).isReg() && NewInstr->getOperand(0).isDef()) {
-      Register DstReg = NewInstr->getOperand(0).getReg();
-      assert(DstReg.isVirtual());
-
-      // Update the destination register class.
-      const TargetRegisterClass *NewDstRC =
-          getDestEquivalentVGPRClass(*NewInstr);
-      assert(NewDstRC);
-
-      NewDstReg = MRI.createVirtualRegister(NewDstRC);
-      MRI.replaceRegWith(DstReg, NewDstReg);
-    }
-
-    if (Opcode == AMDGPU::S_SEXT_I32_I8 || Opcode == AMDGPU::S_SEXT_I32_I16) {
-      // We are converting these to a BFE, so we need to add the missing
-      // operands for the size and offset.
-      unsigned Size = (Opcode == AMDGPU::S_SEXT_I32_I8) ? 8 : 16;
-      NewInstr.addImm(0);
-      NewInstr.addImm(Size);
-    } else if (Opcode == AMDGPU::S_BCNT1_I32_B32) {
-      // The VALU version adds the second operand to the result, so insert an
-      // extra 0 operand.
-      NewInstr.addImm(0);
-    }
-
-    if (Opcode == AMDGPU::S_BFE_I32 || Opcode == AMDGPU::S_BFE_U32) {
-      const MachineOperand &OffsetWidthOp = NewInstr->getOperand(2);
-      // If we need to move this to VGPRs, we need to unpack the second operand
-      // back into the 2 separate ones for bit offset and width.
-      assert(OffsetWidthOp.isImm() &&
-             "Scalar BFE is only implemented for constant width and offset");
-      uint32_t Imm = OffsetWidthOp.getImm();
-
-      uint32_t Offset = Imm & 0x3f; // Extract bits [5:0].
-      uint32_t BitWidth = (Imm & 0x7f0000) >> 16; // Extract bits [22:16].
-      NewInstr->removeOperand(2);
-      NewInstr.addImm(Offset);
-      NewInstr.addImm(BitWidth);
-    }
-
-    fixImplicitOperands(*NewInstr);
-
-    // Legalize the operands
-    CreatedBBTmp = legalizeOperands(*NewInstr, MDT);
-    if (CreatedBBTmp && TopInst.getParent() == CreatedBBTmp)
-      CreatedBB = CreatedBBTmp;
-
-    if (NewDstReg)
-      addUsersToMoveToVALUWorklist(NewDstReg, MRI, Worklist);
-  }
-  return CreatedBB;
+    const TargetRegisterClass *NewDstRC = getDestEquivalentVGPRClass(*NewInstr);
+    uint32_t Offset = Imm & 0x3f;               // Extract bits [5:0].
+  legalizeOperands(*NewInstr, MDT);
+  return;
 }
 
 // Add/sub require special handling to deal with carry outs.
 std::pair<bool, MachineBasicBlock *>
-SIInstrInfo::moveScalarAddSub(SetVectorType &Worklist, MachineInstr &Inst,
+SIInstrInfo::moveScalarAddSub(SIInstrWorklist &Worklist, MachineInstr &Inst,
                               MachineDominatorTree *MDT) const {
   if (ST.hasAddNoCarry()) {
     // Assume there is no user of scc since we don't select this in that case.
     // Since scc isn't used, it doesn't really matter if the i32 or u32 variant
     // is used.
 
     MachineBasicBlock &MBB = *Inst.getParent();
     MachineRegisterInfo &MRI = MBB.getParent()->getRegInfo();
if (ST.hasAddNoCarry()) {
 
     addUsersToMoveToVALUWorklist(ResultReg, MRI, Worklist);
     return std::pair(true, NewBB);
   }
 
   return std::pair(false, nullptr);
 }
 
-void SIInstrInfo::lowerSelect(SetVectorType &Worklist, MachineInstr &Inst,
+void SIInstrInfo::lowerSelect(SIInstrWorklist &Worklist, MachineInstr &Inst,
                               MachineDominatorTree *MDT) const {
 
   MachineBasicBlock &MBB = *Inst.getParent();
   MachineRegisterInfo &MRI = MBB.getParent()->getRegInfo();
   MachineBasicBlock::iterator MII = Inst;
   DebugLoc DL = Inst.getDebugLoc();
 
   MachineOperand &Dest = Inst.getOperand(0);
auto UpdatedInst =
           .add(Src0) // True
           .addReg(IsSCC ? CopySCC : SCCSource);
 
   MRI.replaceRegWith(Dest.getReg(), ResultReg);
   legalizeOperands(*UpdatedInst, MDT);
   addUsersToMoveToVALUWorklist(ResultReg, MRI, Worklist);
 }
 
-void SIInstrInfo::lowerScalarAbs(SetVectorType &Worklist,
+void SIInstrInfo::lowerScalarAbs(SIInstrWorklist &Worklist,
                                  MachineInstr &Inst) const {
   MachineBasicBlock &MBB = *Inst.getParent();
   MachineRegisterInfo &MRI = MBB.getParent()->getRegInfo();
   MachineBasicBlock::iterator MII = Inst;
   DebugLoc DL = Inst.getDebugLoc();
 
   MachineOperand &Dest = Inst.getOperand(0);
   MachineOperand &Src = Inst.getOperand(1);
void SIInstrInfo::lowerScalarAbs(SIInstrWorklist &Worklist,
   BuildMI(MBB, MII, DL, get(AMDGPU::V_MAX_I32_e64), ResultReg)
     .addReg(Src.getReg())
     .addReg(TmpReg);
 
   MRI.replaceRegWith(Dest.getReg(), ResultReg);
   addUsersToMoveToVALUWorklist(ResultReg, MRI, Worklist);
 }
 
-void SIInstrInfo::lowerScalarXnor(SetVectorType &Worklist,
+void SIInstrInfo::lowerScalarXnor(SIInstrWorklist &Worklist,
                                   MachineInstr &Inst) const {
   MachineBasicBlock &MBB = *Inst.getParent();
   MachineRegisterInfo &MRI = MBB.getParent()->getRegInfo();
   MachineBasicBlock::iterator MII = Inst;
   const DebugLoc &DL = Inst.getDebugLoc();
 
   MachineOperand &Dest = Inst.getOperand(0);
   MachineOperand &Src0 = Inst.getOperand(1);
if (ST.hasDLInsts()) {
     MRI.replaceRegWith(Dest.getReg(), NewDest);
 
     Worklist.insert(Xor);
 
     addUsersToMoveToVALUWorklist(NewDest, MRI, Worklist);
   }
 }
 
-void SIInstrInfo::splitScalarNotBinop(SetVectorType &Worklist,
+void SIInstrInfo::splitScalarNotBinop(SIInstrWorklist &Worklist,
                                       MachineInstr &Inst,
                                       unsigned Opcode) const {
   MachineBasicBlock &MBB = *Inst.getParent();
   MachineRegisterInfo &MRI = MBB.getParent()->getRegInfo();
   MachineBasicBlock::iterator MII = Inst;
   const DebugLoc &DL = Inst.getDebugLoc();
 
   MachineOperand &Dest = Inst.getOperand(0);
void SIInstrInfo::splitScalarNotBinop(SIInstrWorklist &Worklist,
 
   Worklist.insert(&Op);
   Worklist.insert(&Not);
 
   MRI.replaceRegWith(Dest.getReg(), NewDest);
   addUsersToMoveToVALUWorklist(NewDest, MRI, Worklist);
 }
 
-void SIInstrInfo::splitScalarBinOpN2(SetVectorType& Worklist,
+void SIInstrInfo::splitScalarBinOpN2(SIInstrWorklist &Worklist,
                                      MachineInstr &Inst,
                                      unsigned Opcode) const {
   MachineBasicBlock &MBB = *Inst.getParent();
   MachineRegisterInfo &MRI = MBB.getParent()->getRegInfo();
   MachineBasicBlock::iterator MII = Inst;
   const DebugLoc &DL = Inst.getDebugLoc();
 
   MachineOperand &Dest = Inst.getOperand(0);
void SIInstrInfo::splitScalarBinOpN2(SIInstrWorklist &Worklist,
 
   Worklist.insert(&Not);
   Worklist.insert(&Op);
 
   MRI.replaceRegWith(Dest.getReg(), NewDest);
   addUsersToMoveToVALUWorklist(NewDest, MRI, Worklist);
 }
 
-void SIInstrInfo::splitScalar64BitUnaryOp(
-    SetVectorType &Worklist, MachineInstr &Inst,
-    unsigned Opcode, bool Swap) const {
+void SIInstrInfo::splitScalar64BitUnaryOp(SIInstrWorklist &Worklist,
+                                          MachineInstr &Inst, unsigned Opcode,
+                                          bool Swap) const {
   MachineBasicBlock &MBB = *Inst.getParent();
   MachineRegisterInfo &MRI = MBB.getParent()->getRegInfo();
 
   MachineOperand &Dest = Inst.getOperand(0);
   MachineOperand &Src0 = Inst.getOperand(1);
   DebugLoc DL = Inst.getDebugLoc();
 
   MachineBasicBlock::iterator MII = Inst;
void SIInstrInfo::splitScalar64BitUnaryOp(SIInstrWorklist &Worklist,
 
   // We don't need to legalizeOperands here because for a single operand, src0
   // will support any kind of input.
 
   // Move all users of this moved value.
   addUsersToMoveToVALUWorklist(FullDestReg, MRI, Worklist);
 }
 
-void SIInstrInfo::splitScalar64BitAddSub(SetVectorType &Worklist,
+void SIInstrInfo::splitScalar64BitAddSub(SIInstrWorklist &Worklist,
                                          MachineInstr &Inst,
                                          MachineDominatorTree *MDT) const {
   bool IsAdd = (Inst.getOpcode() == AMDGPU::S_ADD_U64_PSEUDO);
 
   MachineBasicBlock &MBB = *Inst.getParent();
   MachineRegisterInfo &MRI = MBB.getParent()->getRegInfo();
   const auto *CarryRC = RI.getRegClass(AMDGPU::SReg_1_XEXECRegClassID);
 
void SIInstrInfo::splitScalar64BitAddSub(SIInstrWorklist &Worklist,
   // valid.
   legalizeOperands(*LoHalf, MDT);
   legalizeOperands(*HiHalf, MDT);
 
   // Move all users of this moved value.
   addUsersToMoveToVALUWorklist(FullDestReg, MRI, Worklist);
 }
 
-void SIInstrInfo::splitScalar64BitBinaryOp(SetVectorType &Worklist,
+void SIInstrInfo::splitScalar64BitBinaryOp(SIInstrWorklist &Worklist,
                                            MachineInstr &Inst, unsigned Opcode,
                                            MachineDominatorTree *MDT) const {
   MachineBasicBlock &MBB = *Inst.getParent();
   MachineRegisterInfo &MRI = MBB.getParent()->getRegInfo();
 
   MachineOperand &Dest = Inst.getOperand(0);
   MachineOperand &Src0 = Inst.getOperand(1);
   MachineOperand &Src1 = Inst.getOperand(2);
void SIInstrInfo::splitScalar64BitBinaryOp(SIInstrWorklist &Worklist,
 
   Worklist.insert(&LoHalf);
   Worklist.insert(&HiHalf);
 
   // Move all users of this moved value.
   addUsersToMoveToVALUWorklist(FullDestReg, MRI, Worklist);
 }
 
-void SIInstrInfo::splitScalar64BitXnor(SetVectorType &Worklist,
+void SIInstrInfo::splitScalar64BitXnor(SIInstrWorklist &Worklist,
                                        MachineInstr &Inst,
                                        MachineDominatorTree *MDT) const {
   MachineBasicBlock &MBB = *Inst.getParent();
   MachineRegisterInfo &MRI = MBB.getParent()->getRegInfo();
 
   MachineOperand &Dest = Inst.getOperand(0);
   MachineOperand &Src0 = Inst.getOperand(1);
   MachineOperand &Src1 = Inst.getOperand(2);
MachineInstr &Xor = *BuildMI(MBB, MII, DL, get(AMDGPU::S_XOR_B64), NewDest)
     .addReg(Interm)
     .add(*Op1);
 
   MRI.replaceRegWith(Dest.getReg(), NewDest);
 
   Worklist.insert(&Xor);
 }
 
-void SIInstrInfo::splitScalar64BitBCNT(
-    SetVectorType &Worklist, MachineInstr &Inst) const {
+void SIInstrInfo::splitScalar64BitBCNT(SIInstrWorklist &Worklist,
+                                       MachineInstr &Inst) const {
   MachineBasicBlock &MBB = *Inst.getParent();
   MachineRegisterInfo &MRI = MBB.getParent()->getRegInfo();
 
   MachineBasicBlock::iterator MII = Inst;
   const DebugLoc &DL = Inst.getDebugLoc();
 
   MachineOperand &Dest = Inst.getOperand(0);
   MachineOperand &Src = Inst.getOperand(1);
void SIInstrInfo::splitScalar64BitBCNT(SIInstrWorklist &Worklist,
 
   MRI.replaceRegWith(Dest.getReg(), ResultReg);
 
   // We don't need to legalize operands here. src0 for either instruction can be
   // an SGPR, and the second input is unused or determined here.
   addUsersToMoveToVALUWorklist(ResultReg, MRI, Worklist);
 }
 
-void SIInstrInfo::splitScalar64BitBFE(SetVectorType &Worklist,
+void SIInstrInfo::splitScalar64BitBFE(SIInstrWorklist &Worklist,
                                       MachineInstr &Inst) const {
   MachineBasicBlock &MBB = *Inst.getParent();
   MachineRegisterInfo &MRI = MBB.getParent()->getRegInfo();
   MachineBasicBlock::iterator MII = Inst;
   const DebugLoc &DL = Inst.getDebugLoc();
 
   MachineOperand &Dest = Inst.getOperand(0);
   uint32_t Imm = Inst.getOperand(2).getImm();
BuildMI(MBB, MII, DL, get(TargetOpcode::REG_SEQUENCE), ResultReg)
     .addReg(TmpReg)
     .addImm(AMDGPU::sub1);
 
   MRI.replaceRegWith(Dest.getReg(), ResultReg);
   addUsersToMoveToVALUWorklist(ResultReg, MRI, Worklist);
 }
 
 void SIInstrInfo::addUsersToMoveToVALUWorklist(
-  Register DstReg,
-  MachineRegisterInfo &MRI,
-  SetVectorType &Worklist) const {
+    Register DstReg, MachineRegisterInfo &MRI,
+    SIInstrWorklist &Worklist) const {
   for (MachineRegisterInfo::use_iterator I = MRI.use_begin(DstReg),
          E = MRI.use_end(); I != E;) {
     MachineInstr &UseMI = *I->getParent();
 
     unsigned OpNo = 0;
 
     switch (UseMI.getOpcode()) {
     case AMDGPU::COPY:
if (!RI.hasVectorRegisters(getOpRegClass(UseMI, OpNo))) {
         ++I;
       } while (I != E && I->getParent() == &UseMI);
     } else {
       ++I;
     }
   }
 }
 
-void SIInstrInfo::movePackToVALU(SetVectorType &Worklist,
+void SIInstrInfo::movePackToVALU(SIInstrWorklist &Worklist,
                                  MachineRegisterInfo &MRI,
                                  MachineInstr &Inst) const {
   Register ResultReg = MRI.createVirtualRegister(&AMDGPU::VGPR_32RegClass);
   MachineBasicBlock *MBB = Inst.getParent();
   MachineOperand &Src0 = Inst.getOperand(1);
   MachineOperand &Src1 = Inst.getOperand(2);
   const DebugLoc &DL = Inst.getDebugLoc();
 
void SIInstrInfo::movePackToVALU(SIInstrWorklist &Worklist,
 
   MachineOperand &Dest = Inst.getOperand(0);
   MRI.replaceRegWith(Dest.getReg(), ResultReg);
   addUsersToMoveToVALUWorklist(ResultReg, MRI, Worklist);
 }
 
 void SIInstrInfo::addSCCDefUsersToVALUWorklist(MachineOperand &Op,
                                                MachineInstr &SCCDefInst,
-                                               SetVectorType &Worklist,
+                                               SIInstrWorklist &Worklist,
                                                Register NewCond) const {
 
   // Ensure that def inst defines SCC, which is still live.
   assert(Op.isReg() && Op.getReg() == AMDGPU::SCC && Op.isDef() &&
          !Op.isDead() && Op.getParent() == &SCCDefInst);
   SmallVector<MachineInstr *, 4> CopyToDelete;
   // This assumes that all the users of SCC are in the same block
   // as the SCC def.

 }
 
 // Instructions that use SCC may be converted to VALU instructions. When that
 // happens, the SCC register is changed to VCC_LO. The instruction that defines
 // SCC must be changed to an instruction that defines VCC. This function makes
 // sure that the instruction that defines SCC is added to the moveToVALU
 // worklist.
 void SIInstrInfo::addSCCDefsToVALUWorklist(MachineInstr *SCCUseInst,
-                                           SetVectorType &Worklist) const {
+                                           SIInstrWorklist &Worklist) const {
   // Look for a preceding instruction that either defines VCC or SCC. If VCC
   // then there is nothing to do because the defining instruction has been
   // converted to a VALU already. If SCC then that instruction needs to be
   // converted to a VALU.
   for (MachineInstr &MI :
        make_range(std::next(MachineBasicBlock::reverse_iterator(SCCUseInst)),
                   SCCUseInst->getParent()->rend())) {
     if (MI.modifiesRegister(AMDGPU::VCC, &RI))

llvm/test/CodeGen/AMDGPU/add3.ll

 ; but we don't.
 define amdgpu_ps float @add3_uniform_vgpr(float inreg %a, float inreg %b, float inreg %c) {
 ; VI-LABEL: add3_uniform_vgpr:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    v_add_f32_e64 v0, s2, 1.0
 ; VI-NEXT:    v_add_f32_e64 v1, s3, 2.0
 ; VI-NEXT:    v_mov_b32_e32 v2, 0x40400000
 ; VI-NEXT:    v_add_f32_e32 v2, s4, v2
-; VI-NEXT:    v_add_u32_e32 v0, vcc, v1, v0
+; VI-NEXT:    v_add_u32_e32 v0, vcc, v0, v1
 ; VI-NEXT:    v_add_u32_e32 v0, vcc, v0, v2
 ; VI-NEXT:    ; return to shader part epilog
 ;
 ; GFX9-LABEL: add3_uniform_vgpr:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    v_add_f32_e64 v0, s2, 1.0
 ; GFX9-NEXT:    v_add_f32_e64 v1, s3, 2.0
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0x40400000

llvm/test/CodeGen/AMDGPU/amdgpu-codegenprepare-idiv.ll

-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --force-update
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: opt -S -mtriple=amdgcn-- -mcpu=tahiti -amdgpu-codegenprepare -amdgpu-bypass-slow-div=0 %s | FileCheck %s
 ; RUN: llc -mtriple=amdgcn-- -mcpu=tahiti -amdgpu-bypass-slow-div=0 < %s | FileCheck -check-prefix=GFX6 %s
 ; RUN: llc -mtriple=amdgcn-- -mcpu=gfx900 -amdgpu-bypass-slow-div=0 < %s | FileCheck -check-prefix=GFX9 %s
 
 define amdgpu_kernel void @udiv_i32(ptr addrspace(1) %out, i32 %x, i32 %y) {
 ; CHECK-LABEL: @udiv_i32(
 ; CHECK-NEXT:    [[TMP1:%.*]] = uitofp i32 [[Y:%.*]] to float
 ; CHECK-NEXT:    [[TMP2:%.*]] = call fast float @llvm.amdgcn.rcp.f32(float [[TMP1]])

 ; GFX6-NEXT:    s_sub_i32 s4, 0, s3
 ; GFX6-NEXT:    s_mov_b32 s5, s1
 ; GFX6-NEXT:    v_rcp_iflag_f32_e32 v0, v0
 ; GFX6-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; GFX6-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v1, s4, v0
 ; GFX6-NEXT:    s_mov_b32 s4, s0
 ; GFX6-NEXT:    v_mul_hi_u32 v1, v0, v1
-; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
+; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; GFX6-NEXT:    v_mul_hi_u32 v0, s2, v0
 ; GFX6-NEXT:    v_readfirstlane_b32 s0, v0
 ; GFX6-NEXT:    s_mul_i32 s0, s0, s3
 ; GFX6-NEXT:    s_sub_i32 s0, s2, s0
 ; GFX6-NEXT:    s_sub_i32 s1, s0, s3
 ; GFX6-NEXT:    s_cmp_ge_u32 s0, s3
 ; GFX6-NEXT:    s_cselect_b32 s0, s1, s0
 ; GFX6-NEXT:    s_sub_i32 s1, s0, s3

 ; GFX6-NEXT:    s_xor_b32 s2, s2, s9
 ; GFX6-NEXT:    s_mov_b32 s5, s1
 ; GFX6-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; GFX6-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v1, s4, v0
 ; GFX6-NEXT:    s_mov_b32 s4, s0
 ; GFX6-NEXT:    s_xor_b32 s0, s9, s8
 ; GFX6-NEXT:    v_mul_hi_u32 v1, v0, v1
-; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
+; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; GFX6-NEXT:    v_mul_hi_u32 v0, s2, v0
 ; GFX6-NEXT:    v_readfirstlane_b32 s1, v0
 ; GFX6-NEXT:    s_mul_i32 s1, s1, s3
 ; GFX6-NEXT:    s_sub_i32 s1, s2, s1
 ; GFX6-NEXT:    s_sub_i32 s2, s1, s3
 ; GFX6-NEXT:    v_add_i32_e32 v1, vcc, 1, v0
 ; GFX6-NEXT:    s_cmp_ge_u32 s1, s3
 ; GFX6-NEXT:    s_cselect_b64 vcc, -1, 0

 ; GFX6-NEXT:    s_cmp_ge_u32 s2, s12
 ; GFX6-NEXT:    s_cselect_b64 s[2:3], -1, 0
 ; GFX6-NEXT:    s_sub_i32 s4, 0, s13
 ; GFX6-NEXT:    v_mul_lo_u32 v3, s4, v1
 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; GFX6-NEXT:    v_add_i32_e32 v2, vcc, 1, v0
 ; GFX6-NEXT:    v_mul_hi_u32 v3, v1, v3
 ; GFX6-NEXT:    v_cndmask_b32_e64 v0, v0, v2, s[2:3]
-; GFX6-NEXT:    v_add_i32_e32 v1, vcc, v3, v1
+; GFX6-NEXT:    v_add_i32_e32 v1, vcc, v1, v3
 ; GFX6-NEXT:    v_mul_hi_u32 v1, s9, v1
 ; GFX6-NEXT:    v_mul_f32_e32 v3, 0x4f7ffffe, v4
 ; GFX6-NEXT:    v_cvt_u32_f32_e32 v3, v3
 ; GFX6-NEXT:    v_readfirstlane_b32 s4, v1
 ; GFX6-NEXT:    s_mul_i32 s4, s4, s13
 ; GFX6-NEXT:    s_sub_i32 s4, s9, s4
 ; GFX6-NEXT:    s_sub_i32 s5, s4, s13
 ; GFX6-NEXT:    s_cmp_ge_u32 s4, s13

 ; GFX6-NEXT:    s_cmp_ge_u32 s6, s14
 ; GFX6-NEXT:    s_cselect_b64 s[6:7], -1, 0
 ; GFX6-NEXT:    s_sub_i32 s8, 0, s15
 ; GFX6-NEXT:    v_mul_lo_u32 v7, s8, v5
 ; GFX6-NEXT:    v_cndmask_b32_e32 v3, v3, v6, vcc
 ; GFX6-NEXT:    v_add_i32_e32 v6, vcc, 1, v3
 ; GFX6-NEXT:    v_mul_hi_u32 v7, v5, v7
 ; GFX6-NEXT:    v_cndmask_b32_e64 v2, v3, v6, s[6:7]
-; GFX6-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
+; GFX6-NEXT:    v_add_i32_e32 v5, vcc, v5, v7
 ; GFX6-NEXT:    v_mul_hi_u32 v5, s11, v5
 ; GFX6-NEXT:    v_readfirstlane_b32 s0, v5
 ; GFX6-NEXT:    s_mul_i32 s0, s0, s15
 ; GFX6-NEXT:    s_sub_i32 s0, s11, s0
 ; GFX6-NEXT:    s_sub_i32 s1, s0, s15
 ; GFX6-NEXT:    v_add_i32_e32 v3, vcc, 1, v5
 ; GFX6-NEXT:    s_cmp_ge_u32 s0, s15
 ; GFX6-NEXT:    s_cselect_b64 vcc, -1, 0

 ; GFX6-NEXT:    s_cselect_b32 s2, s3, s2
 ; GFX6-NEXT:    s_sub_i32 s3, s2, s8
 ; GFX6-NEXT:    s_cmp_ge_u32 s2, s8
 ; GFX6-NEXT:    s_cselect_b32 s4, s3, s2
 ; GFX6-NEXT:    s_sub_i32 s2, 0, s9
 ; GFX6-NEXT:    v_mul_lo_u32 v0, s2, v1
 ; GFX6-NEXT:    v_rcp_iflag_f32_e32 v2, v2
 ; GFX6-NEXT:    v_mul_hi_u32 v0, v1, v0
-; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
+; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
 ; GFX6-NEXT:    v_mul_hi_u32 v0, s5, v0
 ; GFX6-NEXT:    v_mul_f32_e32 v1, 0x4f7ffffe, v2
 ; GFX6-NEXT:    v_cvt_u32_f32_e32 v1, v1
 ; GFX6-NEXT:    v_cvt_f32_u32_e32 v2, s11
 ; GFX6-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX6-NEXT:    s_mul_i32 s2, s2, s9
 ; GFX6-NEXT:    s_sub_i32 s2, s5, s2
 ; GFX6-NEXT:    s_sub_i32 s3, s2, s9
 ; GFX6-NEXT:    s_cmp_ge_u32 s2, s9
 ; GFX6-NEXT:    s_cselect_b32 s2, s3, s2
 ; GFX6-NEXT:    s_sub_i32 s3, s2, s9
 ; GFX6-NEXT:    s_cmp_ge_u32 s2, s9
 ; GFX6-NEXT:    s_cselect_b32 s5, s3, s2
 ; GFX6-NEXT:    s_sub_i32 s2, 0, s10
 ; GFX6-NEXT:    v_mul_lo_u32 v0, s2, v1
 ; GFX6-NEXT:    v_rcp_iflag_f32_e32 v2, v2
 ; GFX6-NEXT:    v_mul_hi_u32 v0, v1, v0
-; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
+; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
 ; GFX6-NEXT:    v_mul_hi_u32 v0, s6, v0
 ; GFX6-NEXT:    v_mul_f32_e32 v1, 0x4f7ffffe, v2
 ; GFX6-NEXT:    v_cvt_u32_f32_e32 v1, v1
 ; GFX6-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX6-NEXT:    s_mul_i32 s2, s2, s10
 ; GFX6-NEXT:    s_sub_i32 s2, s6, s2
 ; GFX6-NEXT:    s_sub_i32 s3, s2, s10
 ; GFX6-NEXT:    s_cmp_ge_u32 s2, s10

 ; GFX6-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; GFX6-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v1, s4, v0
 ; GFX6-NEXT:    s_ashr_i32 s4, s8, 31
 ; GFX6-NEXT:    s_add_i32 s5, s8, s4
 ; GFX6-NEXT:    s_xor_b32 s5, s5, s4
 ; GFX6-NEXT:    v_mul_hi_u32 v1, v0, v1
 ; GFX6-NEXT:    s_xor_b32 s8, s4, s2
-; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
+; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; GFX6-NEXT:    v_mul_hi_u32 v0, s5, v0
 ; GFX6-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX6-NEXT:    s_mul_i32 s2, s2, s3
 ; GFX6-NEXT:    s_sub_i32 s2, s5, s2
 ; GFX6-NEXT:    s_sub_i32 s4, s2, s3
 ; GFX6-NEXT:    s_cmp_ge_u32 s2, s3
 ; GFX6-NEXT:    v_add_i32_e32 v1, vcc, 1, v0
 ; GFX6-NEXT:    s_cselect_b32 s2, s4, s2

 ; GFX6-NEXT:    v_mul_f32_e32 v2, 0x4f7ffffe, v2
 ; GFX6-NEXT:    v_cvt_u32_f32_e32 v2, v2
 ; GFX6-NEXT:    v_mul_lo_u32 v3, s6, v2
 ; GFX6-NEXT:    s_ashr_i32 s6, s9, 31
 ; GFX6-NEXT:    s_add_i32 s7, s9, s6
 ; GFX6-NEXT:    s_xor_b32 s7, s7, s6
 ; GFX6-NEXT:    v_mul_hi_u32 v3, v2, v3
 ; GFX6-NEXT:    s_xor_b32 s9, s6, s4
-; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
+; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; GFX6-NEXT:    v_mul_hi_u32 v2, s7, v2
 ; GFX6-NEXT:    v_readfirstlane_b32 s4, v2
 ; GFX6-NEXT:    s_mul_i32 s4, s4, s5
 ; GFX6-NEXT:    s_sub_i32 s4, s7, s4
 ; GFX6-NEXT:    s_sub_i32 s6, s4, s5
 ; GFX6-NEXT:    s_cmp_ge_u32 s4, s5
 ; GFX6-NEXT:    v_add_i32_e32 v3, vcc, 1, v2
 ; GFX6-NEXT:    s_cselect_b32 s4, s6, s4

 ; GFX6-NEXT:    v_cvt_u32_f32_e32 v3, v1
 ; GFX6-NEXT:    v_subrev_i32_e32 v1, vcc, s9, v2
 ; GFX6-NEXT:    v_mul_lo_u32 v2, s0, v3
 ; GFX6-NEXT:    s_ashr_i32 s0, s11, 31
 ; GFX6-NEXT:    s_add_i32 s1, s11, s0
 ; GFX6-NEXT:    s_xor_b32 s1, s1, s0
 ; GFX6-NEXT:    v_mul_hi_u32 v2, v3, v2
 ; GFX6-NEXT:    s_xor_b32 s0, s0, s10
-; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
+; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
 ; GFX6-NEXT:    v_mul_hi_u32 v3, s1, v2
 ; GFX6-NEXT:    v_subrev_i32_e32 v2, vcc, s12, v4
 ; GFX6-NEXT:    v_readfirstlane_b32 s2, v3
 ; GFX6-NEXT:    s_mul_i32 s2, s2, s13
 ; GFX6-NEXT:    s_sub_i32 s1, s1, s2
 ; GFX6-NEXT:    s_sub_i32 s2, s1, s13
 ; GFX6-NEXT:    v_add_i32_e32 v4, vcc, 1, v3
 ; GFX6-NEXT:    s_cmp_ge_u32 s1, s13

 ; GFX6-NEXT:    v_rcp_iflag_f32_e32 v0, v0
 ; GFX6-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; GFX6-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v1, s8, v0
 ; GFX6-NEXT:    s_ashr_i32 s8, s5, 31
 ; GFX6-NEXT:    s_add_i32 s5, s5, s8
 ; GFX6-NEXT:    s_xor_b32 s5, s5, s8
 ; GFX6-NEXT:    v_mul_hi_u32 v1, v0, v1
-; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
+; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; GFX6-NEXT:    v_mul_hi_u32 v0, s5, v0
 ; GFX6-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX6-NEXT:    s_mul_i32 s2, s2, s4
 ; GFX6-NEXT:    s_sub_i32 s2, s5, s2
 ; GFX6-NEXT:    s_sub_i32 s3, s2, s4
 ; GFX6-NEXT:    s_cmp_ge_u32 s2, s4
 ; GFX6-NEXT:    s_cselect_b32 s2, s3, s2
 ; GFX6-NEXT:    s_sub_i32 s3, s2, s4

 ; GFX6-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; GFX6-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v1, s4, v0
 ; GFX6-NEXT:    s_ashr_i32 s4, s6, 31
 ; GFX6-NEXT:    s_add_i32 s5, s6, s4
 ; GFX6-NEXT:    s_xor_b32 s5, s5, s4
 ; GFX6-NEXT:    v_mul_hi_u32 v1, v0, v1
 ; GFX6-NEXT:    s_sub_i32 s6, s2, s8
-; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
+; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; GFX6-NEXT:    v_mul_hi_u32 v0, s5, v0
 ; GFX6-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX6-NEXT:    s_mul_i32 s2, s2, s3
 ; GFX6-NEXT:    s_sub_i32 s2, s5, s2
 ; GFX6-NEXT:    s_sub_i32 s5, s2, s3
 ; GFX6-NEXT:    s_cmp_ge_u32 s2, s3
 ; GFX6-NEXT:    s_cselect_b32 s2, s5, s2
 ; GFX6-NEXT:    s_sub_i32 s5, s2, s3

 ; GFX6-NEXT:    v_or_b32_e32 v0, 1, v0
 ; GFX6-NEXT:    v_mul_f32_e32 v2, v7, v8
 ; GFX6-NEXT:    v_trunc_f32_e32 v2, v2
 ; GFX6-NEXT:    v_mad_f32 v7, -v2, v6, v7
 ; GFX6-NEXT:    v_cvt_i32_f32_e32 v2, v2
 ; GFX6-NEXT:    v_cmp_ge_f32_e64 vcc, |v7|, |v6|
 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, 0, v0, vcc
 ; GFX6-NEXT:    v_mul_lo_u32 v5, v5, s9
-; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v2, v0
+; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GFX6-NEXT:    v_mul_lo_u32 v0, v0, v3
 ; GFX6-NEXT:    v_sub_i32_e32 v2, vcc, s7, v5
 ; GFX6-NEXT:    v_and_b32_e32 v2, 0x7fff, v2
 ; GFX6-NEXT:    v_sub_i32_e32 v0, vcc, v1, v0
 ; GFX6-NEXT:    v_lshl_b64 v[0:1], v[0:1], 30
 ; GFX6-NEXT:    v_and_b32_e32 v3, 0x7fff, v4
 ; GFX6-NEXT:    v_lshlrev_b32_e32 v2, 15, v2
 ; GFX6-NEXT:    v_or_b32_e32 v2, v2, v3

 ; GFX6-NEXT:    s_cmp_ge_u32 s3, s2
 ; GFX6-NEXT:    s_cselect_b64 s[2:3], -1, 0
 ; GFX6-NEXT:    s_sub_i32 s4, 0, s6
 ; GFX6-NEXT:    v_mul_lo_u32 v3, s4, v1
 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, v0, v2, vcc
 ; GFX6-NEXT:    v_add_i32_e32 v2, vcc, 1, v0
 ; GFX6-NEXT:    v_mul_hi_u32 v3, v1, v3
 ; GFX6-NEXT:    v_cndmask_b32_e64 v0, v0, v2, s[2:3]
-; GFX6-NEXT:    v_add_i32_e32 v1, vcc, v3, v1
+; GFX6-NEXT:    v_add_i32_e32 v1, vcc, v1, v3
 ; GFX6-NEXT:    v_mul_hi_u32 v1, s5, v1
 ; GFX6-NEXT:    v_readfirstlane_b32 s0, v1
 ; GFX6-NEXT:    s_mul_i32 s0, s0, s6
 ; GFX6-NEXT:    s_sub_i32 s0, s5, s0
 ; GFX6-NEXT:    s_sub_i32 s1, s0, s6
 ; GFX6-NEXT:    v_add_i32_e32 v2, vcc, 1, v1
 ; GFX6-NEXT:    s_cmp_ge_u32 s0, s6
 ; GFX6-NEXT:    s_cselect_b64 vcc, -1, 0

 ; GFX6-NEXT:    s_sub_i32 s4, s3, s2
 ; GFX6-NEXT:    s_cmp_ge_u32 s3, s2
 ; GFX6-NEXT:    s_cselect_b32 s4, s4, s3
 ; GFX6-NEXT:    s_sub_i32 s2, 0, s6
 ; GFX6-NEXT:    v_mul_lo_u32 v0, s2, v1
 ; GFX6-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX6-NEXT:    s_mov_b32 s2, -1
 ; GFX6-NEXT:    v_mul_hi_u32 v0, v1, v0
-; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
+; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
 ; GFX6-NEXT:    v_mul_hi_u32 v0, s5, v0
 ; GFX6-NEXT:    v_readfirstlane_b32 s7, v0
 ; GFX6-NEXT:    s_mul_i32 s7, s7, s6
 ; GFX6-NEXT:    s_sub_i32 s5, s5, s7
 ; GFX6-NEXT:    s_sub_i32 s7, s5, s6
 ; GFX6-NEXT:    s_cmp_ge_u32 s5, s6
 ; GFX6-NEXT:    s_cselect_b32 s5, s7, s5
 ; GFX6-NEXT:    s_sub_i32 s7, s5, s6

 ; GFX6-NEXT:    v_mov_b32_e32 v0, 0xd9528441
 ; GFX6-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX6-NEXT:    s_mov_b32 s2, -1
 ; GFX6-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX6-NEXT:    v_mul_hi_i32 v0, s4, v0
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, s4, v0
 ; GFX6-NEXT:    v_lshrrev_b32_e32 v1, 31, v0
 ; GFX6-NEXT:    v_ashrrev_i32_e32 v0, 20, v0
-; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
+; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; GFX6-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX6-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: sdiv_i32_oddk_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dword s4, s[0:1], 0x2c
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    v_mov_b32_e32 v0, 0

 ; GFX6-NEXT:    v_rcp_iflag_f32_e32 v0, v0
 ; GFX6-NEXT:    s_xor_b32 s2, s2, s9
 ; GFX6-NEXT:    s_mov_b32 s5, s1
 ; GFX6-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; GFX6-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v1, s4, v0
 ; GFX6-NEXT:    s_mov_b32 s4, s0
 ; GFX6-NEXT:    v_mul_hi_u32 v1, v0, v1
-; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
+; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; GFX6-NEXT:    v_mul_hi_u32 v0, s2, v0
 ; GFX6-NEXT:    v_readfirstlane_b32 s0, v0
 ; GFX6-NEXT:    s_mul_i32 s0, s0, s3
 ; GFX6-NEXT:    s_sub_i32 s0, s2, s0
 ; GFX6-NEXT:    s_sub_i32 s1, s0, s3
 ; GFX6-NEXT:    v_add_i32_e32 v1, vcc, 1, v0
 ; GFX6-NEXT:    s_cmp_ge_u32 s0, s3
 ; GFX6-NEXT:    s_cselect_b64 vcc, -1, 0

 ; GFX6-NEXT:    s_ashr_i32 s0, s2, 31
 ; GFX6-NEXT:    s_lshr_b32 s0, s0, 20
 ; GFX6-NEXT:    s_add_i32 s0, s2, s0
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, s3, v0
 ; GFX6-NEXT:    s_ashr_i32 s0, s0, 12
 ; GFX6-NEXT:    v_lshrrev_b32_e32 v1, 31, v0
 ; GFX6-NEXT:    v_ashrrev_i32_e32 v0, 11, v0
 ; GFX6-NEXT:    s_mov_b32 s5, s1
-; GFX6-NEXT:    v_add_i32_e32 v1, vcc, v1, v0
+; GFX6-NEXT:    v_add_i32_e32 v1, vcc, v0, v1
 ; GFX6-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX6-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; GFX6-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: ssdiv_v2i32_mixed_pow2k_denom:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0

 ; GFX6-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; GFX6-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v1, s6, v0
 ; GFX6-NEXT:    s_ashr_i32 s6, s4, 31
 ; GFX6-NEXT:    s_add_i32 s4, s4, s6
 ; GFX6-NEXT:    s_xor_b32 s4, s4, s6
 ; GFX6-NEXT:    v_mul_hi_u32 v1, v0, v1
 ; GFX6-NEXT:    s_xor_b32 s6, s6, s3
-; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
+; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; GFX6-NEXT:    v_mul_hi_u32 v0, s4, v0
 ; GFX6-NEXT:    v_readfirstlane_b32 s3, v0
 ; GFX6-NEXT:    s_mul_i32 s3, s3, s2
 ; GFX6-NEXT:    s_sub_i32 s3, s4, s3
 ; GFX6-NEXT:    s_sub_i32 s4, s3, s2
 ; GFX6-NEXT:    v_add_i32_e32 v1, vcc, 1, v0
 ; GFX6-NEXT:    s_cmp_ge_u32 s3, s2
 ; GFX6-NEXT:    s_cselect_b64 vcc, -1, 0

 ; GFX6-NEXT:    v_cvt_u32_f32_e32 v2, v2
 ; GFX6-NEXT:    s_mov_b32 s2, -1
 ; GFX6-NEXT:    v_mul_lo_u32 v3, s8, v2
 ; GFX6-NEXT:    s_ashr_i32 s8, s5, 31
 ; GFX6-NEXT:    s_add_i32 s5, s5, s8
 ; GFX6-NEXT:    s_xor_b32 s5, s5, s8
 ; GFX6-NEXT:    v_mul_hi_u32 v1, v2, v3
 ; GFX6-NEXT:    s_xor_b32 s4, s8, s4
-; GFX6-NEXT:    v_add_i32_e32 v1, vcc, v1, v2
+; GFX6-NEXT:    v_add_i32_e32 v1, vcc, v2, v1
 ; GFX6-NEXT:    v_mul_hi_u32 v1, s5, v1
 ; GFX6-NEXT:    v_readfirstlane_b32 s6, v1
 ; GFX6-NEXT:    s_mul_i32 s6, s6, s7
 ; GFX6-NEXT:    s_sub_i32 s5, s5, s6
 ; GFX6-NEXT:    s_sub_i32 s6, s5, s7
 ; GFX6-NEXT:    v_add_i32_e32 v2, vcc, 1, v1
 ; GFX6-NEXT:    s_cmp_ge_u32 s5, s7
 ; GFX6-NEXT:    s_cselect_b64 vcc, -1, 0

 ; GFX6-NEXT:    v_rcp_iflag_f32_e32 v0, v0
 ; GFX6-NEXT:    s_xor_b32 s6, s2, s5
 ; GFX6-NEXT:    s_mov_b32 s2, -1
 ; GFX6-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; GFX6-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v1, s3, v0
 ; GFX6-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX6-NEXT:    v_mul_hi_u32 v1, v0, v1
-; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
+; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; GFX6-NEXT:    v_mul_hi_u32 v0, s6, v0
 ; GFX6-NEXT:    v_readfirstlane_b32 s7, v0
 ; GFX6-NEXT:    s_mul_i32 s7, s7, s4
 ; GFX6-NEXT:    s_sub_i32 s6, s6, s7
 ; GFX6-NEXT:    s_sub_i32 s7, s6, s4
 ; GFX6-NEXT:    s_cmp_ge_u32 s6, s4
 ; GFX6-NEXT:    s_cselect_b32 s6, s7, s6
 ; GFX6-NEXT:    s_sub_i32 s7, s6, s4

 ; GFX6-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX6-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; GFX6-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v1, s2, v0
 ; GFX6-NEXT:    s_add_i32 s2, s5, s8
 ; GFX6-NEXT:    s_xor_b32 s5, s2, s8
 ; GFX6-NEXT:    s_mov_b32 s2, -1
 ; GFX6-NEXT:    v_mul_hi_u32 v1, v0, v1
-; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
+; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; GFX6-NEXT:    v_mul_hi_u32 v0, s5, v0
 ; GFX6-NEXT:    v_readfirstlane_b32 s7, v0
 ; GFX6-NEXT:    s_mul_i32 s7, s7, s4
 ; GFX6-NEXT:    s_sub_i32 s5, s5, s7
 ; GFX6-NEXT:    s_sub_i32 s7, s5, s4
 ; GFX6-NEXT:    s_cmp_ge_u32 s5, s4
 ; GFX6-NEXT:    s_cselect_b32 s5, s7, s5
 ; GFX6-NEXT:    s_sub_i32 s7, s5, s4

 ; GFX6-NEXT:    v_addc_u32_e32 v0, vcc, v3, v0, vcc
 ; GFX6-NEXT:    v_addc_u32_e32 v2, vcc, 0, v5, vcc
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; GFX6-NEXT:    v_addc_u32_e32 v1, vcc, 0, v2, vcc
 ; GFX6-NEXT:    v_mul_lo_u32 v2, v0, s8
 ; GFX6-NEXT:    v_mul_hi_u32 v3, v0, s9
 ; GFX6-NEXT:    v_mul_lo_u32 v4, v1, s9
 ; GFX6-NEXT:    v_mov_b32_e32 v5, 0x11f
-; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
+; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; GFX6-NEXT:    v_mul_lo_u32 v3, v0, s9
-; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
+; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v4, v2
 ; GFX6-NEXT:    v_sub_i32_e32 v4, vcc, s3, v2
 ; GFX6-NEXT:    v_sub_i32_e32 v3, vcc, s2, v3
 ; GFX6-NEXT:    v_subb_u32_e64 v4, s[0:1], v4, v5, vcc
 ; GFX6-NEXT:    v_subrev_i32_e64 v5, s[0:1], s9, v3
 ; GFX6-NEXT:    v_subbrev_u32_e64 v4, s[0:1], 0, v4, s[0:1]
 ; GFX6-NEXT:    s_movk_i32 s2, 0x11e
 ; GFX6-NEXT:    v_cmp_lt_u32_e64 s[0:1], s2, v4
 ; GFX6-NEXT:    s_mov_b32 s9, 0x976a7376

 ; GFX6-NEXT:    v_addc_u32_e32 v3, vcc, 0, v4, vcc
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GFX6-NEXT:    v_addc_u32_e32 v1, vcc, v1, v3, vcc
 ; GFX6-NEXT:    v_mul_hi_u32 v2, v0, s6
 ; GFX6-NEXT:    v_mul_lo_u32 v3, v1, s6
 ; GFX6-NEXT:    v_mul_lo_u32 v4, v0, s6
 ; GFX6-NEXT:    s_mov_b32 s6, -1
 ; GFX6-NEXT:    v_sub_i32_e32 v2, vcc, v2, v0
-; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
+; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
 ; GFX6-NEXT:    v_mul_lo_u32 v3, v0, v2
 ; GFX6-NEXT:    v_mul_hi_u32 v5, v0, v4
 ; GFX6-NEXT:    v_mul_hi_u32 v6, v0, v2
 ; GFX6-NEXT:    v_mul_hi_u32 v7, v1, v2
 ; GFX6-NEXT:    v_mul_lo_u32 v2, v1, v2
 ; GFX6-NEXT:    v_add_i32_e32 v3, vcc, v5, v3
 ; GFX6-NEXT:    v_addc_u32_e32 v5, vcc, 0, v6, vcc
 ; GFX6-NEXT:    v_mul_lo_u32 v6, v1, v4

 ; GFX6-NEXT:    v_addc_u32_e32 v0, vcc, v3, v0, vcc
 ; GFX6-NEXT:    v_addc_u32_e32 v2, vcc, 0, v5, vcc
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; GFX6-NEXT:    v_addc_u32_e32 v1, vcc, 0, v2, vcc
 ; GFX6-NEXT:    v_mul_lo_u32 v2, v0, s4
 ; GFX6-NEXT:    v_mul_hi_u32 v3, v0, s12
 ; GFX6-NEXT:    v_mul_lo_u32 v1, v1, s12
 ; GFX6-NEXT:    v_mul_lo_u32 v0, v0, s12
-; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
-; GFX6-NEXT:    v_add_i32_e32 v1, vcc, v2, v1
+; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
+; GFX6-NEXT:    v_add_i32_e32 v1, vcc, v1, v2
 ; GFX6-NEXT:    v_sub_i32_e32 v2, vcc, s7, v1
 ; GFX6-NEXT:    v_mov_b32_e32 v3, 0x11f
 ; GFX6-NEXT:    v_sub_i32_e32 v0, vcc, s6, v0
 ; GFX6-NEXT:    v_subb_u32_e64 v2, s[0:1], v2, v3, vcc
 ; GFX6-NEXT:    v_subrev_i32_e64 v4, s[0:1], s12, v0
 ; GFX6-NEXT:    v_subbrev_u32_e64 v5, s[2:3], 0, v2, s[0:1]
 ; GFX6-NEXT:    v_cmp_lt_u32_e64 s[2:3], s5, v5
 ; GFX6-NEXT:    s_mov_b32 s6, 0x9761f7c8

 ; GFX6-NEXT:    s_mov_b32 s0, s4
 ; GFX6-NEXT:    s_mov_b32 s4, 0x12d8fb
 ; GFX6-NEXT:    v_mul_lo_u32 v2, v1, s8
 ; GFX6-NEXT:    v_mul_hi_u32 v3, v0, s8
 ; GFX6-NEXT:    v_mul_lo_u32 v4, v0, s8
 ; GFX6-NEXT:    s_mov_b32 s2, -1
 ; GFX6-NEXT:    s_mov_b32 s1, s5
 ; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
-; GFX6-NEXT:    v_subrev_i32_e32 v2, vcc, v0, v2
+; GFX6-NEXT:    v_sub_i32_e32 v2, vcc, v2, v0
 ; GFX6-NEXT:    v_mul_hi_u32 v3, v0, v4
 ; GFX6-NEXT:    v_mul_lo_u32 v5, v0, v2
 ; GFX6-NEXT:    v_mul_hi_u32 v6, v0, v2
 ; GFX6-NEXT:    v_mul_hi_u32 v7, v1, v2
 ; GFX6-NEXT:    v_mul_lo_u32 v2, v1, v2
 ; GFX6-NEXT:    v_add_i32_e32 v3, vcc, v3, v5
 ; GFX6-NEXT:    v_addc_u32_e32 v5, vcc, 0, v6, vcc
 ; GFX6-NEXT:    v_mul_lo_u32 v6, v1, v4
 ; GFX6-NEXT:    v_mul_hi_u32 v4, v1, v4
 ; GFX6-NEXT:    v_add_i32_e32 v3, vcc, v3, v6
 ; GFX6-NEXT:    v_addc_u32_e32 v3, vcc, v5, v4, vcc
 ; GFX6-NEXT:    v_addc_u32_e32 v4, vcc, 0, v7, vcc
 ; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
 ; GFX6-NEXT:    v_addc_u32_e32 v3, vcc, 0, v4, vcc
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GFX6-NEXT:    v_addc_u32_e32 v1, vcc, v1, v3, vcc
 ; GFX6-NEXT:    v_mul_lo_u32 v2, v1, s8
 ; GFX6-NEXT:    v_mul_hi_u32 v3, v0, s8
-; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
+; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; GFX6-NEXT:    v_mul_lo_u32 v3, v0, s8
-; GFX6-NEXT:    v_subrev_i32_e32 v2, vcc, v0, v2
+; GFX6-NEXT:    v_sub_i32_e32 v2, vcc, v2, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v6, v0, v2
 ; GFX6-NEXT:    v_mul_hi_u32 v7, v0, v3
 ; GFX6-NEXT:    v_mul_hi_u32 v8, v0, v2
 ; GFX6-NEXT:    v_mul_hi_u32 v5, v1, v3
 ; GFX6-NEXT:    v_mul_lo_u32 v3, v1, v3
 ; GFX6-NEXT:    v_mul_hi_u32 v4, v1, v2
 ; GFX6-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
 ; GFX6-NEXT:    v_addc_u32_e32 v7, vcc, 0, v8, vcc

 ; GFX6-NEXT:    v_addc_u32_e32 v1, vcc, 0, v2, vcc
 ; GFX6-NEXT:    v_mul_lo_u32 v4, v1, s4
 ; GFX6-NEXT:    v_mul_hi_u32 v5, v0, s4
 ; GFX6-NEXT:    v_add_i32_e32 v2, vcc, 1, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v8, v0, s4
 ; GFX6-NEXT:    v_addc_u32_e32 v3, vcc, 0, v1, vcc
 ; GFX6-NEXT:    v_add_i32_e32 v6, vcc, 2, v0
 ; GFX6-NEXT:    v_addc_u32_e32 v7, vcc, 0, v1, vcc
-; GFX6-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
+; GFX6-NEXT:    v_add_i32_e32 v4, vcc, v4, v5
 ; GFX6-NEXT:    v_mov_b32_e32 v5, s7
 ; GFX6-NEXT:    v_sub_i32_e32 v8, vcc, s6, v8
 ; GFX6-NEXT:    v_subb_u32_e32 v4, vcc, v5, v4, vcc
 ; GFX6-NEXT:    v_subrev_i32_e32 v5, vcc, s4, v8
 ; GFX6-NEXT:    v_subbrev_u32_e32 v9, vcc, 0, v4, vcc
 ; GFX6-NEXT:    s_mov_b32 s4, 0x12d8fa
 ; GFX6-NEXT:    v_cmp_lt_u32_e32 vcc, s4, v5
 ; GFX6-NEXT:    v_cndmask_b32_e64 v5, 0, -1, vcc

 ; GFX6-NEXT:    s_add_u32 s2, s2, s12
 ; GFX6-NEXT:    s_mov_b32 s13, s12
 ; GFX6-NEXT:    v_mul_lo_u32 v2, s4, v1
 ; GFX6-NEXT:    v_mul_hi_u32 v3, s4, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v5, s5, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v4, s4, v0
 ; GFX6-NEXT:    s_addc_u32 s3, s3, s12
 ; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
-; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v5, v2
+; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v5
 ; GFX6-NEXT:    v_mul_hi_u32 v3, v0, v4
 ; GFX6-NEXT:    v_mul_lo_u32 v5, v0, v2
 ; GFX6-NEXT:    v_mul_hi_u32 v6, v0, v2
 ; GFX6-NEXT:    v_mul_hi_u32 v7, v1, v2
 ; GFX6-NEXT:    v_mul_lo_u32 v2, v1, v2
 ; GFX6-NEXT:    v_add_i32_e32 v3, vcc, v3, v5
 ; GFX6-NEXT:    v_addc_u32_e32 v5, vcc, 0, v6, vcc
 ; GFX6-NEXT:    v_mul_lo_u32 v6, v1, v4

 ; GFX6-NEXT:    v_addc_u32_e32 v0, vcc, v3, v0, vcc
 ; GFX6-NEXT:    v_addc_u32_e32 v2, vcc, 0, v5, vcc
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; GFX6-NEXT:    v_addc_u32_e32 v1, vcc, 0, v2, vcc
 ; GFX6-NEXT:    v_mul_lo_u32 v2, s10, v1
 ; GFX6-NEXT:    v_mul_hi_u32 v3, s10, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v4, s11, v0
 ; GFX6-NEXT:    v_mov_b32_e32 v5, s11
-; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
+; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; GFX6-NEXT:    v_mul_lo_u32 v3, s10, v0
-; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
+; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v4, v2
 ; GFX6-NEXT:    v_sub_i32_e32 v4, vcc, s3, v2
 ; GFX6-NEXT:    v_sub_i32_e32 v3, vcc, s2, v3
 ; GFX6-NEXT:    v_subb_u32_e64 v4, s[0:1], v4, v5, vcc
 ; GFX6-NEXT:    v_subrev_i32_e64 v5, s[0:1], s10, v3
 ; GFX6-NEXT:    v_subbrev_u32_e64 v4, s[0:1], 0, v4, s[0:1]
 ; GFX6-NEXT:    v_cmp_le_u32_e64 s[0:1], s11, v4
 ; GFX6-NEXT:    v_cndmask_b32_e64 v6, 0, -1, s[0:1]
 ; GFX6-NEXT:    v_cmp_le_u32_e64 s[0:1], s10, v5

 ; GFX6-NEXT:    s_lshr_b32 s8, s8, 20
 ; GFX6-NEXT:    v_mul_hi_u32 v2, v0, s6
 ; GFX6-NEXT:    v_mul_lo_u32 v3, v1, s6
 ; GFX6-NEXT:    s_add_u32 s0, s0, s8
 ; GFX6-NEXT:    s_addc_u32 s1, s1, 0
 ; GFX6-NEXT:    s_ashr_i64 s[0:1], s[0:1], 12
 ; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
 ; GFX6-NEXT:    v_mul_lo_u32 v3, v0, s6
-; GFX6-NEXT:    v_subrev_i32_e32 v2, vcc, v0, v2
+; GFX6-NEXT:    v_sub_i32_e32 v2, vcc, v2, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v4, v0, v2
 ; GFX6-NEXT:    v_mul_hi_u32 v5, v0, v3
 ; GFX6-NEXT:    v_mul_hi_u32 v6, v0, v2
 ; GFX6-NEXT:    v_mul_hi_u32 v7, v1, v2
 ; GFX6-NEXT:    v_mul_lo_u32 v2, v1, v2
 ; GFX6-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
 ; GFX6-NEXT:    v_addc_u32_e32 v5, vcc, 0, v6, vcc
 ; GFX6-NEXT:    v_mul_lo_u32 v6, v1, v3
 ; GFX6-NEXT:    v_mul_hi_u32 v3, v1, v3
 ; GFX6-NEXT:    s_ashr_i32 s8, s3, 31
 ; GFX6-NEXT:    s_add_u32 s2, s2, s8
 ; GFX6-NEXT:    v_add_i32_e32 v4, vcc, v4, v6
 ; GFX6-NEXT:    v_addc_u32_e32 v3, vcc, v5, v3, vcc
 ; GFX6-NEXT:    v_addc_u32_e32 v4, vcc, 0, v7, vcc
 ; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
 ; GFX6-NEXT:    v_addc_u32_e32 v3, vcc, 0, v4, vcc
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GFX6-NEXT:    v_addc_u32_e32 v1, vcc, v1, v3, vcc
 ; GFX6-NEXT:    v_mul_lo_u32 v2, v1, s6
 ; GFX6-NEXT:    v_mul_hi_u32 v3, v0, s6
 ; GFX6-NEXT:    s_mov_b32 s9, s8
 ; GFX6-NEXT:    s_addc_u32 s3, s3, s8
 ; GFX6-NEXT:    s_xor_b64 s[2:3], s[2:3], s[8:9]
-; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
+; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; GFX6-NEXT:    v_mul_lo_u32 v3, v0, s6
-; GFX6-NEXT:    v_subrev_i32_e32 v2, vcc, v0, v2
+; GFX6-NEXT:    v_sub_i32_e32 v2, vcc, v2, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v6, v0, v2
 ; GFX6-NEXT:    v_mul_hi_u32 v7, v0, v3
 ; GFX6-NEXT:    v_mul_hi_u32 v8, v0, v2
 ; GFX6-NEXT:    v_mul_hi_u32 v5, v1, v3
 ; GFX6-NEXT:    v_mul_lo_u32 v3, v1, v3
 ; GFX6-NEXT:    v_mul_hi_u32 v4, v1, v2
 ; GFX6-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
 ; GFX6-NEXT:    v_addc_u32_e32 v7, vcc, 0, v8, vcc

 ; GFX6-NEXT:    v_addc_u32_e32 v1, vcc, 0, v2, vcc
 ; GFX6-NEXT:    v_mul_lo_u32 v4, v1, s9
 ; GFX6-NEXT:    v_mul_hi_u32 v5, v0, s9
 ; GFX6-NEXT:    v_add_i32_e32 v2, vcc, 1, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v8, v0, s9
 ; GFX6-NEXT:    v_addc_u32_e32 v3, vcc, 0, v1, vcc
 ; GFX6-NEXT:    v_add_i32_e32 v6, vcc, 2, v0
 ; GFX6-NEXT:    v_addc_u32_e32 v7, vcc, 0, v1, vcc
-; GFX6-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
+; GFX6-NEXT:    v_add_i32_e32 v4, vcc, v4, v5
 ; GFX6-NEXT:    v_mov_b32_e32 v5, s3
 ; GFX6-NEXT:    v_sub_i32_e32 v8, vcc, s2, v8
 ; GFX6-NEXT:    v_subb_u32_e32 v4, vcc, v5, v4, vcc
 ; GFX6-NEXT:    v_subrev_i32_e32 v5, vcc, s9, v8
 ; GFX6-NEXT:    v_subbrev_u32_e32 v9, vcc, 0, v4, vcc
 ; GFX6-NEXT:    s_movk_i32 s2, 0xffe
 ; GFX6-NEXT:    v_cmp_lt_u32_e32 vcc, s2, v5
 ; GFX6-NEXT:    v_cndmask_b32_e64 v5, 0, -1, vcc

 ; GFX6-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; GFX6-NEXT:    s_addc_u32 s1, s5, s16
 ; GFX6-NEXT:    s_xor_b64 s[4:5], s[0:1], s[16:17]
 ; GFX6-NEXT:    v_mul_lo_u32 v2, s10, v1
 ; GFX6-NEXT:    v_mul_hi_u32 v3, s10, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v5, s11, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v4, s10, v0
 ; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
-; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v5, v2
+; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v5
 ; GFX6-NEXT:    v_mul_hi_u32 v3, v0, v4
 ; GFX6-NEXT:    v_mul_lo_u32 v5, v0, v2
 ; GFX6-NEXT:    v_mul_hi_u32 v7, v0, v2
 ; GFX6-NEXT:    v_mul_lo_u32 v6, v1, v4
 ; GFX6-NEXT:    v_mul_hi_u32 v4, v1, v4
 ; GFX6-NEXT:    v_add_i32_e32 v3, vcc, v3, v5
 ; GFX6-NEXT:    v_addc_u32_e32 v5, vcc, 0, v7, vcc
 ; GFX6-NEXT:    v_mul_hi_u32 v7, v1, v2
 ; GFX6-NEXT:    v_mul_lo_u32 v2, v1, v2
 ; GFX6-NEXT:    v_add_i32_e32 v3, vcc, v3, v6
 ; GFX6-NEXT:    v_addc_u32_e32 v3, vcc, v5, v4, vcc
 ; GFX6-NEXT:    v_addc_u32_e32 v4, vcc, 0, v7, vcc
 ; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
 ; GFX6-NEXT:    v_addc_u32_e32 v3, vcc, 0, v4, vcc
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GFX6-NEXT:    v_addc_u32_e32 v1, vcc, v1, v3, vcc
 ; GFX6-NEXT:    v_mul_lo_u32 v2, s10, v1
 ; GFX6-NEXT:    v_mul_hi_u32 v3, s10, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v4, s11, v0
 ; GFX6-NEXT:    s_mov_b32 s11, 0xf000
-; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
+; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; GFX6-NEXT:    v_mul_lo_u32 v3, s10, v0
-; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v4, v2
+; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
 ; GFX6-NEXT:    v_mul_lo_u32 v6, v0, v2
 ; GFX6-NEXT:    v_mul_hi_u32 v7, v0, v3
 ; GFX6-NEXT:    v_mul_hi_u32 v8, v0, v2
 ; GFX6-NEXT:    v_mul_hi_u32 v5, v1, v3
 ; GFX6-NEXT:    v_mul_lo_u32 v3, v1, v3
 ; GFX6-NEXT:    v_mul_hi_u32 v4, v1, v2
 ; GFX6-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
 ; GFX6-NEXT:    v_addc_u32_e32 v7, vcc, 0, v8, vcc

 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, v0, v4, vcc
 ; GFX6-NEXT:    v_mul_hi_u32 v4, s12, v2
 ; GFX6-NEXT:    v_mul_lo_u32 v5, s12, v3
 ; GFX6-NEXT:    s_subb_u32 s13, 0, s3
 ; GFX6-NEXT:    v_mul_lo_u32 v6, s13, v2
 ; GFX6-NEXT:    v_xor_b32_e32 v0, s0, v0
 ; GFX6-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
 ; GFX6-NEXT:    v_mul_lo_u32 v5, s12, v2
-; GFX6-NEXT:    v_add_i32_e32 v4, vcc, v6, v4
+; GFX6-NEXT:    v_add_i32_e32 v4, vcc, v4, v6
 ; GFX6-NEXT:    v_mul_lo_u32 v6, v2, v4
 ; GFX6-NEXT:    v_mul_hi_u32 v7, v2, v5
 ; GFX6-NEXT:    v_mul_hi_u32 v8, v2, v4
 ; GFX6-NEXT:    v_mul_hi_u32 v9, v3, v4
 ; GFX6-NEXT:    v_mul_lo_u32 v4, v3, v4
 ; GFX6-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
 ; GFX6-NEXT:    v_addc_u32_e32 v7, vcc, 0, v8, vcc
 ; GFX6-NEXT:    v_mul_lo_u32 v8, v3, v5
 ; GFX6-NEXT:    v_mul_hi_u32 v5, v3, v5
 ; GFX6-NEXT:    v_xor_b32_e32 v1, s1, v1
 ; GFX6-NEXT:    v_add_i32_e32 v6, vcc, v6, v8
 ; GFX6-NEXT:    v_addc_u32_e32 v5, vcc, v7, v5, vcc
 ; GFX6-NEXT:    v_addc_u32_e32 v6, vcc, 0, v9, vcc
 ; GFX6-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
 ; GFX6-NEXT:    v_addc_u32_e32 v5, vcc, 0, v6, vcc
 ; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
 ; GFX6-NEXT:    v_addc_u32_e32 v3, vcc, v3, v5, vcc
 ; GFX6-NEXT:    v_mul_lo_u32 v4, s12, v3
 ; GFX6-NEXT:    v_mul_hi_u32 v5, s12, v2
 ; GFX6-NEXT:    v_mul_lo_u32 v6, s13, v2
-; GFX6-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
+; GFX6-NEXT:    v_add_i32_e32 v4, vcc, v4, v5
 ; GFX6-NEXT:    v_mul_lo_u32 v5, s12, v2
-; GFX6-NEXT:    v_add_i32_e32 v4, vcc, v6, v4
+; GFX6-NEXT:    v_add_i32_e32 v4, vcc, v4, v6
 ; GFX6-NEXT:    v_mul_lo_u32 v8, v2, v4
 ; GFX6-NEXT:    v_mul_hi_u32 v9, v2, v5
 ; GFX6-NEXT:    v_mul_hi_u32 v10, v2, v4
 ; GFX6-NEXT:    v_mul_hi_u32 v7, v3, v5
 ; GFX6-NEXT:    v_mul_lo_u32 v5, v3, v5
 ; GFX6-NEXT:    v_mul_hi_u32 v6, v3, v4
 ; GFX6-NEXT:    v_add_i32_e32 v8, vcc, v9, v8
 ; GFX6-NEXT:    v_addc_u32_e32 v9, vcc, 0, v10, vcc

 ; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
 ; GFX6-NEXT:    v_addc_u32_e32 v3, vcc, 0, v4, vcc
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GFX6-NEXT:    v_addc_u32_e32 v1, vcc, v1, v3, vcc
 ; GFX6-NEXT:    v_mul_lo_u32 v2, v1, s8
 ; GFX6-NEXT:    v_mul_hi_u32 v3, v0, s8
 ; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; GFX6-NEXT:    v_mul_lo_u32 v3, v0, s8
-; GFX6-NEXT:    v_subrev_i32_e32 v2, vcc, v0, v2
+; GFX6-NEXT:    v_sub_i32_e32 v2, vcc, v2, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v6, v0, v2
 ; GFX6-NEXT:    v_mul_hi_u32 v7, v0, v3
 ; GFX6-NEXT:    v_mul_hi_u32 v8, v0, v2
 ; GFX6-NEXT:    v_mul_hi_u32 v5, v1, v3
 ; GFX6-NEXT:    v_mul_lo_u32 v3, v1, v3
 ; GFX6-NEXT:    v_mul_hi_u32 v4, v1, v2
 ; GFX6-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
 ; GFX6-NEXT:    v_addc_u32_e32 v7, vcc, 0, v8, vcc

 ; GFX6-NEXT:    v_addc_u32_e32 v0, vcc, v3, v0, vcc
 ; GFX6-NEXT:    v_addc_u32_e32 v2, vcc, 0, v5, vcc
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; GFX6-NEXT:    v_addc_u32_e32 v1, vcc, 0, v2, vcc
 ; GFX6-NEXT:    v_mul_lo_u32 v1, v1, s4
 ; GFX6-NEXT:    v_mul_hi_u32 v2, v0, s4
 ; GFX6-NEXT:    v_mul_lo_u32 v0, v0, s4
 ; GFX6-NEXT:    s_mov_b32 s1, s5
-; GFX6-NEXT:    v_add_i32_e32 v1, vcc, v2, v1
+; GFX6-NEXT:    v_add_i32_e32 v1, vcc, v1, v2
 ; GFX6-NEXT:    v_mov_b32_e32 v2, s7
 ; GFX6-NEXT:    v_sub_i32_e32 v0, vcc, s6, v0
 ; GFX6-NEXT:    v_subb_u32_e32 v1, vcc, v2, v1, vcc
 ; GFX6-NEXT:    v_subrev_i32_e32 v2, vcc, s4, v0
 ; GFX6-NEXT:    v_subbrev_u32_e32 v3, vcc, 0, v1, vcc
 ; GFX6-NEXT:    v_subrev_i32_e32 v4, vcc, s4, v2
 ; GFX6-NEXT:    v_subbrev_u32_e32 v5, vcc, 0, v3, vcc
 ; GFX6-NEXT:    s_mov_b32 s4, 0x12d8fa

 ; GFX6-NEXT:    s_add_u32 s2, s2, s10
 ; GFX6-NEXT:    s_mov_b32 s11, s10
 ; GFX6-NEXT:    v_mul_lo_u32 v2, s4, v1
 ; GFX6-NEXT:    v_mul_hi_u32 v3, s4, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v5, s5, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v4, s4, v0
 ; GFX6-NEXT:    s_addc_u32 s3, s3, s10
 ; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
-; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v5, v2
+; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v5
 ; GFX6-NEXT:    v_mul_hi_u32 v3, v0, v4
 ; GFX6-NEXT:    v_mul_lo_u32 v5, v0, v2
 ; GFX6-NEXT:    v_mul_hi_u32 v6, v0, v2
 ; GFX6-NEXT:    v_mul_hi_u32 v7, v1, v2
 ; GFX6-NEXT:    v_mul_lo_u32 v2, v1, v2
 ; GFX6-NEXT:    v_add_i32_e32 v3, vcc, v3, v5
 ; GFX6-NEXT:    v_addc_u32_e32 v5, vcc, 0, v6, vcc
 ; GFX6-NEXT:    v_mul_lo_u32 v6, v1, v4
 ; GFX6-NEXT:    v_mul_hi_u32 v4, v1, v4
 ; GFX6-NEXT:    s_xor_b64 s[12:13], s[2:3], s[10:11]
 ; GFX6-NEXT:    v_add_i32_e32 v3, vcc, v3, v6
 ; GFX6-NEXT:    v_addc_u32_e32 v3, vcc, v5, v4, vcc
 ; GFX6-NEXT:    v_addc_u32_e32 v4, vcc, 0, v7, vcc
 ; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
 ; GFX6-NEXT:    v_addc_u32_e32 v3, vcc, 0, v4, vcc
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GFX6-NEXT:    v_addc_u32_e32 v1, vcc, v1, v3, vcc
 ; GFX6-NEXT:    v_mul_lo_u32 v2, s4, v1
 ; GFX6-NEXT:    v_mul_hi_u32 v3, s4, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v4, s5, v0
 ; GFX6-NEXT:    s_mov_b32 s5, s1
-; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
+; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; GFX6-NEXT:    v_mul_lo_u32 v3, s4, v0
-; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v4, v2
+; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
 ; GFX6-NEXT:    v_mul_lo_u32 v6, v0, v2
 ; GFX6-NEXT:    v_mul_hi_u32 v7, v0, v3
 ; GFX6-NEXT:    v_mul_hi_u32 v8, v0, v2
 ; GFX6-NEXT:    v_mul_hi_u32 v5, v1, v3
 ; GFX6-NEXT:    v_mul_lo_u32 v3, v1, v3
 ; GFX6-NEXT:    v_mul_hi_u32 v4, v1, v2
 ; GFX6-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
 ; GFX6-NEXT:    v_addc_u32_e32 v7, vcc, 0, v8, vcc

 ; GFX6-NEXT:    s_addc_u32 s1, s5, s12
 ; GFX6-NEXT:    s_xor_b64 s[4:5], s[0:1], s[12:13]
 ; GFX6-NEXT:    v_mul_lo_u32 v2, s2, v1
 ; GFX6-NEXT:    v_mul_hi_u32 v3, s2, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v5, s3, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v4, s2, v0
 ; GFX6-NEXT:    s_mov_b32 s10, -1
 ; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
-; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v5, v2
+; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v5
 ; GFX6-NEXT:    v_mul_hi_u32 v3, v0, v4
 ; GFX6-NEXT:    v_mul_lo_u32 v5, v0, v2
 ; GFX6-NEXT:    v_mul_hi_u32 v7, v0, v2
 ; GFX6-NEXT:    v_mul_lo_u32 v6, v1, v4
 ; GFX6-NEXT:    v_mul_hi_u32 v4, v1, v4
 ; GFX6-NEXT:    v_add_i32_e32 v3, vcc, v3, v5
 ; GFX6-NEXT:    v_addc_u32_e32 v5, vcc, 0, v7, vcc
 ; GFX6-NEXT:    v_mul_hi_u32 v7, v1, v2
 ; GFX6-NEXT:    v_mul_lo_u32 v2, v1, v2
 ; GFX6-NEXT:    v_add_i32_e32 v3, vcc, v3, v6
 ; GFX6-NEXT:    v_addc_u32_e32 v3, vcc, v5, v4, vcc
 ; GFX6-NEXT:    v_addc_u32_e32 v4, vcc, 0, v7, vcc
 ; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
 ; GFX6-NEXT:    v_addc_u32_e32 v3, vcc, 0, v4, vcc
 ; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GFX6-NEXT:    v_addc_u32_e32 v1, vcc, v1, v3, vcc
 ; GFX6-NEXT:    v_mul_lo_u32 v2, s2, v1
 ; GFX6-NEXT:    v_mul_hi_u32 v3, s2, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v4, s3, v0
-; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
+; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; GFX6-NEXT:    v_mul_lo_u32 v3, s2, v0
-; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v4, v2
+; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
 ; GFX6-NEXT:    v_mul_lo_u32 v6, v0, v2
 ; GFX6-NEXT:    v_mul_hi_u32 v7, v0, v3
 ; GFX6-NEXT:    v_mul_hi_u32 v8, v0, v2
 ; GFX6-NEXT:    v_mul_hi_u32 v5, v1, v3
 ; GFX6-NEXT:    v_mul_lo_u32 v3, v1, v3
 ; GFX6-NEXT:    v_mul_hi_u32 v4, v1, v2
 ; GFX6-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
 ; GFX6-NEXT:    v_addc_u32_e32 v7, vcc, 0, v8, vcc

 ; GFX6-NEXT:    v_cndmask_b32_e32 v0, v0, v3, vcc
 ; GFX6-NEXT:    v_mul_hi_u32 v3, s0, v2
 ; GFX6-NEXT:    v_mul_lo_u32 v5, s0, v4
 ; GFX6-NEXT:    s_subb_u32 s1, 0, s5
 ; GFX6-NEXT:    v_mul_lo_u32 v6, s1, v2
 ; GFX6-NEXT:    s_ashr_i32 s14, s7, 31
 ; GFX6-NEXT:    v_add_i32_e32 v3, vcc, v5, v3
 ; GFX6-NEXT:    v_mul_lo_u32 v5, s0, v2
-; GFX6-NEXT:    v_add_i32_e32 v3, vcc, v6, v3
+; GFX6-NEXT:    v_add_i32_e32 v3, vcc, v3, v6
 ; GFX6-NEXT:    v_mul_lo_u32 v6, v2, v3
 ; GFX6-NEXT:    v_mul_hi_u32 v7, v2, v5
 ; GFX6-NEXT:    v_mul_hi_u32 v8, v2, v3
 ; GFX6-NEXT:    v_mul_hi_u32 v9, v4, v3
 ; GFX6-NEXT:    v_mul_lo_u32 v3, v4, v3
 ; GFX6-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
 ; GFX6-NEXT:    v_addc_u32_e32 v7, vcc, 0, v8, vcc
 ; GFX6-NEXT:    v_mul_lo_u32 v8, v4, v5
 ; GFX6-NEXT:    v_mul_hi_u32 v5, v4, v5
 ; GFX6-NEXT:    s_mov_b32 s15, s14
 ; GFX6-NEXT:    v_xor_b32_e32 v0, s12, v0
 ; GFX6-NEXT:    v_add_i32_e32 v6, vcc, v6, v8
 ; GFX6-NEXT:    v_addc_u32_e32 v5, vcc, v7, v5, vcc
 ; GFX6-NEXT:    v_addc_u32_e32 v6, vcc, 0, v9, vcc
 ; GFX6-NEXT:    v_add_i32_e32 v3, vcc, v5, v3
 ; GFX6-NEXT:    v_addc_u32_e32 v5, vcc, 0, v6, vcc
 ; GFX6-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; GFX6-NEXT:    v_addc_u32_e32 v3, vcc, v4, v5, vcc
 ; GFX6-NEXT:    v_mul_lo_u32 v4, s0, v3
 ; GFX6-NEXT:    v_mul_hi_u32 v5, s0, v2
 ; GFX6-NEXT:    v_mul_lo_u32 v6, s1, v2
 ; GFX6-NEXT:    v_xor_b32_e32 v1, s12, v1
-; GFX6-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
+; GFX6-NEXT:    v_add_i32_e32 v4, vcc, v4, v5
 ; GFX6-NEXT:    v_mul_lo_u32 v5, s0, v2
-; GFX6-NEXT:    v_add_i32_e32 v4, vcc, v6, v4
+; GFX6-NEXT:    v_add_i32_e32 v4, vcc, v4, v6
 ; GFX6-NEXT:    v_mul_lo_u32 v8, v2, v4
 ; GFX6-NEXT:    v_mul_hi_u32 v9, v2, v5
 ; GFX6-NEXT:    v_mul_hi_u32 v10, v2, v4
 ; GFX6-NEXT:    v_mul_hi_u32 v7, v3, v5
 ; GFX6-NEXT:    v_mul_lo_u32 v5, v3, v5
 ; GFX6-NEXT:    v_mul_hi_u32 v6, v3, v4
 ; GFX6-NEXT:    v_add_i32_e32 v8, vcc, v9, v8
 ; GFX6-NEXT:    v_addc_u32_e32 v9, vcc, 0, v10, vcc

llvm/test/CodeGen/AMDGPU/carryout-selection.ll

 ; CISI-NEXT:    v_mac_f32_e32 v0, 0xcf800000, v1
 ; CISI-NEXT:    v_cvt_u32_f32_e32 v1, v1
 ; CISI-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; CISI-NEXT:    v_mul_lo_u32 v2, s0, v1
 ; CISI-NEXT:    v_mul_hi_u32 v3, s0, v0
 ; CISI-NEXT:    v_mul_lo_u32 v5, s1, v0
 ; CISI-NEXT:    v_mul_lo_u32 v4, s0, v0
 ; CISI-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
-; CISI-NEXT:    v_add_i32_e32 v2, vcc, v5, v2
+; CISI-NEXT:    v_add_i32_e32 v2, vcc, v2, v5
 ; CISI-NEXT:    v_mul_hi_u32 v3, v0, v4
 ; CISI-NEXT:    v_mul_lo_u32 v5, v0, v2
 ; CISI-NEXT:    v_mul_hi_u32 v7, v0, v2
 ; CISI-NEXT:    v_mul_lo_u32 v6, v1, v4
 ; CISI-NEXT:    v_mul_hi_u32 v4, v1, v4
 ; CISI-NEXT:    v_add_i32_e32 v3, vcc, v3, v5
 ; CISI-NEXT:    v_addc_u32_e32 v5, vcc, 0, v7, vcc
 ; CISI-NEXT:    v_mul_hi_u32 v7, v1, v2

 ; CISI-NEXT:    v_addc_u32_e32 v0, vcc, v3, v0, vcc
 ; CISI-NEXT:    v_addc_u32_e32 v2, vcc, 0, v5, vcc
 ; CISI-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; CISI-NEXT:    v_addc_u32_e32 v1, vcc, 0, v2, vcc
 ; CISI-NEXT:    v_mul_lo_u32 v2, s2, v1
 ; CISI-NEXT:    v_mul_hi_u32 v3, s2, v0
 ; CISI-NEXT:    v_mul_lo_u32 v4, s3, v0
 ; CISI-NEXT:    v_mov_b32_e32 v5, s3
-; CISI-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
+; CISI-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; CISI-NEXT:    v_mul_lo_u32 v3, s2, v0
-; CISI-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
+; CISI-NEXT:    v_add_i32_e32 v2, vcc, v4, v2
 ; CISI-NEXT:    v_sub_i32_e32 v4, vcc, s7, v2
 ; CISI-NEXT:    v_sub_i32_e32 v3, vcc, s6, v3
 ; CISI-NEXT:    v_subb_u32_e64 v4, s[0:1], v4, v5, vcc
 ; CISI-NEXT:    v_subrev_i32_e64 v5, s[0:1], s2, v3
 ; CISI-NEXT:    v_subbrev_u32_e64 v4, s[0:1], 0, v4, s[0:1]
 ; CISI-NEXT:    v_cmp_le_u32_e64 s[0:1], s3, v4
 ; CISI-NEXT:    v_cndmask_b32_e64 v6, 0, -1, s[0:1]
 ; CISI-NEXT:    v_cmp_le_u32_e64 s[0:1], s2, v5

 ; VI-NEXT:    v_trunc_f32_e32 v1, v1
 ; VI-NEXT:    v_mac_f32_e32 v0, 0xcf800000, v1
 ; VI-NEXT:    v_cvt_u32_f32_e32 v4, v1
 ; VI-NEXT:    v_cvt_u32_f32_e32 v5, v0
 ; VI-NEXT:    v_mul_lo_u32 v2, s8, v4
 ; VI-NEXT:    v_mad_u64_u32 v[0:1], s[0:1], s8, v5, 0
 ; VI-NEXT:    v_mul_lo_u32 v3, s9, v5
 ; VI-NEXT:    v_add_u32_e32 v1, vcc, v2, v1
-; VI-NEXT:    v_add_u32_e32 v3, vcc, v3, v1
+; VI-NEXT:    v_add_u32_e32 v3, vcc, v1, v3
 ; VI-NEXT:    v_mul_hi_u32 v6, v5, v0
 ; VI-NEXT:    v_mad_u64_u32 v[1:2], s[0:1], v5, v3, 0
 ; VI-NEXT:    v_add_u32_e32 v6, vcc, v6, v1
 ; VI-NEXT:    v_mad_u64_u32 v[0:1], s[0:1], v4, v0, 0
 ; VI-NEXT:    v_addc_u32_e32 v7, vcc, 0, v2, vcc
 ; VI-NEXT:    v_mad_u64_u32 v[2:3], s[0:1], v4, v3, 0
 ; VI-NEXT:    v_add_u32_e32 v0, vcc, v6, v0
 ; VI-NEXT:    v_addc_u32_e32 v0, vcc, v7, v1, vcc
 ; VI-NEXT:    v_addc_u32_e32 v1, vcc, 0, v3, vcc
 ; VI-NEXT:    v_add_u32_e32 v0, vcc, v0, v2
 ; VI-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
 ; VI-NEXT:    v_add_u32_e32 v6, vcc, v5, v0
 ; VI-NEXT:    v_addc_u32_e32 v7, vcc, v4, v1, vcc
 ; VI-NEXT:    v_mad_u64_u32 v[0:1], s[0:1], s8, v6, 0
 ; VI-NEXT:    v_mul_lo_u32 v4, s8, v7
 ; VI-NEXT:    v_mul_lo_u32 v5, s9, v6
 ; VI-NEXT:    v_mul_hi_u32 v8, v6, v0
 ; VI-NEXT:    v_mad_u64_u32 v[2:3], s[0:1], v7, v0, 0
-; VI-NEXT:    v_add_u32_e32 v1, vcc, v1, v4
-; VI-NEXT:    v_add_u32_e32 v1, vcc, v5, v1
+; VI-NEXT:    v_add_u32_e32 v1, vcc, v4, v1
+; VI-NEXT:    v_add_u32_e32 v1, vcc, v1, v5
 ; VI-NEXT:    v_mad_u64_u32 v[4:5], s[0:1], v6, v1, 0
 ; VI-NEXT:    v_mad_u64_u32 v[0:1], s[0:1], v7, v1, 0
 ; VI-NEXT:    v_add_u32_e32 v4, vcc, v8, v4
 ; VI-NEXT:    v_addc_u32_e32 v5, vcc, 0, v5, vcc
 ; VI-NEXT:    v_add_u32_e32 v2, vcc, v4, v2
 ; VI-NEXT:    v_addc_u32_e32 v2, vcc, v5, v3, vcc
 ; VI-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
 ; VI-NEXT:    v_add_u32_e32 v0, vcc, v2, v0

llvm/test/CodeGen/AMDGPU/control-flow-fastregalloc.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 2
 ; RUN: llc -O0 -mtriple=amdgcn--amdhsa -march=amdgcn -amdgpu-spill-sgpr-to-vgpr=0 -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefix=VMEM -check-prefix=GCN %s
 ; RUN: llc -O0 -mtriple=amdgcn--amdhsa -march=amdgcn -amdgpu-spill-sgpr-to-vgpr=1 -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefix=VGPR -check-prefix=GCN %s
 
 ; Verify registers used for tracking exec mask changes when all
 ; registers are spilled at the end of the block. The SGPR spill
 ; placement relative to the exec modifications are important.
 
 ; FIXME: This checks with SGPR to VGPR spilling disabled, but this may
 ; not work correctly in cases where no workitems take a branch.
 
-
-; GCN-LABEL: {{^}}divergent_if_endif:
-; VGPR: workitem_private_segment_byte_size = 12{{$}}
-
-
-; GCN: {{^}}; %bb.0:
-; GCN: s_mov_b32 m0, -1
-; GCN: ds_read_b32 [[LOAD0:v[0-9]+]]
-
-; Spill load
-; GCN: buffer_store_dword [[LOAD0]], off, s[0:3], 0 offset:[[LOAD0_OFFSET:[0-9]+]] ; 4-byte Folded Spill
-; GCN: v_cmp_eq_u32_e64 [[CMP0:s\[[0-9]+:[0-9]\]]], v0, s{{[0-9]+}}
-
-; Spill saved exec
-; GCN: s_mov_b64 s[[[SAVEEXEC_LO:[0-9]+]]:[[SAVEEXEC_HI:[0-9]+]]], exec
-; VGPR: v_writelane_b32 [[SPILL_VGPR:v[0-9]+]], s[[SAVEEXEC_LO]], [[SAVEEXEC_LO_LANE:[0-9]+]]
-; VGPR: v_writelane_b32 [[SPILL_VGPR]], s[[SAVEEXEC_HI]], [[SAVEEXEC_HI_LANE:[0-9]+]]
-
-; VMEM: v_writelane_b32 v[[V_SAVEEXEC:[0-9]+]], s[[SAVEEXEC_LO]], 0
-; VMEM: v_writelane_b32 v[[V_SAVEEXEC]], s[[SAVEEXEC_HI]], 1
-; VMEM: buffer_store_dword v[[V_SAVEEXEC]], off, s[0:3], 0 offset:[[V_EXEC_SPILL_OFFSET:[0-9]+]] ; 4-byte Folded Spill
-
-; GCN: s_and_b64 s[[[ANDEXEC_LO:[0-9]+]]:[[ANDEXEC_HI:[0-9]+]]], s[[[SAVEEXEC_LO]]:[[SAVEEXEC_HI]]], [[CMP0]]
-; GCN: s_mov_b64 exec, s[[[ANDEXEC_LO]]:[[ANDEXEC_HI]]]
-
-; GCN: s_cbranch_execz [[ENDIF:.LBB[0-9]+_[0-9]+]]
-
-; GCN: ; %bb.{{[0-9]+}}: ; %if
-; GCN: buffer_load_dword [[RELOAD_LOAD0:v[0-9]+]], off, s[0:3], 0 offset:[[LOAD0_OFFSET]] ; 4-byte Folded Reload
-; GCN: s_mov_b32 m0, -1
-; GCN: ds_read_b32 [[LOAD1:v[0-9]+]]
-; GCN: s_waitcnt vmcnt(0) lgkmcnt(0)
-
-
-; Spill val register
-; GCN: v_add_i32_e32 [[VAL:v[0-9]+]], vcc, [[LOAD1]], [[RELOAD_LOAD0]]
-; GCN: buffer_store_dword [[VAL]], off, s[0:3], 0 offset:[[VAL_OFFSET:[0-9]+]] ; 4-byte Folded Spill
-
-; VMEM: [[ENDIF]]:
-
-; Restore val
-; GCN: buffer_load_dword [[RELOAD_VAL:v[0-9]+]], off, s[0:3], 0 offset:[[VAL_OFFSET]] ; 4-byte Folded Reload
-
-; Reload and restore exec mask
-; VGPR: v_readlane_b32 s[[S_RELOAD_SAVEEXEC_LO:[0-9]+]], [[SPILL_VGPR]], [[SAVEEXEC_LO_LANE]]
-; VGPR: v_readlane_b32 s[[S_RELOAD_SAVEEXEC_HI:[0-9]+]], [[SPILL_VGPR]], [[SAVEEXEC_HI_LANE]]
-
-; VMEM: buffer_load_dword v[[V_RELOAD_SAVEEXEC:[0-9]+]], off, s[0:3], 0 offset:[[V_EXEC_SPILL_OFFSET]] ; 4-byte Folded Reload
-; VMEM: s_waitcnt vmcnt(0)
-; VMEM: v_readlane_b32 s[[S_RELOAD_SAVEEXEC_LO:[0-9]+]], v[[V_RELOAD_SAVEEXEC]], 0
-; VMEM: v_readlane_b32 s[[S_RELOAD_SAVEEXEC_HI:[0-9]+]], v[[V_RELOAD_SAVEEXEC]], 1
-
-; GCN: s_or_b64 exec, exec, s[[[S_RELOAD_SAVEEXEC_LO]]:[[S_RELOAD_SAVEEXEC_HI]]]
-
-; GCN: flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, [[RELOAD_VAL]]
 define amdgpu_kernel void @divergent_if_endif(ptr addrspace(1) %out) #0 {
+; VMEM-LABEL: divergent_if_endif:
+; VMEM:         .amd_kernel_code_t
+; VMEM-NEXT:     amd_code_version_major = 1
+; VMEM-NEXT:     amd_code_version_minor = 2
+; VMEM-NEXT:     amd_machine_kind = 1
+; VMEM-NEXT:     amd_machine_version_major = 7
+; VMEM-NEXT:     amd_machine_version_minor = 0
+; VMEM-NEXT:     amd_machine_version_stepping = 0
+; VMEM-NEXT:     kernel_code_entry_byte_offset = 256
+; VMEM-NEXT:     kernel_code_prefetch_byte_size = 0
+; VMEM-NEXT:     granulated_workitem_vgpr_count = 1
+; VMEM-NEXT:     granulated_wavefront_sgpr_count = 1
+; VMEM-NEXT:     priority = 0
+; VMEM-NEXT:     float_mode = 240
+; VMEM-NEXT:     priv = 0
+; VMEM-NEXT:     enable_dx10_clamp = 1
+; VMEM-NEXT:     debug_mode = 0
+; VMEM-NEXT:     enable_ieee_mode = 1
+; VMEM-NEXT:     enable_wgp_mode = 0
+; VMEM-NEXT:     enable_mem_ordered = 0
+; VMEM-NEXT:     enable_fwd_progress = 0
+; VMEM-NEXT:     enable_sgpr_private_segment_wave_byte_offset = 1
+; VMEM-NEXT:     user_sgpr_count = 6
+; VMEM-NEXT:     enable_trap_handler = 0
+; VMEM-NEXT:     enable_sgpr_workgroup_id_x = 1
+; VMEM-NEXT:     enable_sgpr_workgroup_id_y = 0
+; VMEM-NEXT:     enable_sgpr_workgroup_id_z = 0
+; VMEM-NEXT:     enable_sgpr_workgroup_info = 0
+; VMEM-NEXT:     enable_vgpr_workitem_id = 0
+; VMEM-NEXT:     enable_exception_msb = 0
+; VMEM-NEXT:     granulated_lds_size = 0
+; VMEM-NEXT:     enable_exception = 0
+; VMEM-NEXT:     enable_sgpr_private_segment_buffer = 1
+; VMEM-NEXT:     enable_sgpr_dispatch_ptr = 0
+; VMEM-NEXT:     enable_sgpr_queue_ptr = 0
+; VMEM-NEXT:     enable_sgpr_kernarg_segment_ptr = 1
+; VMEM-NEXT:     enable_sgpr_dispatch_id = 0
+; VMEM-NEXT:     enable_sgpr_flat_scratch_init = 0
+; VMEM-NEXT:     enable_sgpr_private_segment_size = 0
+; VMEM-NEXT:     enable_sgpr_grid_workgroup_count_x = 0
+; VMEM-NEXT:     enable_sgpr_grid_workgroup_count_y = 0
+; VMEM-NEXT:     enable_sgpr_grid_workgroup_count_z = 0
+; VMEM-NEXT:     enable_wavefront_size32 = 0
+; VMEM-NEXT:     enable_ordered_append_gds = 0
+; VMEM-NEXT:     private_element_size = 1
+; VMEM-NEXT:     is_ptr64 = 1
+; VMEM-NEXT:     is_dynamic_callstack = 0
+; VMEM-NEXT:     is_debug_enabled = 0
+; VMEM-NEXT:     is_xnack_enabled = 0
+; VMEM-NEXT:     workitem_private_segment_byte_size = 28
+; VMEM-NEXT:     workgroup_group_segment_byte_size = 0
+; VMEM-NEXT:     gds_segment_byte_size = 0
+; VMEM-NEXT:     kernarg_segment_byte_size = 8
+; VMEM-NEXT:     workgroup_fbarrier_count = 0
+; VMEM-NEXT:     wavefront_sgpr_count = 14
+; VMEM-NEXT:     workitem_vgpr_count = 5
+; VMEM-NEXT:     reserved_vgpr_first = 0
+; VMEM-NEXT:     reserved_vgpr_count = 0
+; VMEM-NEXT:     reserved_sgpr_first = 0
+; VMEM-NEXT:     reserved_sgpr_count = 0
+; VMEM-NEXT:     debug_wavefront_private_segment_offset_sgpr = 0
+; VMEM-NEXT:     debug_private_segment_buffer_sgpr = 0
+; VMEM-NEXT:     kernarg_segment_alignment = 4
+; VMEM-NEXT:     group_segment_alignment = 4
+; VMEM-NEXT:     private_segment_alignment = 4
+; VMEM-NEXT:     wavefront_size = 6
+; VMEM-NEXT:     call_convention = -1
+; VMEM-NEXT:     runtime_loader_kernel_symbol = 0
+; VMEM-NEXT:    .end_amd_kernel_code_t
+; VMEM-NEXT:  ; %bb.0: ; %entry
+; VMEM-NEXT:    s_add_u32 s0, s0, s7
+; VMEM-NEXT:    s_addc_u32 s1, s1, 0
+; VMEM-NEXT:    s_load_dwordx2 s[4:5], s[4:5], 0x0
+; VMEM-NEXT:    s_mov_b64 s[10:11], exec
+; VMEM-NEXT:    s_mov_b64 exec, 3
+; VMEM-NEXT:    buffer_store_dword v2, off, s[0:3], 0
+; VMEM-NEXT:    s_waitcnt lgkmcnt(0)
+; VMEM-NEXT:    v_writelane_b32 v2, s4, 0
+; VMEM-NEXT:    v_writelane_b32 v2, s5, 1
+; VMEM-NEXT:    buffer_store_dword v2, off, s[0:3], 0 offset:12 ; 4-byte Folded Spill
+; VMEM-NEXT:    buffer_load_dword v2, off, s[0:3], 0
+; VMEM-NEXT:    s_waitcnt vmcnt(0)
+; VMEM-NEXT:    s_mov_b64 exec, s[10:11]
+; VMEM-NEXT:    s_mov_b32 m0, -1
+; VMEM-NEXT:    ; implicit-def: $sgpr4
+; VMEM-NEXT:    v_mov_b32_e32 v1, s4
+; VMEM-NEXT:    ds_read_b32 v1, v1
+; VMEM-NEXT:    s_waitcnt lgkmcnt(0)
+; VMEM-NEXT:    buffer_store_dword v1, off, s[0:3], 0 offset:24 ; 4-byte Folded Spill
+; VMEM-NEXT:    ; implicit-def: $sgpr4
+; VMEM-NEXT:    s_mov_b32 s4, 0
+; VMEM-NEXT:    v_cmp_eq_u32_e64 s[6:7], v0, s4
+; VMEM-NEXT:    v_mov_b32_e32 v0, s4
+; VMEM-NEXT:    buffer_store_dword v0, off, s[0:3], 0 offset:20 ; 4-byte Folded Spill
+; VMEM-NEXT:    s_mov_b64 s[4:5], exec
+; VMEM-NEXT:    s_mov_b64 s[8:9], exec
+; VMEM-NEXT:    s_mov_b64 exec, 3
+; VMEM-NEXT:    buffer_store_dword v0, off, s[0:3], 0
+; VMEM-NEXT:    v_writelane_b32 v0, s4, 0
+; VMEM-NEXT:    v_writelane_b32 v0, s5, 1
+; VMEM-NEXT:    buffer_store_dword v0, off, s[0:3], 0 offset:4 ; 4-byte Folded Spill
+; VMEM-NEXT:    buffer_load_dword v0, off, s[0:3], 0
+; VMEM-NEXT:    s_waitcnt vmcnt(0)
+; VMEM-NEXT:    s_mov_b64 exec, s[8:9]
+; VMEM-NEXT:    s_and_b64 s[4:5], s[4:5], s[6:7]
+; VMEM-NEXT:    s_mov_b64 exec, s[4:5]
+; VMEM-NEXT:    s_cbranch_execz .LBB0_2
+; VMEM-NEXT:  ; %bb.1: ; %if
+; VMEM-NEXT:    buffer_load_dword v0, off, s[0:3], 0 offset:24 ; 4-byte Folded Reload
+; VMEM-NEXT:    s_mov_b32 m0, -1
+; VMEM-NEXT:    ; implicit-def: $sgpr4
+; VMEM-NEXT:    v_mov_b32_e32 v1, s4
+; VMEM-NEXT:    ds_read_b32 v1, v1
+; VMEM-NEXT:    ; implicit-def: $sgpr4
+; VMEM-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
+; VMEM-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
+; VMEM-NEXT:    buffer_store_dword v0, off, s[0:3], 0 offset:20 ; 4-byte Folded Spill
+; VMEM-NEXT:  .LBB0_2: ; %endif
+; VMEM-NEXT:    buffer_load_dword v2, off, s[0:3], 0 offset:20 ; 4-byte Folded Reload
+; VMEM-NEXT:    s_mov_b64 s[8:9], exec
+; VMEM-NEXT:    s_mov_b64 exec, 3
+; VMEM-NEXT:    buffer_store_dword v4, off, s[0:3], 0
+; VMEM-NEXT:    buffer_load_dword v4, off, s[0:3], 0 offset:4 ; 4-byte Folded Reload
+; VMEM-NEXT:    s_waitcnt vmcnt(0)
+; VMEM-NEXT:    v_readlane_b32 s6, v4, 0
+; VMEM-NEXT:    v_readlane_b32 s7, v4, 1
+; VMEM-NEXT:    buffer_load_dword v4, off, s[0:3], 0
+; VMEM-NEXT:    s_waitcnt vmcnt(0)
+; VMEM-NEXT:    s_mov_b64 exec, s[8:9]
+; VMEM-NEXT:    s_or_b64 exec, exec, s[6:7]
+; VMEM-NEXT:    s_mov_b64 s[6:7], exec
+; VMEM-NEXT:    s_mov_b64 exec, 3
+; VMEM-NEXT:    buffer_store_dword v3, off, s[0:3], 0
+; VMEM-NEXT:    buffer_load_dword v3, off, s[0:3], 0 offset:12 ; 4-byte Folded Reload
+; VMEM-NEXT:    s_waitcnt vmcnt(0)
+; VMEM-NEXT:    v_readlane_b32 s4, v3, 0
+; VMEM-NEXT:    v_readlane_b32 s5, v3, 1
+; VMEM-NEXT:    buffer_load_dword v3, off, s[0:3], 0
+; VMEM-NEXT:    s_waitcnt vmcnt(0)
+; VMEM-NEXT:    s_mov_b64 exec, s[6:7]
+; VMEM-NEXT:    v_mov_b32_e32 v0, s4
+; VMEM-NEXT:    v_mov_b32_e32 v1, s5
+; VMEM-NEXT:    flat_store_dword v[0:1], v2
+; VMEM-NEXT:    s_endpgm
+;
+; VGPR-LABEL: divergent_if_endif:
+; VGPR:         .amd_kernel_code_t
+; VGPR-NEXT:     amd_code_version_major = 1
+; VGPR-NEXT:     amd_code_version_minor = 2
+; VGPR-NEXT:     amd_machine_kind = 1
+; VGPR-NEXT:     amd_machine_version_major = 7
+; VGPR-NEXT:     amd_machine_version_minor = 0
+; VGPR-NEXT:     amd_machine_version_stepping = 0
+; VGPR-NEXT:     kernel_code_entry_byte_offset = 256
+; VGPR-NEXT:     kernel_code_prefetch_byte_size = 0
+; VGPR-NEXT:     granulated_workitem_vgpr_count = 0
+; VGPR-NEXT:     granulated_wavefront_sgpr_count = 1
+; VGPR-NEXT:     priority = 0
+; VGPR-NEXT:     float_mode = 240
+; VGPR-NEXT:     priv = 0
+; VGPR-NEXT:     enable_dx10_clamp = 1
+; VGPR-NEXT:     debug_mode = 0
+; VGPR-NEXT:     enable_ieee_mode = 1
+; VGPR-NEXT:     enable_wgp_mode = 0
+; VGPR-NEXT:     enable_mem_ordered = 0
+; VGPR-NEXT:     enable_fwd_progress = 0
+; VGPR-NEXT:     enable_sgpr_private_segment_wave_byte_offset = 1
+; VGPR-NEXT:     user_sgpr_count = 6
+; VGPR-NEXT:     enable_trap_handler = 0
+; VGPR-NEXT:     enable_sgpr_workgroup_id_x = 1
+; VGPR-NEXT:     enable_sgpr_workgroup_id_y = 0
+; VGPR-NEXT:     enable_sgpr_workgroup_id_z = 0
+; VGPR-NEXT:     enable_sgpr_workgroup_info = 0
+; VGPR-NEXT:     enable_vgpr_workitem_id = 0
+; VGPR-NEXT:     enable_exception_msb = 0
+; VGPR-NEXT:     granulated_lds_size = 0
+; VGPR-NEXT:     enable_exception = 0
+; VGPR-NEXT:     enable_sgpr_private_segment_buffer = 1
+; VGPR-NEXT:     enable_sgpr_dispatch_ptr = 0
+; VGPR-NEXT:     enable_sgpr_queue_ptr = 0
+; VGPR-NEXT:     enable_sgpr_kernarg_segment_ptr = 1
+; VGPR-NEXT:     enable_sgpr_dispatch_id = 0
+; VGPR-NEXT:     enable_sgpr_flat_scratch_init = 0
+; VGPR-NEXT:     enable_sgpr_private_segment_size = 0
+; VGPR-NEXT:     enable_sgpr_grid_workgroup_count_x = 0
+; VGPR-NEXT:     enable_sgpr_grid_workgroup_count_y = 0
+; VGPR-NEXT:     enable_sgpr_grid_workgroup_count_z = 0
+; VGPR-NEXT:     enable_wavefront_size32 = 0
+; VGPR-NEXT:     enable_ordered_append_gds = 0
+; VGPR-NEXT:     private_element_size = 1
+; VGPR-NEXT:     is_ptr64 = 1
+; VGPR-NEXT:     is_dynamic_callstack = 0
+; VGPR-NEXT:     is_debug_enabled = 0
+; VGPR-NEXT:     is_xnack_enabled = 0
+; VGPR-NEXT:     workitem_private_segment_byte_size = 12
+; VGPR-NEXT:     workgroup_group_segment_byte_size = 0
+; VGPR-NEXT:     gds_segment_byte_size = 0
+; VGPR-NEXT:     kernarg_segment_byte_size = 8
+; VGPR-NEXT:     workgroup_fbarrier_count = 0
+; VGPR-NEXT:     wavefront_sgpr_count = 10
+; VGPR-NEXT:     workitem_vgpr_count = 4
+; VGPR-NEXT:     reserved_vgpr_first = 0
+; VGPR-NEXT:     reserved_vgpr_count = 0
+; VGPR-NEXT:     reserved_sgpr_first = 0
+; VGPR-NEXT:     reserved_sgpr_count = 0
+; VGPR-NEXT:     debug_wavefront_private_segment_offset_sgpr = 0
+; VGPR-NEXT:     debug_private_segment_buffer_sgpr = 0
+; VGPR-NEXT:     kernarg_segment_alignment = 4
+; VGPR-NEXT:     group_segment_alignment = 4
+; VGPR-NEXT:     private_segment_alignment = 4
+; VGPR-NEXT:     wavefront_size = 6
+; VGPR-NEXT:     call_convention = -1
+; VGPR-NEXT:     runtime_loader_kernel_symbol = 0
+; VGPR-NEXT:    .end_amd_kernel_code_t
+; VGPR-NEXT:  ; %bb.0: ; %entry
+; VGPR-NEXT:    s_add_u32 s0, s0, s7
+; VGPR-NEXT:    s_addc_u32 s1, s1, 0
+; VGPR-NEXT:    s_load_dwordx2 s[4:5], s[4:5], 0x0
+; VGPR-NEXT:    s_waitcnt lgkmcnt(0)
+; VGPR-NEXT:    v_writelane_b32 v1, s4, 0
+; VGPR-NEXT:    v_writelane_b32 v1, s5, 1
+; VGPR-NEXT:    s_mov_b32 m0, -1
+; VGPR-NEXT:    ; implicit-def: $sgpr4
+; VGPR-NEXT:    v_mov_b32_e32 v2, s4
+; VGPR-NEXT:    ds_read_b32 v2, v2
+; VGPR-NEXT:    s_waitcnt lgkmcnt(0)
+; VGPR-NEXT:    buffer_store_dword v2, off, s[0:3], 0 offset:8 ; 4-byte Folded Spill
+; VGPR-NEXT:    ; implicit-def: $sgpr4
+; VGPR-NEXT:    s_mov_b32 s4, 0
+; VGPR-NEXT:    v_cmp_eq_u32_e64 s[6:7], v0, s4
+; VGPR-NEXT:    v_mov_b32_e32 v0, s4
+; VGPR-NEXT:    buffer_store_dword v0, off, s[0:3], 0 offset:4 ; 4-byte Folded Spill
+; VGPR-NEXT:    s_mov_b64 s[4:5], exec
+; VGPR-NEXT:    v_writelane_b32 v1, s4, 2
+; VGPR-NEXT:    v_writelane_b32 v1, s5, 3
+; VGPR-NEXT:    s_and_b64 s[4:5], s[4:5], s[6:7]
+; VGPR-NEXT:    s_mov_b64 exec, s[4:5]
+; VGPR-NEXT:    s_cbranch_execz .LBB0_2
+; VGPR-NEXT:  ; %bb.1: ; %if
+; VGPR-NEXT:    buffer_load_dword v0, off, s[0:3], 0 offset:8 ; 4-byte Folded Reload
+; VGPR-NEXT:    s_mov_b32 m0, -1
+; VGPR-NEXT:    ; implicit-def: $sgpr4
+; VGPR-NEXT:    v_mov_b32_e32 v2, s4
+; VGPR-NEXT:    ds_read_b32 v2, v2
+; VGPR-NEXT:    ; implicit-def: $sgpr4
+; VGPR-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
+; VGPR-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
+; VGPR-NEXT:    buffer_store_dword v0, off, s[0:3], 0 offset:4 ; 4-byte Folded Spill
+; VGPR-NEXT:  .LBB0_2: ; %endif
+; VGPR-NEXT:    buffer_load_dword v0, off, s[0:3], 0 offset:4 ; 4-byte Folded Reload
+; VGPR-NEXT:    v_readlane_b32 s6, v1, 2
+; VGPR-NEXT:    v_readlane_b32 s7, v1, 3
+; VGPR-NEXT:    s_or_b64 exec, exec, s[6:7]
+; VGPR-NEXT:    v_readlane_b32 s4, v1, 0
+; VGPR-NEXT:    v_readlane_b32 s5, v1, 1
+; VGPR-NEXT:    v_mov_b32_e32 v2, s4
+; VGPR-NEXT:    v_mov_b32_e32 v3, s5
+; VGPR-NEXT:    s_waitcnt vmcnt(0)
+; VGPR-NEXT:    flat_store_dword v[2:3], v0
+; VGPR-NEXT:    s_endpgm
 entry:
   %tid = call i32 @llvm.amdgcn.workitem.id.x()
   %load0 = load volatile i32, ptr addrspace(3) undef
   %cmp0 = icmp eq i32 %tid, 0
   br i1 %cmp0, label %if, label %endif
 
 if:
   %load1 = load volatile i32, ptr addrspace(3) undef
   %val = add i32 %load0, %load1
   br label %endif
 
 endif:
   %tmp4 = phi i32 [ %val, %if ], [ 0, %entry ]
   store i32 %tmp4, ptr addrspace(1) %out
   ret void
 }
 
-; GCN-LABEL: {{^}}divergent_loop:
-; VGPR: workitem_private_segment_byte_size = 16{{$}}
-
-; GCN: {{^}}; %bb.0:
-; GCN-DAG: s_mov_b32 m0, -1
-; GCN-DAG: v_mov_b32_e32 [[PTR0:v[0-9]+]], 0{{$}}
-; GCN: ds_read_b32 [[LOAD0:v[0-9]+]], [[PTR0]]
-; GCN: v_cmp_eq_u32_e64 [[CMP0:s\[[0-9]+:[0-9]+\]]], v0, s{{[0-9]+}}
-
-; Spill load
-; GCN: buffer_store_dword [[LOAD0]], off, s[0:3], 0 offset:[[LOAD0_OFFSET:[0-9]+]] ; 4-byte Folded Spill
-; GCN: s_mov_b64 s[[[SAVEEXEC_LO:[0-9]+]]:[[SAVEEXEC_HI:[0-9]+]]], exec
-
-; Spill saved exec
-; VGPR: v_writelane_b32 [[SPILL_VGPR:v[0-9]+]], s[[SAVEEXEC_LO]], [[SAVEEXEC_LO_LANE:[0-9]+]]
-; VGPR: v_writelane_b32 [[SPILL_VGPR]], s[[SAVEEXEC_HI]], [[SAVEEXEC_HI_LANE:[0-9]+]]
-
-; VMEM: v_writelane_b32 v[[V_SAVEEXEC:[0-9]+]], s[[SAVEEXEC_LO]], 0
-; VMEM: v_writelane_b32 v[[V_SAVEEXEC]], s[[SAVEEXEC_HI]], 1
-; VMEM: buffer_store_dword v[[V_SAVEEXEC]], off, s[0:3], 0 offset:[[V_EXEC_SPILL_OFFSET:[0-9]+]] ; 4-byte Folded Spill
-
-
-; GCN: s_and_b64 s[[[ANDEXEC_LO:[0-9]+]]:[[ANDEXEC_HI:[0-9]+]]], s[[[SAVEEXEC_LO:[0-9]+]]:[[SAVEEXEC_HI:[0-9]+]]], [[CMP0]]
-; GCN: s_mov_b64 exec, s[[[ANDEXEC_LO]]:[[ANDEXEC_HI]]]
-; GCN-NEXT: s_cbranch_execz [[END:.LBB[0-9]+_[0-9]+]]
-
-
-; GCN: [[LOOP:.LBB[0-9]+_[0-9]+]]:
-; GCN: buffer_load_dword v[[VAL_LOOP_RELOAD:[0-9]+]], off, s[0:3], 0 offset:[[LOAD0_OFFSET]] ; 4-byte Folded Reload
-; GCN: v_subrev_i32_e32 [[VAL_LOOP:v[0-9]+]], vcc, v{{[0-9]+}}, v[[VAL_LOOP_RELOAD]]
-; GCN: s_cmp_lg_u32
-; VMEM: buffer_store_dword
-; VMEM: buffer_store_dword
-; VMEM: buffer_store_dword
-; GCN: buffer_store_dword [[VAL_LOOP]], off, s[0:3], 0 offset:{{[0-9]+}} ; 4-byte Folded Spill
-; GCN-NEXT: s_cbranch_scc1 [[LOOP]]
-
-; GCN: buffer_store_dword [[VAL_LOOP]], off, s[0:3], 0 offset:[[VAL_SUB_OFFSET:[0-9]+]] ; 4-byte Folded Spill
-
-; GCN: [[END]]:
-; GCN: buffer_load_dword v[[VAL_END:[0-9]+]], off, s[0:3], 0 offset:[[VAL_SUB_OFFSET]] ; 4-byte Folded Reload
-; VGPR: v_readlane_b32 s[[S_RELOAD_SAVEEXEC_LO:[0-9]+]], [[SPILL_VGPR]], [[SAVEEXEC_LO_LANE]]
-; VGPR: v_readlane_b32 s[[S_RELOAD_SAVEEXEC_HI:[0-9]+]], [[SPILL_VGPR]], [[SAVEEXEC_HI_LANE]]
-
-; VMEM: buffer_load_dword v[[V_RELOAD_SAVEEXEC:[0-9]+]], off, s[0:3], 0 offset:[[V_EXEC_SPILL_OFFSET]] ; 4-byte Folded Reload
-; VMEM: s_waitcnt vmcnt(0)
-; VMEM: v_readlane_b32 s[[S_RELOAD_SAVEEXEC_LO:[0-9]+]], v[[V_RELOAD_SAVEEXEC]], 0
-; VMEM: v_readlane_b32 s[[S_RELOAD_SAVEEXEC_HI:[0-9]+]], v[[V_RELOAD_SAVEEXEC]], 1
-
-; GCN: s_or_b64 exec, exec, s[[[S_RELOAD_SAVEEXEC_LO]]:[[S_RELOAD_SAVEEXEC_HI]]]
-
-; GCN: flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, v[[VAL_END]]
 define amdgpu_kernel void @divergent_loop(ptr addrspace(1) %out) #0 {
+; VMEM-LABEL: divergent_loop:
+; VMEM:         .amd_kernel_code_t
+; VMEM-NEXT:     amd_code_version_major = 1
+; VMEM-NEXT:     amd_code_version_minor = 2
+; VMEM-NEXT:     amd_machine_kind = 1
+; VMEM-NEXT:     amd_machine_version_major = 7
+; VMEM-NEXT:     amd_machine_version_minor = 0
+; VMEM-NEXT:     amd_machine_version_stepping = 0
+; VMEM-NEXT:     kernel_code_entry_byte_offset = 256
+; VMEM-NEXT:     kernel_code_prefetch_byte_size = 0
+; VMEM-NEXT:     granulated_workitem_vgpr_count = 1
+; VMEM-NEXT:     granulated_wavefront_sgpr_count = 1
+; VMEM-NEXT:     priority = 0
+; VMEM-NEXT:     float_mode = 240
+; VMEM-NEXT:     priv = 0
+; VMEM-NEXT:     enable_dx10_clamp = 1
+; VMEM-NEXT:     debug_mode = 0
+; VMEM-NEXT:     enable_ieee_mode = 1
+; VMEM-NEXT:     enable_wgp_mode = 0
+; VMEM-NEXT:     enable_mem_ordered = 0
+; VMEM-NEXT:     enable_fwd_progress = 0
+; VMEM-NEXT:     enable_sgpr_private_segment_wave_byte_offset = 1
+; VMEM-NEXT:     user_sgpr_count = 6
+; VMEM-NEXT:     enable_trap_handler = 0
+; VMEM-NEXT:     enable_sgpr_workgroup_id_x = 1
+; VMEM-NEXT:     enable_sgpr_workgroup_id_y = 0
+; VMEM-NEXT:     enable_sgpr_workgroup_id_z = 0
+; VMEM-NEXT:     enable_sgpr_workgroup_info = 0
+; VMEM-NEXT:     enable_vgpr_workitem_id = 0
+; VMEM-NEXT:     enable_exception_msb = 0
+; VMEM-NEXT:     granulated_lds_size = 0
+; VMEM-NEXT:     enable_exception = 0
+; VMEM-NEXT:     enable_sgpr_private_segment_buffer = 1
+; VMEM-NEXT:     enable_sgpr_dispatch_ptr = 0
+; VMEM-NEXT:     enable_sgpr_queue_ptr = 0
+; VMEM-NEXT:     enable_sgpr_kernarg_segment_ptr = 1
+; VMEM-NEXT:     enable_sgpr_dispatch_id = 0
+; VMEM-NEXT:     enable_sgpr_flat_scratch_init = 0
+; VMEM-NEXT:     enable_sgpr_private_segment_size = 0
+; VMEM-NEXT:     enable_sgpr_grid_workgroup_count_x = 0
+; VMEM-NEXT:     enable_sgpr_grid_workgroup_count_y = 0
+; VMEM-NEXT:     enable_sgpr_grid_workgroup_count_z = 0
+; VMEM-NEXT:     enable_wavefront_size32 = 0
+; VMEM-NEXT:     enable_ordered_append_gds = 0
+; VMEM-NEXT:     private_element_size = 1
+; VMEM-NEXT:     is_ptr64 = 1
+; VMEM-NEXT:     is_dynamic_callstack = 0
+; VMEM-NEXT:     is_debug_enabled = 0
+; VMEM-NEXT:     is_xnack_enabled = 0
+; VMEM-NEXT:     workitem_private_segment_byte_size = 36
+; VMEM-NEXT:     workgroup_group_segment_byte_size = 0
+; VMEM-NEXT:     gds_segment_byte_size = 0
+; VMEM-NEXT:     kernarg_segment_byte_size = 8
+; VMEM-NEXT:     workgroup_fbarrier_count = 0
+; VMEM-NEXT:     wavefront_sgpr_count = 16
+; VMEM-NEXT:     workitem_vgpr_count = 5
+; VMEM-NEXT:     reserved_vgpr_first = 0
+; VMEM-NEXT:     reserved_vgpr_count = 0
+; VMEM-NEXT:     reserved_sgpr_first = 0
+; VMEM-NEXT:     reserved_sgpr_count = 0
+; VMEM-NEXT:     debug_wavefront_private_segment_offset_sgpr = 0
+; VMEM-NEXT:     debug_private_segment_buffer_sgpr = 0
+; VMEM-NEXT:     kernarg_segment_alignment = 4
+; VMEM-NEXT:     group_segment_alignment = 4
+; VMEM-NEXT:     private_segment_alignment = 4
+; VMEM-NEXT:     wavefront_size = 6
+; VMEM-NEXT:     call_convention = -1
+; VMEM-NEXT:     runtime_loader_kernel_symbol = 0
+; VMEM-NEXT:    .end_amd_kernel_code_t
+; VMEM-NEXT:  ; %bb.0: ; %entry
+; VMEM-NEXT:    s_add_u32 s0, s0, s7
+; VMEM-NEXT:    s_addc_u32 s1, s1, 0
+; VMEM-NEXT:    s_load_dwordx2 s[4:5], s[4:5], 0x0
+; VMEM-NEXT:    s_mov_b64 s[12:13], exec
+; VMEM-NEXT:    s_mov_b64 exec, 3
+; VMEM-NEXT:    buffer_store_dword v3, off, s[0:3], 0
+; VMEM-NEXT:    s_waitcnt lgkmcnt(0)
+; VMEM-NEXT:    v_writelane_b32 v3, s4, 0
+; VMEM-NEXT:    v_writelane_b32 v3, s5, 1
+; VMEM-NEXT:    buffer_store_dword v3, off, s[0:3], 0 offset:16 ; 4-byte Folded Spill
+; VMEM-NEXT:    buffer_load_dword v3, off, s[0:3], 0
+; VMEM-NEXT:    s_waitcnt vmcnt(0)
+; VMEM-NEXT:    s_mov_b64 exec, s[12:13]
+; VMEM-NEXT:    v_mov_b32_e32 v1, 0
+; VMEM-NEXT:    s_mov_b32 m0, -1
+; VMEM-NEXT:    ds_read_b32 v1, v1
+; VMEM-NEXT:    ; implicit-def: $sgpr4
+; VMEM-NEXT:    s_mov_b32 s4, 0
+; VMEM-NEXT:    v_cmp_eq_u32_e64 s[6:7], v0, s4
+; VMEM-NEXT:    v_mov_b32_e32 v0, s4
+; VMEM-NEXT:    s_mov_b64 s[10:11], exec
+; VMEM-NEXT:    s_mov_b64 exec, 1
+; VMEM-NEXT:    buffer_store_dword v2, off, s[0:3], 0
+; VMEM-NEXT:    v_writelane_b32 v2, s4, 0
+; VMEM-NEXT:    buffer_store_dword v2, off, s[0:3], 0 offset:12 ; 4-byte Folded Spill
+; VMEM-NEXT:    buffer_load_dword v2, off, s[0:3], 0
+; VMEM-NEXT:    s_waitcnt vmcnt(0)
+; VMEM-NEXT:    s_mov_b64 exec, s[10:11]
+; VMEM-NEXT:    s_waitcnt lgkmcnt(0)
+; VMEM-NEXT:    buffer_store_dword v1, off, s[0:3], 0 offset:28 ; 4-byte Folded Spill
+; VMEM-NEXT:    buffer_store_dword v0, off, s[0:3], 0 offset:24 ; 4-byte Folded Spill
+; VMEM-NEXT:    s_mov_b64 s[4:5], exec
+; VMEM-NEXT:    s_mov_b64 s[8:9], exec
+; VMEM-NEXT:    s_mov_b64 exec, 3
+; VMEM-NEXT:    buffer_store_dword v0, off, s[0:3], 0
+; VMEM-NEXT:    v_writelane_b32 v0, s4, 0
+; VMEM-NEXT:    v_writelane_b32 v0, s5, 1
+; VMEM-NEXT:    buffer_store_dword v0, off, s[0:3], 0 offset:4 ; 4-byte Folded Spill
+; VMEM-NEXT:    buffer_load_dword v0, off, s[0:3], 0
+; VMEM-NEXT:    s_waitcnt vmcnt(0)
+; VMEM-NEXT:    s_mov_b64 exec, s[8:9]
+; VMEM-NEXT:    s_and_b64 s[4:5], s[4:5], s[6:7]
+; VMEM-NEXT:    s_mov_b64 exec, s[4:5]
+; VMEM-NEXT:    s_cbranch_execz .LBB1_3
+; VMEM-NEXT:  .LBB1_1: ; %loop
+; VMEM-NEXT:    ; =>This Inner Loop Header: Depth=1
+; VMEM-NEXT:    buffer_load_dword v0, off, s[0:3], 0 offset:28 ; 4-byte Folded Reload
+; VMEM-NEXT:    s_mov_b64 s[8:9], exec
+; VMEM-NEXT:    s_mov_b64 exec, 1
+; VMEM-NEXT:    buffer_store_dword v3, off, s[0:3], 0
+; VMEM-NEXT:    buffer_load_dword v3, off, s[0:3], 0 offset:12 ; 4-byte Folded Reload
+; VMEM-NEXT:    s_waitcnt vmcnt(0)
+; VMEM-NEXT:    v_readlane_b32 s5, v3, 0
+; VMEM-NEXT:    buffer_load_dword v3, off, s[0:3], 0
+; VMEM-NEXT:    s_waitcnt vmcnt(0)
+; VMEM-NEXT:    s_mov_b64 exec, s[8:9]
+; VMEM-NEXT:    s_mov_b32 m0, -1
+; VMEM-NEXT:    ; implicit-def: $sgpr4
+; VMEM-NEXT:    v_mov_b32_e32 v1, s4
+; VMEM-NEXT:    ds_read_b32 v1, v1
+; VMEM-NEXT:    s_mov_b32 s4, 1
+; VMEM-NEXT:    s_add_i32 s4, s5, s4
+; VMEM-NEXT:    ; implicit-def: $sgpr6
+; VMEM-NEXT:    s_waitcnt lgkmcnt(0)
+; VMEM-NEXT:    v_sub_i32_e32 v0, vcc, v0, v1
+; VMEM-NEXT:    s_mov_b32 s6, 0x100
+; VMEM-NEXT:    s_cmp_lg_u32 s5, s6
+; VMEM-NEXT:    v_mov_b32_e32 v1, v0
+; VMEM-NEXT:    s_mov_b64 s[6:7], exec
+; VMEM-NEXT:    s_mov_b64 exec, 1
+; VMEM-NEXT:    buffer_store_dword v2, off, s[0:3], 0
+; VMEM-NEXT:    v_writelane_b32 v2, s4, 0
+; VMEM-NEXT:    buffer_store_dword v2, off, s[0:3], 0 offset:12 ; 4-byte Folded Spill
+; VMEM-NEXT:    buffer_load_dword v2, off, s[0:3], 0
+; VMEM-NEXT:    s_waitcnt vmcnt(0)
+; VMEM-NEXT:    s_mov_b64 exec, s[6:7]
+; VMEM-NEXT:    buffer_store_dword v1, off, s[0:3], 0 offset:28 ; 4-byte Folded Spill
+; VMEM-NEXT:    buffer_store_dword v0, off, s[0:3], 0 offset:32 ; 4-byte Folded Spill
+; VMEM-NEXT:    s_cbranch_scc1 .LBB1_1
+; VMEM-NEXT:  ; %bb.2: ; %Flow
+; VMEM-NEXT:    buffer_load_dword v0, off, s[0:3], 0 offset:32 ; 4-byte Folded Reload
+; VMEM-NEXT:    s_waitcnt vmcnt(0)
+; VMEM-NEXT:    buffer_store_dword v0, off, s[0:3], 0 offset:24 ; 4-byte Folded Spill
+; VMEM-NEXT:  .LBB1_3: ; %end
+; VMEM-NEXT:    buffer_load_dword v2, off, s[0:3], 0 offset:24 ; 4-byte Folded Reload
+; VMEM-NEXT:    s_mov_b64 s[8:9], exec
+; VMEM-NEXT:    s_mov_b64 exec, 3
+; VMEM-NEXT:    buffer_store_dword v4, off, s[0:3], 0
+; VMEM-NEXT:    buffer_load_dword v4, off, s[0:3], 0 offset:4 ; 4-byte Folded Reload
+; VMEM-NEXT:    s_waitcnt vmcnt(0)
+; VMEM-NEXT:    v_readlane_b32 s6, v4, 0
+; VMEM-NEXT:    v_readlane_b32 s7, v4, 1
+; VMEM-NEXT:    buffer_load_dword v4, off, s[0:3], 0
+; VMEM-NEXT:    s_waitcnt vmcnt(0)
+; VMEM-NEXT:    s_mov_b64 exec, s[8:9]
+; VMEM-NEXT:    s_or_b64 exec, exec, s[6:7]
+; VMEM-NEXT:    s_mov_b64 s[6:7], exec
+; VMEM-NEXT:    s_mov_b64 exec, 3
+; VMEM-NEXT:    buffer_store_dword v3, off, s[0:3], 0
+; VMEM-NEXT:    buffer_load_dword v3, off, s[0:3], 0 offset:16 ; 4-byte Folded Reload
+; VMEM-NEXT:    s_waitcnt vmcnt(0)
+; VMEM-NEXT:    v_readlane_b32 s4, v3, 0
+; VMEM-NEXT:    v_readlane_b32 s5, v3, 1
+; VMEM-NEXT:    buffer_load_dword v3, off, s[0:3], 0
+; VMEM-NEXT:    s_waitcnt vmcnt(0)
+; VMEM-NEXT:    s_mov_b64 exec, s[6:7]
+; VMEM-NEXT:    v_mov_b32_e32 v0, s4
+; VMEM-NEXT:    v_mov_b32_e32 v1, s5
+; VMEM-NEXT:    flat_store_dword v[0:1], v2
+; VMEM-NEXT:    s_endpgm
+;
+; VGPR-LABEL: divergent_loop:
+; VGPR:         .amd_kernel_code_t
+; VGPR-NEXT:     amd_code_version_major = 1
+; VGPR-NEXT:     amd_code_version_minor = 2
+; VGPR-NEXT:     amd_machine_kind = 1
+; VGPR-NEXT:     amd_machine_version_major = 7
+; VGPR-NEXT:     amd_machine_version_minor = 0
+; VGPR-NEXT:     amd_machine_version_stepping = 0
+; VGPR-NEXT:     kernel_code_entry_byte_offset = 256
+; VGPR-NEXT:     kernel_code_prefetch_byte_size = 0
+; VGPR-NEXT:     granulated_workitem_vgpr_count = 0
+; VGPR-NEXT:     granulated_wavefront_sgpr_count = 1
+; VGPR-NEXT:     priority = 0
+; VGPR-NEXT:     float_mode = 240
+; VGPR-NEXT:     priv = 0
+; VGPR-NEXT:     enable_dx10_clamp = 1
+; VGPR-NEXT:     debug_mode = 0
+; VGPR-NEXT:     enable_ieee_mode = 1
+; VGPR-NEXT:     enable_wgp_mode = 0
+; VGPR-NEXT:     enable_mem_ordered = 0
+; VGPR-NEXT:     enable_fwd_progress = 0
+; VGPR-NEXT:     enable_sgpr_private_segment_wave_byte_offset = 1
+; VGPR-NEXT:     user_sgpr_count = 6
+; VGPR-NEXT:     enable_trap_handler = 0
+; VGPR-NEXT:     enable_sgpr_workgroup_id_x = 1
+; VGPR-NEXT:     enable_sgpr_workgroup_id_y = 0
+; VGPR-NEXT:     enable_sgpr_workgroup_id_z = 0
+; VGPR-NEXT:     enable_sgpr_workgroup_info = 0
+; VGPR-NEXT:     enable_vgpr_workitem_id = 0
+; VGPR-NEXT:     enable_exception_msb = 0
+; VGPR-NEXT:     granulated_lds_size = 0
+; VGPR-NEXT:     enable_exception = 0
+; VGPR-NEXT:     enable_sgpr_private_segment_buffer = 1
+; VGPR-NEXT:     enable_sgpr_dispatch_ptr = 0
+; VGPR-NEXT:     enable_sgpr_queue_ptr = 0
+; VGPR-NEXT:     enable_sgpr_kernarg_segment_ptr = 1
+; VGPR-NEXT:     enable_sgpr_dispatch_id = 0
+; VGPR-NEXT:     enable_sgpr_flat_scratch_init = 0
+; VGPR-NEXT:     enable_sgpr_private_segment_size = 0
+; VGPR-NEXT:     enable_sgpr_grid_workgroup_count_x = 0
+; VGPR-NEXT:     enable_sgpr_grid_workgroup_count_y = 0
+; VGPR-NEXT:     enable_sgpr_grid_workgroup_count_z = 0
+; VGPR-NEXT:     enable_wavefront_size32 = 0
+; VGPR-NEXT:     enable_ordered_append_gds = 0
+; VGPR-NEXT:     private_element_size = 1
+; VGPR-NEXT:     is_ptr64 = 1
+; VGPR-NEXT:     is_dynamic_callstack = 0
+; VGPR-NEXT:     is_debug_enabled = 0
+; VGPR-NEXT:     is_xnack_enabled = 0
+; VGPR-NEXT:     workitem_private_segment_byte_size = 16
+; VGPR-NEXT:     workgroup_group_segment_byte_size = 0
+; VGPR-NEXT:     gds_segment_byte_size = 0
+; VGPR-NEXT:     kernarg_segment_byte_size = 8
+; VGPR-NEXT:     workgroup_fbarrier_count = 0
+; VGPR-NEXT:     wavefront_sgpr_count = 10
+; VGPR-NEXT:     workitem_vgpr_count = 4
+; VGPR-NEXT:     reserved_vgpr_first = 0
+; VGPR-NEXT:     reserved_vgpr_count = 0
+; VGPR-NEXT:     reserved_sgpr_first = 0
+; VGPR-NEXT:     reserved_sgpr_count = 0
+; VGPR-NEXT:     debug_wavefront_private_segment_offset_sgpr = 0
+; VGPR-NEXT:     debug_private_segment_buffer_sgpr = 0
+; VGPR-NEXT:     kernarg_segment_alignment = 4
+; VGPR-NEXT:     group_segment_alignment = 4
+; VGPR-NEXT:     private_segment_alignment = 4
+; VGPR-NEXT:     wavefront_size = 6
+; VGPR-NEXT:     call_convention = -1
+; VGPR-NEXT:     runtime_loader_kernel_symbol = 0
+; VGPR-NEXT:    .end_amd_kernel_code_t
+; VGPR-NEXT:  ; %bb.0: ; %entry
+; VGPR-NEXT:    s_add_u32 s0, s0, s7
+; VGPR-NEXT:    s_addc_u32 s1, s1, 0
+; VGPR-NEXT:    s_load_dwordx2 s[4:5], s[4:5], 0x0
+; VGPR-NEXT:    s_waitcnt lgkmcnt(0)
+; VGPR-NEXT:    v_writelane_b32 v1, s4, 0
+; VGPR-NEXT:    v_writelane_b32 v1, s5, 1
+; VGPR-NEXT:    v_mov_b32_e32 v2, 0
+; VGPR-NEXT:    s_mov_b32 m0, -1
+; VGPR-NEXT:    ds_read_b32 v2, v2
+; VGPR-NEXT:    ; implicit-def: $sgpr4
+; VGPR-NEXT:    s_mov_b32 s4, 0
+; VGPR-NEXT:    v_cmp_eq_u32_e64 s[6:7], v0, s4
+; VGPR-NEXT:    v_mov_b32_e32 v0, s4
+; VGPR-NEXT:    v_writelane_b32 v1, s4, 2
+; VGPR-NEXT:    s_waitcnt lgkmcnt(0)
+; VGPR-NEXT:    buffer_store_dword v2, off, s[0:3], 0 offset:8 ; 4-byte Folded Spill
+; VGPR-NEXT:    buffer_store_dword v0, off, s[0:3], 0 offset:4 ; 4-byte Folded Spill
+; VGPR-NEXT:    s_mov_b64 s[4:5], exec
+; VGPR-NEXT:    v_writelane_b32 v1, s4, 3
+; VGPR-NEXT:    v_writelane_b32 v1, s5, 4
+; VGPR-NEXT:    s_and_b64 s[4:5], s[4:5], s[6:7]
+; VGPR-NEXT:    s_mov_b64 exec, s[4:5]
+; VGPR-NEXT:    s_cbranch_execz .LBB1_3
+; VGPR-NEXT:  .LBB1_1: ; %loop
+; VGPR-NEXT:    ; =>This Inner Loop Header: Depth=1
+; VGPR-NEXT:    buffer_load_dword v0, off, s[0:3], 0 offset:8 ; 4-byte Folded Reload
+; VGPR-NEXT:    v_readlane_b32 s5, v1, 2
+; VGPR-NEXT:    s_mov_b32 m0, -1
+; VGPR-NEXT:    ; implicit-def: $sgpr4
+; VGPR-NEXT:    v_mov_b32_e32 v2, s4
+; VGPR-NEXT:    ds_read_b32 v2, v2
+; VGPR-NEXT:    s_mov_b32 s4, 1
+; VGPR-NEXT:    s_add_i32 s4, s5, s4
+; VGPR-NEXT:    ; implicit-def: $sgpr6
+; VGPR-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
+; VGPR-NEXT:    v_sub_i32_e32 v0, vcc, v0, v2
+; VGPR-NEXT:    s_mov_b32 s6, 0x100
+; VGPR-NEXT:    s_cmp_lg_u32 s5, s6
+; VGPR-NEXT:    v_mov_b32_e32 v2, v0
+; VGPR-NEXT:    v_writelane_b32 v1, s4, 2
+; VGPR-NEXT:    buffer_store_dword v2, off, s[0:3], 0 offset:8 ; 4-byte Folded Spill
+; VGPR-NEXT:    buffer_store_dword v0, off, s[0:3], 0 offset:12 ; 4-byte Folded Spill
+; VGPR-NEXT:    s_cbranch_scc1 .LBB1_1
+; VGPR-NEXT:  ; %bb.2: ; %Flow
+; VGPR-NEXT:    buffer_load_dword v0, off, s[0:3], 0 offset:12 ; 4-byte Folded Reload
+; VGPR-NEXT:    s_waitcnt vmcnt(0)
+; VGPR-NEXT:    buffer_store_dword v0, off, s[0:3], 0 offset:4 ; 4-byte Folded Spill
+; VGPR-NEXT:  .LBB1_3: ; %end
+; VGPR-NEXT:    buffer_load_dword v0, off, s[0:3], 0 offset:4 ; 4-byte Folded Reload
+; VGPR-NEXT:    v_readlane_b32 s6, v1, 3
+; VGPR-NEXT:    v_readlane_b32 s7, v1, 4
+; VGPR-NEXT:    s_or_b64 exec, exec, s[6:7]
+; VGPR-NEXT:    v_readlane_b32 s4, v1, 0
+; VGPR-NEXT:    v_readlane_b32 s5, v1, 1
+; VGPR-NEXT:    v_mov_b32_e32 v2, s4
+; VGPR-NEXT:    v_mov_b32_e32 v3, s5
+; VGPR-NEXT:    s_waitcnt vmcnt(0)
+; VGPR-NEXT:    flat_store_dword v[2:3], v0
+; VGPR-NEXT:    s_endpgm
 entry:
   %tid = call i32 @llvm.amdgcn.workitem.id.x()
   %load0 = load volatile i32, ptr addrspace(3) null
   %cmp0 = icmp eq i32 %tid, 0
   br i1 %cmp0, label %loop, label %end
 
 loop:
   %i = phi i32 [ %i.inc, %loop ], [ 0, %entry ]
   %val = phi i32 [ %val.sub, %loop ], [ %load0, %entry ]
   %load1 = load volatile i32, ptr addrspace(3) undef
   %i.inc = add i32 %i, 1
   %val.sub = sub i32 %val, %load1
   %cmp1 = icmp ne i32 %i, 256
   br i1 %cmp1, label %loop, label %end
 
 end:
   %tmp4 = phi i32 [ %val.sub, %loop ], [ 0, %entry ]
   store i32 %tmp4, ptr addrspace(1) %out
   ret void
 }
 
-; GCN-LABEL: {{^}}divergent_if_else_endif:
-; GCN: {{^}}; %bb.0:
-
-; GCN-DAG: s_mov_b32 m0, -1
-; GCN-DAG: v_mov_b32_e32 [[PTR0:v[0-9]+]], 0{{$}}
-; GCN: ds_read_b32 [[LOAD0:v[0-9]+]], [[PTR0]]
-
-; Spill load
-; GCN: buffer_store_dword [[LOAD0]], off, s[0:3], 0 offset:[[LOAD0_OFFSET:[0-9]+]] ; 4-byte Folded Spill
-
-; GCN: s_mov_b32 [[ZERO:s[0-9]+]], 0
-; GCN: v_cmp_ne_u32_e64 [[CMP0:s\[[0-9]+:[0-9]\]]], v0, [[ZERO]]
-
-; GCN: s_mov_b64 s[[[SAVEEXEC_LO:[0-9]+]]:[[SAVEEXEC_HI:[0-9]+]]], exec
-; GCN: s_and_b64 s[[[ANDEXEC_LO:[0-9]+]]:[[ANDEXEC_HI:[0-9]+]]], s[[[SAVEEXEC_LO:[0-9]+]]:[[SAVEEXEC_HI:[0-9]+]]], [[CMP0]]
-; GCN: s_xor_b64 s[[[SAVEEXEC_LO]]:[[SAVEEXEC_HI]]], s[[[ANDEXEC_LO]]:[[ANDEXEC_HI]]], s[[[SAVEEXEC_LO]]:[[SAVEEXEC_HI]]]
-
-; Spill saved exec
-; VGPR: v_writelane_b32 [[SPILL_VGPR:v[0-9]+]], s[[SAVEEXEC_LO]], [[SAVEEXEC_LO_LANE:[0-9]+]]
-; VGPR: v_writelane_b32 [[SPILL_VGPR]], s[[SAVEEXEC_HI]], [[SAVEEXEC_HI_LANE:[0-9]+]]
-
-; VMEM: v_writelane_b32 v[[V_SAVEEXEC:[0-9]+]], s[[SAVEEXEC_LO]], 0
-; VMEM: v_writelane_b32 v[[V_SAVEEXEC]], s[[SAVEEXEC_HI]], 1
-; VMEM: buffer_store_dword v[[V_SAVEEXEC]], off, s[0:3], 0 offset:[[SAVEEXEC_OFFSET:[0-9]+]] ; 4-byte Folded Spill
-
-; GCN: s_mov_b64 exec, [[CMP0]]
-
-; FIXME: It makes no sense to put this skip here
-; GCN: s_cbranch_execz [[FLOW:.LBB[0-9]+_[0-9]+]]
-; GCN-NEXT: s_branch [[ELSE:.LBB[0-9]+_[0-9]+]]
-
-; GCN: [[FLOW]]: ; %Flow
-; GCN: buffer_load_dword [[FLOW_VAL:v[0-9]+]], off, s[0:3], 0 offset:[[FLOW_VAL_OFFSET:[0-9]+]] ; 4-byte Folded Reload
-; VGPR: v_readlane_b32 s[[FLOW_S_RELOAD_SAVEEXEC_LO:[0-9]+]], [[SPILL_VGPR]], [[SAVEEXEC_LO_LANE]]
-; VGPR: v_readlane_b32 s[[FLOW_S_RELOAD_SAVEEXEC_HI:[0-9]+]], [[SPILL_VGPR]], [[SAVEEXEC_HI_LANE]]
-
-; VMEM: buffer_load_dword v[[FLOW_V_RELOAD_SAVEEXEC:[0-9]+]], off, s[0:3], 0 offset:[[SAVEEXEC_OFFSET]]
-; VMEM: s_waitcnt vmcnt(0)
-; VMEM: v_readlane_b32 s[[FLOW_S_RELOAD_SAVEEXEC_LO:[0-9]+]], v[[FLOW_V_RELOAD_SAVEEXEC]], 0
-; VMEM: v_readlane_b32 s[[FLOW_S_RELOAD_SAVEEXEC_HI:[0-9]+]], v[[FLOW_V_RELOAD_SAVEEXEC]], 1
-
-; GCN: s_or_saveexec_b64 s[[[FLOW_S_RELOAD_SAVEEXEC_LO_SAVEEXEC:[0-9]+]]:[[FLOW_S_RELOAD_SAVEEXEC_HI_SAVEEXEC:[0-9]+]]], s[[[FLOW_S_RELOAD_SAVEEXEC_LO]]:[[FLOW_S_RELOAD_SAVEEXEC_HI]]]
-
-; Regular spill value restored after exec modification
-; Followed by spill
-; GCN: buffer_store_dword [[FLOW_VAL]], off, s[0:3], 0 offset:[[RESULT_OFFSET:[0-9]+]] ; 4-byte Folded Spill
-
-; GCN: s_and_b64 s[[[FLOW_AND_EXEC_LO:[0-9]+]]:[[FLOW_AND_EXEC_HI:[0-9]+]]], exec, s[[[FLOW_S_RELOAD_SAVEEXEC_LO_SAVEEXEC]]:[[FLOW_S_RELOAD_SAVEEXEC_HI_SAVEEXEC]]]
-
-; Spill saved exec
-; VGPR: v_writelane_b32 [[SPILL_VGPR]], s[[FLOW_AND_EXEC_LO]], [[FLOW_SAVEEXEC_LO_LANE:[0-9]+]]
-; VGPR: v_writelane_b32 [[SPILL_VGPR]], s[[FLOW_AND_EXEC_HI]], [[FLOW_SAVEEXEC_HI_LANE:[0-9]+]]
-
-; VMEM: v_writelane_b32 v[[FLOW_V_SAVEEXEC:[0-9]+]], s[[FLOW_AND_EXEC_LO]], 0
-; VMEM: v_writelane_b32 v[[FLOW_V_SAVEEXEC]], s[[FLOW_AND_EXEC_HI]], 1
-; VMEM: buffer_store_dword v[[FLOW_V_SAVEEXEC]], off, s[0:3], 0 offset:[[FLOW_SAVEEXEC_OFFSET:[0-9]+]] ; 4-byte Folded Spill
-
-; GCN: s_xor_b64 exec, exec, s[[[FLOW_AND_EXEC_LO]]:[[FLOW_AND_EXEC_HI]]]
-; GCN-NEXT: s_cbranch_execz [[ENDIF:.LBB[0-9]+_[0-9]+]]
-
-
-; GCN: ; %bb.{{[0-9]+}}: ; %if
-; GCN: buffer_load_dword v[[LOAD0_RELOAD:[0-9]+]], off, s[0:3], 0 offset:[[LOAD0_OFFSET]] ; 4-byte Folded Reload
-; GCN: ds_read_b32
-; GCN: v_add_i32_e32 [[ADD:v[0-9]+]], vcc, v{{[0-9]+}}, v[[LOAD0_RELOAD]]
-; GCN: buffer_store_dword [[ADD]], off, s[0:3], 0 offset:[[RESULT_OFFSET]] ; 4-byte Folded Spill
-; GCN-NEXT: s_branch [[ENDIF:.LBB[0-9]+_[0-9]+]]
-
-; GCN: [[ELSE]]: ; %else
-; GCN: buffer_load_dword v[[LOAD0_RELOAD:[0-9]+]], off, s[0:3], 0 offset:[[LOAD0_OFFSET]] ; 4-byte Folded Reload
-; GCN: v_subrev_i32_e32 [[SUB:v[0-9]+]], vcc, v{{[0-9]+}}, v[[LOAD0_RELOAD]]
-; GCN: buffer_store_dword [[ADD]], off, s[0:3], 0 offset:[[FLOW_RESULT_OFFSET:[0-9]+]] ; 4-byte Folded Spill
-; GCN-NEXT: s_branch [[FLOW]]
-
-; GCN: [[ENDIF]]:
-; GCN: buffer_load_dword v[[RESULT:[0-9]+]], off, s[0:3], 0 offset:[[RESULT_OFFSET]] ; 4-byte Folded Reload
-; VGPR: v_readlane_b32 s[[S_RELOAD_SAVEEXEC_LO:[0-9]+]], [[SPILL_VGPR]], [[FLOW_SAVEEXEC_LO_LANE]]
-; VGPR: v_readlane_b32 s[[S_RELOAD_SAVEEXEC_HI:[0-9]+]], [[SPILL_VGPR]], [[FLOW_SAVEEXEC_HI_LANE]]
-
-
-; VMEM: buffer_load_dword v[[V_RELOAD_SAVEEXEC:[0-9]+]], off, s[0:3], 0 offset:[[FLOW_SAVEEXEC_OFFSET]] ; 4-byte Folded Reload
-; VMEM: s_waitcnt vmcnt(0)
-; VMEM: v_readlane_b32 s[[S_RELOAD_SAVEEXEC_LO:[0-9]+]], v[[V_RELOAD_SAVEEXEC]], 0
-; VMEM: v_readlane_b32 s[[S_RELOAD_SAVEEXEC_HI:[0-9]+]], v[[V_RELOAD_SAVEEXEC]], 1
-
-; GCN: s_or_b64 exec, exec, s[[[S_RELOAD_SAVEEXEC_LO]]:[[S_RELOAD_SAVEEXEC_HI]]]
-
-; GCN: flat_store_dword v{{\[[0-9]+:[0-9]+\]}}, v[[RESULT]]
 define amdgpu_kernel void @divergent_if_else_endif(ptr addrspace(1) %out) #0 {
+; VMEM-LABEL: divergent_if_else_endif:
+; VMEM:         .amd_kernel_code_t
+; VMEM-NEXT:     amd_code_version_major = 1
+; VMEM-NEXT:     amd_code_version_minor = 2
+; VMEM-NEXT:     amd_machine_kind = 1
+; VMEM-NEXT:     amd_machine_version_major = 7
+; VMEM-NEXT:     amd_machine_version_minor = 0
+; VMEM-NEXT:     amd_machine_version_stepping = 0
+; VMEM-NEXT:     kernel_code_entry_byte_offset = 256
+; VMEM-NEXT:     kernel_code_prefetch_byte_size = 0
+; VMEM-NEXT:     granulated_workitem_vgpr_count = 1
+; VMEM-NEXT:     granulated_wavefront_sgpr_count = 1
+; VMEM-NEXT:     priority = 0
+; VMEM-NEXT:     float_mode = 240
+; VMEM-NEXT:     priv = 0
+; VMEM-NEXT:     enable_dx10_clamp = 1
+; VMEM-NEXT:     debug_mode = 0
+; VMEM-NEXT:     enable_ieee_mode = 1
+; VMEM-NEXT:     enable_wgp_mode = 0
+; VMEM-NEXT:     enable_mem_ordered = 0
+; VMEM-NEXT:     enable_fwd_progress = 0
+; VMEM-NEXT:     enable_sgpr_private_segment_wave_byte_offset = 1
+; VMEM-NEXT:     user_sgpr_count = 6
+; VMEM-NEXT:     enable_trap_handler = 0
+; VMEM-NEXT:     enable_sgpr_workgroup_id_x = 1
+; VMEM-NEXT:     enable_sgpr_workgroup_id_y = 0
+; VMEM-NEXT:     enable_sgpr_workgroup_id_z = 0
+; VMEM-NEXT:     enable_sgpr_workgroup_info = 0
+; VMEM-NEXT:     enable_vgpr_workitem_id = 0
+; VMEM-NEXT:     enable_exception_msb = 0
+; VMEM-NEXT:     granulated_lds_size = 0
+; VMEM-NEXT:     enable_exception = 0
+; VMEM-NEXT:     enable_sgpr_private_segment_buffer = 1
+; VMEM-NEXT:     enable_sgpr_dispatch_ptr = 0
+; VMEM-NEXT:     enable_sgpr_queue_ptr = 0
+; VMEM-NEXT:     enable_sgpr_kernarg_segment_ptr = 1
+; VMEM-NEXT:     enable_sgpr_dispatch_id = 0
+; VMEM-NEXT:     enable_sgpr_flat_scratch_init = 0
+; VMEM-NEXT:     enable_sgpr_private_segment_size = 0
+; VMEM-NEXT:     enable_sgpr_grid_workgroup_count_x = 0
+; VMEM-NEXT:     enable_sgpr_grid_workgroup_count_y = 0
+; VMEM-NEXT:     enable_sgpr_grid_workgroup_count_z = 0
+; VMEM-NEXT:     enable_wavefront_size32 = 0
+; VMEM-NEXT:     enable_ordered_append_gds = 0
+; VMEM-NEXT:     private_element_size = 1
+; VMEM-NEXT:     is_ptr64 = 1
+; VMEM-NEXT:     is_dynamic_callstack = 0
+; VMEM-NEXT:     is_debug_enabled = 0
+; VMEM-NEXT:     is_xnack_enabled = 0
+; VMEM-NEXT:     workitem_private_segment_byte_size = 40
+; VMEM-NEXT:     workgroup_group_segment_byte_size = 0
+; VMEM-NEXT:     gds_segment_byte_size = 0
+; VMEM-NEXT:     kernarg_segment_byte_size = 8
+; VMEM-NEXT:     workgroup_fbarrier_count = 0
+; VMEM-NEXT:     wavefront_sgpr_count = 14
+; VMEM-NEXT:     workitem_vgpr_count = 5
+; VMEM-NEXT:     reserved_vgpr_first = 0
+; VMEM-NEXT:     reserved_vgpr_count = 0
+; VMEM-NEXT:     reserved_sgpr_first = 0
+; VMEM-NEXT:     reserved_sgpr_count = 0
+; VMEM-NEXT:     debug_wavefront_private_segment_offset_sgpr = 0
+; VMEM-NEXT:     debug_private_segment_buffer_sgpr = 0
+; VMEM-NEXT:     kernarg_segment_alignment = 4
+; VMEM-NEXT:     group_segment_alignment = 4
+; VMEM-NEXT:     private_segment_alignment = 4
+; VMEM-NEXT:     wavefront_size = 6
+; VMEM-NEXT:     call_convention = -1
+; VMEM-NEXT:     runtime_loader_kernel_symbol = 0
+; VMEM-NEXT:    .end_amd_kernel_code_t
+; VMEM-NEXT:  ; %bb.0: ; %entry
+; VMEM-NEXT:    s_add_u32 s0, s0, s7
+; VMEM-NEXT:    s_addc_u32 s1, s1, 0
+; VMEM-NEXT:    s_load_dwordx2 s[4:5], s[4:5], 0x0
+; VMEM-NEXT:    s_mov_b64 s[10:11], exec
+; VMEM-NEXT:    s_mov_b64 exec, 3
+; VMEM-NEXT:    buffer_store_dword v2, off, s[0:3], 0
+; VMEM-NEXT:    s_waitcnt lgkmcnt(0)
+; VMEM-NEXT:    v_writelane_b32 v2, s4, 0
+; VMEM-NEXT:    v_writelane_b32 v2, s5, 1
+; VMEM-NEXT:    buffer_store_dword v2, off, s[0:3], 0 offset:12 ; 4-byte Folded Spill
+; VMEM-NEXT:    buffer_load_dword v2, off, s[0:3], 0
+; VMEM-NEXT:    s_waitcnt vmcnt(0)
+; VMEM-NEXT:    s_mov_b64 exec, s[10:11]
+; VMEM-NEXT:    v_mov_b32_e32 v1, 0
+; VMEM-NEXT:    s_mov_b32 m0, -1
+; VMEM-NEXT:    ds_read_b32 v1, v1
+; VMEM-NEXT:    s_waitcnt lgkmcnt(0)
+; VMEM-NEXT:    buffer_store_dword v1, off, s[0:3], 0 offset:32 ; 4-byte Folded Spill
+; VMEM-NEXT:    ; implicit-def: $sgpr4
+; VMEM-NEXT:    s_mov_b32 s4, 0
+; VMEM-NEXT:    v_cmp_ne_u32_e64 s[4:5], v0, s4
+; VMEM-NEXT:    ; implicit-def: $sgpr6
+; VMEM-NEXT:    v_mov_b32_e32 v0, s6
+; VMEM-NEXT:    buffer_store_dword v0, off, s[0:3], 0 offset:28 ; 4-byte Folded Spill
+; VMEM-NEXT:    s_mov_b64 s[6:7], exec
+; VMEM-NEXT:    s_and_b64 s[4:5], s[6:7], s[4:5]
+; VMEM-NEXT:    s_xor_b64 s[6:7], s[4:5], s[6:7]
+; VMEM-NEXT:    s_mov_b64 s[8:9], exec
+; VMEM-NEXT:    s_mov_b64 exec, 3
+; VMEM-NEXT:    buffer_store_dword v0, off, s[0:3], 0
+; VMEM-NEXT:    v_writelane_b32 v0, s6, 0
+; VMEM-NEXT:    v_writelane_b32 v0, s7, 1
+; VMEM-NEXT:    buffer_store_dword v0, off, s[0:3], 0 offset:4 ; 4-byte Folded Spill
+; VMEM-NEXT:    buffer_load_dword v0, off, s[0:3], 0
+; VMEM-NEXT:    s_waitcnt vmcnt(0)
+; VMEM-NEXT:    s_mov_b64 exec, s[8:9]
+; VMEM-NEXT:    s_mov_b64 exec, s[4:5]
+; VMEM-NEXT:    s_cbranch_execz .LBB2_1
+; VMEM-NEXT:    s_branch .LBB2_3
+; VMEM-NEXT:  .LBB2_1: ; %Flow
+; VMEM-NEXT:    buffer_load_dword v0, off, s[0:3], 0 offset:28 ; 4-byte Folded Reload
+; VMEM-NEXT:    s_mov_b64 s[8:9], exec
+; VMEM-NEXT:    s_mov_b64 exec, 3
+; VMEM-NEXT:    buffer_store_dword v1, off, s[0:3], 0
+; VMEM-NEXT:    buffer_load_dword v1, off, s[0:3], 0 offset:4 ; 4-byte Folded Reload
+; VMEM-NEXT:    s_waitcnt vmcnt(0)
+; VMEM-NEXT:    v_readlane_b32 s4, v1, 0
+; VMEM-NEXT:    v_readlane_b32 s5, v1, 1
+; VMEM-NEXT:    buffer_load_dword v1, off, s[0:3], 0
+; VMEM-NEXT:    s_waitcnt vmcnt(0)
+; VMEM-NEXT:    s_mov_b64 exec, s[8:9]
+; VMEM-NEXT:    s_or_saveexec_b64 s[4:5], s[4:5]
+; VMEM-NEXT:    buffer_store_dword v0, off, s[0:3], 0 offset:36 ; 4-byte Folded Spill
+; VMEM-NEXT:    s_and_b64 s[4:5], exec, s[4:5]
+; VMEM-NEXT:    s_mov_b64 s[6:7], exec
+; VMEM-NEXT:    s_mov_b64 exec, 3
+; VMEM-NEXT:    buffer_store_dword v0, off, s[0:3], 0
+; VMEM-NEXT:    v_writelane_b32 v0, s4, 0
+; VMEM-NEXT:    v_writelane_b32 v0, s5, 1
+; VMEM-NEXT:    buffer_store_dword v0, off, s[0:3], 0 offset:20 ; 4-byte Folded Spill
+; VMEM-NEXT:    buffer_load_dword v0, off, s[0:3], 0
+; VMEM-NEXT:    s_waitcnt vmcnt(0)
+; VMEM-NEXT:    s_mov_b64 exec, s[6:7]
+; VMEM-NEXT:    s_xor_b64 exec, exec, s[4:5]
+; VMEM-NEXT:    s_cbranch_execz .LBB2_4
+; VMEM-NEXT:  ; %bb.2: ; %if
+; VMEM-NEXT:    buffer_load_dword v0, off, s[0:3], 0 offset:32 ; 4-byte Folded Reload
+; VMEM-NEXT:    s_mov_b32 m0, -1
+; VMEM-NEXT:    ; implicit-def: $sgpr4
+; VMEM-NEXT:    v_mov_b32_e32 v1, s4
+; VMEM-NEXT:    ds_read_b32 v1, v1
+; VMEM-NEXT:    ; implicit-def: $sgpr4
+; VMEM-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
+; VMEM-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
+; VMEM-NEXT:    buffer_store_dword v0, off, s[0:3], 0 offset:36 ; 4-byte Folded Spill
+; VMEM-NEXT:    s_branch .LBB2_4
+; VMEM-NEXT:  .LBB2_3: ; %else
+; VMEM-NEXT:    buffer_load_dword v0, off, s[0:3], 0 offset:32 ; 4-byte Folded Reload
+; VMEM-NEXT:    s_mov_b32 m0, -1
+; VMEM-NEXT:    ; implicit-def: $sgpr4
+; VMEM-NEXT:    v_mov_b32_e32 v1, s4
+; VMEM-NEXT:    ds_read_b32 v1, v1
+; VMEM-NEXT:    ; implicit-def: $sgpr4
+; VMEM-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
+; VMEM-NEXT:    v_sub_i32_e32 v0, vcc, v0, v1
+; VMEM-NEXT:    buffer_store_dword v0, off, s[0:3], 0 offset:28 ; 4-byte Folded Spill
+; VMEM-NEXT:    s_branch .LBB2_1
+; VMEM-NEXT:  .LBB2_4: ; %endif
+; VMEM-NEXT:    buffer_load_dword v2, off, s[0:3], 0 offset:36 ; 4-byte Folded Reload
+; VMEM-NEXT:    s_mov_b64 s[8:9], exec
+; VMEM-NEXT:    s_mov_b64 exec, 3
+; VMEM-NEXT:    buffer_store_dword v4, off, s[0:3], 0
+; VMEM-NEXT:    buffer_load_dword v4, off, s[0:3], 0 offset:20 ; 4-byte Folded Reload
+; VMEM-NEXT:    s_waitcnt vmcnt(0)
+; VMEM-NEXT:    v_readlane_b32 s6, v4, 0
+; VMEM-NEXT:    v_readlane_b32 s7, v4, 1
+; VMEM-NEXT:    buffer_load_dword v4, off, s[0:3], 0
+; VMEM-NEXT:    s_waitcnt vmcnt(0)
+; VMEM-NEXT:    s_mov_b64 exec, s[8:9]
+; VMEM-NEXT:    s_or_b64 exec, exec, s[6:7]
+; VMEM-NEXT:    s_mov_b64 s[6:7], exec
+; VMEM-NEXT:    s_mov_b64 exec, 3
+; VMEM-NEXT:    buffer_store_dword v3, off, s[0:3], 0
+; VMEM-NEXT:    buffer_load_dword v3, off, s[0:3], 0 offset:12 ; 4-byte Folded Reload
+; VMEM-NEXT:    s_waitcnt vmcnt(0)
+; VMEM-NEXT:    v_readlane_b32 s4, v3, 0
+; VMEM-NEXT:    v_readlane_b32 s5, v3, 1
+; VMEM-NEXT:    buffer_load_dword v3, off, s[0:3], 0
+; VMEM-NEXT:    s_waitcnt vmcnt(0)
+; VMEM-NEXT:    s_mov_b64 exec, s[6:7]
+; VMEM-NEXT:    v_mov_b32_e32 v0, s4
+; VMEM-NEXT:    v_mov_b32_e32 v1, s5
+; VMEM-NEXT:    flat_store_dword v[0:1], v2
+; VMEM-NEXT:    s_endpgm
+;
+; VGPR-LABEL: divergent_if_else_endif:
+; VGPR:         .amd_kernel_code_t
+; VGPR-NEXT:     amd_code_version_major = 1
+; VGPR-NEXT:     amd_code_version_minor = 2
+; VGPR-NEXT:     amd_machine_kind = 1
+; VGPR-NEXT:     amd_machine_version_major = 7
+; VGPR-NEXT:     amd_machine_version_minor = 0
+; VGPR-NEXT:     amd_machine_version_stepping = 0
+; VGPR-NEXT:     kernel_code_entry_byte_offset = 256
+; VGPR-NEXT:     kernel_code_prefetch_byte_size = 0
+; VGPR-NEXT:     granulated_workitem_vgpr_count = 0
+; VGPR-NEXT:     granulated_wavefront_sgpr_count = 1
+; VGPR-NEXT:     priority = 0
+; VGPR-NEXT:     float_mode = 240
+; VGPR-NEXT:     priv = 0
+; VGPR-NEXT:     enable_dx10_clamp = 1
+; VGPR-NEXT:     debug_mode = 0
+; VGPR-NEXT:     enable_ieee_mode = 1
+; VGPR-NEXT:     enable_wgp_mode = 0
+; VGPR-NEXT:     enable_mem_ordered = 0
+; VGPR-NEXT:     enable_fwd_progress = 0
+; VGPR-NEXT:     enable_sgpr_private_segment_wave_byte_offset = 1
+; VGPR-NEXT:     user_sgpr_count = 6
+; VGPR-NEXT:     enable_trap_handler = 0
+; VGPR-NEXT:     enable_sgpr_workgroup_id_x = 1
+; VGPR-NEXT:     enable_sgpr_workgroup_id_y = 0
+; VGPR-NEXT:     enable_sgpr_workgroup_id_z = 0
+; VGPR-NEXT:     enable_sgpr_workgroup_info = 0
+; VGPR-NEXT:     enable_vgpr_workitem_id = 0
+; VGPR-NEXT:     enable_exception_msb = 0
+; VGPR-NEXT:     granulated_lds_size = 0
+; VGPR-NEXT:     enable_exception = 0
+; VGPR-NEXT:     enable_sgpr_private_segment_buffer = 1
+; VGPR-NEXT:     enable_sgpr_dispatch_ptr = 0
+; VGPR-NEXT:     enable_sgpr_queue_ptr = 0
+; VGPR-NEXT:     enable_sgpr_kernarg_segment_ptr = 1
+; VGPR-NEXT:     enable_sgpr_dispatch_id = 0
+; VGPR-NEXT:     enable_sgpr_flat_scratch_init = 0
+; VGPR-NEXT:     enable_sgpr_private_segment_size = 0
+; VGPR-NEXT:     enable_sgpr_grid_workgroup_count_x = 0
+; VGPR-NEXT:     enable_sgpr_grid_workgroup_count_y = 0
+; VGPR-NEXT:     enable_sgpr_grid_workgroup_count_z = 0
+; VGPR-NEXT:     enable_wavefront_size32 = 0
+; VGPR-NEXT:     enable_ordered_append_gds = 0
+; VGPR-NEXT:     private_element_size = 1
+; VGPR-NEXT:     is_ptr64 = 1
+; VGPR-NEXT:     is_dynamic_callstack = 0
+; VGPR-NEXT:     is_debug_enabled = 0
+; VGPR-NEXT:     is_xnack_enabled = 0
+; VGPR-NEXT:     workitem_private_segment_byte_size = 16
+; VGPR-NEXT:     workgroup_group_segment_byte_size = 0
+; VGPR-NEXT:     gds_segment_byte_size = 0
+; VGPR-NEXT:     kernarg_segment_byte_size = 8
+; VGPR-NEXT:     workgroup_fbarrier_count = 0
+; VGPR-NEXT:     wavefront_sgpr_count = 10
+; VGPR-NEXT:     workitem_vgpr_count = 4
+; VGPR-NEXT:     reserved_vgpr_first = 0
+; VGPR-NEXT:     reserved_vgpr_count = 0
+; VGPR-NEXT:     reserved_sgpr_first = 0
+; VGPR-NEXT:     reserved_sgpr_count = 0
+; VGPR-NEXT:     debug_wavefront_private_segment_offset_sgpr = 0
+; VGPR-NEXT:     debug_private_segment_buffer_sgpr = 0
+; VGPR-NEXT:     kernarg_segment_alignment = 4
+; VGPR-NEXT:     group_segment_alignment = 4
+; VGPR-NEXT:     private_segment_alignment = 4
+; VGPR-NEXT:     wavefront_size = 6
+; VGPR-NEXT:     call_convention = -1
+; VGPR-NEXT:     runtime_loader_kernel_symbol = 0
+; VGPR-NEXT:    .end_amd_kernel_code_t
+; VGPR-NEXT:  ; %bb.0: ; %entry
+; VGPR-NEXT:    s_add_u32 s0, s0, s7
+; VGPR-NEXT:    s_addc_u32 s1, s1, 0
+; VGPR-NEXT:    s_load_dwordx2 s[4:5], s[4:5], 0x0
+; VGPR-NEXT:    s_waitcnt lgkmcnt(0)
+; VGPR-NEXT:    v_writelane_b32 v1, s4, 0
+; VGPR-NEXT:    v_writelane_b32 v1, s5, 1
+; VGPR-NEXT:    v_mov_b32_e32 v2, 0
+; VGPR-NEXT:    s_mov_b32 m0, -1
+; VGPR-NEXT:    ds_read_b32 v2, v2
+; VGPR-NEXT:    s_waitcnt lgkmcnt(0)
+; VGPR-NEXT:    buffer_store_dword v2, off, s[0:3], 0 offset:8 ; 4-byte Folded Spill
+; VGPR-NEXT:    ; implicit-def: $sgpr4
+; VGPR-NEXT:    s_mov_b32 s4, 0
+; VGPR-NEXT:    v_cmp_ne_u32_e64 s[4:5], v0, s4
+; VGPR-NEXT:    ; implicit-def: $sgpr6
+; VGPR-NEXT:    v_mov_b32_e32 v0, s6
+; VGPR-NEXT:    buffer_store_dword v0, off, s[0:3], 0 offset:4 ; 4-byte Folded Spill
+; VGPR-NEXT:    s_mov_b64 s[6:7], exec
+; VGPR-NEXT:    s_and_b64 s[4:5], s[6:7], s[4:5]
+; VGPR-NEXT:    s_xor_b64 s[6:7], s[4:5], s[6:7]
+; VGPR-NEXT:    v_writelane_b32 v1, s6, 2
+; VGPR-NEXT:    v_writelane_b32 v1, s7, 3
+; VGPR-NEXT:    s_mov_b64 exec, s[4:5]
+; VGPR-NEXT:    s_cbranch_execz .LBB2_1
+; VGPR-NEXT:    s_branch .LBB2_3
+; VGPR-NEXT:  .LBB2_1: ; %Flow
+; VGPR-NEXT:    buffer_load_dword v0, off, s[0:3], 0 offset:4 ; 4-byte Folded Reload
+; VGPR-NEXT:    v_readlane_b32 s4, v1, 2
+; VGPR-NEXT:    v_readlane_b32 s5, v1, 3
+; VGPR-NEXT:    s_or_saveexec_b64 s[4:5], s[4:5]
+; VGPR-NEXT:    s_waitcnt vmcnt(0)
+; VGPR-NEXT:    buffer_store_dword v0, off, s[0:3], 0 offset:12 ; 4-byte Folded Spill
+; VGPR-NEXT:    s_and_b64 s[4:5], exec, s[4:5]
+; VGPR-NEXT:    v_writelane_b32 v1, s4, 4
+; VGPR-NEXT:    v_writelane_b32 v1, s5, 5
+; VGPR-NEXT:    s_xor_b64 exec, exec, s[4:5]
+; VGPR-NEXT:    s_cbranch_execz .LBB2_4
+; VGPR-NEXT:  ; %bb.2: ; %if
+; VGPR-NEXT:    buffer_load_dword v0, off, s[0:3], 0 offset:8 ; 4-byte Folded Reload
+; VGPR-NEXT:    s_mov_b32 m0, -1
+; VGPR-NEXT:    ; implicit-def: $sgpr4
+; VGPR-NEXT:    v_mov_b32_e32 v2, s4
+; VGPR-NEXT:    ds_read_b32 v2, v2
+; VGPR-NEXT:    ; implicit-def: $sgpr4
+; VGPR-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
+; VGPR-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
+; VGPR-NEXT:    buffer_store_dword v0, off, s[0:3], 0 offset:12 ; 4-byte Folded Spill
+; VGPR-NEXT:    s_branch .LBB2_4
+; VGPR-NEXT:  .LBB2_3: ; %else
+; VGPR-NEXT:    buffer_load_dword v0, off, s[0:3], 0 offset:8 ; 4-byte Folded Reload
+; VGPR-NEXT:    s_mov_b32 m0, -1
+; VGPR-NEXT:    ; implicit-def: $sgpr4
+; VGPR-NEXT:    v_mov_b32_e32 v2, s4
+; VGPR-NEXT:    ds_read_b32 v2, v2
+; VGPR-NEXT:    ; implicit-def: $sgpr4
+; VGPR-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
+; VGPR-NEXT:    v_sub_i32_e32 v0, vcc, v0, v2
+; VGPR-NEXT:    buffer_store_dword v0, off, s[0:3], 0 offset:4 ; 4-byte Folded Spill
+; VGPR-NEXT:    s_branch .LBB2_1
+; VGPR-NEXT:  .LBB2_4: ; %endif
+; VGPR-NEXT:    buffer_load_dword v0, off, s[0:3], 0 offset:12 ; 4-byte Folded Reload
+; VGPR-NEXT:    v_readlane_b32 s6, v1, 4
+; VGPR-NEXT:    v_readlane_b32 s7, v1, 5
+; VGPR-NEXT:    s_or_b64 exec, exec, s[6:7]
+; VGPR-NEXT:    v_readlane_b32 s4, v1, 0
+; VGPR-NEXT:    v_readlane_b32 s5, v1, 1
+; VGPR-NEXT:    v_mov_b32_e32 v2, s4
+; VGPR-NEXT:    v_mov_b32_e32 v3, s5
+; VGPR-NEXT:    s_waitcnt vmcnt(0)
+; VGPR-NEXT:    flat_store_dword v[2:3], v0
+; VGPR-NEXT:    s_endpgm
 entry:
   %tid = call i32 @llvm.amdgcn.workitem.id.x()
   %load0 = load volatile i32, ptr addrspace(3) null
   %cmp0 = icmp eq i32 %tid, 0
   br i1 %cmp0, label %if, label %else
 
 if:
   %load1 = load volatile i32, ptr addrspace(3) undef

 
 declare i32 @llvm.amdgcn.workitem.id.x() #1
 
 attributes #0 = { nounwind }
 attributes #1 = { nounwind readnone }
 
 !llvm.module.flags = !{!0}
 !0 = !{i32 1, !"amdgpu_code_object_version", i32 200}
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GCN: {{.*}}

llvm/test/CodeGen/AMDGPU/ds_read2.ll

 ; CI:       ; %bb.0:
 ; CI-NEXT:    v_mov_b32_e32 v0, 0
 ; CI-NEXT:    s_mov_b32 m0, -1
 ; CI-NEXT:    ds_read_b64 v[0:1], v0
 ; CI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x0
 ; CI-NEXT:    s_mov_b32 s3, 0xf000
 ; CI-NEXT:    s_mov_b32 s2, -1
 ; CI-NEXT:    s_waitcnt lgkmcnt(0)
-; CI-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
+; CI-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; CI-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; CI-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: load_constant_adjacent_offsets:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    ds_read_b64 v[0:1], v2
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x0

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.sbfe.ll

define amdgpu_kernel void @bfe_sext_in_reg_i24(ptr addrspace(1) %out, ptr addrspace(1) %in) #0 {
   store i32 %ashr, ptr addrspace(1) %out, align 4
   ret void
 }
 
 ; GCN-LABEL: @simplify_demanded_bfe_sdiv
 ; GCN: buffer_load_dword [[LOAD:v[0-9]+]]
 ; GCN: v_bfe_i32 [[BFE:v[0-9]+]], [[LOAD]], 1, 16
 ; GCN: v_lshrrev_b32_e32 [[TMP0:v[0-9]+]], 31, [[BFE]]
-; GCN: v_add_{{[iu]}}32_e32 [[TMP1:v[0-9]+]], vcc, [[TMP0]], [[BFE]]
+; GCN: v_add_{{[iu]}}32_e32 [[TMP1:v[0-9]+]], vcc, [[BFE]], [[TMP0]]
 ; GCN: v_ashrrev_i32_e32 [[TMP2:v[0-9]+]], 1, [[TMP1]]
 ; GCN: buffer_store_dword [[TMP2]]
 define amdgpu_kernel void @simplify_demanded_bfe_sdiv(ptr addrspace(1) %out, ptr addrspace(1) %in) #0 {
   %src = load i32, ptr addrspace(1) %in, align 4
   %bfe = call i32 @llvm.amdgcn.sbfe.i32(i32 %src, i32 1, i32 16)
   %div = sdiv i32 %bfe, 2
   store i32 %div, ptr addrspace(1) %out, align 4
   ret void

llvm/test/CodeGen/AMDGPU/mubuf-legalize-operands.ll

 ; W64-O0-NEXT:    buffer_store_dword v4, off, s[0:3], s32 offset:16 ; 4-byte Folded Spill
 ; W64-O0-NEXT:    v_mov_b32_e32 v4, v3
 ; W64-O0-NEXT:    v_mov_b32_e32 v6, v2
 ; W64-O0-NEXT:    v_mov_b32_e32 v7, v1
 ; W64-O0-NEXT:    ; implicit-def: $sgpr4
 ; W64-O0-NEXT:    ; implicit-def: $sgpr4
 ; W64-O0-NEXT:    ; implicit-def: $sgpr4
 ; W64-O0-NEXT:    ; implicit-def: $sgpr4
-; W64-O0-NEXT:    ; kill: def $vgpr0 killed $vgpr0 killed $exec
-; W64-O0-NEXT:    ; kill: def $vgpr7 killed $vgpr7 killed $exec
-; W64-O0-NEXT:    ; kill: def $vgpr4 killed $vgpr4 killed $exec
 ; W64-O0-NEXT:    ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1_vgpr2_vgpr3 killed $exec
 ; W64-O0-NEXT:    v_mov_b32_e32 v1, v7
 ; W64-O0-NEXT:    v_mov_b32_e32 v2, v6
 ; W64-O0-NEXT:    v_mov_b32_e32 v3, v4
 ; W64-O0-NEXT:    buffer_store_dword v0, off, s[0:3], s32 ; 4-byte Folded Spill
 ; W64-O0-NEXT:    s_waitcnt vmcnt(0)
 ; W64-O0-NEXT:    buffer_store_dword v1, off, s[0:3], s32 offset:4 ; 4-byte Folded Spill
 ; W64-O0-NEXT:    buffer_store_dword v2, off, s[0:3], s32 offset:8 ; 4-byte Folded Spill

 ; W64-O0-NEXT:    v_mov_b32_e32 v8, v1
 ; W64-O0-NEXT:    buffer_load_dword v1, off, s[0:3], s32 offset:52 ; 4-byte Folded Reload
 ; W64-O0-NEXT:    v_mov_b32_e32 v2, v0
 ; W64-O0-NEXT:    buffer_load_dword v0, off, s[0:3], s32 offset:48 ; 4-byte Folded Reload
 ; W64-O0-NEXT:    ; implicit-def: $sgpr4
 ; W64-O0-NEXT:    ; implicit-def: $sgpr4
 ; W64-O0-NEXT:    ; implicit-def: $sgpr4
 ; W64-O0-NEXT:    ; implicit-def: $sgpr4
-; W64-O0-NEXT:    ; kill: def $vgpr5 killed $vgpr5 killed $exec
-; W64-O0-NEXT:    ; kill: def $vgpr4 killed $vgpr4 killed $exec
-; W64-O0-NEXT:    ; kill: def $vgpr3 killed $vgpr3 killed $exec
 ; W64-O0-NEXT:    ; kill: def $vgpr14 killed $vgpr14 def $vgpr14_vgpr15_vgpr16_vgpr17 killed $exec
 ; W64-O0-NEXT:    v_mov_b32_e32 v15, v5
 ; W64-O0-NEXT:    s_waitcnt vmcnt(3)
 ; W64-O0-NEXT:    v_mov_b32_e32 v16, v4
 ; W64-O0-NEXT:    s_waitcnt vmcnt(2)
 ; W64-O0-NEXT:    v_mov_b32_e32 v17, v3
 ; W64-O0-NEXT:    buffer_store_dword v14, off, s[0:3], s32 offset:32 ; 4-byte Folded Spill
 ; W64-O0-NEXT:    s_waitcnt vmcnt(0)
 ; W64-O0-NEXT:    buffer_store_dword v15, off, s[0:3], s32 offset:36 ; 4-byte Folded Spill
 ; W64-O0-NEXT:    buffer_store_dword v16, off, s[0:3], s32 offset:40 ; 4-byte Folded Spill
 ; W64-O0-NEXT:    buffer_store_dword v17, off, s[0:3], s32 offset:44 ; 4-byte Folded Spill
 ; W64-O0-NEXT:    ; implicit-def: $sgpr4
 ; W64-O0-NEXT:    ; implicit-def: $sgpr4
 ; W64-O0-NEXT:    ; implicit-def: $sgpr4
 ; W64-O0-NEXT:    ; implicit-def: $sgpr4
-; W64-O0-NEXT:    ; kill: def $vgpr2 killed $vgpr2 killed $exec
-; W64-O0-NEXT:    ; kill: def $vgpr8 killed $vgpr8 killed $exec
-; W64-O0-NEXT:    ; kill: def $vgpr7 killed $vgpr7 killed $exec
 ; W64-O0-NEXT:    ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3_vgpr4_vgpr5 killed $exec
 ; W64-O0-NEXT:    v_mov_b32_e32 v3, v8
 ; W64-O0-NEXT:    v_mov_b32_e32 v4, v7
 ; W64-O0-NEXT:    v_mov_b32_e32 v5, v6
 ; W64-O0-NEXT:    buffer_store_dword v2, off, s[0:3], s32 offset:16 ; 4-byte Folded Spill
 ; W64-O0-NEXT:    s_waitcnt vmcnt(0)
 ; W64-O0-NEXT:    buffer_store_dword v3, off, s[0:3], s32 offset:20 ; 4-byte Folded Spill
 ; W64-O0-NEXT:    buffer_store_dword v4, off, s[0:3], s32 offset:24 ; 4-byte Folded Spill
 ; W64-O0-NEXT:    buffer_store_dword v5, off, s[0:3], s32 offset:28 ; 4-byte Folded Spill
 ; W64-O0-NEXT:    ; implicit-def: $sgpr4
 ; W64-O0-NEXT:    ; implicit-def: $sgpr4
-; W64-O0-NEXT:    ; kill: def $vgpr12 killed $vgpr12 killed $exec
 ; W64-O0-NEXT:    ; kill: def $vgpr1 killed $vgpr1 def $vgpr1_vgpr2 killed $exec
 ; W64-O0-NEXT:    v_mov_b32_e32 v2, v12
 ; W64-O0-NEXT:    buffer_store_dword v1, off, s[0:3], s32 offset:8 ; 4-byte Folded Spill
 ; W64-O0-NEXT:    s_waitcnt vmcnt(0)
 ; W64-O0-NEXT:    buffer_store_dword v2, off, s[0:3], s32 offset:12 ; 4-byte Folded Spill
 ; W64-O0-NEXT:    ; implicit-def: $sgpr4
 ; W64-O0-NEXT:    ; implicit-def: $sgpr4
-; W64-O0-NEXT:    ; kill: def $vgpr10 killed $vgpr10 killed $exec
 ; W64-O0-NEXT:    ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
 ; W64-O0-NEXT:    v_mov_b32_e32 v1, v10
 ; W64-O0-NEXT:    buffer_store_dword v0, off, s[0:3], s32 ; 4-byte Folded Spill
 ; W64-O0-NEXT:    s_waitcnt vmcnt(0)
 ; W64-O0-NEXT:    buffer_store_dword v1, off, s[0:3], s32 offset:4 ; 4-byte Folded Spill
 ; W64-O0-NEXT:    ; implicit-def: $sgpr4_sgpr5
 ; W64-O0-NEXT:    ; implicit-def: $sgpr4_sgpr5
 ; W64-O0-NEXT:    ; implicit-def: $sgpr4_sgpr5_sgpr6_sgpr7

 ; W64-O0-NEXT:    v_mov_b32_e32 v6, v2
 ; W64-O0-NEXT:    v_mov_b32_e32 v7, v1
 ; W64-O0-NEXT:    v_mov_b32_e32 v13, v0
 ; W64-O0-NEXT:    buffer_load_dword v0, off, s[0:3], s32 offset:44 ; 4-byte Folded Reload
 ; W64-O0-NEXT:    ; implicit-def: $sgpr4
 ; W64-O0-NEXT:    ; implicit-def: $sgpr4
 ; W64-O0-NEXT:    ; implicit-def: $sgpr4
 ; W64-O0-NEXT:    ; implicit-def: $sgpr4
-; W64-O0-NEXT:    ; kill: def $vgpr0 killed $vgpr0 killed $exec
-; W64-O0-NEXT:    ; kill: def $vgpr11 killed $vgpr11 killed $exec
-; W64-O0-NEXT:    ; kill: def $vgpr9 killed $vgpr9 killed $exec
 ; W64-O0-NEXT:    ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1_vgpr2_vgpr3 killed $exec
 ; W64-O0-NEXT:    v_mov_b32_e32 v1, v11
 ; W64-O0-NEXT:    v_mov_b32_e32 v2, v10
 ; W64-O0-NEXT:    v_mov_b32_e32 v3, v9
 ; W64-O0-NEXT:    ; implicit-def: $sgpr4
 ; W64-O0-NEXT:    ; implicit-def: $sgpr4
 ; W64-O0-NEXT:    ; implicit-def: $sgpr4
 ; W64-O0-NEXT:    ; implicit-def: $sgpr4
-; W64-O0-NEXT:    ; kill: def $vgpr13 killed $vgpr13 killed $exec
-; W64-O0-NEXT:    ; kill: def $vgpr7 killed $vgpr7 killed $exec
-; W64-O0-NEXT:    ; kill: def $vgpr6 killed $vgpr6 killed $exec
 ; W64-O0-NEXT:    ; kill: def $vgpr13 killed $vgpr13 def $vgpr13_vgpr14_vgpr15_vgpr16 killed $exec
 ; W64-O0-NEXT:    v_mov_b32_e32 v14, v7
 ; W64-O0-NEXT:    v_mov_b32_e32 v15, v6
 ; W64-O0-NEXT:    v_mov_b32_e32 v16, v5
 ; W64-O0-NEXT:    buffer_store_dword v13, off, s[0:3], s32 offset:28 ; 4-byte Folded Spill
 ; W64-O0-NEXT:    s_waitcnt vmcnt(0)
 ; W64-O0-NEXT:    buffer_store_dword v14, off, s[0:3], s32 offset:32 ; 4-byte Folded Spill
 ; W64-O0-NEXT:    buffer_store_dword v15, off, s[0:3], s32 offset:36 ; 4-byte Folded Spill
 ; W64-O0-NEXT:    buffer_store_dword v16, off, s[0:3], s32 offset:40 ; 4-byte Folded Spill
 ; W64-O0-NEXT:    ; implicit-def: $sgpr4
 ; W64-O0-NEXT:    ; implicit-def: $sgpr4
-; W64-O0-NEXT:    ; kill: def $vgpr12 killed $vgpr12 killed $exec
 ; W64-O0-NEXT:    ; kill: def $vgpr4 killed $vgpr4 def $vgpr4_vgpr5 killed $exec
 ; W64-O0-NEXT:    v_mov_b32_e32 v5, v12
 ; W64-O0-NEXT:    buffer_store_dword v4, off, s[0:3], s32 offset:20 ; 4-byte Folded Spill
 ; W64-O0-NEXT:    s_waitcnt vmcnt(0)
 ; W64-O0-NEXT:    buffer_store_dword v5, off, s[0:3], s32 offset:24 ; 4-byte Folded Spill
 ; W64-O0-NEXT:    buffer_store_dword v0, off, s[0:3], s32 offset:4 ; 4-byte Folded Spill
 ; W64-O0-NEXT:    s_waitcnt vmcnt(0)
 ; W64-O0-NEXT:    buffer_store_dword v1, off, s[0:3], s32 offset:8 ; 4-byte Folded Spill

llvm/test/CodeGen/AMDGPU/mubuf-legalize-operands.mir

bb.0:
     ; W64-NEXT: [[COPY2:%[0-9]+]]:vgpr_32 = COPY $vgpr3
     ; W64-NEXT: [[COPY3:%[0-9]+]]:vgpr_32 = COPY $vgpr2
     ; W64-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY $vgpr1
     ; W64-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; W64-NEXT: [[DEF:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; W64-NEXT: [[DEF1:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; W64-NEXT: [[DEF2:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; W64-NEXT: [[DEF3:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
-    ; W64-NEXT: [[COPY6:%[0-9]+]]:vgpr_32 = COPY [[COPY5]], implicit $exec
-    ; W64-NEXT: [[COPY7:%[0-9]+]]:vgpr_32 = COPY [[COPY4]], implicit $exec
-    ; W64-NEXT: [[COPY8:%[0-9]+]]:vgpr_32 = COPY [[COPY2]], implicit $exec
-    ; W64-NEXT: [[REG_SEQUENCE:%[0-9]+]]:vreg_128 = REG_SEQUENCE killed [[COPY6]], %subreg.sub0, killed [[COPY7]], %subreg.sub1, [[COPY3]], %subreg.sub2, killed [[COPY8]], %subreg.sub3
+    ; W64-NEXT: [[REG_SEQUENCE:%[0-9]+]]:vreg_128 = REG_SEQUENCE [[COPY5]], %subreg.sub0, [[COPY4]], %subreg.sub1, [[COPY3]], %subreg.sub2, [[COPY2]], %subreg.sub3
     ; W64-NEXT: [[S_MOV_B64_:%[0-9]+]]:sreg_64_xexec = S_MOV_B64 $exec
     ; W64-NEXT: {{  $}}
     ; W64-NEXT: .1:
     ; W64-NEXT: successors: %bb.2(0x80000000)
     ; W64-NEXT: {{  $}}
     ; W64-NEXT: [[V_READFIRSTLANE_B32_:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[REG_SEQUENCE]].sub0, implicit $exec
     ; W64-NEXT: [[V_READFIRSTLANE_B32_1:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[REG_SEQUENCE]].sub1, implicit $exec
     ; W64-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:sgpr_64 = REG_SEQUENCE [[V_READFIRSTLANE_B32_]], %subreg.sub0, [[V_READFIRSTLANE_B32_1]], %subreg.sub1
bb.0:
     ; W32-NEXT: [[COPY2:%[0-9]+]]:vgpr_32 = COPY $vgpr3
     ; W32-NEXT: [[COPY3:%[0-9]+]]:vgpr_32 = COPY $vgpr2
     ; W32-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY $vgpr1
     ; W32-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; W32-NEXT: [[DEF:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; W32-NEXT: [[DEF1:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; W32-NEXT: [[DEF2:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; W32-NEXT: [[DEF3:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
-    ; W32-NEXT: [[COPY6:%[0-9]+]]:vgpr_32 = COPY [[COPY5]], implicit $exec
-    ; W32-NEXT: [[COPY7:%[0-9]+]]:vgpr_32 = COPY [[COPY4]], implicit $exec
-    ; W32-NEXT: [[COPY8:%[0-9]+]]:vgpr_32 = COPY [[COPY2]], implicit $exec
-    ; W32-NEXT: [[REG_SEQUENCE:%[0-9]+]]:vreg_128 = REG_SEQUENCE killed [[COPY6]], %subreg.sub0, killed [[COPY7]], %subreg.sub1, [[COPY3]], %subreg.sub2, killed [[COPY8]], %subreg.sub3
+    ; W32-NEXT: [[REG_SEQUENCE:%[0-9]+]]:vreg_128 = REG_SEQUENCE [[COPY5]], %subreg.sub0, [[COPY4]], %subreg.sub1, [[COPY3]], %subreg.sub2, [[COPY2]], %subreg.sub3
     ; W32-NEXT: [[S_MOV_B32_:%[0-9]+]]:sreg_32_xm0_xexec = S_MOV_B32 $exec_lo
     ; W32-NEXT: {{  $}}
     ; W32-NEXT: .1:
     ; W32-NEXT: successors: %bb.2(0x80000000)
     ; W32-NEXT: {{  $}}
     ; W32-NEXT: [[V_READFIRSTLANE_B32_:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[REG_SEQUENCE]].sub0, implicit $exec
     ; W32-NEXT: [[V_READFIRSTLANE_B32_1:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[REG_SEQUENCE]].sub1, implicit $exec
     ; W32-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:sgpr_64 = REG_SEQUENCE [[V_READFIRSTLANE_B32_]], %subreg.sub0, [[V_READFIRSTLANE_B32_1]], %subreg.sub1
bb.0:
     ; W64-NEXT: [[COPY2:%[0-9]+]]:vgpr_32 = COPY $vgpr3
     ; W64-NEXT: [[COPY3:%[0-9]+]]:vgpr_32 = COPY $vgpr2
     ; W64-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY $vgpr1
     ; W64-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; W64-NEXT: [[DEF:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; W64-NEXT: [[DEF1:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; W64-NEXT: [[DEF2:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; W64-NEXT: [[DEF3:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
-    ; W64-NEXT: [[COPY6:%[0-9]+]]:vgpr_32 = COPY [[COPY4]], implicit $exec
-    ; W64-NEXT: [[COPY7:%[0-9]+]]:vgpr_32 = COPY [[COPY3]], implicit $exec
-    ; W64-NEXT: [[COPY8:%[0-9]+]]:vgpr_32 = COPY [[COPY2]], implicit $exec
-    ; W64-NEXT: [[REG_SEQUENCE:%[0-9]+]]:vreg_128 = REG_SEQUENCE [[COPY5]], %subreg.sub0, killed [[COPY6]], %subreg.sub1, killed [[COPY7]], %subreg.sub2, killed [[COPY8]], %subreg.sub3
+    ; W64-NEXT: [[REG_SEQUENCE:%[0-9]+]]:vreg_128 = REG_SEQUENCE [[COPY5]], %subreg.sub0, [[COPY4]], %subreg.sub1, [[COPY3]], %subreg.sub2, [[COPY2]], %subreg.sub3
     ; W64-NEXT: [[S_MOV_B64_:%[0-9]+]]:sreg_64_xexec = S_MOV_B64 $exec
     ; W64-NEXT: {{  $}}
     ; W64-NEXT: .1:
     ; W64-NEXT: successors: %bb.2(0x80000000)
     ; W64-NEXT: {{  $}}
     ; W64-NEXT: [[V_READFIRSTLANE_B32_:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[REG_SEQUENCE]].sub0, implicit $exec
     ; W64-NEXT: [[V_READFIRSTLANE_B32_1:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[REG_SEQUENCE]].sub1, implicit $exec
     ; W64-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:sgpr_64 = REG_SEQUENCE [[V_READFIRSTLANE_B32_]], %subreg.sub0, [[V_READFIRSTLANE_B32_1]], %subreg.sub1
bb.0:
     ; W32-NEXT: [[COPY2:%[0-9]+]]:vgpr_32 = COPY $vgpr3
     ; W32-NEXT: [[COPY3:%[0-9]+]]:vgpr_32 = COPY $vgpr2
     ; W32-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY $vgpr1
     ; W32-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; W32-NEXT: [[DEF:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; W32-NEXT: [[DEF1:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; W32-NEXT: [[DEF2:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; W32-NEXT: [[DEF3:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
-    ; W32-NEXT: [[COPY6:%[0-9]+]]:vgpr_32 = COPY [[COPY4]], implicit $exec
-    ; W32-NEXT: [[COPY7:%[0-9]+]]:vgpr_32 = COPY [[COPY3]], implicit $exec
-    ; W32-NEXT: [[COPY8:%[0-9]+]]:vgpr_32 = COPY [[COPY2]], implicit $exec
-    ; W32-NEXT: [[REG_SEQUENCE:%[0-9]+]]:vreg_128 = REG_SEQUENCE [[COPY5]], %subreg.sub0, killed [[COPY6]], %subreg.sub1, killed [[COPY7]], %subreg.sub2, killed [[COPY8]], %subreg.sub3
+    ; W32-NEXT: [[REG_SEQUENCE:%[0-9]+]]:vreg_128 = REG_SEQUENCE [[COPY5]], %subreg.sub0, [[COPY4]], %subreg.sub1, [[COPY3]], %subreg.sub2, [[COPY2]], %subreg.sub3
     ; W32-NEXT: [[S_MOV_B32_:%[0-9]+]]:sreg_32_xm0_xexec = S_MOV_B32 $exec_lo
     ; W32-NEXT: {{  $}}
     ; W32-NEXT: .1:
     ; W32-NEXT: successors: %bb.2(0x80000000)
     ; W32-NEXT: {{  $}}
     ; W32-NEXT: [[V_READFIRSTLANE_B32_:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[REG_SEQUENCE]].sub0, implicit $exec
     ; W32-NEXT: [[V_READFIRSTLANE_B32_1:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[REG_SEQUENCE]].sub1, implicit $exec
     ; W32-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:sgpr_64 = REG_SEQUENCE [[V_READFIRSTLANE_B32_]], %subreg.sub0, [[V_READFIRSTLANE_B32_1]], %subreg.sub1
bb.0:
     ; W64-NEXT: [[COPY2:%[0-9]+]]:vgpr_32 = COPY $vgpr3
     ; W64-NEXT: [[COPY3:%[0-9]+]]:vgpr_32 = COPY $vgpr2
     ; W64-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY $vgpr1
     ; W64-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; W64-NEXT: [[DEF:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; W64-NEXT: [[DEF1:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; W64-NEXT: [[DEF2:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; W64-NEXT: [[DEF3:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
-    ; W64-NEXT: [[COPY6:%[0-9]+]]:vgpr_32 = COPY [[COPY5]], implicit $exec
-    ; W64-NEXT: [[COPY7:%[0-9]+]]:vgpr_32 = COPY [[COPY4]], implicit $exec
-    ; W64-NEXT: [[COPY8:%[0-9]+]]:vgpr_32 = COPY [[COPY3]], implicit $exec
-    ; W64-NEXT: [[REG_SEQUENCE:%[0-9]+]]:vreg_128 = REG_SEQUENCE killed [[COPY6]], %subreg.sub0, killed [[COPY7]], %subreg.sub1, killed [[COPY8]], %subreg.sub2, [[COPY2]], %subreg.sub3
+    ; W64-NEXT: [[REG_SEQUENCE:%[0-9]+]]:vreg_128 = REG_SEQUENCE [[COPY5]], %subreg.sub0, [[COPY4]], %subreg.sub1, [[COPY3]], %subreg.sub2, [[COPY2]], %subreg.sub3
     ; W64-NEXT: [[S_MOV_B64_:%[0-9]+]]:sreg_64_xexec = S_MOV_B64 $exec
     ; W64-NEXT: {{  $}}
     ; W64-NEXT: .1:
     ; W64-NEXT: successors: %bb.2(0x80000000)
     ; W64-NEXT: {{  $}}
     ; W64-NEXT: [[V_READFIRSTLANE_B32_:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[REG_SEQUENCE]].sub0, implicit $exec
     ; W64-NEXT: [[V_READFIRSTLANE_B32_1:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[REG_SEQUENCE]].sub1, implicit $exec
     ; W64-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:sgpr_64 = REG_SEQUENCE [[V_READFIRSTLANE_B32_]], %subreg.sub0, [[V_READFIRSTLANE_B32_1]], %subreg.sub1
bb.0:
     ; W32-NEXT: [[COPY2:%[0-9]+]]:vgpr_32 = COPY $vgpr3
     ; W32-NEXT: [[COPY3:%[0-9]+]]:vgpr_32 = COPY $vgpr2
     ; W32-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY $vgpr1
     ; W32-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; W32-NEXT: [[DEF:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; W32-NEXT: [[DEF1:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; W32-NEXT: [[DEF2:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; W32-NEXT: [[DEF3:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
-    ; W32-NEXT: [[COPY6:%[0-9]+]]:vgpr_32 = COPY [[COPY5]], implicit $exec
-    ; W32-NEXT: [[COPY7:%[0-9]+]]:vgpr_32 = COPY [[COPY4]], implicit $exec
-    ; W32-NEXT: [[COPY8:%[0-9]+]]:vgpr_32 = COPY [[COPY3]], implicit $exec
-    ; W32-NEXT: [[REG_SEQUENCE:%[0-9]+]]:vreg_128 = REG_SEQUENCE killed [[COPY6]], %subreg.sub0, killed [[COPY7]], %subreg.sub1, killed [[COPY8]], %subreg.sub2, [[COPY2]], %subreg.sub3
+    ; W32-NEXT: [[REG_SEQUENCE:%[0-9]+]]:vreg_128 = REG_SEQUENCE [[COPY5]], %subreg.sub0, [[COPY4]], %subreg.sub1, [[COPY3]], %subreg.sub2, [[COPY2]], %subreg.sub3
     ; W32-NEXT: [[S_MOV_B32_:%[0-9]+]]:sreg_32_xm0_xexec = S_MOV_B32 $exec_lo
     ; W32-NEXT: {{  $}}
     ; W32-NEXT: .1:
     ; W32-NEXT: successors: %bb.2(0x80000000)
     ; W32-NEXT: {{  $}}
     ; W32-NEXT: [[V_READFIRSTLANE_B32_:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[REG_SEQUENCE]].sub0, implicit $exec
     ; W32-NEXT: [[V_READFIRSTLANE_B32_1:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[REG_SEQUENCE]].sub1, implicit $exec
     ; W32-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:sgpr_64 = REG_SEQUENCE [[V_READFIRSTLANE_B32_]], %subreg.sub0, [[V_READFIRSTLANE_B32_1]], %subreg.sub1
bb.0:
     ; ADDR64-NEXT: [[COPY2:%[0-9]+]]:vgpr_32 = COPY $vgpr3
     ; ADDR64-NEXT: [[COPY3:%[0-9]+]]:vgpr_32 = COPY $vgpr2
     ; ADDR64-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY $vgpr1
     ; ADDR64-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; ADDR64-NEXT: [[DEF:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; ADDR64-NEXT: [[DEF1:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; ADDR64-NEXT: [[DEF2:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; ADDR64-NEXT: [[DEF3:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
-    ; ADDR64-NEXT: [[COPY6:%[0-9]+]]:vgpr_32 = COPY [[COPY5]], implicit $exec
-    ; ADDR64-NEXT: [[COPY7:%[0-9]+]]:vgpr_32 = COPY [[COPY4]], implicit $exec
-    ; ADDR64-NEXT: [[COPY8:%[0-9]+]]:vgpr_32 = COPY [[COPY2]], implicit $exec
-    ; ADDR64-NEXT: [[REG_SEQUENCE:%[0-9]+]]:vreg_128 = REG_SEQUENCE killed [[COPY6]], %subreg.sub0, killed [[COPY7]], %subreg.sub1, [[COPY3]], %subreg.sub2, killed [[COPY8]], %subreg.sub3
-    ; ADDR64-NEXT: [[COPY9:%[0-9]+]]:vreg_64 = COPY [[REG_SEQUENCE]].sub0_sub1
+    ; ADDR64-NEXT: [[REG_SEQUENCE:%[0-9]+]]:vreg_128 = REG_SEQUENCE [[COPY5]], %subreg.sub0, [[COPY4]], %subreg.sub1, [[COPY3]], %subreg.sub2, [[COPY2]], %subreg.sub3
+    ; ADDR64-NEXT: [[COPY6:%[0-9]+]]:vreg_64 = COPY [[REG_SEQUENCE]].sub0_sub1
     ; ADDR64-NEXT: [[S_MOV_B64_:%[0-9]+]]:sreg_64 = S_MOV_B64 0
     ; ADDR64-NEXT: [[S_MOV_B32_:%[0-9]+]]:sgpr_32 = S_MOV_B32 0
     ; ADDR64-NEXT: [[S_MOV_B32_1:%[0-9]+]]:sgpr_32 = S_MOV_B32 61440
     ; ADDR64-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:sgpr_128 = REG_SEQUENCE [[S_MOV_B64_]], %subreg.sub0_sub1, [[S_MOV_B32_]], %subreg.sub2, [[S_MOV_B32_1]], %subreg.sub3
-    ; ADDR64-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADD_CO_U32_e64 [[COPY9]].sub0, [[COPY1]].sub0, 0, implicit $exec
-    ; ADDR64-NEXT: %17:vgpr_32, dead %20:sreg_64_xexec = V_ADDC_U32_e64 [[COPY9]].sub1, [[COPY1]].sub1, killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; ADDR64-NEXT: [[REG_SEQUENCE2:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %17, %subreg.sub1
+    ; ADDR64-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADD_CO_U32_e64 [[COPY6]].sub0, [[COPY1]].sub0, 0, implicit $exec
+    ; ADDR64-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_64_xexec = V_ADDC_U32_e64 [[COPY6]].sub1, [[COPY1]].sub1, killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; ADDR64-NEXT: [[REG_SEQUENCE2:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; ADDR64-NEXT: [[BUFFER_LOAD_FORMAT_X_ADDR64_:%[0-9]+]]:vgpr_32 = BUFFER_LOAD_FORMAT_X_ADDR64 [[REG_SEQUENCE2]], killed [[REG_SEQUENCE1]], 0, 0, 0, 0, implicit $exec
     ; ADDR64-NEXT: $sgpr30_sgpr31 = COPY [[COPY]]
     ; ADDR64-NEXT: $vgpr0 = COPY [[BUFFER_LOAD_FORMAT_X_ADDR64_]]
     ; ADDR64-NEXT: S_SETPC_B64_return $sgpr30_sgpr31, implicit $vgpr0
     ; W32-LABEL: name: addr64
     ; W32: liveins: $vgpr0, $vgpr1, $vgpr2, $vgpr3, $vgpr4, $sgpr30_sgpr31
     ; W32-NEXT: {{  $}}
     ; W32-NEXT: [[COPY:%[0-9]+]]:sreg_64 = COPY $sgpr30_sgpr31
     ; W32-NEXT: [[COPY1:%[0-9]+]]:vreg_64 = COPY $vgpr4_vgpr5
     ; W32-NEXT: [[COPY2:%[0-9]+]]:vgpr_32 = COPY $vgpr3
     ; W32-NEXT: [[COPY3:%[0-9]+]]:vgpr_32 = COPY $vgpr2
     ; W32-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY $vgpr1
     ; W32-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; W32-NEXT: [[DEF:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; W32-NEXT: [[DEF1:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; W32-NEXT: [[DEF2:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; W32-NEXT: [[DEF3:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
-    ; W32-NEXT: [[COPY6:%[0-9]+]]:vgpr_32 = COPY [[COPY5]], implicit $exec
-    ; W32-NEXT: [[COPY7:%[0-9]+]]:vgpr_32 = COPY [[COPY4]], implicit $exec
-    ; W32-NEXT: [[COPY8:%[0-9]+]]:vgpr_32 = COPY [[COPY2]], implicit $exec
-    ; W32-NEXT: [[REG_SEQUENCE:%[0-9]+]]:vreg_128 = REG_SEQUENCE killed [[COPY6]], %subreg.sub0, killed [[COPY7]], %subreg.sub1, [[COPY3]], %subreg.sub2, killed [[COPY8]], %subreg.sub3
-    ; W32-NEXT: [[COPY9:%[0-9]+]]:vreg_64 = COPY [[REG_SEQUENCE]].sub0_sub1
+    ; W32-NEXT: [[REG_SEQUENCE:%[0-9]+]]:vreg_128 = REG_SEQUENCE [[COPY5]], %subreg.sub0, [[COPY4]], %subreg.sub1, [[COPY3]], %subreg.sub2, [[COPY2]], %subreg.sub3
+    ; W32-NEXT: [[COPY6:%[0-9]+]]:vreg_64 = COPY [[REG_SEQUENCE]].sub0_sub1
     ; W32-NEXT: [[S_MOV_B64_:%[0-9]+]]:sreg_64 = S_MOV_B64 0
     ; W32-NEXT: [[S_MOV_B32_:%[0-9]+]]:sgpr_32 = S_MOV_B32 0
     ; W32-NEXT: [[S_MOV_B32_1:%[0-9]+]]:sgpr_32 = S_MOV_B32 822173696
     ; W32-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:sgpr_128 = REG_SEQUENCE [[S_MOV_B64_]], %subreg.sub0_sub1, [[S_MOV_B32_]], %subreg.sub2, [[S_MOV_B32_1]], %subreg.sub3
-    ; W32-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADD_CO_U32_e64 [[COPY9]].sub0, [[COPY1]].sub0, 0, implicit $exec
-    ; W32-NEXT: %17:vgpr_32, dead %20:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY9]].sub1, [[COPY1]].sub1, killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-    ; W32-NEXT: [[REG_SEQUENCE2:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, %17, %subreg.sub1
+    ; W32-NEXT: [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADD_CO_U32_e64 [[COPY6]].sub0, [[COPY1]].sub0, 0, implicit $exec
+    ; W32-NEXT: [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADDC_U32_e64 [[COPY6]].sub1, [[COPY1]].sub1, killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+    ; W32-NEXT: [[REG_SEQUENCE2:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[V_ADD_CO_U32_e64_]], %subreg.sub0, [[V_ADDC_U32_e64_]], %subreg.sub1
     ; W32-NEXT: [[BUFFER_LOAD_FORMAT_X_ADDR64_:%[0-9]+]]:vgpr_32 = BUFFER_LOAD_FORMAT_X_ADDR64 [[REG_SEQUENCE2]], killed [[REG_SEQUENCE1]], 0, 0, 0, 0, implicit $exec
     ; W32-NEXT: $sgpr30_sgpr31 = COPY [[COPY]]
     ; W32-NEXT: $vgpr0 = COPY [[BUFFER_LOAD_FORMAT_X_ADDR64_]]
     ; W32-NEXT: S_SETPC_B64_return $sgpr30_sgpr31, implicit $vgpr0
     %5:sreg_64 = COPY $sgpr30_sgpr31
     %4:vreg_64 = COPY $vgpr4_vgpr5
     %3:vgpr_32 = COPY $vgpr3
     %2:vgpr_32 = COPY $vgpr2
bb.0:
     ; ADDR64-NEXT: [[COPY2:%[0-9]+]]:vgpr_32 = COPY $vgpr3
     ; ADDR64-NEXT: [[COPY3:%[0-9]+]]:vgpr_32 = COPY $vgpr2
     ; ADDR64-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY $vgpr1
     ; ADDR64-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; ADDR64-NEXT: [[DEF:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; ADDR64-NEXT: [[DEF1:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; ADDR64-NEXT: [[DEF2:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; ADDR64-NEXT: [[DEF3:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
-    ; ADDR64-NEXT: [[COPY6:%[0-9]+]]:vgpr_32 = COPY [[COPY5]], implicit $exec
-    ; ADDR64-NEXT: [[COPY7:%[0-9]+]]:vgpr_32 = COPY [[COPY4]], implicit $exec
-    ; ADDR64-NEXT: [[COPY8:%[0-9]+]]:vgpr_32 = COPY [[COPY2]], implicit $exec
-    ; ADDR64-NEXT: [[REG_SEQUENCE:%[0-9]+]]:vreg_128 = REG_SEQUENCE killed [[COPY6]], %subreg.sub0, killed [[COPY7]], %subreg.sub1, [[COPY3]], %subreg.sub2, killed [[COPY8]], %subreg.sub3
-    ; ADDR64-NEXT: [[COPY9:%[0-9]+]]:vreg_64 = COPY [[REG_SEQUENCE]].sub0_sub1
+    ; ADDR64-NEXT: [[REG_SEQUENCE:%[0-9]+]]:vreg_128 = REG_SEQUENCE [[COPY5]], %subreg.sub0, [[COPY4]], %subreg.sub1, [[COPY3]], %subreg.sub2, [[COPY2]], %subreg.sub3
+    ; ADDR64-NEXT: [[COPY6:%[0-9]+]]:vreg_64 = COPY [[REG_SEQUENCE]].sub0_sub1
     ; ADDR64-NEXT: [[S_MOV_B64_:%[0-9]+]]:sreg_64 = S_MOV_B64 0
     ; ADDR64-NEXT: [[S_MOV_B32_:%[0-9]+]]:sgpr_32 = S_MOV_B32 0
     ; ADDR64-NEXT: [[S_MOV_B32_1:%[0-9]+]]:sgpr_32 = S_MOV_B32 61440
     ; ADDR64-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:sgpr_128 = REG_SEQUENCE [[S_MOV_B64_]], %subreg.sub0_sub1, [[S_MOV_B32_]], %subreg.sub2, [[S_MOV_B32_1]], %subreg.sub3
-    ; ADDR64-NEXT: [[REG_SEQUENCE2:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[COPY9]].sub0, %subreg.sub0, [[COPY9]].sub1, %subreg.sub1
+    ; ADDR64-NEXT: [[REG_SEQUENCE2:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[COPY6]].sub0, %subreg.sub0, [[COPY6]].sub1, %subreg.sub1
     ; ADDR64-NEXT: [[BUFFER_LOAD_FORMAT_X_ADDR64_:%[0-9]+]]:vgpr_32 = BUFFER_LOAD_FORMAT_X_ADDR64 [[REG_SEQUENCE2]], [[REG_SEQUENCE1]], 0, 0, 0, 0, implicit $exec
     ; ADDR64-NEXT: $sgpr30_sgpr31 = COPY [[COPY]]
     ; ADDR64-NEXT: $vgpr0 = COPY [[BUFFER_LOAD_FORMAT_X_ADDR64_]]
     ; ADDR64-NEXT: S_SETPC_B64_return $sgpr30_sgpr31, implicit $vgpr0
     ; W64-NO-ADDR64-LABEL: name: offset
     ; W64-NO-ADDR64: successors: %bb.1(0x80000000)
     ; W64-NO-ADDR64-NEXT: liveins: $vgpr0, $vgpr1, $vgpr2, $vgpr3, $vgpr4, $sgpr30_sgpr31
     ; W64-NO-ADDR64-NEXT: {{  $}}
     ; W64-NO-ADDR64-NEXT: [[COPY:%[0-9]+]]:sreg_64 = COPY $sgpr30_sgpr31
     ; W64-NO-ADDR64-NEXT: [[COPY1:%[0-9]+]]:vreg_64 = COPY $vgpr4_vgpr5
     ; W64-NO-ADDR64-NEXT: [[COPY2:%[0-9]+]]:vgpr_32 = COPY $vgpr3
     ; W64-NO-ADDR64-NEXT: [[COPY3:%[0-9]+]]:vgpr_32 = COPY $vgpr2
     ; W64-NO-ADDR64-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY $vgpr1
     ; W64-NO-ADDR64-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; W64-NO-ADDR64-NEXT: [[DEF:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; W64-NO-ADDR64-NEXT: [[DEF1:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; W64-NO-ADDR64-NEXT: [[DEF2:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; W64-NO-ADDR64-NEXT: [[DEF3:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
-    ; W64-NO-ADDR64-NEXT: [[COPY6:%[0-9]+]]:vgpr_32 = COPY [[COPY5]], implicit $exec
-    ; W64-NO-ADDR64-NEXT: [[COPY7:%[0-9]+]]:vgpr_32 = COPY [[COPY4]], implicit $exec
-    ; W64-NO-ADDR64-NEXT: [[COPY8:%[0-9]+]]:vgpr_32 = COPY [[COPY2]], implicit $exec
-    ; W64-NO-ADDR64-NEXT: [[REG_SEQUENCE:%[0-9]+]]:vreg_128 = REG_SEQUENCE killed [[COPY6]], %subreg.sub0, killed [[COPY7]], %subreg.sub1, [[COPY3]], %subreg.sub2, killed [[COPY8]], %subreg.sub3
+    ; W64-NO-ADDR64-NEXT: [[REG_SEQUENCE:%[0-9]+]]:vreg_128 = REG_SEQUENCE [[COPY5]], %subreg.sub0, [[COPY4]], %subreg.sub1, [[COPY3]], %subreg.sub2, [[COPY2]], %subreg.sub3
     ; W64-NO-ADDR64-NEXT: [[S_MOV_B64_:%[0-9]+]]:sreg_64_xexec = S_MOV_B64 $exec
     ; W64-NO-ADDR64-NEXT: {{  $}}
     ; W64-NO-ADDR64-NEXT: .1:
     ; W64-NO-ADDR64-NEXT: successors: %bb.2(0x80000000)
     ; W64-NO-ADDR64-NEXT: {{  $}}
     ; W64-NO-ADDR64-NEXT: [[V_READFIRSTLANE_B32_:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[REG_SEQUENCE]].sub0, implicit $exec
     ; W64-NO-ADDR64-NEXT: [[V_READFIRSTLANE_B32_1:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[REG_SEQUENCE]].sub1, implicit $exec
     ; W64-NO-ADDR64-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:sgpr_64 = REG_SEQUENCE [[V_READFIRSTLANE_B32_]], %subreg.sub0, [[V_READFIRSTLANE_B32_1]], %subreg.sub1
bb.0:
     ; W32-NEXT: [[COPY2:%[0-9]+]]:vgpr_32 = COPY $vgpr3
     ; W32-NEXT: [[COPY3:%[0-9]+]]:vgpr_32 = COPY $vgpr2
     ; W32-NEXT: [[COPY4:%[0-9]+]]:vgpr_32 = COPY $vgpr1
     ; W32-NEXT: [[COPY5:%[0-9]+]]:vgpr_32 = COPY $vgpr0
     ; W32-NEXT: [[DEF:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; W32-NEXT: [[DEF1:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; W32-NEXT: [[DEF2:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
     ; W32-NEXT: [[DEF3:%[0-9]+]]:sgpr_32 = IMPLICIT_DEF
-    ; W32-NEXT: [[COPY6:%[0-9]+]]:vgpr_32 = COPY [[COPY5]], implicit $exec
-    ; W32-NEXT: [[COPY7:%[0-9]+]]:vgpr_32 = COPY [[COPY4]], implicit $exec
-    ; W32-NEXT: [[COPY8:%[0-9]+]]:vgpr_32 = COPY [[COPY2]], implicit $exec
-    ; W32-NEXT: [[REG_SEQUENCE:%[0-9]+]]:vreg_128 = REG_SEQUENCE killed [[COPY6]], %subreg.sub0, killed [[COPY7]], %subreg.sub1, [[COPY3]], %subreg.sub2, killed [[COPY8]], %subreg.sub3
+    ; W32-NEXT: [[REG_SEQUENCE:%[0-9]+]]:vreg_128 = REG_SEQUENCE [[COPY5]], %subreg.sub0, [[COPY4]], %subreg.sub1, [[COPY3]], %subreg.sub2, [[COPY2]], %subreg.sub3
     ; W32-NEXT: [[S_MOV_B32_:%[0-9]+]]:sreg_32_xm0_xexec = S_MOV_B32 $exec_lo
     ; W32-NEXT: {{  $}}
     ; W32-NEXT: .1:
     ; W32-NEXT: successors: %bb.2(0x80000000)
     ; W32-NEXT: {{  $}}
     ; W32-NEXT: [[V_READFIRSTLANE_B32_:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[REG_SEQUENCE]].sub0, implicit $exec
     ; W32-NEXT: [[V_READFIRSTLANE_B32_1:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[REG_SEQUENCE]].sub1, implicit $exec
     ; W32-NEXT: [[REG_SEQUENCE1:%[0-9]+]]:sgpr_64 = REG_SEQUENCE [[V_READFIRSTLANE_B32_]], %subreg.sub0, [[V_READFIRSTLANE_B32_1]], %subreg.sub1

llvm/test/CodeGen/AMDGPU/mul.ll

 ; SI-NEXT:    s_mov_b32 s0, s4
 ; SI-NEXT:    s_mov_b32 s1, s5
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    v_mul_lo_u32 v1, v2, v1
 ; SI-NEXT:    v_mul_hi_u32 v4, v2, v0
 ; SI-NEXT:    v_mul_lo_u32 v3, v3, v0
 ; SI-NEXT:    v_mul_lo_u32 v0, v2, v0
 ; SI-NEXT:    v_add_i32_e32 v1, vcc, v4, v1
-; SI-NEXT:    v_add_i32_e32 v1, vcc, v3, v1
+; SI-NEXT:    v_add_i32_e32 v1, vcc, v1, v3
 ; SI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: v_mul_i64:
 ; VI:       ; %bb.0: ; %entry
 ; VI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; VI-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x34
 ; VI-NEXT:    s_mov_b32 s3, 0xf000

 ; VI-NEXT:    buffer_load_dwordx2 v[2:3], off, s[12:15], 0
 ; VI-NEXT:    s_mov_b32 s0, s4
 ; VI-NEXT:    s_mov_b32 s1, s5
 ; VI-NEXT:    s_waitcnt vmcnt(0)
 ; VI-NEXT:    v_mul_lo_u32 v4, v2, v1
 ; VI-NEXT:    v_mad_u64_u32 v[1:2], s[6:7], v2, v0, 0
 ; VI-NEXT:    v_mul_lo_u32 v0, v3, v0
 ; VI-NEXT:    v_add_u32_e32 v2, vcc, v4, v2
-; VI-NEXT:    v_add_u32_e32 v2, vcc, v0, v2
+; VI-NEXT:    v_add_u32_e32 v2, vcc, v2, v0
 ; VI-NEXT:    buffer_store_dwordx2 v[1:2], off, s[0:3], 0
 ; VI-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: v_mul_i64:
 ; GFX9:       ; %bb.0: ; %entry
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x34
 ; GFX9-NEXT:    s_mov_b32 s3, 0xf000

llvm/test/CodeGen/AMDGPU/opt-sgpr-to-vgpr-copy.mir

body:             |
   ; GCN-NEXT:   [[S_LOAD_DWORDX2_IMM:%[0-9]+]]:sreg_64_xexec = S_LOAD_DWORDX2_IMM [[COPY]], 9, 0
   ; GCN-NEXT:   [[S_LOAD_DWORDX2_IMM1:%[0-9]+]]:sreg_64_xexec = S_LOAD_DWORDX2_IMM [[COPY]], 11, 0
   ; GCN-NEXT:   [[COPY2:%[0-9]+]]:sreg_64 = COPY [[S_LOAD_DWORDX2_IMM]]
   ; GCN-NEXT:   [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 0, implicit $exec
   ; GCN-NEXT:   [[REG_SEQUENCE:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[COPY1]], %subreg.sub0, killed [[V_MOV_B32_e32_]], %subreg.sub1
   ; GCN-NEXT:   [[V_ADD_CO_U32_e32_:%[0-9]+]]:vgpr_32 = V_ADD_CO_U32_e32 [[S_LOAD_DWORDX2_IMM1]].sub0, [[REG_SEQUENCE]].sub0, implicit-def $vcc, implicit $exec
   ; GCN-NEXT:   [[COPY3:%[0-9]+]]:vgpr_32 = COPY [[S_LOAD_DWORDX2_IMM1]].sub1
   ; GCN-NEXT:   [[V_ADDC_U32_e32_:%[0-9]+]]:vgpr_32 = V_ADDC_U32_e32 0, [[COPY3]], implicit-def $vcc, implicit $vcc, implicit $exec
-  ; GCN-NEXT:   [[COPY4:%[0-9]+]]:vgpr_32 = COPY killed [[V_ADD_CO_U32_e32_]], implicit $exec
-  ; GCN-NEXT:   [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE killed [[COPY4]], %subreg.sub0, killed [[V_ADDC_U32_e32_]], %subreg.sub1
+  ; GCN-NEXT:   [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE killed [[V_ADD_CO_U32_e32_]], %subreg.sub0, killed [[V_ADDC_U32_e32_]], %subreg.sub1
   ; GCN-NEXT:   [[S_MOV_B32_:%[0-9]+]]:sreg_32_xm0 = S_MOV_B32 0
   ; GCN-NEXT:   [[S_MOV_B32_1:%[0-9]+]]:sreg_32_xm0 = S_MOV_B32 1048576
   ; GCN-NEXT:   [[REG_SEQUENCE2:%[0-9]+]]:sreg_64 = REG_SEQUENCE killed [[S_MOV_B32_1]], %subreg.sub0, killed [[S_MOV_B32_]], %subreg.sub1
   ; GCN-NEXT:   [[V_CMP_LT_U64_e64_:%[0-9]+]]:sreg_64 = V_CMP_LT_U64_e64 killed [[REG_SEQUENCE1]], [[REG_SEQUENCE2]], implicit $exec
   ; GCN-NEXT:   [[SI_IF:%[0-9]+]]:sreg_64 = SI_IF killed [[V_CMP_LT_U64_e64_]], %bb.2, implicit-def dead $exec, implicit-def dead $scc, implicit $exec
   ; GCN-NEXT:   S_BRANCH %bb.1
   ; GCN-NEXT: {{  $}}
   ; GCN-NEXT: bb.1.bb1:
body:             |
   ; GCN-NEXT:   [[S_LOAD_DWORDX2_IMM:%[0-9]+]]:sreg_64_xexec = S_LOAD_DWORDX2_IMM [[COPY]], 9, 0
   ; GCN-NEXT:   [[S_LOAD_DWORDX2_IMM1:%[0-9]+]]:sreg_64_xexec = S_LOAD_DWORDX2_IMM [[COPY]], 11, 0
   ; GCN-NEXT:   [[COPY2:%[0-9]+]]:sreg_64 = COPY [[S_LOAD_DWORDX2_IMM]]
   ; GCN-NEXT:   [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 0, implicit $exec
   ; GCN-NEXT:   [[REG_SEQUENCE:%[0-9]+]]:vreg_64 = REG_SEQUENCE [[COPY1]], %subreg.sub0, killed [[V_MOV_B32_e32_]], %subreg.sub1
   ; GCN-NEXT:   [[V_ADD_CO_U32_e32_:%[0-9]+]]:vgpr_32 = V_ADD_CO_U32_e32 [[S_LOAD_DWORDX2_IMM1]].sub0, [[REG_SEQUENCE]].sub0, implicit-def $vcc, implicit $exec
   ; GCN-NEXT:   [[COPY3:%[0-9]+]]:vgpr_32 = COPY [[S_LOAD_DWORDX2_IMM1]].sub1
   ; GCN-NEXT:   [[V_ADDC_U32_e32_:%[0-9]+]]:vgpr_32 = V_ADDC_U32_e32 0, [[COPY3]], implicit-def $vcc, implicit $vcc, implicit $exec
-  ; GCN-NEXT:   [[COPY4:%[0-9]+]]:vgpr_32 = COPY killed [[V_ADD_CO_U32_e32_]], implicit $exec
-  ; GCN-NEXT:   [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE killed [[COPY4]], %subreg.sub0, killed [[V_ADDC_U32_e32_]], %subreg.sub1
+  ; GCN-NEXT:   [[REG_SEQUENCE1:%[0-9]+]]:vreg_64 = REG_SEQUENCE killed [[V_ADD_CO_U32_e32_]], %subreg.sub0, killed [[V_ADDC_U32_e32_]], %subreg.sub1
   ; GCN-NEXT:   [[V_CMP_LT_U32_e64_:%[0-9]+]]:sreg_64 = V_CMP_LT_U32_e64 killed [[REG_SEQUENCE1]].sub0, 12, implicit $exec
   ; GCN-NEXT:   [[SI_IF:%[0-9]+]]:sreg_64 = SI_IF killed [[V_CMP_LT_U32_e64_]], %bb.2, implicit-def dead $exec, implicit-def dead $scc, implicit $exec
   ; GCN-NEXT:   S_BRANCH %bb.1
   ; GCN-NEXT: {{  $}}
   ; GCN-NEXT: bb.1.bb1:
   ; GCN-NEXT:   successors: %bb.2(0x80000000)
   ; GCN-NEXT: {{  $}}
   ; GCN-NEXT:   [[V_LSHL_B64_e64_:%[0-9]+]]:vreg_64 = V_LSHL_B64_e64 [[REG_SEQUENCE]], 2, implicit $exec

llvm/test/CodeGen/AMDGPU/sdiv.ll

 ; GCN-NEXT:    s_waitcnt lgkmcnt(0)
 ; GCN-NEXT:    s_mov_b32 s8, s2
 ; GCN-NEXT:    s_mov_b32 s9, s3
 ; GCN-NEXT:    buffer_load_dwordx2 v[0:1], off, s[8:11], 0
 ; GCN-NEXT:    s_mov_b32 s4, s0
 ; GCN-NEXT:    s_mov_b32 s5, s1
 ; GCN-NEXT:    s_waitcnt vmcnt(0)
 ; GCN-NEXT:    v_ashrrev_i32_e32 v2, 31, v1
-; GCN-NEXT:    v_add_i32_e32 v1, vcc, v2, v1
+; GCN-NEXT:    v_add_i32_e32 v1, vcc, v1, v2
 ; GCN-NEXT:    v_xor_b32_e32 v1, v1, v2
 ; GCN-NEXT:    v_cvt_f32_u32_e32 v3, v1
 ; GCN-NEXT:    v_sub_i32_e32 v4, vcc, 0, v1
 ; GCN-NEXT:    v_ashrrev_i32_e32 v5, 31, v0
 ; GCN-NEXT:    v_rcp_iflag_f32_e32 v3, v3
-; GCN-NEXT:    v_add_i32_e32 v0, vcc, v5, v0
+; GCN-NEXT:    v_add_i32_e32 v0, vcc, v0, v5
 ; GCN-NEXT:    v_xor_b32_e32 v0, v0, v5
 ; GCN-NEXT:    v_mul_f32_e32 v3, 0x4f7ffffe, v3
 ; GCN-NEXT:    v_cvt_u32_f32_e32 v3, v3
 ; GCN-NEXT:    v_xor_b32_e32 v2, v5, v2
 ; GCN-NEXT:    v_mul_lo_u32 v4, v4, v3
 ; GCN-NEXT:    v_mul_hi_u32 v4, v3, v4
 ; GCN-NEXT:    v_add_i32_e32 v3, vcc, v3, v4
 ; GCN-NEXT:    v_mul_hi_u32 v3, v0, v3
 ; GCN-NEXT:    v_mul_lo_u32 v4, v3, v1
 ; GCN-NEXT:    v_add_i32_e32 v5, vcc, 1, v3
 ; GCN-NEXT:    v_sub_i32_e32 v0, vcc, v0, v4
-; GCN-NEXT:    v_subrev_i32_e32 v4, vcc, v1, v0
+; GCN-NEXT:    v_sub_i32_e32 v4, vcc, v0, v1
 ; GCN-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v1
 ; GCN-NEXT:    v_cndmask_b32_e32 v3, v3, v5, vcc
 ; GCN-NEXT:    v_cndmask_b32_e32 v0, v0, v4, vcc
 ; GCN-NEXT:    v_add_i32_e32 v4, vcc, 1, v3
 ; GCN-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v1
 ; GCN-NEXT:    v_cndmask_b32_e32 v0, v3, v4, vcc
 ; GCN-NEXT:    v_xor_b32_e32 v0, v0, v2
-; GCN-NEXT:    v_subrev_i32_e32 v0, vcc, v2, v0
+; GCN-NEXT:    v_sub_i32_e32 v0, vcc, v0, v2
 ; GCN-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GCN-NEXT:    s_endpgm
 ;
 ; TONGA-LABEL: sdiv_i32:
 ; TONGA:       ; %bb.0:
 ; TONGA-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; TONGA-NEXT:    s_mov_b32 s7, 0xf000
 ; TONGA-NEXT:    s_mov_b32 s6, -1
 ; TONGA-NEXT:    s_mov_b32 s10, s6
 ; TONGA-NEXT:    s_mov_b32 s11, s7
 ; TONGA-NEXT:    s_waitcnt lgkmcnt(0)
 ; TONGA-NEXT:    s_mov_b32 s8, s2
 ; TONGA-NEXT:    s_mov_b32 s9, s3
 ; TONGA-NEXT:    buffer_load_dwordx2 v[0:1], off, s[8:11], 0
 ; TONGA-NEXT:    s_mov_b32 s4, s0
 ; TONGA-NEXT:    s_mov_b32 s5, s1
 ; TONGA-NEXT:    s_waitcnt vmcnt(0)
 ; TONGA-NEXT:    v_ashrrev_i32_e32 v2, 31, v1
-; TONGA-NEXT:    v_add_u32_e32 v1, vcc, v2, v1
+; TONGA-NEXT:    v_add_u32_e32 v1, vcc, v1, v2
 ; TONGA-NEXT:    v_xor_b32_e32 v1, v1, v2
 ; TONGA-NEXT:    v_cvt_f32_u32_e32 v3, v1
 ; TONGA-NEXT:    v_sub_u32_e32 v4, vcc, 0, v1
 ; TONGA-NEXT:    v_ashrrev_i32_e32 v5, 31, v0
 ; TONGA-NEXT:    v_rcp_iflag_f32_e32 v3, v3
-; TONGA-NEXT:    v_add_u32_e32 v0, vcc, v5, v0
+; TONGA-NEXT:    v_add_u32_e32 v0, vcc, v0, v5
 ; TONGA-NEXT:    v_xor_b32_e32 v0, v0, v5
 ; TONGA-NEXT:    v_mul_f32_e32 v3, 0x4f7ffffe, v3
 ; TONGA-NEXT:    v_cvt_u32_f32_e32 v3, v3
 ; TONGA-NEXT:    v_xor_b32_e32 v2, v5, v2
 ; TONGA-NEXT:    v_mul_lo_u32 v4, v4, v3
 ; TONGA-NEXT:    v_mul_hi_u32 v4, v3, v4
 ; TONGA-NEXT:    v_add_u32_e32 v3, vcc, v3, v4
 ; TONGA-NEXT:    v_mul_hi_u32 v3, v0, v3
 ; TONGA-NEXT:    v_mul_lo_u32 v4, v3, v1
 ; TONGA-NEXT:    v_add_u32_e32 v5, vcc, 1, v3
 ; TONGA-NEXT:    v_sub_u32_e32 v0, vcc, v0, v4
-; TONGA-NEXT:    v_subrev_u32_e32 v4, vcc, v1, v0
+; TONGA-NEXT:    v_sub_u32_e32 v4, vcc, v0, v1
 ; TONGA-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v1
 ; TONGA-NEXT:    v_cndmask_b32_e32 v3, v3, v5, vcc
 ; TONGA-NEXT:    v_cndmask_b32_e32 v0, v0, v4, vcc
 ; TONGA-NEXT:    v_add_u32_e32 v4, vcc, 1, v3
 ; TONGA-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v1
 ; TONGA-NEXT:    v_cndmask_b32_e32 v0, v3, v4, vcc
 ; TONGA-NEXT:    v_xor_b32_e32 v0, v0, v2
-; TONGA-NEXT:    v_subrev_u32_e32 v0, vcc, v2, v0
+; TONGA-NEXT:    v_sub_u32_e32 v0, vcc, v0, v2
 ; TONGA-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; TONGA-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: sdiv_i32:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_mov_b32 s7, 0xf000
 ; GFX9-NEXT:    s_mov_b32 s6, -1

 ; GCN-NEXT:    s_mov_b32 s2, 0x98a1930b
 ; GCN-NEXT:    s_mov_b32 s4, s0
 ; GCN-NEXT:    s_mov_b32 s5, s1
 ; GCN-NEXT:    s_waitcnt vmcnt(0)
 ; GCN-NEXT:    v_mul_hi_i32 v1, v0, s2
 ; GCN-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
 ; GCN-NEXT:    v_lshrrev_b32_e32 v1, 31, v0
 ; GCN-NEXT:    v_ashrrev_i32_e32 v0, 11, v0
-; GCN-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
+; GCN-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; GCN-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GCN-NEXT:    s_endpgm
 ;
 ; TONGA-LABEL: slow_sdiv_i32_3435:
 ; TONGA:       ; %bb.0:
 ; TONGA-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; TONGA-NEXT:    s_mov_b32 s7, 0xf000
 ; TONGA-NEXT:    s_mov_b32 s6, -1
 ; TONGA-NEXT:    s_mov_b32 s10, s6
 ; TONGA-NEXT:    s_mov_b32 s11, s7
 ; TONGA-NEXT:    s_waitcnt lgkmcnt(0)
 ; TONGA-NEXT:    s_mov_b32 s8, s2
 ; TONGA-NEXT:    s_mov_b32 s9, s3
 ; TONGA-NEXT:    buffer_load_dword v0, off, s[8:11], 0
 ; TONGA-NEXT:    s_mov_b32 s2, 0x98a1930b
 ; TONGA-NEXT:    s_mov_b32 s4, s0
 ; TONGA-NEXT:    s_mov_b32 s5, s1
 ; TONGA-NEXT:    s_waitcnt vmcnt(0)
 ; TONGA-NEXT:    v_mul_hi_i32 v1, v0, s2
 ; TONGA-NEXT:    v_add_u32_e32 v0, vcc, v1, v0
 ; TONGA-NEXT:    v_lshrrev_b32_e32 v1, 31, v0
 ; TONGA-NEXT:    v_ashrrev_i32_e32 v0, 11, v0
-; TONGA-NEXT:    v_add_u32_e32 v0, vcc, v1, v0
+; TONGA-NEXT:    v_add_u32_e32 v0, vcc, v0, v1
 ; TONGA-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; TONGA-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: slow_sdiv_i32_3435:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_mov_b32 s7, 0xf000
 ; GFX9-NEXT:    s_mov_b32 s6, -1

 ; GCN-NEXT:    s_mov_b32 s8, s2
 ; GCN-NEXT:    s_mov_b32 s9, s3
 ; GCN-NEXT:    buffer_load_dwordx4 v[0:3], off, s[8:11], 0
 ; GCN-NEXT:    s_mov_b32 s4, s0
 ; GCN-NEXT:    s_mov_b32 s5, s1
 ; GCN-NEXT:    s_waitcnt vmcnt(0)
 ; GCN-NEXT:    v_ashrrev_i32_e32 v5, 31, v2
 ; GCN-NEXT:    v_ashrrev_i32_e32 v7, 31, v3
-; GCN-NEXT:    v_add_i32_e32 v2, vcc, v5, v2
-; GCN-NEXT:    v_add_i32_e32 v3, vcc, v7, v3
+; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v5
+; GCN-NEXT:    v_add_i32_e32 v3, vcc, v3, v7
 ; GCN-NEXT:    v_ashrrev_i32_e32 v4, 31, v0
 ; GCN-NEXT:    v_ashrrev_i32_e32 v6, 31, v1
 ; GCN-NEXT:    v_xor_b32_e32 v2, v2, v5
 ; GCN-NEXT:    v_xor_b32_e32 v3, v3, v7
 ; GCN-NEXT:    v_xor_b32_e32 v8, v4, v5
 ; GCN-NEXT:    v_xor_b32_e32 v9, v6, v7
 ; GCN-NEXT:    v_cvt_f32_u32_e32 v5, v2
 ; GCN-NEXT:    v_cvt_f32_u32_e32 v7, v3
 ; GCN-NEXT:    v_sub_i32_e32 v10, vcc, 0, v2
 ; GCN-NEXT:    v_rcp_iflag_f32_e32 v5, v5
 ; GCN-NEXT:    v_rcp_iflag_f32_e32 v7, v7
 ; GCN-NEXT:    v_sub_i32_e32 v11, vcc, 0, v3
 ; GCN-NEXT:    v_mul_f32_e32 v5, 0x4f7ffffe, v5
 ; GCN-NEXT:    v_mul_f32_e32 v7, 0x4f7ffffe, v7
 ; GCN-NEXT:    v_cvt_u32_f32_e32 v5, v5
 ; GCN-NEXT:    v_cvt_u32_f32_e32 v7, v7
-; GCN-NEXT:    v_add_i32_e32 v0, vcc, v4, v0
+; GCN-NEXT:    v_add_i32_e32 v0, vcc, v0, v4
 ; GCN-NEXT:    v_mul_lo_u32 v10, v10, v5
 ; GCN-NEXT:    v_mul_lo_u32 v11, v11, v7
-; GCN-NEXT:    v_add_i32_e32 v1, vcc, v6, v1
+; GCN-NEXT:    v_add_i32_e32 v1, vcc, v1, v6
 ; GCN-NEXT:    v_xor_b32_e32 v0, v0, v4
 ; GCN-NEXT:    v_mul_hi_u32 v4, v5, v10
 ; GCN-NEXT:    v_xor_b32_e32 v1, v1, v6
 ; GCN-NEXT:    v_mul_hi_u32 v6, v7, v11
 ; GCN-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
-; GCN-NEXT:    v_add_i32_e32 v5, vcc, v6, v7
+; GCN-NEXT:    v_add_i32_e32 v5, vcc, v7, v6
 ; GCN-NEXT:    v_mul_hi_u32 v4, v0, v4
 ; GCN-NEXT:    v_mul_hi_u32 v5, v1, v5
 ; GCN-NEXT:    v_mul_lo_u32 v6, v4, v2
 ; GCN-NEXT:    v_mul_lo_u32 v10, v5, v3
 ; GCN-NEXT:    v_add_i32_e32 v7, vcc, 1, v4
 ; GCN-NEXT:    v_sub_i32_e32 v0, vcc, v0, v6
 ; GCN-NEXT:    v_sub_i32_e32 v1, vcc, v1, v10
 ; GCN-NEXT:    v_add_i32_e32 v11, vcc, 1, v5
 ; GCN-NEXT:    v_cmp_ge_u32_e64 s[0:1], v0, v2
 ; GCN-NEXT:    v_cmp_ge_u32_e64 s[2:3], v1, v3
-; GCN-NEXT:    v_subrev_i32_e32 v6, vcc, v2, v0
+; GCN-NEXT:    v_sub_i32_e32 v6, vcc, v0, v2
 ; GCN-NEXT:    v_cndmask_b32_e64 v4, v4, v7, s[0:1]
-; GCN-NEXT:    v_subrev_i32_e32 v7, vcc, v3, v1
+; GCN-NEXT:    v_sub_i32_e32 v7, vcc, v1, v3
 ; GCN-NEXT:    v_cndmask_b32_e64 v5, v5, v11, s[2:3]
 ; GCN-NEXT:    v_cndmask_b32_e64 v0, v0, v6, s[0:1]
 ; GCN-NEXT:    v_add_i32_e32 v6, vcc, 1, v4
 ; GCN-NEXT:    v_cndmask_b32_e64 v1, v1, v7, s[2:3]
 ; GCN-NEXT:    v_add_i32_e32 v7, vcc, 1, v5
 ; GCN-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v2
 ; GCN-NEXT:    v_cndmask_b32_e32 v0, v4, v6, vcc
 ; GCN-NEXT:    v_cmp_ge_u32_e32 vcc, v1, v3
 ; GCN-NEXT:    v_cndmask_b32_e32 v1, v5, v7, vcc
 ; GCN-NEXT:    v_xor_b32_e32 v0, v0, v8
 ; GCN-NEXT:    v_xor_b32_e32 v1, v1, v9
-; GCN-NEXT:    v_subrev_i32_e32 v0, vcc, v8, v0
-; GCN-NEXT:    v_subrev_i32_e32 v1, vcc, v9, v1
+; GCN-NEXT:    v_sub_i32_e32 v0, vcc, v0, v8
+; GCN-NEXT:    v_sub_i32_e32 v1, vcc, v1, v9
 ; GCN-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; GCN-NEXT:    s_endpgm
 ;
 ; TONGA-LABEL: sdiv_v2i32:
 ; TONGA:       ; %bb.0:
 ; TONGA-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; TONGA-NEXT:    s_mov_b32 s7, 0xf000
 ; TONGA-NEXT:    s_mov_b32 s6, -1
 ; TONGA-NEXT:    s_mov_b32 s10, s6
 ; TONGA-NEXT:    s_mov_b32 s11, s7
 ; TONGA-NEXT:    s_waitcnt lgkmcnt(0)
 ; TONGA-NEXT:    s_mov_b32 s8, s2
 ; TONGA-NEXT:    s_mov_b32 s9, s3
 ; TONGA-NEXT:    buffer_load_dwordx4 v[0:3], off, s[8:11], 0
 ; TONGA-NEXT:    s_mov_b32 s4, s0
 ; TONGA-NEXT:    s_mov_b32 s5, s1
 ; TONGA-NEXT:    s_waitcnt vmcnt(0)
 ; TONGA-NEXT:    v_ashrrev_i32_e32 v5, 31, v2
 ; TONGA-NEXT:    v_ashrrev_i32_e32 v7, 31, v3
-; TONGA-NEXT:    v_add_u32_e32 v2, vcc, v5, v2
-; TONGA-NEXT:    v_add_u32_e32 v3, vcc, v7, v3
+; TONGA-NEXT:    v_add_u32_e32 v2, vcc, v2, v5
+; TONGA-NEXT:    v_add_u32_e32 v3, vcc, v3, v7
 ; TONGA-NEXT:    v_ashrrev_i32_e32 v4, 31, v0
 ; TONGA-NEXT:    v_ashrrev_i32_e32 v6, 31, v1
 ; TONGA-NEXT:    v_xor_b32_e32 v2, v2, v5
 ; TONGA-NEXT:    v_xor_b32_e32 v3, v3, v7
 ; TONGA-NEXT:    v_xor_b32_e32 v8, v4, v5
 ; TONGA-NEXT:    v_xor_b32_e32 v9, v6, v7
 ; TONGA-NEXT:    v_cvt_f32_u32_e32 v5, v2
 ; TONGA-NEXT:    v_cvt_f32_u32_e32 v7, v3
 ; TONGA-NEXT:    v_sub_u32_e32 v10, vcc, 0, v2
 ; TONGA-NEXT:    v_rcp_iflag_f32_e32 v5, v5
 ; TONGA-NEXT:    v_rcp_iflag_f32_e32 v7, v7
 ; TONGA-NEXT:    v_sub_u32_e32 v11, vcc, 0, v3
 ; TONGA-NEXT:    v_mul_f32_e32 v5, 0x4f7ffffe, v5
 ; TONGA-NEXT:    v_mul_f32_e32 v7, 0x4f7ffffe, v7
 ; TONGA-NEXT:    v_cvt_u32_f32_e32 v5, v5
 ; TONGA-NEXT:    v_cvt_u32_f32_e32 v7, v7
-; TONGA-NEXT:    v_add_u32_e32 v0, vcc, v4, v0
+; TONGA-NEXT:    v_add_u32_e32 v0, vcc, v0, v4
 ; TONGA-NEXT:    v_mul_lo_u32 v10, v10, v5
 ; TONGA-NEXT:    v_mul_lo_u32 v11, v11, v7
-; TONGA-NEXT:    v_add_u32_e32 v1, vcc, v6, v1
+; TONGA-NEXT:    v_add_u32_e32 v1, vcc, v1, v6
 ; TONGA-NEXT:    v_xor_b32_e32 v0, v0, v4
 ; TONGA-NEXT:    v_mul_hi_u32 v4, v5, v10
 ; TONGA-NEXT:    v_xor_b32_e32 v1, v1, v6
 ; TONGA-NEXT:    v_mul_hi_u32 v6, v7, v11
 ; TONGA-NEXT:    v_add_u32_e32 v4, vcc, v5, v4
-; TONGA-NEXT:    v_add_u32_e32 v5, vcc, v6, v7
+; TONGA-NEXT:    v_add_u32_e32 v5, vcc, v7, v6
 ; TONGA-NEXT:    v_mul_hi_u32 v4, v0, v4
 ; TONGA-NEXT:    v_mul_hi_u32 v5, v1, v5
 ; TONGA-NEXT:    v_mul_lo_u32 v6, v4, v2
 ; TONGA-NEXT:    v_mul_lo_u32 v10, v5, v3
 ; TONGA-NEXT:    v_add_u32_e32 v7, vcc, 1, v4
 ; TONGA-NEXT:    v_sub_u32_e32 v0, vcc, v0, v6
 ; TONGA-NEXT:    v_sub_u32_e32 v1, vcc, v1, v10
 ; TONGA-NEXT:    v_add_u32_e32 v11, vcc, 1, v5
 ; TONGA-NEXT:    v_cmp_ge_u32_e64 s[0:1], v0, v2
 ; TONGA-NEXT:    v_cmp_ge_u32_e64 s[2:3], v1, v3
-; TONGA-NEXT:    v_subrev_u32_e32 v6, vcc, v2, v0
+; TONGA-NEXT:    v_sub_u32_e32 v6, vcc, v0, v2
 ; TONGA-NEXT:    v_cndmask_b32_e64 v4, v4, v7, s[0:1]
-; TONGA-NEXT:    v_subrev_u32_e32 v7, vcc, v3, v1
+; TONGA-NEXT:    v_sub_u32_e32 v7, vcc, v1, v3
 ; TONGA-NEXT:    v_cndmask_b32_e64 v5, v5, v11, s[2:3]
 ; TONGA-NEXT:    v_cndmask_b32_e64 v0, v0, v6, s[0:1]
 ; TONGA-NEXT:    v_add_u32_e32 v6, vcc, 1, v4
 ; TONGA-NEXT:    v_cndmask_b32_e64 v1, v1, v7, s[2:3]
 ; TONGA-NEXT:    v_add_u32_e32 v7, vcc, 1, v5
 ; TONGA-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v2
 ; TONGA-NEXT:    v_cndmask_b32_e32 v0, v4, v6, vcc
 ; TONGA-NEXT:    v_cmp_ge_u32_e32 vcc, v1, v3
 ; TONGA-NEXT:    v_cndmask_b32_e32 v1, v5, v7, vcc
 ; TONGA-NEXT:    v_xor_b32_e32 v0, v0, v8
 ; TONGA-NEXT:    v_xor_b32_e32 v1, v1, v9
-; TONGA-NEXT:    v_subrev_u32_e32 v0, vcc, v8, v0
-; TONGA-NEXT:    v_subrev_u32_e32 v1, vcc, v9, v1
+; TONGA-NEXT:    v_sub_u32_e32 v0, vcc, v0, v8
+; TONGA-NEXT:    v_sub_u32_e32 v1, vcc, v1, v9
 ; TONGA-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
 ; TONGA-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: sdiv_v2i32:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX9-NEXT:    s_mov_b32 s2, -1

 ; GCN-NEXT:    s_mov_b32 s8, s0
 ; GCN-NEXT:    s_mov_b32 s9, s1
 ; GCN-NEXT:    s_waitcnt vmcnt(1)
 ; GCN-NEXT:    v_ashrrev_i32_e32 v8, 31, v0
 ; GCN-NEXT:    s_waitcnt vmcnt(0)
 ; GCN-NEXT:    v_ashrrev_i32_e32 v9, 31, v4
 ; GCN-NEXT:    v_ashrrev_i32_e32 v11, 31, v5
 ; GCN-NEXT:    v_ashrrev_i32_e32 v10, 31, v1
-; GCN-NEXT:    v_add_i32_e32 v4, vcc, v9, v4
-; GCN-NEXT:    v_add_i32_e32 v5, vcc, v11, v5
+; GCN-NEXT:    v_add_i32_e32 v4, vcc, v4, v9
+; GCN-NEXT:    v_add_i32_e32 v5, vcc, v5, v11
 ; GCN-NEXT:    v_ashrrev_i32_e32 v13, 31, v6
-; GCN-NEXT:    v_add_i32_e32 v0, vcc, v8, v0
-; GCN-NEXT:    v_add_i32_e32 v1, vcc, v10, v1
+; GCN-NEXT:    v_add_i32_e32 v0, vcc, v0, v8
+; GCN-NEXT:    v_add_i32_e32 v1, vcc, v1, v10
 ; GCN-NEXT:    v_xor_b32_e32 v4, v4, v9
 ; GCN-NEXT:    v_xor_b32_e32 v5, v5, v11
 ; GCN-NEXT:    v_ashrrev_i32_e32 v12, 31, v2
 ; GCN-NEXT:    v_xor_b32_e32 v15, v8, v9
 ; GCN-NEXT:    v_xor_b32_e32 v16, v10, v11
-; GCN-NEXT:    v_add_i32_e32 v6, vcc, v13, v6
+; GCN-NEXT:    v_add_i32_e32 v6, vcc, v6, v13
 ; GCN-NEXT:    v_xor_b32_e32 v0, v0, v8
 ; GCN-NEXT:    v_xor_b32_e32 v1, v1, v10
 ; GCN-NEXT:    v_cvt_f32_u32_e32 v8, v4
 ; GCN-NEXT:    v_cvt_f32_u32_e32 v10, v5
-; GCN-NEXT:    v_add_i32_e32 v2, vcc, v12, v2
+; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v12
 ; GCN-NEXT:    v_xor_b32_e32 v6, v6, v13
 ; GCN-NEXT:    v_xor_b32_e32 v17, v12, v13
 ; GCN-NEXT:    v_xor_b32_e32 v2, v2, v12
 ; GCN-NEXT:    v_cvt_f32_u32_e32 v12, v6
 ; GCN-NEXT:    v_rcp_iflag_f32_e32 v8, v8
 ; GCN-NEXT:    v_rcp_iflag_f32_e32 v10, v10
 ; GCN-NEXT:    v_sub_i32_e32 v9, vcc, 0, v4
 ; GCN-NEXT:    v_rcp_iflag_f32_e32 v12, v12
 ; GCN-NEXT:    v_mul_f32_e32 v8, 0x4f7ffffe, v8
 ; GCN-NEXT:    v_mul_f32_e32 v10, 0x4f7ffffe, v10
 ; GCN-NEXT:    v_cvt_u32_f32_e32 v8, v8
 ; GCN-NEXT:    v_cvt_u32_f32_e32 v10, v10
 ; GCN-NEXT:    v_mul_f32_e32 v12, 0x4f7ffffe, v12
 ; GCN-NEXT:    v_cvt_u32_f32_e32 v12, v12
 ; GCN-NEXT:    v_sub_i32_e32 v11, vcc, 0, v5
 ; GCN-NEXT:    v_mul_lo_u32 v9, v9, v8
 ; GCN-NEXT:    v_mul_lo_u32 v11, v11, v10
 ; GCN-NEXT:    v_sub_i32_e32 v13, vcc, 0, v6
 ; GCN-NEXT:    v_mul_lo_u32 v13, v13, v12
 ; GCN-NEXT:    v_mul_hi_u32 v9, v8, v9
 ; GCN-NEXT:    v_mul_hi_u32 v11, v10, v11
 ; GCN-NEXT:    v_ashrrev_i32_e32 v14, 31, v7
-; GCN-NEXT:    v_add_i32_e32 v7, vcc, v14, v7
+; GCN-NEXT:    v_add_i32_e32 v7, vcc, v7, v14
 ; GCN-NEXT:    v_mul_hi_u32 v13, v12, v13
 ; GCN-NEXT:    v_xor_b32_e32 v7, v7, v14
 ; GCN-NEXT:    v_cvt_f32_u32_e32 v18, v7
 ; GCN-NEXT:    v_add_i32_e32 v8, vcc, v8, v9
-; GCN-NEXT:    v_add_i32_e32 v9, vcc, v11, v10
+; GCN-NEXT:    v_add_i32_e32 v9, vcc, v10, v11
 ; GCN-NEXT:    v_mul_hi_u32 v8, v0, v8
 ; GCN-NEXT:    v_mul_hi_u32 v9, v1, v9
 ; GCN-NEXT:    v_add_i32_e32 v10, vcc, v12, v13
 ; GCN-NEXT:    v_mul_hi_u32 v10, v2, v10
 ; GCN-NEXT:    v_rcp_iflag_f32_e32 v18, v18
 ; GCN-NEXT:    v_mul_lo_u32 v11, v8, v4
 ; GCN-NEXT:    v_mul_lo_u32 v13, v9, v5
 ; GCN-NEXT:    v_mul_lo_u32 v21, v10, v6
 ; GCN-NEXT:    v_mul_f32_e32 v18, 0x4f7ffffe, v18
 ; GCN-NEXT:    v_cvt_u32_f32_e32 v18, v18
 ; GCN-NEXT:    v_sub_i32_e32 v0, vcc, v0, v11
 ; GCN-NEXT:    v_sub_i32_e32 v1, vcc, v1, v13
 ; GCN-NEXT:    v_add_i32_e32 v12, vcc, 1, v8
 ; GCN-NEXT:    v_add_i32_e32 v20, vcc, 1, v9
 ; GCN-NEXT:    v_cmp_ge_u32_e64 s[0:1], v0, v4
 ; GCN-NEXT:    v_cmp_ge_u32_e64 s[2:3], v1, v5
 ; GCN-NEXT:    v_sub_i32_e32 v2, vcc, v2, v21
-; GCN-NEXT:    v_subrev_i32_e32 v11, vcc, v4, v0
+; GCN-NEXT:    v_sub_i32_e32 v11, vcc, v0, v4
 ; GCN-NEXT:    v_cndmask_b32_e64 v8, v8, v12, s[0:1]
-; GCN-NEXT:    v_subrev_i32_e32 v12, vcc, v5, v1
+; GCN-NEXT:    v_sub_i32_e32 v12, vcc, v1, v5
 ; GCN-NEXT:    v_cndmask_b32_e64 v9, v9, v20, s[2:3]
 ; GCN-NEXT:    v_sub_i32_e32 v19, vcc, 0, v7
 ; GCN-NEXT:    v_add_i32_e32 v22, vcc, 1, v10
-; GCN-NEXT:    v_subrev_i32_e32 v13, vcc, v6, v2
+; GCN-NEXT:    v_sub_i32_e32 v13, vcc, v2, v6
 ; GCN-NEXT:    v_cndmask_b32_e64 v0, v0, v11, s[0:1]
 ; GCN-NEXT:    v_add_i32_e32 v11, vcc, 1, v8
 ; GCN-NEXT:    v_cndmask_b32_e64 v1, v1, v12, s[2:3]
 ; GCN-NEXT:    v_add_i32_e32 v12, vcc, 1, v9
 ; GCN-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v4
 ; GCN-NEXT:    v_mul_lo_u32 v4, v19, v18
 ; GCN-NEXT:    v_cndmask_b32_e32 v0, v8, v11, vcc
 ; GCN-NEXT:    v_cmp_ge_u32_e32 vcc, v1, v5
 ; GCN-NEXT:    v_ashrrev_i32_e32 v8, 31, v3
 ; GCN-NEXT:    v_mul_hi_u32 v4, v18, v4
 ; GCN-NEXT:    v_cndmask_b32_e32 v1, v9, v12, vcc
-; GCN-NEXT:    v_add_i32_e32 v3, vcc, v8, v3
+; GCN-NEXT:    v_add_i32_e32 v3, vcc, v3, v8
 ; GCN-NEXT:    v_xor_b32_e32 v3, v3, v8
-; GCN-NEXT:    v_add_i32_e32 v4, vcc, v4, v18
+; GCN-NEXT:    v_add_i32_e32 v4, vcc, v18, v4
 ; GCN-NEXT:    v_cmp_ge_u32_e64 s[4:5], v2, v6
 ; GCN-NEXT:    v_mul_hi_u32 v4, v3, v4
 ; GCN-NEXT:    v_cndmask_b32_e64 v10, v10, v22, s[4:5]
 ; GCN-NEXT:    v_xor_b32_e32 v0, v0, v15
 ; GCN-NEXT:    v_xor_b32_e32 v1, v1, v16
 ; GCN-NEXT:    v_cndmask_b32_e64 v2, v2, v13, s[4:5]
-; GCN-NEXT:    v_subrev_i32_e32 v0, vcc, v15, v0
-; GCN-NEXT:    v_subrev_i32_e32 v1, vcc, v16, v1
+; GCN-NEXT:    v_sub_i32_e32 v0, vcc, v0, v15
+; GCN-NEXT:    v_sub_i32_e32 v1, vcc, v1, v16
 ; GCN-NEXT:    v_add_i32_e32 v5, vcc, 1, v10
 ; GCN-NEXT:    v_cmp_ge_u32_e32 vcc, v2, v6
 ; GCN-NEXT:    v_cndmask_b32_e32 v2, v10, v5, vcc
 ; GCN-NEXT:    v_mul_lo_u32 v5, v4, v7
 ; GCN-NEXT:    v_xor_b32_e32 v2, v2, v17
-; GCN-NEXT:    v_subrev_i32_e32 v2, vcc, v17, v2
+; GCN-NEXT:    v_sub_i32_e32 v2, vcc, v2, v17
 ; GCN-NEXT:    v_sub_i32_e32 v3, vcc, v3, v5
 ; GCN-NEXT:    v_xor_b32_e32 v6, v8, v14
 ; GCN-NEXT:    v_add_i32_e32 v5, vcc, 1, v4
-; GCN-NEXT:    v_subrev_i32_e32 v8, vcc, v7, v3
+; GCN-NEXT:    v_sub_i32_e32 v8, vcc, v3, v7
 ; GCN-NEXT:    v_cmp_ge_u32_e32 vcc, v3, v7
 ; GCN-NEXT:    v_cndmask_b32_e32 v4, v4, v5, vcc
 ; GCN-NEXT:    v_cndmask_b32_e32 v3, v3, v8, vcc
 ; GCN-NEXT:    v_add_i32_e32 v5, vcc, 1, v4
 ; GCN-NEXT:    v_cmp_ge_u32_e32 vcc, v3, v7
 ; GCN-NEXT:    v_cndmask_b32_e32 v3, v4, v5, vcc
 ; GCN-NEXT:    v_xor_b32_e32 v3, v3, v6
-; GCN-NEXT:    v_subrev_i32_e32 v3, vcc, v6, v3
+; GCN-NEXT:    v_sub_i32_e32 v3, vcc, v3, v6
 ; GCN-NEXT:    buffer_store_dwordx4 v[0:3], off, s[8:11], 0
 ; GCN-NEXT:    s_endpgm
 ;
 ; TONGA-LABEL: sdiv_v4i32:
 ; TONGA:       ; %bb.0:
 ; TONGA-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; TONGA-NEXT:    s_mov_b32 s11, 0xf000
 ; TONGA-NEXT:    s_mov_b32 s10, -1
 ; TONGA-NEXT:    s_mov_b32 s6, s10
 ; TONGA-NEXT:    s_mov_b32 s7, s11
 ; TONGA-NEXT:    s_waitcnt lgkmcnt(0)
 ; TONGA-NEXT:    s_mov_b32 s4, s2
 ; TONGA-NEXT:    s_mov_b32 s5, s3
 ; TONGA-NEXT:    buffer_load_dwordx4 v[0:3], off, s[4:7], 0
 ; TONGA-NEXT:    buffer_load_dwordx4 v[4:7], off, s[4:7], 0 offset:16
 ; TONGA-NEXT:    s_mov_b32 s8, s0
 ; TONGA-NEXT:    s_mov_b32 s9, s1
 ; TONGA-NEXT:    s_waitcnt vmcnt(1)
 ; TONGA-NEXT:    v_ashrrev_i32_e32 v8, 31, v0
 ; TONGA-NEXT:    s_waitcnt vmcnt(0)
 ; TONGA-NEXT:    v_ashrrev_i32_e32 v9, 31, v4
 ; TONGA-NEXT:    v_ashrrev_i32_e32 v11, 31, v5
 ; TONGA-NEXT:    v_ashrrev_i32_e32 v10, 31, v1
-; TONGA-NEXT:    v_add_u32_e32 v4, vcc, v9, v4
-; TONGA-NEXT:    v_add_u32_e32 v5, vcc, v11, v5
+; TONGA-NEXT:    v_add_u32_e32 v4, vcc, v4, v9
+; TONGA-NEXT:    v_add_u32_e32 v5, vcc, v5, v11
 ; TONGA-NEXT:    v_ashrrev_i32_e32 v13, 31, v6
-; TONGA-NEXT:    v_add_u32_e32 v0, vcc, v8, v0
-; TONGA-NEXT:    v_add_u32_e32 v1, vcc, v10, v1
+; TONGA-NEXT:    v_add_u32_e32 v0, vcc, v0, v8
+; TONGA-NEXT:    v_add_u32_e32 v1, vcc, v1, v10
 ; TONGA-NEXT:    v_xor_b32_e32 v4, v4, v9
 ; TONGA-NEXT:    v_xor_b32_e32 v5, v5, v11
 ; TONGA-NEXT:    v_ashrrev_i32_e32 v12, 31, v2
 ; TONGA-NEXT:    v_xor_b32_e32 v15, v8, v9
 ; TONGA-NEXT:    v_xor_b32_e32 v16, v10, v11
-; TONGA-NEXT:    v_add_u32_e32 v6, vcc, v13, v6
+; TONGA-NEXT:    v_add_u32_e32 v6, vcc, v6, v13
 ; TONGA-NEXT:    v_xor_b32_e32 v0, v0, v8
 ; TONGA-NEXT:    v_xor_b32_e32 v1, v1, v10
 ; TONGA-NEXT:    v_cvt_f32_u32_e32 v8, v4
 ; TONGA-NEXT:    v_cvt_f32_u32_e32 v10, v5
-; TONGA-NEXT:    v_add_u32_e32 v2, vcc, v12, v2
+; TONGA-NEXT:    v_add_u32_e32 v2, vcc, v2, v12
 ; TONGA-NEXT:    v_xor_b32_e32 v6, v6, v13
 ; TONGA-NEXT:    v_xor_b32_e32 v17, v12, v13
 ; TONGA-NEXT:    v_xor_b32_e32 v2, v2, v12
 ; TONGA-NEXT:    v_cvt_f32_u32_e32 v12, v6
 ; TONGA-NEXT:    v_rcp_iflag_f32_e32 v8, v8
 ; TONGA-NEXT:    v_rcp_iflag_f32_e32 v10, v10
 ; TONGA-NEXT:    v_sub_u32_e32 v9, vcc, 0, v4
 ; TONGA-NEXT:    v_rcp_iflag_f32_e32 v12, v12
 ; TONGA-NEXT:    v_mul_f32_e32 v8, 0x4f7ffffe, v8
 ; TONGA-NEXT:    v_mul_f32_e32 v10, 0x4f7ffffe, v10
 ; TONGA-NEXT:    v_cvt_u32_f32_e32 v8, v8
 ; TONGA-NEXT:    v_cvt_u32_f32_e32 v10, v10
 ; TONGA-NEXT:    v_mul_f32_e32 v12, 0x4f7ffffe, v12
 ; TONGA-NEXT:    v_cvt_u32_f32_e32 v12, v12
 ; TONGA-NEXT:    v_sub_u32_e32 v11, vcc, 0, v5
 ; TONGA-NEXT:    v_mul_lo_u32 v9, v9, v8
 ; TONGA-NEXT:    v_mul_lo_u32 v11, v11, v10
 ; TONGA-NEXT:    v_sub_u32_e32 v13, vcc, 0, v6
 ; TONGA-NEXT:    v_mul_lo_u32 v13, v13, v12
 ; TONGA-NEXT:    v_mul_hi_u32 v9, v8, v9
 ; TONGA-NEXT:    v_mul_hi_u32 v11, v10, v11
 ; TONGA-NEXT:    v_ashrrev_i32_e32 v14, 31, v7
-; TONGA-NEXT:    v_add_u32_e32 v7, vcc, v14, v7
+; TONGA-NEXT:    v_add_u32_e32 v7, vcc, v7, v14
 ; TONGA-NEXT:    v_mul_hi_u32 v13, v12, v13
 ; TONGA-NEXT:    v_xor_b32_e32 v7, v7, v14
 ; TONGA-NEXT:    v_cvt_f32_u32_e32 v18, v7
 ; TONGA-NEXT:    v_add_u32_e32 v8, vcc, v8, v9
-; TONGA-NEXT:    v_add_u32_e32 v9, vcc, v11, v10
+; TONGA-NEXT:    v_add_u32_e32 v9, vcc, v10, v11
 ; TONGA-NEXT:    v_mul_hi_u32 v8, v0, v8
 ; TONGA-NEXT:    v_mul_hi_u32 v9, v1, v9
 ; TONGA-NEXT:    v_add_u32_e32 v10, vcc, v12, v13
 ; TONGA-NEXT:    v_mul_hi_u32 v10, v2, v10
 ; TONGA-NEXT:    v_rcp_iflag_f32_e32 v18, v18
 ; TONGA-NEXT:    v_mul_lo_u32 v11, v8, v4
 ; TONGA-NEXT:    v_mul_lo_u32 v13, v9, v5
 ; TONGA-NEXT:    v_mul_lo_u32 v21, v10, v6
 ; TONGA-NEXT:    v_mul_f32_e32 v18, 0x4f7ffffe, v18
 ; TONGA-NEXT:    v_cvt_u32_f32_e32 v18, v18
 ; TONGA-NEXT:    v_sub_u32_e32 v0, vcc, v0, v11
 ; TONGA-NEXT:    v_sub_u32_e32 v1, vcc, v1, v13
 ; TONGA-NEXT:    v_add_u32_e32 v12, vcc, 1, v8
 ; TONGA-NEXT:    v_add_u32_e32 v20, vcc, 1, v9
 ; TONGA-NEXT:    v_cmp_ge_u32_e64 s[0:1], v0, v4
 ; TONGA-NEXT:    v_cmp_ge_u32_e64 s[2:3], v1, v5
 ; TONGA-NEXT:    v_sub_u32_e32 v2, vcc, v2, v21
-; TONGA-NEXT:    v_subrev_u32_e32 v11, vcc, v4, v0
+; TONGA-NEXT:    v_sub_u32_e32 v11, vcc, v0, v4
 ; TONGA-NEXT:    v_cndmask_b32_e64 v8, v8, v12, s[0:1]
-; TONGA-NEXT:    v_subrev_u32_e32 v12, vcc, v5, v1
+; TONGA-NEXT:    v_sub_u32_e32 v12, vcc, v1, v5
 ; TONGA-NEXT:    v_cndmask_b32_e64 v9, v9, v20, s[2:3]
 ; TONGA-NEXT:    v_sub_u32_e32 v19, vcc, 0, v7
 ; TONGA-NEXT:    v_add_u32_e32 v22, vcc, 1, v10
-; TONGA-NEXT:    v_subrev_u32_e32 v13, vcc, v6, v2
+; TONGA-NEXT:    v_sub_u32_e32 v13, vcc, v2, v6
 ; TONGA-NEXT:    v_cndmask_b32_e64 v0, v0, v11, s[0:1]
 ; TONGA-NEXT:    v_add_u32_e32 v11, vcc, 1, v8
 ; TONGA-NEXT:    v_cndmask_b32_e64 v1, v1, v12, s[2:3]
 ; TONGA-NEXT:    v_add_u32_e32 v12, vcc, 1, v9
 ; TONGA-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v4
 ; TONGA-NEXT:    v_mul_lo_u32 v4, v19, v18
 ; TONGA-NEXT:    v_cndmask_b32_e32 v0, v8, v11, vcc
 ; TONGA-NEXT:    v_cmp_ge_u32_e32 vcc, v1, v5
 ; TONGA-NEXT:    v_ashrrev_i32_e32 v8, 31, v3
 ; TONGA-NEXT:    v_mul_hi_u32 v4, v18, v4
 ; TONGA-NEXT:    v_cndmask_b32_e32 v1, v9, v12, vcc
-; TONGA-NEXT:    v_add_u32_e32 v3, vcc, v8, v3
+; TONGA-NEXT:    v_add_u32_e32 v3, vcc, v3, v8
 ; TONGA-NEXT:    v_xor_b32_e32 v3, v3, v8
-; TONGA-NEXT:    v_add_u32_e32 v4, vcc, v4, v18
+; TONGA-NEXT:    v_add_u32_e32 v4, vcc, v18, v4
 ; TONGA-NEXT:    v_cmp_ge_u32_e64 s[4:5], v2, v6
 ; TONGA-NEXT:    v_mul_hi_u32 v4, v3, v4
 ; TONGA-NEXT:    v_cndmask_b32_e64 v10, v10, v22, s[4:5]
 ; TONGA-NEXT:    v_xor_b32_e32 v0, v0, v15
 ; TONGA-NEXT:    v_xor_b32_e32 v1, v1, v16
 ; TONGA-NEXT:    v_cndmask_b32_e64 v2, v2, v13, s[4:5]
-; TONGA-NEXT:    v_subrev_u32_e32 v0, vcc, v15, v0
-; TONGA-NEXT:    v_subrev_u32_e32 v1, vcc, v16, v1
+; TONGA-NEXT:    v_sub_u32_e32 v0, vcc, v0, v15
+; TONGA-NEXT:    v_sub_u32_e32 v1, vcc, v1, v16
 ; TONGA-NEXT:    v_add_u32_e32 v5, vcc, 1, v10
 ; TONGA-NEXT:    v_cmp_ge_u32_e32 vcc, v2, v6
 ; TONGA-NEXT:    v_cndmask_b32_e32 v2, v10, v5, vcc
 ; TONGA-NEXT:    v_mul_lo_u32 v5, v4, v7
 ; TONGA-NEXT:    v_xor_b32_e32 v2, v2, v17
-; TONGA-NEXT:    v_subrev_u32_e32 v2, vcc, v17, v2
+; TONGA-NEXT:    v_sub_u32_e32 v2, vcc, v2, v17
 ; TONGA-NEXT:    v_sub_u32_e32 v3, vcc, v3, v5
 ; TONGA-NEXT:    v_xor_b32_e32 v6, v8, v14
 ; TONGA-NEXT:    v_add_u32_e32 v5, vcc, 1, v4
-; TONGA-NEXT:    v_subrev_u32_e32 v8, vcc, v7, v3
+; TONGA-NEXT:    v_sub_u32_e32 v8, vcc, v3, v7
 ; TONGA-NEXT:    v_cmp_ge_u32_e32 vcc, v3, v7
 ; TONGA-NEXT:    v_cndmask_b32_e32 v4, v4, v5, vcc
 ; TONGA-NEXT:    v_cndmask_b32_e32 v3, v3, v8, vcc
 ; TONGA-NEXT:    v_add_u32_e32 v5, vcc, 1, v4
 ; TONGA-NEXT:    v_cmp_ge_u32_e32 vcc, v3, v7
 ; TONGA-NEXT:    v_cndmask_b32_e32 v3, v4, v5, vcc
 ; TONGA-NEXT:    v_xor_b32_e32 v3, v3, v6
-; TONGA-NEXT:    v_subrev_u32_e32 v3, vcc, v6, v3
+; TONGA-NEXT:    v_sub_u32_e32 v3, vcc, v3, v6
 ; TONGA-NEXT:    buffer_store_dwordx4 v[0:3], off, s[8:11], 0
 ; TONGA-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: sdiv_v4i32:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX9-NEXT:    s_mov_b32 s2, -1

 ; GCN-NEXT:    v_xor_b32_e32 v0, v0, v1
 ; GCN-NEXT:    v_mul_lo_u32 v4, v4, v3
 ; GCN-NEXT:    v_mul_hi_u32 v4, v3, v4
 ; GCN-NEXT:    v_add_i32_e32 v3, vcc, v3, v4
 ; GCN-NEXT:    v_mul_hi_u32 v3, v5, v3
 ; GCN-NEXT:    v_mul_lo_u32 v1, v3, v2
 ; GCN-NEXT:    v_add_i32_e32 v4, vcc, 1, v3
 ; GCN-NEXT:    v_sub_i32_e32 v1, vcc, v5, v1
-; GCN-NEXT:    v_subrev_i32_e32 v5, vcc, v2, v1
+; GCN-NEXT:    v_sub_i32_e32 v5, vcc, v1, v2
 ; GCN-NEXT:    v_cmp_ge_u32_e32 vcc, v1, v2
 ; GCN-NEXT:    v_cndmask_b32_e32 v3, v3, v4, vcc
 ; GCN-NEXT:    v_cndmask_b32_e32 v1, v1, v5, vcc
 ; GCN-NEXT:    v_add_i32_e32 v4, vcc, 1, v3
 ; GCN-NEXT:    v_cmp_ge_u32_e32 vcc, v1, v2
 ; GCN-NEXT:    v_cndmask_b32_e32 v1, v3, v4, vcc
 ; GCN-NEXT:    v_xor_b32_e32 v1, v1, v0
-; GCN-NEXT:    v_subrev_i32_e32 v0, vcc, v0, v1
+; GCN-NEXT:    v_sub_i32_e32 v0, vcc, v1, v0
 ; GCN-NEXT:    v_bfe_i32 v0, v0, 0, 25
 ; GCN-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GCN-NEXT:    s_endpgm
 ;
 ; TONGA-LABEL: v_sdiv_i25:
 ; TONGA:       ; %bb.0:
 ; TONGA-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; TONGA-NEXT:    s_mov_b32 s7, 0xf000

 ; TONGA-NEXT:    v_xor_b32_e32 v0, v0, v1
 ; TONGA-NEXT:    v_mul_lo_u32 v4, v4, v3
 ; TONGA-NEXT:    v_mul_hi_u32 v4, v3, v4
 ; TONGA-NEXT:    v_add_u32_e32 v3, vcc, v3, v4
 ; TONGA-NEXT:    v_mul_hi_u32 v3, v5, v3
 ; TONGA-NEXT:    v_mul_lo_u32 v1, v3, v2
 ; TONGA-NEXT:    v_add_u32_e32 v4, vcc, 1, v3
 ; TONGA-NEXT:    v_sub_u32_e32 v1, vcc, v5, v1
-; TONGA-NEXT:    v_subrev_u32_e32 v5, vcc, v2, v1
+; TONGA-NEXT:    v_sub_u32_e32 v5, vcc, v1, v2
 ; TONGA-NEXT:    v_cmp_ge_u32_e32 vcc, v1, v2
 ; TONGA-NEXT:    v_cndmask_b32_e32 v3, v3, v4, vcc
 ; TONGA-NEXT:    v_cndmask_b32_e32 v1, v1, v5, vcc
 ; TONGA-NEXT:    v_add_u32_e32 v4, vcc, 1, v3
 ; TONGA-NEXT:    v_cmp_ge_u32_e32 vcc, v1, v2
 ; TONGA-NEXT:    v_cndmask_b32_e32 v1, v3, v4, vcc
 ; TONGA-NEXT:    v_xor_b32_e32 v1, v1, v0
-; TONGA-NEXT:    v_subrev_u32_e32 v0, vcc, v0, v1
+; TONGA-NEXT:    v_sub_u32_e32 v0, vcc, v1, v0
 ; TONGA-NEXT:    v_bfe_i32 v0, v0, 0, 25
 ; TONGA-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; TONGA-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: v_sdiv_i25:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_mov_b32 s7, 0xf000

 ; GCN-NEXT:    v_lshrrev_b32_e32 v5, 31, v1
 ; GCN-NEXT:    v_ashrrev_i32_e32 v1, 12, v1
 ; GCN-NEXT:    v_lshrrev_b32_e32 v6, 31, v2
 ; GCN-NEXT:    v_ashrrev_i32_e32 v2, 12, v2
 ; GCN-NEXT:    v_lshrrev_b32_e32 v7, 31, v3
 ; GCN-NEXT:    v_ashrrev_i32_e32 v3, 12, v3
 ; GCN-NEXT:    v_add_i32_e32 v0, vcc, v0, v4
 ; GCN-NEXT:    v_add_i32_e32 v1, vcc, v1, v5
-; GCN-NEXT:    v_add_i32_e32 v2, vcc, v6, v2
+; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v6
 ; GCN-NEXT:    v_add_i32_e32 v3, vcc, v3, v7
 ; GCN-NEXT:    buffer_store_dwordx4 v[0:3], off, s[4:7], 0
 ; GCN-NEXT:    s_endpgm
 ;
 ; TONGA-LABEL: scalarize_mulhs_4xi32:
 ; TONGA:       ; %bb.0:
 ; TONGA-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; TONGA-NEXT:    s_mov_b32 s7, 0xf000

 ; TONGA-NEXT:    v_lshrrev_b32_e32 v5, 31, v1
 ; TONGA-NEXT:    v_ashrrev_i32_e32 v1, 12, v1
 ; TONGA-NEXT:    v_lshrrev_b32_e32 v6, 31, v2
 ; TONGA-NEXT:    v_ashrrev_i32_e32 v2, 12, v2
 ; TONGA-NEXT:    v_lshrrev_b32_e32 v7, 31, v3
 ; TONGA-NEXT:    v_ashrrev_i32_e32 v3, 12, v3
 ; TONGA-NEXT:    v_add_u32_e32 v0, vcc, v0, v4
 ; TONGA-NEXT:    v_add_u32_e32 v1, vcc, v1, v5
-; TONGA-NEXT:    v_add_u32_e32 v2, vcc, v6, v2
+; TONGA-NEXT:    v_add_u32_e32 v2, vcc, v2, v6
 ; TONGA-NEXT:    v_add_u32_e32 v3, vcc, v3, v7
 ; TONGA-NEXT:    buffer_store_dwordx4 v[0:3], off, s[4:7], 0
 ; TONGA-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: scalarize_mulhs_4xi32:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_mov_b32 s7, 0xf000

llvm/test/CodeGen/AMDGPU/sdiv64.ll

 ; GCN-NEXT:    s_add_u32 s2, s2, s12
 ; GCN-NEXT:    s_mov_b32 s13, s12
 ; GCN-NEXT:    v_mul_lo_u32 v2, s4, v1
 ; GCN-NEXT:    v_mul_hi_u32 v3, s4, v0
 ; GCN-NEXT:    v_mul_lo_u32 v5, s5, v0
 ; GCN-NEXT:    v_mul_lo_u32 v4, s4, v0
 ; GCN-NEXT:    s_addc_u32 s3, s3, s12
 ; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
-; GCN-NEXT:    v_add_i32_e32 v2, vcc, v5, v2
+; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v5
 ; GCN-NEXT:    v_mul_hi_u32 v3, v0, v4
 ; GCN-NEXT:    v_mul_lo_u32 v5, v0, v2
 ; GCN-NEXT:    v_mul_hi_u32 v7, v0, v2
 ; GCN-NEXT:    v_mul_lo_u32 v6, v1, v4
 ; GCN-NEXT:    v_mul_hi_u32 v4, v1, v4
 ; GCN-NEXT:    v_mul_hi_u32 v8, v1, v2
 ; GCN-NEXT:    v_add_i32_e32 v3, vcc, v3, v5
 ; GCN-NEXT:    v_addc_u32_e32 v5, vcc, 0, v7, vcc

 ; GCN-NEXT:    v_addc_u32_e32 v0, vcc, v3, v0, vcc
 ; GCN-NEXT:    v_addc_u32_e32 v2, vcc, 0, v5, vcc
 ; GCN-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; GCN-NEXT:    v_addc_u32_e32 v1, vcc, 0, v2, vcc
 ; GCN-NEXT:    v_mul_lo_u32 v2, s10, v1
 ; GCN-NEXT:    v_mul_hi_u32 v3, s10, v0
 ; GCN-NEXT:    v_mul_lo_u32 v4, s11, v0
 ; GCN-NEXT:    v_mov_b32_e32 v5, s11
-; GCN-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
+; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; GCN-NEXT:    v_mul_lo_u32 v3, s10, v0
-; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
+; GCN-NEXT:    v_add_i32_e32 v2, vcc, v4, v2
 ; GCN-NEXT:    v_sub_i32_e32 v4, vcc, s3, v2
 ; GCN-NEXT:    v_sub_i32_e32 v3, vcc, s2, v3
 ; GCN-NEXT:    v_subb_u32_e64 v4, s[0:1], v4, v5, vcc
 ; GCN-NEXT:    v_subrev_i32_e64 v5, s[0:1], s10, v3
 ; GCN-NEXT:    v_subbrev_u32_e64 v4, s[0:1], 0, v4, s[0:1]
 ; GCN-NEXT:    v_cmp_le_u32_e64 s[0:1], s11, v4
 ; GCN-NEXT:    v_cndmask_b32_e64 v6, 0, -1, s[0:1]
 ; GCN-NEXT:    v_cmp_le_u32_e64 s[0:1], s10, v5

 ; GCN-NEXT:    v_ashrrev_i32_e32 v0, 30, v0
 ; GCN-NEXT:    v_or_b32_e32 v0, 1, v0
 ; GCN-NEXT:    v_mul_f32_e32 v2, v3, v4
 ; GCN-NEXT:    v_trunc_f32_e32 v2, v2
 ; GCN-NEXT:    v_mad_f32 v3, -v2, v1, v3
 ; GCN-NEXT:    v_cvt_i32_f32_e32 v2, v2
 ; GCN-NEXT:    v_cmp_ge_f32_e64 vcc, |v3|, |v1|
 ; GCN-NEXT:    v_cndmask_b32_e32 v0, 0, v0, vcc
-; GCN-NEXT:    v_add_i32_e32 v0, vcc, v2, v0
+; GCN-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GCN-NEXT:    v_bfe_i32 v0, v0, 0, 24
 ; GCN-NEXT:    v_ashrrev_i32_e32 v1, 31, v0
 ; GCN-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GCN-NEXT:    buffer_store_short v1, off, s[0:3], 0 offset:4
 ; GCN-NEXT:    s_endpgm
 ;
 ; GCN-IR-LABEL: s_test_sdiv24_48:
 ; GCN-IR:       ; %bb.0: ; %_udiv-special-cases

 ; GCN-NEXT:    v_mac_f32_e32 v0, 0xcf800000, v1
 ; GCN-NEXT:    v_cvt_u32_f32_e32 v1, v1
 ; GCN-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; GCN-NEXT:    v_mul_lo_u32 v2, s4, v1
 ; GCN-NEXT:    v_mul_hi_u32 v3, s4, v0
 ; GCN-NEXT:    v_mul_lo_u32 v5, s5, v0
 ; GCN-NEXT:    v_mul_lo_u32 v4, s4, v0
 ; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
-; GCN-NEXT:    v_add_i32_e32 v2, vcc, v5, v2
+; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v5
 ; GCN-NEXT:    v_mul_hi_u32 v3, v0, v4
 ; GCN-NEXT:    v_mul_lo_u32 v5, v0, v2
 ; GCN-NEXT:    v_mul_hi_u32 v7, v0, v2
 ; GCN-NEXT:    v_mul_hi_u32 v6, v1, v4
 ; GCN-NEXT:    v_mul_lo_u32 v4, v1, v4
 ; GCN-NEXT:    v_mul_hi_u32 v8, v1, v2
 ; GCN-NEXT:    v_add_i32_e32 v3, vcc, v3, v5
 ; GCN-NEXT:    v_addc_u32_e32 v5, vcc, 0, v7, vcc
 ; GCN-NEXT:    v_mul_lo_u32 v2, v1, v2
 ; GCN-NEXT:    v_add_i32_e32 v3, vcc, v3, v4
 ; GCN-NEXT:    v_addc_u32_e32 v3, vcc, v5, v6, vcc
 ; GCN-NEXT:    v_addc_u32_e32 v4, vcc, 0, v8, vcc
 ; GCN-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
 ; GCN-NEXT:    v_addc_u32_e32 v3, vcc, 0, v4, vcc
 ; GCN-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GCN-NEXT:    v_addc_u32_e32 v1, vcc, v1, v3, vcc
 ; GCN-NEXT:    v_mul_lo_u32 v2, s4, v1
 ; GCN-NEXT:    v_mul_hi_u32 v3, s4, v0
 ; GCN-NEXT:    v_mul_lo_u32 v4, s5, v0
 ; GCN-NEXT:    s_mov_b32 s5, s1
-; GCN-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
+; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; GCN-NEXT:    v_mul_lo_u32 v3, s4, v0
-; GCN-NEXT:    v_add_i32_e32 v2, vcc, v4, v2
+; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
 ; GCN-NEXT:    v_mul_lo_u32 v6, v0, v2
 ; GCN-NEXT:    v_mul_hi_u32 v7, v0, v3
 ; GCN-NEXT:    v_mul_hi_u32 v8, v0, v2
 ; GCN-NEXT:    v_mul_hi_u32 v5, v1, v3
 ; GCN-NEXT:    v_mul_lo_u32 v3, v1, v3
 ; GCN-NEXT:    v_mul_hi_u32 v4, v1, v2
 ; GCN-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
 ; GCN-NEXT:    v_addc_u32_e32 v7, vcc, 0, v8, vcc

 ; GCN-NEXT:    v_mul_hi_u32 v0, v0, 24
 ; GCN-NEXT:    v_mul_hi_u32 v1, v1, 24
 ; GCN-NEXT:    v_mov_b32_e32 v4, s3
 ; GCN-NEXT:    s_mov_b32 s4, s0
 ; GCN-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GCN-NEXT:    v_addc_u32_e32 v0, vcc, 0, v1, vcc
 ; GCN-NEXT:    v_mul_lo_u32 v1, s3, v0
 ; GCN-NEXT:    v_mul_hi_u32 v2, s2, v0
-; GCN-NEXT:    v_add_i32_e32 v1, vcc, v2, v1
+; GCN-NEXT:    v_add_i32_e32 v1, vcc, v1, v2
 ; GCN-NEXT:    v_mul_lo_u32 v2, s2, v0
 ; GCN-NEXT:    v_sub_i32_e32 v3, vcc, 0, v1
 ; GCN-NEXT:    v_sub_i32_e32 v2, vcc, 24, v2
 ; GCN-NEXT:    v_subb_u32_e64 v3, s[0:1], v3, v4, vcc
 ; GCN-NEXT:    v_subrev_i32_e64 v4, s[0:1], s2, v2
 ; GCN-NEXT:    v_subbrev_u32_e64 v3, s[0:1], 0, v3, s[0:1]
 ; GCN-NEXT:    v_cmp_le_u32_e64 s[0:1], s3, v3
 ; GCN-NEXT:    v_cndmask_b32_e64 v5, 0, -1, s[0:1]

llvm/test/CodeGen/AMDGPU/sdwa-peephole.ll

 ; NOSDWA-NEXT:    s_waitcnt lgkmcnt(0)
 ; NOSDWA-NEXT:    v_mov_b32_e32 v0, s2
 ; NOSDWA-NEXT:    v_mov_b32_e32 v1, s3
 ; NOSDWA-NEXT:    flat_load_dword v2, v[0:1]
 ; NOSDWA-NEXT:    v_mov_b32_e32 v0, s0
 ; NOSDWA-NEXT:    v_mov_b32_e32 v1, s1
 ; NOSDWA-NEXT:    s_waitcnt vmcnt(0)
 ; NOSDWA-NEXT:    v_lshrrev_b32_e32 v3, 16, v2
-; NOSDWA-NEXT:    v_subrev_u32_e32 v2, vcc, v2, v3
+; NOSDWA-NEXT:    v_sub_u32_e32 v2, vcc, v3, v2
 ; NOSDWA-NEXT:    flat_store_dword v[0:1], v2
 ; NOSDWA-NEXT:    s_endpgm
 ;
 ; GFX89-LABEL: sub_shr_i32:
 ; GFX89:       ; %bb.0:
 ; GFX89-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; GFX89-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX89-NEXT:    v_mov_b32_e32 v0, s2
 ; GFX89-NEXT:    v_mov_b32_e32 v1, s3
 ; GFX89-NEXT:    flat_load_dword v2, v[0:1]
 ; GFX89-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX89-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX89-NEXT:    s_waitcnt vmcnt(0)
-; GFX89-NEXT:    v_subrev_u32_sdwa v2, vcc, v2, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX89-NEXT:    v_sub_u32_sdwa v2, vcc, v2, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
 ; GFX89-NEXT:    flat_store_dword v[0:1], v2
 ; GFX89-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: sub_shr_i32:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; GFX9-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)

llvm/test/CodeGen/AMDGPU/sgpr-spills-split-regalloc.ll

 ; GCN-NEXT:    v_writelane_b32 v4, s34, 0
 ; GCN-NEXT:    v_writelane_b32 v4, s35, 1
 ; GCN-NEXT:    v_writelane_b32 v4, s36, 2
 ; GCN-NEXT:    v_writelane_b32 v4, s37, 3
 ; GCN-NEXT:    v_mov_b32_e32 v5, v3
 ; GCN-NEXT:    v_mov_b32_e32 v3, v1
 ; GCN-NEXT:    ; implicit-def: $sgpr4
 ; GCN-NEXT:    ; implicit-def: $sgpr4
-; GCN-NEXT:    ; kill: def $vgpr3 killed $vgpr3 killed $exec
 ; GCN-NEXT:    ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
 ; GCN-NEXT:    v_mov_b32_e32 v1, v3
 ; GCN-NEXT:    ; implicit-def: $sgpr4
 ; GCN-NEXT:    ; implicit-def: $sgpr4
-; GCN-NEXT:    ; kill: def $vgpr5 killed $vgpr5 killed $exec
 ; GCN-NEXT:    ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
 ; GCN-NEXT:    v_mov_b32_e32 v3, v5
 ; GCN-NEXT:    ; implicit-def: $sgpr4_sgpr5
 ; GCN-NEXT:    ; implicit-def: $sgpr4_sgpr5
 ; GCN-NEXT:    flat_load_dwordx4 v[5:8], v[2:3]
 ; GCN-NEXT:    s_waitcnt vmcnt(0)
 ; GCN-NEXT:    buffer_store_dword v5, off, s[0:3], s32 offset:448 ; 4-byte Folded Spill
 ; GCN-NEXT:    s_waitcnt vmcnt(0)

llvm/test/CodeGen/AMDGPU/shl.ll

 ; SI-NEXT:    s_waitcnt lgkmcnt(0)
 ; SI-NEXT:    s_mov_b32 s8, s2
 ; SI-NEXT:    s_mov_b32 s9, s3
 ; SI-NEXT:    buffer_load_dwordx4 v[0:3], off, s[8:11], 0
 ; SI-NEXT:    buffer_load_dwordx4 v[4:7], off, s[8:11], 0 offset:16
 ; SI-NEXT:    s_mov_b32 s4, s0
 ; SI-NEXT:    s_mov_b32 s5, s1
 ; SI-NEXT:    s_waitcnt vmcnt(0)
-; SI-NEXT:    v_lshlrev_b32_e32 v3, v7, v3
-; SI-NEXT:    v_lshlrev_b32_e32 v2, v6, v2
-; SI-NEXT:    v_lshlrev_b32_e32 v1, v5, v1
+; SI-NEXT:    v_lshl_b32_e32 v3, v3, v7
+; SI-NEXT:    v_lshl_b32_e32 v2, v2, v6
+; SI-NEXT:    v_lshl_b32_e32 v1, v1, v5
 ; SI-NEXT:    v_lshl_b32_e32 v0, v0, v4
 ; SI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[4:7], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: shl_v4i32:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x24
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)

 ; SI-NEXT:    s_waitcnt lgkmcnt(0)
 ; SI-NEXT:    s_mov_b32 s8, s2
 ; SI-NEXT:    s_mov_b32 s9, s3
 ; SI-NEXT:    buffer_load_ushort v0, off, s[8:11], 0
 ; SI-NEXT:    buffer_load_ushort v1, off, s[8:11], 0 offset:2
 ; SI-NEXT:    s_mov_b32 s4, s0
 ; SI-NEXT:    s_mov_b32 s5, s1
 ; SI-NEXT:    s_waitcnt vmcnt(0)
-; SI-NEXT:    v_lshlrev_b32_e32 v0, v1, v0
+; SI-NEXT:    v_lshl_b32_e32 v0, v0, v1
 ; SI-NEXT:    buffer_store_short v0, off, s[4:7], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: shl_i16:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; VI-NEXT:    s_mov_b32 s7, 0xf000
 ; VI-NEXT:    s_mov_b32 s6, -1

llvm/test/CodeGen/AMDGPU/splitkit-getsubrangeformask.ll

define amdgpu_gs void @_amdgpu_gs_main(i32 inreg %primShaderTableAddrLow, <31 x i32> inreg %userData) {
   ; CHECK-NEXT:   [[V_OR_B32_e64_64:%[0-9]+]]:vgpr_32 = V_OR_B32_e64 [[V_OR_B32_e64_63]], [[V_ADD_U32_e64_28]], implicit $exec
   ; CHECK-NEXT:   [[V_ADD_U32_e64_30:%[0-9]+]]:vgpr_32 = V_ADD_U32_e64 -593, [[BUFFER_LOAD_FORMAT_X_IDXEN]], 0, implicit $exec
   ; CHECK-NEXT:   [[V_OR_B32_e64_65:%[0-9]+]]:vgpr_32 = V_OR_B32_e64 [[V_OR_B32_e64_64]], [[V_ADD_U32_e64_29]], implicit $exec
   ; CHECK-NEXT:   [[S_LOAD_DWORDX8_IMM:%[0-9]+]]:sgpr_256 = S_LOAD_DWORDX8_IMM undef %564:sreg_64, 0, 0 :: (invariant load (s256) from `ptr addrspace(4) undef`, addrspace 4)
   ; CHECK-NEXT:   [[V_OR_B32_e64_66:%[0-9]+]]:vgpr_32 = V_OR_B32_e64 [[V_OR_B32_e64_65]], [[V_ADD_U32_e64_30]], implicit $exec
   ; CHECK-NEXT:   [[S_ADD_I32_24:%[0-9]+]]:sreg_32 = S_ADD_I32 [[S_BUFFER_LOAD_DWORD_IMM8]], -594, implicit-def dead $scc
   ; CHECK-NEXT:   [[V_OR_B32_e64_67:%[0-9]+]]:vgpr_32 = V_OR_B32_e64 [[S_ADD_I32_24]], [[V_OR_B32_e64_66]], implicit $exec
   ; CHECK-NEXT:   [[V_CMP_EQ_U32_e64_:%[0-9]+]]:sreg_32_xm0_xexec = V_CMP_EQ_U32_e64 0, [[V_OR_B32_e64_67]], implicit $exec
-  ; CHECK-NEXT:   undef %693.sub3:vreg_128 = V_CNDMASK_B32_e64 0, 0, 0, 1, [[V_CMP_EQ_U32_e64_]], implicit $exec
-  ; CHECK-NEXT:   IMAGE_STORE_V4_V2_gfx10 %693, undef %578:vreg_64, [[S_LOAD_DWORDX8_IMM]], 15, 1, -1, 0, 0, 0, 0, 0, 0, implicit $exec :: (dereferenceable store (s128), addrspace 7)
+  ; CHECK-NEXT:   undef %624.sub3:vreg_128 = V_CNDMASK_B32_e64 0, 0, 0, 1, [[V_CMP_EQ_U32_e64_]], implicit $exec
+  ; CHECK-NEXT:   IMAGE_STORE_V4_V2_gfx10 %624, undef %578:vreg_64, [[S_LOAD_DWORDX8_IMM]], 15, 1, -1, 0, 0, 0, 0, 0, 0, implicit $exec :: (dereferenceable store (s128), addrspace 7)
   ; CHECK-NEXT:   S_ENDPGM 0
 .expVert:
   %0 = extractelement <31 x i32> %userData, i64 2
   %1 = extractelement <31 x i32> %userData, i64 3
   %2 = extractelement <31 x i32> %userData, i64 4
   %3 = extractelement <31 x i32> %userData, i64 7
   %4 = extractelement <31 x i32> %userData, i64 8
   %5 = extractelement <31 x i32> %userData, i64 9

llvm/test/CodeGen/AMDGPU/sra.ll

 ; SI-NEXT:    s_waitcnt lgkmcnt(0)
 ; SI-NEXT:    s_mov_b32 s8, s2
 ; SI-NEXT:    s_mov_b32 s9, s3
 ; SI-NEXT:    buffer_load_dwordx4 v[0:3], off, s[8:11], 0
 ; SI-NEXT:    buffer_load_dwordx4 v[4:7], off, s[8:11], 0 offset:16
 ; SI-NEXT:    s_mov_b32 s4, s0
 ; SI-NEXT:    s_mov_b32 s5, s1
 ; SI-NEXT:    s_waitcnt vmcnt(0)
-; SI-NEXT:    v_ashrrev_i32_e32 v3, v7, v3
-; SI-NEXT:    v_ashrrev_i32_e32 v2, v6, v2
-; SI-NEXT:    v_ashrrev_i32_e32 v1, v5, v1
+; SI-NEXT:    v_ashr_i32_e32 v3, v3, v7
+; SI-NEXT:    v_ashr_i32_e32 v2, v2, v6
+; SI-NEXT:    v_ashr_i32_e32 v1, v1, v5
 ; SI-NEXT:    v_ashr_i32_e32 v0, v0, v4
 ; SI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[4:7], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: ashr_v4i32:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; VI-NEXT:    s_mov_b32 s7, 0xf000

llvm/test/CodeGen/AMDGPU/srem64.ll

 ; GCN-NEXT:    v_mac_f32_e32 v0, 0xcf800000, v1
 ; GCN-NEXT:    v_cvt_u32_f32_e32 v1, v1
 ; GCN-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; GCN-NEXT:    v_mul_lo_u32 v2, s0, v1
 ; GCN-NEXT:    v_mul_hi_u32 v3, s0, v0
 ; GCN-NEXT:    v_mul_lo_u32 v5, s1, v0
 ; GCN-NEXT:    v_mul_lo_u32 v4, s0, v0
 ; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
-; GCN-NEXT:    v_add_i32_e32 v2, vcc, v5, v2
+; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v5
 ; GCN-NEXT:    v_mul_hi_u32 v3, v0, v4
 ; GCN-NEXT:    v_mul_lo_u32 v5, v0, v2
 ; GCN-NEXT:    v_mul_hi_u32 v7, v0, v2
 ; GCN-NEXT:    v_mul_hi_u32 v6, v1, v4
 ; GCN-NEXT:    v_mul_lo_u32 v4, v1, v4
 ; GCN-NEXT:    v_mul_hi_u32 v8, v1, v2
 ; GCN-NEXT:    v_add_i32_e32 v3, vcc, v3, v5
 ; GCN-NEXT:    v_addc_u32_e32 v5, vcc, 0, v7, vcc

 ; GCN-NEXT:    v_addc_u32_e32 v0, vcc, v3, v0, vcc
 ; GCN-NEXT:    v_addc_u32_e32 v2, vcc, 0, v5, vcc
 ; GCN-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; GCN-NEXT:    v_addc_u32_e32 v1, vcc, 0, v2, vcc
 ; GCN-NEXT:    v_mul_lo_u32 v1, s12, v1
 ; GCN-NEXT:    v_mul_hi_u32 v2, s12, v0
 ; GCN-NEXT:    v_mul_lo_u32 v3, s13, v0
 ; GCN-NEXT:    v_mul_lo_u32 v0, s12, v0
-; GCN-NEXT:    v_add_i32_e32 v1, vcc, v2, v1
-; GCN-NEXT:    v_add_i32_e32 v1, vcc, v1, v3
+; GCN-NEXT:    v_add_i32_e32 v1, vcc, v1, v2
+; GCN-NEXT:    v_add_i32_e32 v1, vcc, v3, v1
 ; GCN-NEXT:    v_sub_i32_e32 v2, vcc, s11, v1
 ; GCN-NEXT:    v_mov_b32_e32 v3, s13
 ; GCN-NEXT:    v_sub_i32_e32 v0, vcc, s10, v0
 ; GCN-NEXT:    v_subb_u32_e64 v2, s[0:1], v2, v3, vcc
 ; GCN-NEXT:    v_subrev_i32_e64 v4, s[0:1], s12, v0
 ; GCN-NEXT:    v_subbrev_u32_e64 v5, s[2:3], 0, v2, s[0:1]
 ; GCN-NEXT:    v_cmp_le_u32_e64 s[2:3], s13, v5
 ; GCN-NEXT:    v_cndmask_b32_e64 v6, 0, -1, s[2:3]

 ; GCN-NEXT:    v_mac_f32_e32 v0, 0xcf800000, v1
 ; GCN-NEXT:    v_cvt_u32_f32_e32 v1, v1
 ; GCN-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; GCN-NEXT:    v_mul_lo_u32 v2, s0, v1
 ; GCN-NEXT:    v_mul_hi_u32 v3, s0, v0
 ; GCN-NEXT:    v_mul_lo_u32 v5, s1, v0
 ; GCN-NEXT:    v_mul_lo_u32 v4, s0, v0
 ; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
-; GCN-NEXT:    v_add_i32_e32 v2, vcc, v5, v2
+; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v5
 ; GCN-NEXT:    v_mul_hi_u32 v3, v0, v4
 ; GCN-NEXT:    v_mul_lo_u32 v5, v0, v2
 ; GCN-NEXT:    v_mul_hi_u32 v6, v0, v2
 ; GCN-NEXT:    v_mul_hi_u32 v7, v1, v2
 ; GCN-NEXT:    v_mul_lo_u32 v2, v1, v2
 ; GCN-NEXT:    v_add_i32_e32 v3, vcc, v3, v5
 ; GCN-NEXT:    v_addc_u32_e32 v5, vcc, 0, v6, vcc
 ; GCN-NEXT:    v_mul_lo_u32 v6, v1, v4

 ; GCN-NEXT:    v_addc_u32_e32 v0, vcc, v3, v0, vcc
 ; GCN-NEXT:    v_addc_u32_e32 v2, vcc, 0, v5, vcc
 ; GCN-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; GCN-NEXT:    v_addc_u32_e32 v1, vcc, 0, v2, vcc
 ; GCN-NEXT:    v_mul_lo_u32 v1, s12, v1
 ; GCN-NEXT:    v_mul_hi_u32 v2, s12, v0
 ; GCN-NEXT:    v_mul_lo_u32 v3, s13, v0
 ; GCN-NEXT:    v_mul_lo_u32 v0, s12, v0
-; GCN-NEXT:    v_add_i32_e32 v1, vcc, v2, v1
-; GCN-NEXT:    v_add_i32_e32 v1, vcc, v1, v3
+; GCN-NEXT:    v_add_i32_e32 v1, vcc, v1, v2
+; GCN-NEXT:    v_add_i32_e32 v1, vcc, v3, v1
 ; GCN-NEXT:    v_sub_i32_e32 v2, vcc, s15, v1
 ; GCN-NEXT:    v_mov_b32_e32 v3, s13
 ; GCN-NEXT:    v_sub_i32_e32 v0, vcc, s14, v0
 ; GCN-NEXT:    v_subb_u32_e64 v2, s[0:1], v2, v3, vcc
 ; GCN-NEXT:    v_subrev_i32_e64 v4, s[0:1], s12, v0
 ; GCN-NEXT:    v_subbrev_u32_e64 v5, s[2:3], 0, v2, s[0:1]
 ; GCN-NEXT:    v_cmp_le_u32_e64 s[2:3], s13, v5
 ; GCN-NEXT:    v_cndmask_b32_e64 v6, 0, -1, s[2:3]

 ; GCN-NEXT:    v_or_b32_e32 v5, 1, v5
 ; GCN-NEXT:    v_mul_f32_e32 v4, v3, v4
 ; GCN-NEXT:    v_trunc_f32_e32 v4, v4
 ; GCN-NEXT:    v_mad_f32 v3, -v4, v1, v3
 ; GCN-NEXT:    v_cvt_i32_f32_e32 v4, v4
 ; GCN-NEXT:    v_cmp_ge_f32_e64 vcc, |v3|, |v1|
 ; GCN-NEXT:    v_cndmask_b32_e32 v1, 0, v5, vcc
 ; GCN-NEXT:    s_mov_b32 s0, s4
-; GCN-NEXT:    v_add_i32_e32 v1, vcc, v4, v1
+; GCN-NEXT:    v_add_i32_e32 v1, vcc, v1, v4
 ; GCN-NEXT:    v_mul_lo_u32 v0, v1, v0
 ; GCN-NEXT:    s_mov_b32 s1, s5
 ; GCN-NEXT:    v_subrev_i32_e32 v0, vcc, v0, v2
 ; GCN-NEXT:    v_bfe_i32 v0, v0, 0, 24
 ; GCN-NEXT:    v_ashrrev_i32_e32 v1, 31, v0
 ; GCN-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GCN-NEXT:    buffer_store_short v1, off, s[0:3], 0 offset:4
 ; GCN-NEXT:    s_endpgm

 ; GCN-NEXT:    v_mac_f32_e32 v0, 0xcf800000, v1
 ; GCN-NEXT:    v_cvt_u32_f32_e32 v1, v1
 ; GCN-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; GCN-NEXT:    v_mul_lo_u32 v2, s2, v1
 ; GCN-NEXT:    v_mul_hi_u32 v3, s2, v0
 ; GCN-NEXT:    v_mul_lo_u32 v5, s3, v0
 ; GCN-NEXT:    v_mul_lo_u32 v4, s2, v0
 ; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
-; GCN-NEXT:    v_add_i32_e32 v2, vcc, v5, v2
+; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v5
 ; GCN-NEXT:    v_mul_hi_u32 v3, v0, v4
 ; GCN-NEXT:    v_mul_lo_u32 v5, v0, v2
 ; GCN-NEXT:    v_mul_hi_u32 v7, v0, v2
 ; GCN-NEXT:    v_mul_hi_u32 v6, v1, v4
 ; GCN-NEXT:    v_mul_lo_u32 v4, v1, v4
 ; GCN-NEXT:    v_mul_hi_u32 v8, v1, v2
 ; GCN-NEXT:    v_add_i32_e32 v3, vcc, v3, v5
 ; GCN-NEXT:    v_addc_u32_e32 v5, vcc, 0, v7, vcc

 ; GCN-NEXT:    v_mul_hi_u32 v0, v0, 24
 ; GCN-NEXT:    v_mul_hi_u32 v1, v1, 24
 ; GCN-NEXT:    v_mov_b32_e32 v3, s9
 ; GCN-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GCN-NEXT:    v_addc_u32_e32 v0, vcc, 0, v1, vcc
 ; GCN-NEXT:    v_mul_lo_u32 v1, s9, v0
 ; GCN-NEXT:    v_mul_hi_u32 v2, s8, v0
 ; GCN-NEXT:    v_mul_lo_u32 v0, s8, v0
-; GCN-NEXT:    v_add_i32_e32 v1, vcc, v2, v1
+; GCN-NEXT:    v_add_i32_e32 v1, vcc, v1, v2
 ; GCN-NEXT:    v_sub_i32_e32 v2, vcc, 0, v1
 ; GCN-NEXT:    v_sub_i32_e32 v0, vcc, 24, v0
 ; GCN-NEXT:    v_subb_u32_e64 v2, s[0:1], v2, v3, vcc
 ; GCN-NEXT:    v_subrev_i32_e64 v4, s[0:1], s8, v0
 ; GCN-NEXT:    v_subbrev_u32_e64 v5, s[2:3], 0, v2, s[0:1]
 ; GCN-NEXT:    v_cmp_le_u32_e64 s[2:3], s9, v5
 ; GCN-NEXT:    v_cndmask_b32_e64 v6, 0, -1, s[2:3]
 ; GCN-NEXT:    v_cmp_le_u32_e64 s[2:3], s8, v4

llvm/test/CodeGen/AMDGPU/srl.ll

 ; SI-NEXT:    s_mov_b32 s11, s7
 ; SI-NEXT:    s_waitcnt lgkmcnt(0)
 ; SI-NEXT:    s_mov_b32 s8, s2
 ; SI-NEXT:    s_mov_b32 s9, s3
 ; SI-NEXT:    buffer_load_dwordx2 v[0:1], off, s[8:11], 0
 ; SI-NEXT:    s_mov_b32 s4, s0
 ; SI-NEXT:    s_mov_b32 s5, s1
 ; SI-NEXT:    s_waitcnt vmcnt(0)
-; SI-NEXT:    v_lshrrev_b32_e32 v0, v1, v0
+; SI-NEXT:    v_lshr_b32_e32 v0, v0, v1
 ; SI-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: lshr_i32:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    s_load_dwordx2 s[4:5], s[2:3], 0x0

 ; SI-NEXT:    s_mov_b32 s8, s2
 ; SI-NEXT:    s_mov_b32 s9, s3
 ; SI-NEXT:    buffer_load_dwordx4 v[0:3], off, s[8:11], 0
 ; SI-NEXT:    buffer_load_dwordx4 v[4:7], off, s[8:11], 0 offset:16
 ; SI-NEXT:    s_mov_b32 s4, s0
 ; SI-NEXT:    s_mov_b32 s5, s1
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    v_lshr_b32_e32 v3, v3, v7
-; SI-NEXT:    v_lshrrev_b32_e32 v2, v6, v2
-; SI-NEXT:    v_lshrrev_b32_e32 v1, v5, v1
-; SI-NEXT:    v_lshrrev_b32_e32 v0, v4, v0
+; SI-NEXT:    v_lshr_b32_e32 v2, v2, v6
+; SI-NEXT:    v_lshr_b32_e32 v1, v1, v5
+; SI-NEXT:    v_lshr_b32_e32 v0, v0, v4
 ; SI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[4:7], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: lshr_v4i32:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x24
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    s_load_dwordx8 s[0:7], s[10:11], 0x0

llvm/test/CodeGen/AMDGPU/sub.ll

-; RUN: llc -amdgpu-scalarize-global-loads=false -march=amdgcn -mcpu=verde -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GCN,SI %s
-; RUN: llc -amdgpu-scalarize-global-loads=false -march=amdgcn -mcpu=fiji -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GCN,VI,GFX89 %s
-; RUN: llc -amdgpu-scalarize-global-loads=false -march=amdgcn -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GCN,GFX9,GFX89 %s
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -amdgpu-scalarize-global-loads=false -march=amdgcn -mcpu=verde -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GCN1 %s
+; RUN: llc -amdgpu-scalarize-global-loads=false -march=amdgcn -mcpu=fiji -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GCN2 %s
+; RUN: llc -amdgpu-scalarize-global-loads=false -march=amdgcn -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GCN3 %s
 
 declare i32 @llvm.amdgcn.workitem.id.x() nounwind readnone speculatable
 
-; GCN-LABEL: {{^}}s_sub_i32:
-; GCN: s_load_dwordx4 s[[[#LOAD:]]:{{[0-9]+}}]
-; GCN: s_sub_i32 s{{[0-9]+}}, s[[#LOAD + 2]], s[[#LOAD + 3]]
 define amdgpu_kernel void @s_sub_i32(ptr addrspace(1) %out, i32 %a, i32 %b) {
+; GCN1-LABEL: s_sub_i32:
+; GCN1:       ; %bb.0:
+; GCN1-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
+; GCN1-NEXT:    s_mov_b32 s7, 0xf000
+; GCN1-NEXT:    s_mov_b32 s6, -1
+; GCN1-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN1-NEXT:    s_mov_b32 s4, s0
+; GCN1-NEXT:    s_sub_i32 s0, s2, s3
+; GCN1-NEXT:    s_mov_b32 s5, s1
+; GCN1-NEXT:    v_mov_b32_e32 v0, s0
+; GCN1-NEXT:    buffer_store_dword v0, off, s[4:7], 0
+; GCN1-NEXT:    s_endpgm
+;
+; GCN2-LABEL: s_sub_i32:
+; GCN2:       ; %bb.0:
+; GCN2-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GCN2-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN2-NEXT:    s_sub_i32 s2, s2, s3
+; GCN2-NEXT:    v_mov_b32_e32 v0, s0
+; GCN2-NEXT:    v_mov_b32_e32 v1, s1
+; GCN2-NEXT:    v_mov_b32_e32 v2, s2
+; GCN2-NEXT:    flat_store_dword v[0:1], v2
+; GCN2-NEXT:    s_endpgm
+;
+; GCN3-LABEL: s_sub_i32:
+; GCN3:       ; %bb.0:
+; GCN3-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GCN3-NEXT:    v_mov_b32_e32 v0, 0
+; GCN3-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN3-NEXT:    s_sub_i32 s2, s2, s3
+; GCN3-NEXT:    v_mov_b32_e32 v1, s2
+; GCN3-NEXT:    global_store_dword v0, v1, s[0:1]
+; GCN3-NEXT:    s_endpgm
   %result = sub i32 %a, %b
   store i32 %result, ptr addrspace(1) %out
   ret void
 }
 
-; GCN-LABEL: {{^}}s_sub_imm_i32:
-; GCN: s_load_dword [[A:s[0-9]+]]
-; GCN: s_sub_i32 s{{[0-9]+}}, 0x4d2, [[A]]
 define amdgpu_kernel void @s_sub_imm_i32(ptr addrspace(1) %out, i32 %a) {
+; GCN1-LABEL: s_sub_imm_i32:
+; GCN1:       ; %bb.0:
+; GCN1-NEXT:    s_load_dword s4, s[0:1], 0xb
+; GCN1-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
+; GCN1-NEXT:    s_mov_b32 s3, 0xf000
+; GCN1-NEXT:    s_mov_b32 s2, -1
+; GCN1-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN1-NEXT:    s_sub_i32 s4, 0x4d2, s4
+; GCN1-NEXT:    v_mov_b32_e32 v0, s4
+; GCN1-NEXT:    buffer_store_dword v0, off, s[0:3], 0
+; GCN1-NEXT:    s_endpgm
+;
+; GCN2-LABEL: s_sub_imm_i32:
+; GCN2:       ; %bb.0:
+; GCN2-NEXT:    s_load_dword s2, s[0:1], 0x2c
+; GCN2-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GCN2-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN2-NEXT:    s_sub_i32 s2, 0x4d2, s2
+; GCN2-NEXT:    v_mov_b32_e32 v0, s0
+; GCN2-NEXT:    v_mov_b32_e32 v1, s1
+; GCN2-NEXT:    v_mov_b32_e32 v2, s2
+; GCN2-NEXT:    flat_store_dword v[0:1], v2
+; GCN2-NEXT:    s_endpgm
+;
+; GCN3-LABEL: s_sub_imm_i32:
+; GCN3:       ; %bb.0:
+; GCN3-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GCN3-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GCN3-NEXT:    v_mov_b32_e32 v0, 0
+; GCN3-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN3-NEXT:    s_sub_i32 s0, 0x4d2, s4
+; GCN3-NEXT:    v_mov_b32_e32 v1, s0
+; GCN3-NEXT:    global_store_dword v0, v1, s[2:3]
+; GCN3-NEXT:    s_endpgm
   %result = sub i32 1234, %a
   store i32 %result, ptr addrspace(1) %out
   ret void
 }
 
-; GCN-LABEL: {{^}}test_sub_i32:
-; SI: v_subrev_i32_e32 v{{[0-9]+, vcc, v[0-9]+, v[0-9]+}}
-; GFX9: v_sub_u32_e32 v{{[0-9]+, v[0-9]+, v[0-9]+}}
 define amdgpu_kernel void @test_sub_i32(ptr addrspace(1) %out, ptr addrspace(1) %in) {
+; GCN1-LABEL: test_sub_i32:
+; GCN1:       ; %bb.0:
+; GCN1-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
+; GCN1-NEXT:    s_mov_b32 s7, 0xf000
+; GCN1-NEXT:    s_mov_b32 s6, -1
+; GCN1-NEXT:    s_mov_b32 s10, s6
+; GCN1-NEXT:    s_mov_b32 s11, s7
+; GCN1-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN1-NEXT:    s_mov_b32 s8, s2
+; GCN1-NEXT:    s_mov_b32 s9, s3
+; GCN1-NEXT:    buffer_load_dwordx2 v[0:1], off, s[8:11], 0
+; GCN1-NEXT:    s_mov_b32 s4, s0
+; GCN1-NEXT:    s_mov_b32 s5, s1
+; GCN1-NEXT:    s_waitcnt vmcnt(0)
+; GCN1-NEXT:    v_sub_i32_e32 v0, vcc, v0, v1
+; GCN1-NEXT:    buffer_store_dword v0, off, s[4:7], 0
+; GCN1-NEXT:    s_endpgm
+;
+; GCN2-LABEL: test_sub_i32:
+; GCN2:       ; %bb.0:
+; GCN2-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GCN2-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN2-NEXT:    v_mov_b32_e32 v0, s2
+; GCN2-NEXT:    v_mov_b32_e32 v1, s3
+; GCN2-NEXT:    flat_load_dwordx2 v[0:1], v[0:1]
+; GCN2-NEXT:    v_mov_b32_e32 v2, s0
+; GCN2-NEXT:    v_mov_b32_e32 v3, s1
+; GCN2-NEXT:    s_waitcnt vmcnt(0)
+; GCN2-NEXT:    v_sub_u32_e32 v0, vcc, v0, v1
+; GCN2-NEXT:    flat_store_dword v[2:3], v0
+; GCN2-NEXT:    s_endpgm
+;
+; GCN3-LABEL: test_sub_i32:
+; GCN3:       ; %bb.0:
+; GCN3-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GCN3-NEXT:    v_mov_b32_e32 v2, 0
+; GCN3-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN3-NEXT:    global_load_dwordx2 v[0:1], v2, s[2:3]
+; GCN3-NEXT:    s_waitcnt vmcnt(0)
+; GCN3-NEXT:    v_sub_u32_e32 v0, v0, v1
+; GCN3-NEXT:    global_store_dword v2, v0, s[0:1]
+; GCN3-NEXT:    s_endpgm
   %b_ptr = getelementptr i32, ptr addrspace(1) %in, i32 1
   %a = load i32, ptr addrspace(1) %in
   %b = load i32, ptr addrspace(1) %b_ptr
   %result = sub i32 %a, %b
   store i32 %result, ptr addrspace(1) %out
   ret void
 }
 
-; GCN-LABEL: {{^}}test_sub_imm_i32:
-; SI: v_sub_i32_e32 v{{[0-9]+}}, vcc, 0x7b, v{{[0-9]+}}
-; GFX9: v_sub_u32_e32 v{{[0-9]+}}, 0x7b, v{{[0-9]+}}
 define amdgpu_kernel void @test_sub_imm_i32(ptr addrspace(1) %out, ptr addrspace(1) %in) {
+; GCN1-LABEL: test_sub_imm_i32:
+; GCN1:       ; %bb.0:
+; GCN1-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
+; GCN1-NEXT:    s_mov_b32 s7, 0xf000
+; GCN1-NEXT:    s_mov_b32 s6, -1
+; GCN1-NEXT:    s_mov_b32 s10, s6
+; GCN1-NEXT:    s_mov_b32 s11, s7
+; GCN1-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN1-NEXT:    s_mov_b32 s8, s2
+; GCN1-NEXT:    s_mov_b32 s9, s3
+; GCN1-NEXT:    buffer_load_dword v0, off, s[8:11], 0
+; GCN1-NEXT:    s_mov_b32 s4, s0
+; GCN1-NEXT:    s_mov_b32 s5, s1
+; GCN1-NEXT:    s_waitcnt vmcnt(0)
+; GCN1-NEXT:    v_sub_i32_e32 v0, vcc, 0x7b, v0
+; GCN1-NEXT:    buffer_store_dword v0, off, s[4:7], 0
+; GCN1-NEXT:    s_endpgm
+;
+; GCN2-LABEL: test_sub_imm_i32:
+; GCN2:       ; %bb.0:
+; GCN2-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GCN2-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN2-NEXT:    v_mov_b32_e32 v0, s2
+; GCN2-NEXT:    v_mov_b32_e32 v1, s3
+; GCN2-NEXT:    flat_load_dword v2, v[0:1]
+; GCN2-NEXT:    v_mov_b32_e32 v0, s0
+; GCN2-NEXT:    v_mov_b32_e32 v1, s1
+; GCN2-NEXT:    s_waitcnt vmcnt(0)
+; GCN2-NEXT:    v_sub_u32_e32 v2, vcc, 0x7b, v2
+; GCN2-NEXT:    flat_store_dword v[0:1], v2
+; GCN2-NEXT:    s_endpgm
+;
+; GCN3-LABEL: test_sub_imm_i32:
+; GCN3:       ; %bb.0:
+; GCN3-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GCN3-NEXT:    v_mov_b32_e32 v0, 0
+; GCN3-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN3-NEXT:    global_load_dword v1, v0, s[2:3]
+; GCN3-NEXT:    s_waitcnt vmcnt(0)
+; GCN3-NEXT:    v_sub_u32_e32 v1, 0x7b, v1
+; GCN3-NEXT:    global_store_dword v0, v1, s[0:1]
+; GCN3-NEXT:    s_endpgm
   %a = load i32, ptr addrspace(1) %in
   %result = sub i32 123, %a
   store i32 %result, ptr addrspace(1) %out
   ret void
 }
 
-; GCN-LABEL: {{^}}test_sub_v2i32:
-; SI: v_sub_i32_e32 v{{[0-9]+, vcc, v[0-9]+, v[0-9]+}}
-; SI: v_subrev_i32_e32 v{{[0-9]+, vcc, v[0-9]+, v[0-9]+}}
-
-; GFX9: v_sub_u32_e32 v{{[0-9]+, v[0-9]+, v[0-9]+}}
-; GFX9: v_sub_u32_e32 v{{[0-9]+, v[0-9]+, v[0-9]+}}
 define amdgpu_kernel void @test_sub_v2i32(ptr addrspace(1) %out, ptr addrspace(1) %in) {
+; GCN1-LABEL: test_sub_v2i32:
+; GCN1:       ; %bb.0:
+; GCN1-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
+; GCN1-NEXT:    s_mov_b32 s7, 0xf000
+; GCN1-NEXT:    s_mov_b32 s6, -1
+; GCN1-NEXT:    s_mov_b32 s10, s6
+; GCN1-NEXT:    s_mov_b32 s11, s7
+; GCN1-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN1-NEXT:    s_mov_b32 s8, s2
+; GCN1-NEXT:    s_mov_b32 s9, s3
+; GCN1-NEXT:    buffer_load_dwordx4 v[0:3], off, s[8:11], 0
+; GCN1-NEXT:    s_mov_b32 s4, s0
+; GCN1-NEXT:    s_mov_b32 s5, s1
+; GCN1-NEXT:    s_waitcnt vmcnt(0)
+; GCN1-NEXT:    v_sub_i32_e32 v1, vcc, v1, v3
+; GCN1-NEXT:    v_sub_i32_e32 v0, vcc, v0, v2
+; GCN1-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
+; GCN1-NEXT:    s_endpgm
+;
+; GCN2-LABEL: test_sub_v2i32:
+; GCN2:       ; %bb.0:
+; GCN2-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GCN2-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN2-NEXT:    v_mov_b32_e32 v0, s2
+; GCN2-NEXT:    v_mov_b32_e32 v1, s3
+; GCN2-NEXT:    flat_load_dwordx4 v[0:3], v[0:1]
+; GCN2-NEXT:    v_mov_b32_e32 v4, s0
+; GCN2-NEXT:    v_mov_b32_e32 v5, s1
+; GCN2-NEXT:    s_waitcnt vmcnt(0)
+; GCN2-NEXT:    v_sub_u32_e32 v1, vcc, v1, v3
+; GCN2-NEXT:    v_sub_u32_e32 v0, vcc, v0, v2
+; GCN2-NEXT:    flat_store_dwordx2 v[4:5], v[0:1]
+; GCN2-NEXT:    s_endpgm
+;
+; GCN3-LABEL: test_sub_v2i32:
+; GCN3:       ; %bb.0:
+; GCN3-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GCN3-NEXT:    v_mov_b32_e32 v4, 0
+; GCN3-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN3-NEXT:    global_load_dwordx4 v[0:3], v4, s[2:3]
+; GCN3-NEXT:    s_waitcnt vmcnt(0)
+; GCN3-NEXT:    v_sub_u32_e32 v1, v1, v3
+; GCN3-NEXT:    v_sub_u32_e32 v0, v0, v2
+; GCN3-NEXT:    global_store_dwordx2 v4, v[0:1], s[0:1]
+; GCN3-NEXT:    s_endpgm
   %b_ptr = getelementptr <2 x i32>, ptr addrspace(1) %in, i32 1
   %a = load <2 x i32>, ptr addrspace(1) %in
   %b = load <2 x i32>, ptr addrspace(1) %b_ptr
   %result = sub <2 x i32> %a, %b
   store <2 x i32> %result, ptr addrspace(1) %out
   ret void
 }
 
-; GCN-LABEL: {{^}}test_sub_v4i32:
-; SI: v_subrev_i32_e32 v{{[0-9]+, vcc, v[0-9]+, v[0-9]+}}
-; SI: v_sub_i32_e32 v{{[0-9]+, vcc, v[0-9]+, v[0-9]+}}
-; SI: v_sub_i32_e32 v{{[0-9]+, vcc, v[0-9]+, v[0-9]+}}
-; SI: v_sub_i32_e32 v{{[0-9]+, vcc, v[0-9]+, v[0-9]+}}
-
-; GFX9: v_sub_u32_e32 v{{[0-9]+, v[0-9]+, v[0-9]+}}
-; GFX9: v_sub_u32_e32 v{{[0-9]+, v[0-9]+, v[0-9]+}}
-; GFX9: v_sub_u32_e32 v{{[0-9]+, v[0-9]+, v[0-9]+}}
-; GFX9: v_sub_u32_e32 v{{[0-9]+, v[0-9]+, v[0-9]+}}
 define amdgpu_kernel void @test_sub_v4i32(ptr addrspace(1) %out, ptr addrspace(1) %in) {
+; GCN1-LABEL: test_sub_v4i32:
+; GCN1:       ; %bb.0:
+; GCN1-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
+; GCN1-NEXT:    s_mov_b32 s7, 0xf000
+; GCN1-NEXT:    s_mov_b32 s6, -1
+; GCN1-NEXT:    s_mov_b32 s10, s6
+; GCN1-NEXT:    s_mov_b32 s11, s7
+; GCN1-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN1-NEXT:    s_mov_b32 s8, s2
+; GCN1-NEXT:    s_mov_b32 s9, s3
+; GCN1-NEXT:    buffer_load_dwordx4 v[0:3], off, s[8:11], 0
+; GCN1-NEXT:    buffer_load_dwordx4 v[4:7], off, s[8:11], 0 offset:16
+; GCN1-NEXT:    s_mov_b32 s4, s0
+; GCN1-NEXT:    s_mov_b32 s5, s1
+; GCN1-NEXT:    s_waitcnt vmcnt(0)
+; GCN1-NEXT:    v_sub_i32_e32 v3, vcc, v3, v7
+; GCN1-NEXT:    v_sub_i32_e32 v2, vcc, v2, v6
+; GCN1-NEXT:    v_sub_i32_e32 v1, vcc, v1, v5
+; GCN1-NEXT:    v_sub_i32_e32 v0, vcc, v0, v4
+; GCN1-NEXT:    buffer_store_dwordx4 v[0:3], off, s[4:7], 0
+; GCN1-NEXT:    s_endpgm
+;
+; GCN2-LABEL: test_sub_v4i32:
+; GCN2:       ; %bb.0:
+; GCN2-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GCN2-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN2-NEXT:    v_mov_b32_e32 v0, s2
+; GCN2-NEXT:    v_mov_b32_e32 v1, s3
+; GCN2-NEXT:    s_add_u32 s2, s2, 16
+; GCN2-NEXT:    s_addc_u32 s3, s3, 0
+; GCN2-NEXT:    v_mov_b32_e32 v5, s3
+; GCN2-NEXT:    v_mov_b32_e32 v4, s2
+; GCN2-NEXT:    flat_load_dwordx4 v[0:3], v[0:1]
+; GCN2-NEXT:    flat_load_dwordx4 v[4:7], v[4:5]
+; GCN2-NEXT:    v_mov_b32_e32 v8, s0
+; GCN2-NEXT:    v_mov_b32_e32 v9, s1
+; GCN2-NEXT:    s_waitcnt vmcnt(0)
+; GCN2-NEXT:    v_sub_u32_e32 v3, vcc, v3, v7
+; GCN2-NEXT:    v_sub_u32_e32 v2, vcc, v2, v6
+; GCN2-NEXT:    v_sub_u32_e32 v1, vcc, v1, v5
+; GCN2-NEXT:    v_sub_u32_e32 v0, vcc, v0, v4
+; GCN2-NEXT:    flat_store_dwordx4 v[8:9], v[0:3]
+; GCN2-NEXT:    s_endpgm
+;
+; GCN3-LABEL: test_sub_v4i32:
+; GCN3:       ; %bb.0:
+; GCN3-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GCN3-NEXT:    v_mov_b32_e32 v8, 0
+; GCN3-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN3-NEXT:    global_load_dwordx4 v[0:3], v8, s[2:3] offset:16
+; GCN3-NEXT:    global_load_dwordx4 v[4:7], v8, s[2:3]
+; GCN3-NEXT:    s_waitcnt vmcnt(0)
+; GCN3-NEXT:    v_sub_u32_e32 v3, v7, v3
+; GCN3-NEXT:    v_sub_u32_e32 v2, v6, v2
+; GCN3-NEXT:    v_sub_u32_e32 v1, v5, v1
+; GCN3-NEXT:    v_sub_u32_e32 v0, v4, v0
+; GCN3-NEXT:    global_store_dwordx4 v8, v[0:3], s[0:1]
+; GCN3-NEXT:    s_endpgm
   %b_ptr = getelementptr <4 x i32>, ptr addrspace(1) %in, i32 1
   %a = load <4 x i32>, ptr addrspace(1) %in
   %b = load <4 x i32>, ptr addrspace(1) %b_ptr
   %result = sub <4 x i32> %a, %b
   store <4 x i32> %result, ptr addrspace(1) %out
   ret void
 }
 
-; GCN-LABEL: {{^}}test_sub_i16:
-; SI: v_sub_i32_e32 v{{[0-9]+}}, vcc,
-; GFX89: v_sub_u16_e32 v{{[0-9]+, v[0-9]+, v[0-9]+}}
 define amdgpu_kernel void @test_sub_i16(ptr addrspace(1) %out, ptr addrspace(1) %in) {
+; GCN1-LABEL: test_sub_i16:
+; GCN1:       ; %bb.0:
+; GCN1-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
+; GCN1-NEXT:    s_mov_b32 s7, 0xf000
+; GCN1-NEXT:    s_mov_b32 s10, 0
+; GCN1-NEXT:    s_mov_b32 s11, s7
+; GCN1-NEXT:    v_lshlrev_b32_e32 v0, 1, v0
+; GCN1-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN1-NEXT:    s_mov_b64 s[8:9], s[2:3]
+; GCN1-NEXT:    v_mov_b32_e32 v1, 0
+; GCN1-NEXT:    buffer_load_ushort v2, v[0:1], s[8:11], 0 addr64 glc
+; GCN1-NEXT:    s_waitcnt vmcnt(0)
+; GCN1-NEXT:    buffer_load_ushort v0, v[0:1], s[8:11], 0 addr64 offset:2 glc
+; GCN1-NEXT:    s_waitcnt vmcnt(0)
+; GCN1-NEXT:    s_mov_b32 s6, -1
+; GCN1-NEXT:    s_mov_b32 s4, s0
+; GCN1-NEXT:    s_mov_b32 s5, s1
+; GCN1-NEXT:    v_sub_i32_e32 v0, vcc, v2, v0
+; GCN1-NEXT:    buffer_store_short v0, off, s[4:7], 0
+; GCN1-NEXT:    s_endpgm
+;
+; GCN2-LABEL: test_sub_i16:
+; GCN2:       ; %bb.0:
+; GCN2-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GCN2-NEXT:    v_lshlrev_b32_e32 v0, 1, v0
+; GCN2-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN2-NEXT:    v_mov_b32_e32 v1, s3
+; GCN2-NEXT:    v_add_u32_e32 v0, vcc, s2, v0
+; GCN2-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; GCN2-NEXT:    v_add_u32_e32 v2, vcc, 2, v0
+; GCN2-NEXT:    v_addc_u32_e32 v3, vcc, 0, v1, vcc
+; GCN2-NEXT:    flat_load_ushort v4, v[0:1] glc
+; GCN2-NEXT:    s_waitcnt vmcnt(0)
+; GCN2-NEXT:    flat_load_ushort v2, v[2:3] glc
+; GCN2-NEXT:    s_waitcnt vmcnt(0)
+; GCN2-NEXT:    v_mov_b32_e32 v0, s0
+; GCN2-NEXT:    v_mov_b32_e32 v1, s1
+; GCN2-NEXT:    v_sub_u16_e32 v2, v4, v2
+; GCN2-NEXT:    flat_store_short v[0:1], v2
+; GCN2-NEXT:    s_endpgm
+;
+; GCN3-LABEL: test_sub_i16:
+; GCN3:       ; %bb.0:
+; GCN3-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GCN3-NEXT:    v_lshlrev_b32_e32 v0, 1, v0
+; GCN3-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN3-NEXT:    global_load_ushort v1, v0, s[2:3] glc
+; GCN3-NEXT:    s_waitcnt vmcnt(0)
+; GCN3-NEXT:    global_load_ushort v2, v0, s[2:3] offset:2 glc
+; GCN3-NEXT:    s_waitcnt vmcnt(0)
+; GCN3-NEXT:    v_mov_b32_e32 v0, 0
+; GCN3-NEXT:    v_sub_u16_e32 v1, v1, v2
+; GCN3-NEXT:    global_store_short v0, v1, s[0:1]
+; GCN3-NEXT:    s_endpgm
   %tid = call i32 @llvm.amdgcn.workitem.id.x()
   %gep = getelementptr i16, ptr addrspace(1) %in, i32 %tid
   %b_ptr = getelementptr i16, ptr addrspace(1) %gep, i32 1
   %a = load volatile i16, ptr addrspace(1) %gep
   %b = load volatile i16, ptr addrspace(1) %b_ptr
   %result = sub i16 %a, %b
   store i16 %result, ptr addrspace(1) %out
   ret void
 }
 
-; GCN-LABEL: {{^}}test_sub_v2i16:
-; VI: v_sub_u16_e32 v{{[0-9]+, v[0-9]+, v[0-9]+}}
-; VI: v_sub_u16_sdwa v{{[0-9]+, v[0-9]+, v[0-9]+}}
-
-; GFX9: v_pk_sub_i16
 define amdgpu_kernel void @test_sub_v2i16(ptr addrspace(1) %out, ptr addrspace(1) %in) {
+; GCN1-LABEL: test_sub_v2i16:
+; GCN1:       ; %bb.0:
+; GCN1-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
+; GCN1-NEXT:    s_mov_b32 s7, 0xf000
+; GCN1-NEXT:    s_mov_b32 s10, 0
+; GCN1-NEXT:    s_mov_b32 s11, s7
+; GCN1-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GCN1-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN1-NEXT:    s_mov_b64 s[8:9], s[2:3]
+; GCN1-NEXT:    v_mov_b32_e32 v1, 0
+; GCN1-NEXT:    buffer_load_dwordx2 v[0:1], v[0:1], s[8:11], 0 addr64
+; GCN1-NEXT:    s_mov_b32 s6, -1
+; GCN1-NEXT:    s_mov_b32 s4, s0
+; GCN1-NEXT:    s_mov_b32 s5, s1
+; GCN1-NEXT:    s_waitcnt vmcnt(0)
+; GCN1-NEXT:    v_lshrrev_b32_e32 v2, 16, v0
+; GCN1-NEXT:    v_lshrrev_b32_e32 v3, 16, v1
+; GCN1-NEXT:    v_sub_i32_e32 v0, vcc, v0, v1
+; GCN1-NEXT:    v_sub_i32_e32 v1, vcc, v2, v3
+; GCN1-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GCN1-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
+; GCN1-NEXT:    v_or_b32_e32 v0, v0, v1
+; GCN1-NEXT:    buffer_store_dword v0, off, s[4:7], 0
+; GCN1-NEXT:    s_endpgm
+;
+; GCN2-LABEL: test_sub_v2i16:
+; GCN2:       ; %bb.0:
+; GCN2-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GCN2-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GCN2-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN2-NEXT:    v_mov_b32_e32 v1, s3
+; GCN2-NEXT:    v_add_u32_e32 v0, vcc, s2, v0
+; GCN2-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; GCN2-NEXT:    flat_load_dwordx2 v[0:1], v[0:1]
+; GCN2-NEXT:    v_mov_b32_e32 v2, s0
+; GCN2-NEXT:    v_mov_b32_e32 v3, s1
+; GCN2-NEXT:    s_waitcnt vmcnt(0)
+; GCN2-NEXT:    v_sub_u16_e32 v4, v0, v1
+; GCN2-NEXT:    v_sub_u16_sdwa v0, v0, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
+; GCN2-NEXT:    v_or_b32_e32 v0, v4, v0
+; GCN2-NEXT:    flat_store_dword v[2:3], v0
+; GCN2-NEXT:    s_endpgm
+;
+; GCN3-LABEL: test_sub_v2i16:
+; GCN3:       ; %bb.0:
+; GCN3-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GCN3-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GCN3-NEXT:    v_mov_b32_e32 v2, 0
+; GCN3-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN3-NEXT:    global_load_dwordx2 v[0:1], v0, s[2:3]
+; GCN3-NEXT:    s_waitcnt vmcnt(0)
+; GCN3-NEXT:    v_pk_sub_i16 v0, v0, v1
+; GCN3-NEXT:    global_store_dword v2, v0, s[0:1]
+; GCN3-NEXT:    s_endpgm
   %tid = call i32 @llvm.amdgcn.workitem.id.x()
   %gep = getelementptr <2 x i16>, ptr addrspace(1) %in, i32 %tid
   %b_ptr = getelementptr <2 x i16>, ptr addrspace(1) %gep, i16 1
   %a = load <2 x i16>, ptr addrspace(1) %gep
   %b = load <2 x i16>, ptr addrspace(1) %b_ptr
   %result = sub <2 x i16> %a, %b
   store <2 x i16> %result, ptr addrspace(1) %out
   ret void
 }
 
-; GCN-LABEL: {{^}}test_sub_v4i16:
-; VI: v_sub_u16_e32 v{{[0-9]+, v[0-9]+, v[0-9]+}}
-; VI: v_sub_u16_sdwa v{{[0-9]+, v[0-9]+, v[0-9]+}}
-; VI: v_sub_u16_e32 v{{[0-9]+, v[0-9]+, v[0-9]+}}
-; VI: v_sub_u16_sdwa v{{[0-9]+, v[0-9]+, v[0-9]+}}
-
-; GFX9: v_pk_sub_i16
-; GFX9: v_pk_sub_i16
 define amdgpu_kernel void @test_sub_v4i16(ptr addrspace(1) %out, ptr addrspace(1) %in) {
+; GCN1-LABEL: test_sub_v4i16:
+; GCN1:       ; %bb.0:
+; GCN1-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x9
+; GCN1-NEXT:    s_mov_b32 s7, 0xf000
+; GCN1-NEXT:    s_mov_b32 s10, 0
+; GCN1-NEXT:    s_mov_b32 s11, s7
+; GCN1-NEXT:    v_lshlrev_b32_e32 v0, 3, v0
+; GCN1-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN1-NEXT:    s_mov_b64 s[8:9], s[2:3]
+; GCN1-NEXT:    v_mov_b32_e32 v1, 0
+; GCN1-NEXT:    buffer_load_dwordx4 v[0:3], v[0:1], s[8:11], 0 addr64
+; GCN1-NEXT:    s_mov_b32 s6, -1
+; GCN1-NEXT:    s_mov_b32 s4, s0
+; GCN1-NEXT:    s_mov_b32 s5, s1
+; GCN1-NEXT:    s_waitcnt vmcnt(0)
+; GCN1-NEXT:    v_lshrrev_b32_e32 v4, 16, v0
+; GCN1-NEXT:    v_lshrrev_b32_e32 v5, 16, v1
+; GCN1-NEXT:    v_lshrrev_b32_e32 v6, 16, v2
+; GCN1-NEXT:    v_lshrrev_b32_e32 v7, 16, v3
+; GCN1-NEXT:    v_sub_i32_e32 v1, vcc, v1, v3
+; GCN1-NEXT:    v_sub_i32_e32 v0, vcc, v0, v2
+; GCN1-NEXT:    v_sub_i32_e32 v2, vcc, v5, v7
+; GCN1-NEXT:    v_sub_i32_e32 v3, vcc, v4, v6
+; GCN1-NEXT:    v_and_b32_e32 v1, 0xffff, v1
+; GCN1-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GCN1-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
+; GCN1-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
+; GCN1-NEXT:    v_or_b32_e32 v1, v1, v2
+; GCN1-NEXT:    v_or_b32_e32 v0, v0, v3
+; GCN1-NEXT:    buffer_store_dwordx2 v[0:1], off, s[4:7], 0
+; GCN1-NEXT:    s_endpgm
+;
+; GCN2-LABEL: test_sub_v4i16:
+; GCN2:       ; %bb.0:
+; GCN2-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GCN2-NEXT:    v_lshlrev_b32_e32 v0, 3, v0
+; GCN2-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN2-NEXT:    v_mov_b32_e32 v1, s3
+; GCN2-NEXT:    v_add_u32_e32 v0, vcc, s2, v0
+; GCN2-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; GCN2-NEXT:    flat_load_dwordx4 v[0:3], v[0:1]
+; GCN2-NEXT:    v_mov_b32_e32 v4, s0
+; GCN2-NEXT:    v_mov_b32_e32 v5, s1
+; GCN2-NEXT:    s_waitcnt vmcnt(0)
+; GCN2-NEXT:    v_sub_u16_e32 v6, v1, v3
+; GCN2-NEXT:    v_sub_u16_sdwa v1, v1, v3 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
+; GCN2-NEXT:    v_sub_u16_e32 v3, v0, v2
+; GCN2-NEXT:    v_sub_u16_sdwa v0, v0, v2 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:WORD_1
+; GCN2-NEXT:    v_or_b32_e32 v1, v6, v1
+; GCN2-NEXT:    v_or_b32_e32 v0, v3, v0
+; GCN2-NEXT:    flat_store_dwordx2 v[4:5], v[0:1]
+; GCN2-NEXT:    s_endpgm
+;
+; GCN3-LABEL: test_sub_v4i16:
+; GCN3:       ; %bb.0:
+; GCN3-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
+; GCN3-NEXT:    v_lshlrev_b32_e32 v0, 3, v0
+; GCN3-NEXT:    v_mov_b32_e32 v4, 0
+; GCN3-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN3-NEXT:    global_load_dwordx4 v[0:3], v0, s[2:3]
+; GCN3-NEXT:    s_waitcnt vmcnt(0)
+; GCN3-NEXT:    v_pk_sub_i16 v1, v1, v3
+; GCN3-NEXT:    v_pk_sub_i16 v0, v0, v2
+; GCN3-NEXT:    global_store_dwordx2 v4, v[0:1], s[0:1]
+; GCN3-NEXT:    s_endpgm
   %tid = call i32 @llvm.amdgcn.workitem.id.x()
   %gep = getelementptr <4 x i16>, ptr addrspace(1) %in, i32 %tid
   %b_ptr = getelementptr <4 x i16>, ptr addrspace(1) %gep, i16 1
   %a = load <4 x i16>, ptr addrspace(1) %gep
   %b = load <4 x i16>, ptr addrspace(1) %b_ptr
   %result = sub <4 x i16> %a, %b
   store <4 x i16> %result, ptr addrspace(1) %out
   ret void
 }
 
-; GCN-LABEL: {{^}}s_sub_i64:
-; GCN: s_sub_u32
-; GCN: s_subb_u32
 define amdgpu_kernel void @s_sub_i64(ptr addrspace(1) noalias %out, i64 %a, i64 %b) nounwind {
+; GCN1-LABEL: s_sub_i64:
+; GCN1:       ; %bb.0:
+; GCN1-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0xb
+; GCN1-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x9
+; GCN1-NEXT:    s_mov_b32 s3, 0xf000
+; GCN1-NEXT:    s_mov_b32 s2, -1
+; GCN1-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN1-NEXT:    s_sub_u32 s4, s4, s6
+; GCN1-NEXT:    s_subb_u32 s5, s5, s7
+; GCN1-NEXT:    v_mov_b32_e32 v0, s4
+; GCN1-NEXT:    v_mov_b32_e32 v1, s5
+; GCN1-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
+; GCN1-NEXT:    s_endpgm
+;
+; GCN2-LABEL: s_sub_i64:
+; GCN2:       ; %bb.0:
+; GCN2-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x2c
+; GCN2-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
+; GCN2-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN2-NEXT:    s_sub_u32 s2, s4, s6
+; GCN2-NEXT:    s_subb_u32 s3, s5, s7
+; GCN2-NEXT:    v_mov_b32_e32 v0, s0
+; GCN2-NEXT:    v_mov_b32_e32 v2, s2
+; GCN2-NEXT:    v_mov_b32_e32 v1, s1
+; GCN2-NEXT:    v_mov_b32_e32 v3, s3
+; GCN2-NEXT:    flat_store_dwordx2 v[0:1], v[2:3]
+; GCN2-NEXT:    s_endpgm
+;
+; GCN3-LABEL: s_sub_i64:
+; GCN3:       ; %bb.0:
+; GCN3-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x2c
+; GCN3-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
+; GCN3-NEXT:    v_mov_b32_e32 v2, 0
+; GCN3-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN3-NEXT:    s_sub_u32 s0, s4, s6
+; GCN3-NEXT:    s_subb_u32 s1, s5, s7
+; GCN3-NEXT:    v_mov_b32_e32 v0, s0
+; GCN3-NEXT:    v_mov_b32_e32 v1, s1
+; GCN3-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
+; GCN3-NEXT:    s_endpgm
   %result = sub i64 %a, %b
   store i64 %result, ptr addrspace(1) %out, align 8
   ret void
 }
 
-; GCN-LABEL: {{^}}v_sub_i64:
-; SI: v_sub_i32_e32
-; SI: v_subb_u32_e32
-
-; VI: v_sub_u32_e32
-; VI: v_subb_u32_e32
-
-; GFX9: v_sub_co_u32_e32
-; GFX9: v_subb_co_u32_e32
 define amdgpu_kernel void @v_sub_i64(ptr addrspace(1) noalias %out, ptr addrspace(1) noalias %inA, ptr addrspace(1) noalias %inB) nounwind {
+; GCN1-LABEL: v_sub_i64:
+; GCN1:       ; %bb.0:
+; GCN1-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x9
+; GCN1-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0xd
+; GCN1-NEXT:    s_mov_b32 s11, 0xf000
+; GCN1-NEXT:    s_mov_b32 s14, 0
+; GCN1-NEXT:    s_mov_b32 s15, s11
+; GCN1-NEXT:    v_lshlrev_b32_e32 v0, 3, v0
+; GCN1-NEXT:    v_mov_b32_e32 v1, 0
+; GCN1-NEXT:    s_mov_b64 s[2:3], s[14:15]
+; GCN1-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN1-NEXT:    s_mov_b64 s[12:13], s[6:7]
+; GCN1-NEXT:    buffer_load_dwordx2 v[2:3], v[0:1], s[0:3], 0 addr64
+; GCN1-NEXT:    buffer_load_dwordx2 v[0:1], v[0:1], s[12:15], 0 addr64
+; GCN1-NEXT:    s_mov_b32 s10, -1
+; GCN1-NEXT:    s_mov_b32 s8, s4
+; GCN1-NEXT:    s_mov_b32 s9, s5
+; GCN1-NEXT:    s_waitcnt vmcnt(0)
+; GCN1-NEXT:    v_sub_i32_e32 v0, vcc, v0, v2
+; GCN1-NEXT:    v_subb_u32_e32 v1, vcc, v1, v3, vcc
+; GCN1-NEXT:    buffer_store_dwordx2 v[0:1], off, s[8:11], 0
+; GCN1-NEXT:    s_endpgm
+;
+; GCN2-LABEL: v_sub_i64:
+; GCN2:       ; %bb.0:
+; GCN2-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GCN2-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; GCN2-NEXT:    v_lshlrev_b32_e32 v2, 3, v0
+; GCN2-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN2-NEXT:    v_mov_b32_e32 v1, s7
+; GCN2-NEXT:    v_add_u32_e32 v0, vcc, s6, v2
+; GCN2-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; GCN2-NEXT:    v_mov_b32_e32 v3, s1
+; GCN2-NEXT:    v_add_u32_e32 v2, vcc, s0, v2
+; GCN2-NEXT:    v_addc_u32_e32 v3, vcc, 0, v3, vcc
+; GCN2-NEXT:    flat_load_dwordx2 v[0:1], v[0:1]
+; GCN2-NEXT:    flat_load_dwordx2 v[2:3], v[2:3]
+; GCN2-NEXT:    s_waitcnt vmcnt(0)
+; GCN2-NEXT:    v_sub_u32_e32 v0, vcc, v0, v2
+; GCN2-NEXT:    v_subb_u32_e32 v1, vcc, v1, v3, vcc
+; GCN2-NEXT:    v_mov_b32_e32 v2, s4
+; GCN2-NEXT:    v_mov_b32_e32 v3, s5
+; GCN2-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
+; GCN2-NEXT:    s_endpgm
+;
+; GCN3-LABEL: v_sub_i64:
+; GCN3:       ; %bb.0:
+; GCN3-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GCN3-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GCN3-NEXT:    v_lshlrev_b32_e32 v4, 3, v0
+; GCN3-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN3-NEXT:    global_load_dwordx2 v[0:1], v4, s[6:7]
+; GCN3-NEXT:    global_load_dwordx2 v[2:3], v4, s[2:3]
+; GCN3-NEXT:    v_mov_b32_e32 v4, 0
+; GCN3-NEXT:    s_waitcnt vmcnt(0)
+; GCN3-NEXT:    v_sub_co_u32_e32 v0, vcc, v0, v2
+; GCN3-NEXT:    v_subb_co_u32_e32 v1, vcc, v1, v3, vcc
+; GCN3-NEXT:    global_store_dwordx2 v4, v[0:1], s[4:5]
+; GCN3-NEXT:    s_endpgm
   %tid = call i32 @llvm.amdgcn.workitem.id.x() readnone
   %a_ptr = getelementptr i64, ptr addrspace(1) %inA, i32 %tid
   %b_ptr = getelementptr i64, ptr addrspace(1) %inB, i32 %tid
   %a = load i64, ptr addrspace(1) %a_ptr
   %b = load i64, ptr addrspace(1) %b_ptr
   %result = sub i64 %a, %b
   store i64 %result, ptr addrspace(1) %out, align 8
   ret void
 }
 
-; GCN-LABEL: {{^}}v_test_sub_v2i64:
-; SI: v_sub_i32_e32 v{{[0-9]+}}, vcc,
-; SI: v_subb_u32_e32 v{{[0-9]+}}, vcc,
-; SI: v_sub_i32_e32 v{{[0-9]+}}, vcc,
-; SI: v_subb_u32_e32 v{{[0-9]+}}, vcc,
-
-; VI: v_sub_u32_e32 v{{[0-9]+}}, vcc,
-; VI: v_subb_u32_e32 v{{[0-9]+}}, vcc,
-; VI: v_sub_u32_e32 v{{[0-9]+}}, vcc,
-; VI: v_subb_u32_e32 v{{[0-9]+}}, vcc,
-
-; GFX9: v_sub_co_u32_e32 v{{[0-9]+}}, vcc,
-; GFX9: v_subb_co_u32_e32 v{{[0-9]+}}, vcc,
-; GFX9: v_sub_co_u32_e32 v{{[0-9]+}}, vcc,
-; GFX9: v_subb_co_u32_e32 v{{[0-9]+}}, vcc,
 define amdgpu_kernel void @v_test_sub_v2i64(ptr addrspace(1) %out, ptr addrspace(1) noalias %inA, ptr addrspace(1) noalias %inB) {
+; GCN1-LABEL: v_test_sub_v2i64:
+; GCN1:       ; %bb.0:
+; GCN1-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x9
+; GCN1-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0xd
+; GCN1-NEXT:    s_mov_b32 s11, 0xf000
+; GCN1-NEXT:    s_mov_b32 s14, 0
+; GCN1-NEXT:    s_mov_b32 s15, s11
+; GCN1-NEXT:    v_lshlrev_b32_e32 v4, 4, v0
+; GCN1-NEXT:    v_mov_b32_e32 v5, 0
+; GCN1-NEXT:    s_mov_b64 s[2:3], s[14:15]
+; GCN1-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN1-NEXT:    s_mov_b64 s[12:13], s[6:7]
+; GCN1-NEXT:    buffer_load_dwordx4 v[0:3], v[4:5], s[0:3], 0 addr64
+; GCN1-NEXT:    buffer_load_dwordx4 v[4:7], v[4:5], s[12:15], 0 addr64
+; GCN1-NEXT:    s_mov_b32 s10, -1
+; GCN1-NEXT:    s_mov_b32 s8, s4
+; GCN1-NEXT:    s_mov_b32 s9, s5
+; GCN1-NEXT:    s_waitcnt vmcnt(0)
+; GCN1-NEXT:    v_sub_i32_e32 v2, vcc, v6, v2
+; GCN1-NEXT:    v_subb_u32_e32 v3, vcc, v7, v3, vcc
+; GCN1-NEXT:    v_sub_i32_e32 v0, vcc, v4, v0
+; GCN1-NEXT:    v_subb_u32_e32 v1, vcc, v5, v1, vcc
+; GCN1-NEXT:    buffer_store_dwordx4 v[0:3], off, s[8:11], 0
+; GCN1-NEXT:    s_endpgm
+;
+; GCN2-LABEL: v_test_sub_v2i64:
+; GCN2:       ; %bb.0:
+; GCN2-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GCN2-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; GCN2-NEXT:    v_lshlrev_b32_e32 v2, 4, v0
+; GCN2-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN2-NEXT:    v_mov_b32_e32 v1, s7
+; GCN2-NEXT:    v_add_u32_e32 v0, vcc, s6, v2
+; GCN2-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; GCN2-NEXT:    v_mov_b32_e32 v3, s1
+; GCN2-NEXT:    v_add_u32_e32 v4, vcc, s0, v2
+; GCN2-NEXT:    v_addc_u32_e32 v5, vcc, 0, v3, vcc
+; GCN2-NEXT:    flat_load_dwordx4 v[0:3], v[0:1]
+; GCN2-NEXT:    flat_load_dwordx4 v[4:7], v[4:5]
+; GCN2-NEXT:    s_waitcnt vmcnt(0)
+; GCN2-NEXT:    v_sub_u32_e32 v2, vcc, v2, v6
+; GCN2-NEXT:    v_subb_u32_e32 v3, vcc, v3, v7, vcc
+; GCN2-NEXT:    v_sub_u32_e32 v0, vcc, v0, v4
+; GCN2-NEXT:    v_subb_u32_e32 v1, vcc, v1, v5, vcc
+; GCN2-NEXT:    v_mov_b32_e32 v4, s4
+; GCN2-NEXT:    v_mov_b32_e32 v5, s5
+; GCN2-NEXT:    flat_store_dwordx4 v[4:5], v[0:3]
+; GCN2-NEXT:    s_endpgm
+;
+; GCN3-LABEL: v_test_sub_v2i64:
+; GCN3:       ; %bb.0:
+; GCN3-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GCN3-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GCN3-NEXT:    v_lshlrev_b32_e32 v8, 4, v0
+; GCN3-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN3-NEXT:    global_load_dwordx4 v[0:3], v8, s[6:7]
+; GCN3-NEXT:    global_load_dwordx4 v[4:7], v8, s[2:3]
+; GCN3-NEXT:    v_mov_b32_e32 v8, 0
+; GCN3-NEXT:    s_waitcnt vmcnt(0)
+; GCN3-NEXT:    v_sub_co_u32_e32 v2, vcc, v2, v6
+; GCN3-NEXT:    v_subb_co_u32_e32 v3, vcc, v3, v7, vcc
+; GCN3-NEXT:    v_sub_co_u32_e32 v0, vcc, v0, v4
+; GCN3-NEXT:    v_subb_co_u32_e32 v1, vcc, v1, v5, vcc
+; GCN3-NEXT:    global_store_dwordx4 v8, v[0:3], s[4:5]
+; GCN3-NEXT:    s_endpgm
   %tid = call i32 @llvm.amdgcn.workitem.id.x() readnone
   %a_ptr = getelementptr <2 x i64>, ptr addrspace(1) %inA, i32 %tid
   %b_ptr = getelementptr <2 x i64>, ptr addrspace(1) %inB, i32 %tid
   %a = load <2 x i64>, ptr addrspace(1) %a_ptr
   %b = load <2 x i64>, ptr addrspace(1) %b_ptr
   %result = sub <2 x i64> %a, %b
   store <2 x i64> %result, ptr addrspace(1) %out
   ret void
 }
 
-; GCN-LABEL: {{^}}v_test_sub_v4i64:
-; SI: v_sub_i32_e32 v{{[0-9]+}}, vcc,
-; SI: v_subb_u32_e32 v{{[0-9]+}}, vcc,
-; SI: v_sub_i32_e32 v{{[0-9]+}}, vcc,
-; SI: v_subb_u32_e32 v{{[0-9]+}}, vcc,
-; SI: v_sub_i32_e32 v{{[0-9]+}}, vcc,
-; SI: v_subb_u32_e32 v{{[0-9]+}}, vcc,
-; SI: v_sub_i32_e32 v{{[0-9]+}}, vcc,
-; SI: v_subb_u32_e32 v{{[0-9]+}}, vcc,
-
-; VI: v_sub_u32_e32 v{{[0-9]+}}, vcc,
-; VI: v_subb_u32_e32 v{{[0-9]+}}, vcc,
-; VI: v_sub_u32_e32 v{{[0-9]+}}, vcc,
-; VI: v_subb_u32_e32 v{{[0-9]+}}, vcc,
-; VI: v_sub_u32_e32 v{{[0-9]+}}, vcc,
-; VI: v_subb_u32_e32 v{{[0-9]+}}, vcc,
-; VI: v_sub_u32_e32 v{{[0-9]+}}, vcc,
-; VI: v_subb_u32_e32 v{{[0-9]+}}, vcc,
-
-; GFX9: v_sub_co_u32_e32 v{{[0-9]+}}, vcc,
-; GFX9: v_subb_co_u32_e32 v{{[0-9]+}}, vcc,
-; GFX9: v_sub_co_u32_e32 v{{[0-9]+}}, vcc,
-; GFX9: v_subb_co_u32_e32 v{{[0-9]+}}, vcc,
-; GFX9: v_sub_co_u32_e32 v{{[0-9]+}}, vcc,
-; GFX9: v_subb_co_u32_e32 v{{[0-9]+}}, vcc,
-; GFX9: v_sub_co_u32_e32 v{{[0-9]+}}, vcc,
-; GFX9: v_subb_co_u32_e32 v{{[0-9]+}}, vcc,
 define amdgpu_kernel void @v_test_sub_v4i64(ptr addrspace(1) %out, ptr addrspace(1) noalias %inA, ptr addrspace(1) noalias %inB) {
+; GCN1-LABEL: v_test_sub_v4i64:
+; GCN1:       ; %bb.0:
+; GCN1-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x9
+; GCN1-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0xd
+; GCN1-NEXT:    s_mov_b32 s11, 0xf000
+; GCN1-NEXT:    s_mov_b32 s14, 0
+; GCN1-NEXT:    s_mov_b32 s15, s11
+; GCN1-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN1-NEXT:    s_mov_b64 s[12:13], s[6:7]
+; GCN1-NEXT:    v_lshlrev_b32_e32 v12, 5, v0
+; GCN1-NEXT:    v_mov_b32_e32 v13, 0
+; GCN1-NEXT:    s_mov_b64 s[2:3], s[14:15]
+; GCN1-NEXT:    buffer_load_dwordx4 v[0:3], v[12:13], s[12:15], 0 addr64
+; GCN1-NEXT:    buffer_load_dwordx4 v[4:7], v[12:13], s[0:3], 0 addr64
+; GCN1-NEXT:    buffer_load_dwordx4 v[8:11], v[12:13], s[0:3], 0 addr64 offset:16
+; GCN1-NEXT:    buffer_load_dwordx4 v[12:15], v[12:13], s[12:15], 0 addr64 offset:16
+; GCN1-NEXT:    s_mov_b32 s10, -1
+; GCN1-NEXT:    s_mov_b32 s8, s4
+; GCN1-NEXT:    s_mov_b32 s9, s5
+; GCN1-NEXT:    s_waitcnt vmcnt(2)
+; GCN1-NEXT:    v_sub_i32_e32 v2, vcc, v2, v6
+; GCN1-NEXT:    v_subb_u32_e32 v3, vcc, v3, v7, vcc
+; GCN1-NEXT:    v_sub_i32_e32 v0, vcc, v0, v4
+; GCN1-NEXT:    v_subb_u32_e32 v1, vcc, v1, v5, vcc
+; GCN1-NEXT:    s_waitcnt vmcnt(0)
+; GCN1-NEXT:    v_sub_i32_e32 v6, vcc, v14, v10
+; GCN1-NEXT:    v_subb_u32_e32 v7, vcc, v15, v11, vcc
+; GCN1-NEXT:    v_sub_i32_e32 v4, vcc, v12, v8
+; GCN1-NEXT:    v_subb_u32_e32 v5, vcc, v13, v9, vcc
+; GCN1-NEXT:    buffer_store_dwordx4 v[4:7], off, s[8:11], 0 offset:16
+; GCN1-NEXT:    buffer_store_dwordx4 v[0:3], off, s[8:11], 0
+; GCN1-NEXT:    s_endpgm
+;
+; GCN2-LABEL: v_test_sub_v4i64:
+; GCN2:       ; %bb.0:
+; GCN2-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GCN2-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
+; GCN2-NEXT:    v_lshlrev_b32_e32 v0, 5, v0
+; GCN2-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN2-NEXT:    v_mov_b32_e32 v1, s7
+; GCN2-NEXT:    v_add_u32_e32 v8, vcc, s6, v0
+; GCN2-NEXT:    v_addc_u32_e32 v9, vcc, 0, v1, vcc
+; GCN2-NEXT:    v_mov_b32_e32 v1, s1
+; GCN2-NEXT:    v_add_u32_e32 v12, vcc, s0, v0
+; GCN2-NEXT:    v_addc_u32_e32 v13, vcc, 0, v1, vcc
+; GCN2-NEXT:    flat_load_dwordx4 v[0:3], v[8:9]
+; GCN2-NEXT:    flat_load_dwordx4 v[4:7], v[12:13]
+; GCN2-NEXT:    v_add_u32_e32 v8, vcc, 16, v8
+; GCN2-NEXT:    v_addc_u32_e32 v9, vcc, 0, v9, vcc
+; GCN2-NEXT:    v_add_u32_e32 v12, vcc, 16, v12
+; GCN2-NEXT:    v_addc_u32_e32 v13, vcc, 0, v13, vcc
+; GCN2-NEXT:    flat_load_dwordx4 v[8:11], v[8:9]
+; GCN2-NEXT:    flat_load_dwordx4 v[12:15], v[12:13]
+; GCN2-NEXT:    v_mov_b32_e32 v17, s5
+; GCN2-NEXT:    v_mov_b32_e32 v16, s4
+; GCN2-NEXT:    s_add_u32 s0, s4, 16
+; GCN2-NEXT:    s_addc_u32 s1, s5, 0
+; GCN2-NEXT:    s_waitcnt vmcnt(2)
+; GCN2-NEXT:    v_sub_u32_e32 v2, vcc, v2, v6
+; GCN2-NEXT:    v_subb_u32_e32 v3, vcc, v3, v7, vcc
+; GCN2-NEXT:    v_sub_u32_e32 v0, vcc, v0, v4
+; GCN2-NEXT:    v_subb_u32_e32 v1, vcc, v1, v5, vcc
+; GCN2-NEXT:    flat_store_dwordx4 v[16:17], v[0:3]
+; GCN2-NEXT:    s_waitcnt vmcnt(1)
+; GCN2-NEXT:    v_sub_u32_e32 v6, vcc, v10, v14
+; GCN2-NEXT:    v_subb_u32_e32 v7, vcc, v11, v15, vcc
+; GCN2-NEXT:    v_sub_u32_e32 v4, vcc, v8, v12
+; GCN2-NEXT:    v_mov_b32_e32 v0, s0
+; GCN2-NEXT:    v_subb_u32_e32 v5, vcc, v9, v13, vcc
+; GCN2-NEXT:    v_mov_b32_e32 v1, s1
+; GCN2-NEXT:    flat_store_dwordx4 v[0:1], v[4:7]
+; GCN2-NEXT:    s_endpgm
+;
+; GCN3-LABEL: v_test_sub_v4i64:
+; GCN3:       ; %bb.0:
+; GCN3-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GCN3-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
+; GCN3-NEXT:    v_lshlrev_b32_e32 v16, 5, v0
+; GCN3-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN3-NEXT:    global_load_dwordx4 v[0:3], v16, s[6:7]
+; GCN3-NEXT:    global_load_dwordx4 v[4:7], v16, s[2:3]
+; GCN3-NEXT:    global_load_dwordx4 v[8:11], v16, s[6:7] offset:16
+; GCN3-NEXT:    global_load_dwordx4 v[12:15], v16, s[2:3] offset:16
+; GCN3-NEXT:    v_mov_b32_e32 v16, 0
+; GCN3-NEXT:    s_waitcnt vmcnt(2)
+; GCN3-NEXT:    v_sub_co_u32_e32 v2, vcc, v2, v6
+; GCN3-NEXT:    v_subb_co_u32_e32 v3, vcc, v3, v7, vcc
+; GCN3-NEXT:    v_sub_co_u32_e32 v0, vcc, v0, v4
+; GCN3-NEXT:    v_subb_co_u32_e32 v1, vcc, v1, v5, vcc
+; GCN3-NEXT:    s_waitcnt vmcnt(0)
+; GCN3-NEXT:    v_sub_co_u32_e32 v6, vcc, v10, v14
+; GCN3-NEXT:    v_subb_co_u32_e32 v7, vcc, v11, v15, vcc
+; GCN3-NEXT:    v_sub_co_u32_e32 v4, vcc, v8, v12
+; GCN3-NEXT:    v_subb_co_u32_e32 v5, vcc, v9, v13, vcc
+; GCN3-NEXT:    global_store_dwordx4 v16, v[4:7], s[4:5] offset:16
+; GCN3-NEXT:    global_store_dwordx4 v16, v[0:3], s[4:5]
+; GCN3-NEXT:    s_endpgm
   %tid = call i32 @llvm.amdgcn.workitem.id.x() readnone
   %a_ptr = getelementptr <4 x i64>, ptr addrspace(1) %inA, i32 %tid
   %b_ptr = getelementptr <4 x i64>, ptr addrspace(1) %inB, i32 %tid
   %a = load <4 x i64>, ptr addrspace(1) %a_ptr
   %b = load <4 x i64>, ptr addrspace(1) %b_ptr
   %result = sub <4 x i64> %a, %b
   store <4 x i64> %result, ptr addrspace(1) %out
   ret void
 }
 
-; Make sure the VOP3 form of sub is initially selected. Otherwise pair
-; of opies from/to VCC would be necessary
-
-; GCN-LABEL: {{^}}sub_select_vop3:
-; SI: v_subrev_i32_e64 v0, s[0:1], s0, v0
-; VI: v_subrev_u32_e64 v0, s[0:1], s0, v0
-; GFX9: v_subrev_u32_e32 v0, s0, v0
-
-; GCN: ; def vcc
-; GCN: ds_write_b32
-; GCN: ; use vcc
 define amdgpu_ps void @sub_select_vop3(i32 inreg %s, i32 %v) {
+; GCN1-LABEL: sub_select_vop3:
+; GCN1:       ; %bb.0:
+; GCN1-NEXT:    v_subrev_i32_e64 v0, s[0:1], s0, v0
+; GCN1-NEXT:    s_mov_b32 m0, -1
+; GCN1-NEXT:    ;;#ASMSTART
+; GCN1-NEXT:    ; def vcc
+; GCN1-NEXT:    ;;#ASMEND
+; GCN1-NEXT:    ds_write_b32 v0, v0
+; GCN1-NEXT:    ;;#ASMSTART
+; GCN1-NEXT:    ; use vcc
+; GCN1-NEXT:    ;;#ASMEND
+; GCN1-NEXT:    s_endpgm
+;
+; GCN2-LABEL: sub_select_vop3:
+; GCN2:       ; %bb.0:
+; GCN2-NEXT:    v_subrev_u32_e64 v0, s[0:1], s0, v0
+; GCN2-NEXT:    s_mov_b32 m0, -1
+; GCN2-NEXT:    ;;#ASMSTART
+; GCN2-NEXT:    ; def vcc
+; GCN2-NEXT:    ;;#ASMEND
+; GCN2-NEXT:    ds_write_b32 v0, v0
+; GCN2-NEXT:    ;;#ASMSTART
+; GCN2-NEXT:    ; use vcc
+; GCN2-NEXT:    ;;#ASMEND
+; GCN2-NEXT:    s_endpgm
+;
+; GCN3-LABEL: sub_select_vop3:
+; GCN3:       ; %bb.0:
+; GCN3-NEXT:    v_subrev_u32_e32 v0, s0, v0
+; GCN3-NEXT:    ;;#ASMSTART
+; GCN3-NEXT:    ; def vcc
+; GCN3-NEXT:    ;;#ASMEND
+; GCN3-NEXT:    ds_write_b32 v0, v0
+; GCN3-NEXT:    ;;#ASMSTART
+; GCN3-NEXT:    ; use vcc
+; GCN3-NEXT:    ;;#ASMEND
+; GCN3-NEXT:    s_endpgm
   %vcc = call i64 asm sideeffect "; def vcc", "={vcc}"()
   %sub = sub i32 %v, %s
   store i32 %sub, ptr addrspace(3) undef
   call void asm sideeffect "; use vcc", "{vcc}"(i64 %vcc)
   ret void
 }

llvm/test/CodeGen/AMDGPU/udiv.ll

 ; SI-NEXT:    v_cvt_u32_f32_e32 v2, v2
 ; SI-NEXT:    v_mul_lo_u32 v3, v3, v2
 ; SI-NEXT:    v_mul_hi_u32 v3, v2, v3
 ; SI-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; SI-NEXT:    v_mul_hi_u32 v2, v0, v2
 ; SI-NEXT:    v_mul_lo_u32 v3, v2, v1
 ; SI-NEXT:    v_add_i32_e32 v4, vcc, 1, v2
 ; SI-NEXT:    v_sub_i32_e32 v0, vcc, v0, v3
-; SI-NEXT:    v_subrev_i32_e32 v3, vcc, v1, v0
+; SI-NEXT:    v_sub_i32_e32 v3, vcc, v0, v1
 ; SI-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v1
 ; SI-NEXT:    v_cndmask_b32_e32 v2, v2, v4, vcc
 ; SI-NEXT:    v_cndmask_b32_e32 v0, v0, v3, vcc
 ; SI-NEXT:    v_add_i32_e32 v3, vcc, 1, v2
 ; SI-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v1
 ; SI-NEXT:    v_cndmask_b32_e32 v0, v2, v3, vcc
 ; SI-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; SI-NEXT:    s_endpgm

 ; VI-NEXT:    v_cvt_u32_f32_e32 v2, v2
 ; VI-NEXT:    v_mul_lo_u32 v3, v3, v2
 ; VI-NEXT:    v_mul_hi_u32 v3, v2, v3
 ; VI-NEXT:    v_add_u32_e32 v2, vcc, v2, v3
 ; VI-NEXT:    v_mul_hi_u32 v2, v0, v2
 ; VI-NEXT:    v_mul_lo_u32 v3, v2, v1
 ; VI-NEXT:    v_add_u32_e32 v4, vcc, 1, v2
 ; VI-NEXT:    v_sub_u32_e32 v0, vcc, v0, v3
-; VI-NEXT:    v_subrev_u32_e32 v3, vcc, v1, v0
+; VI-NEXT:    v_sub_u32_e32 v3, vcc, v0, v1
 ; VI-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v1
 ; VI-NEXT:    v_cndmask_b32_e32 v2, v2, v4, vcc
 ; VI-NEXT:    v_cndmask_b32_e32 v0, v0, v3, vcc
 ; VI-NEXT:    v_add_u32_e32 v3, vcc, 1, v2
 ; VI-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v1
 ; VI-NEXT:    v_cndmask_b32_e32 v0, v2, v3, vcc
 ; VI-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; VI-NEXT:    s_endpgm

 ; GCN-NEXT:    v_mul_hi_u32 v3, v2, v3
 ; GCN-NEXT:    v_add_u32_e32 v2, vcc, v2, v3
 ; GCN-NEXT:    v_mul_hi_u32 v4, v0, v2
 ; GCN-NEXT:    v_mov_b32_e32 v2, s0
 ; GCN-NEXT:    v_mov_b32_e32 v3, s1
 ; GCN-NEXT:    v_mul_lo_u32 v5, v4, v1
 ; GCN-NEXT:    v_add_u32_e32 v6, vcc, 1, v4
 ; GCN-NEXT:    v_sub_u32_e32 v0, vcc, v0, v5
-; GCN-NEXT:    v_subrev_u32_e32 v5, vcc, v1, v0
+; GCN-NEXT:    v_sub_u32_e32 v5, vcc, v0, v1
 ; GCN-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v1
 ; GCN-NEXT:    v_cndmask_b32_e32 v4, v4, v6, vcc
 ; GCN-NEXT:    v_cndmask_b32_e32 v0, v0, v5, vcc
 ; GCN-NEXT:    v_add_u32_e32 v5, vcc, 1, v4
 ; GCN-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v1
 ; GCN-NEXT:    v_cndmask_b32_e32 v0, v4, v5, vcc
 ; GCN-NEXT:    flat_store_dword v[2:3], v0
 ; GCN-NEXT:    s_endpgm

 ; SI-NEXT:    s_sub_i32 s4, 0, s3
 ; SI-NEXT:    s_mov_b32 s5, s1
 ; SI-NEXT:    v_rcp_iflag_f32_e32 v0, v0
 ; SI-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; SI-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; SI-NEXT:    v_mul_lo_u32 v1, s4, v0
 ; SI-NEXT:    s_mov_b32 s4, s0
 ; SI-NEXT:    v_mul_hi_u32 v1, v0, v1
-; SI-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
+; SI-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; SI-NEXT:    v_mul_hi_u32 v0, s2, v0
 ; SI-NEXT:    v_readfirstlane_b32 s0, v0
 ; SI-NEXT:    s_mul_i32 s0, s0, s3
 ; SI-NEXT:    s_sub_i32 s0, s2, s0
 ; SI-NEXT:    s_sub_i32 s1, s0, s3
 ; SI-NEXT:    v_add_i32_e32 v1, vcc, 1, v0
 ; SI-NEXT:    s_cmp_ge_u32 s0, s3
 ; SI-NEXT:    s_cselect_b64 vcc, -1, 0

 ; VI-NEXT:    s_sub_i32 s4, 0, s3
 ; VI-NEXT:    s_mov_b32 s5, s1
 ; VI-NEXT:    v_rcp_iflag_f32_e32 v0, v0
 ; VI-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; VI-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; VI-NEXT:    v_mul_lo_u32 v1, s4, v0
 ; VI-NEXT:    s_mov_b32 s4, s0
 ; VI-NEXT:    v_mul_hi_u32 v1, v0, v1
-; VI-NEXT:    v_add_u32_e32 v0, vcc, v1, v0
+; VI-NEXT:    v_add_u32_e32 v0, vcc, v0, v1
 ; VI-NEXT:    v_mul_hi_u32 v0, s2, v0
 ; VI-NEXT:    v_readfirstlane_b32 s0, v0
 ; VI-NEXT:    s_mul_i32 s0, s0, s3
 ; VI-NEXT:    s_sub_i32 s0, s2, s0
 ; VI-NEXT:    s_sub_i32 s1, s0, s3
 ; VI-NEXT:    v_add_u32_e32 v1, vcc, 1, v0
 ; VI-NEXT:    s_cmp_ge_u32 s0, s3
 ; VI-NEXT:    s_cselect_b64 vcc, -1, 0

 ; GCN-NEXT:    s_waitcnt lgkmcnt(0)
 ; GCN-NEXT:    v_cvt_f32_u32_e32 v0, s3
 ; GCN-NEXT:    s_sub_i32 s4, 0, s3
 ; GCN-NEXT:    v_rcp_iflag_f32_e32 v0, v0
 ; GCN-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; GCN-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; GCN-NEXT:    v_mul_lo_u32 v1, s4, v0
 ; GCN-NEXT:    v_mul_hi_u32 v1, v0, v1
-; GCN-NEXT:    v_add_u32_e32 v0, vcc, v1, v0
+; GCN-NEXT:    v_add_u32_e32 v0, vcc, v0, v1
 ; GCN-NEXT:    v_mul_hi_u32 v0, s2, v0
 ; GCN-NEXT:    v_readfirstlane_b32 s4, v0
 ; GCN-NEXT:    s_mul_i32 s4, s4, s3
 ; GCN-NEXT:    s_sub_i32 s2, s2, s4
 ; GCN-NEXT:    s_sub_i32 s4, s2, s3
 ; GCN-NEXT:    v_add_u32_e32 v1, vcc, 1, v0
 ; GCN-NEXT:    s_cmp_ge_u32 s2, s3
 ; GCN-NEXT:    s_cselect_b64 vcc, -1, 0

 ; SI-NEXT:    v_mul_f32_e32 v4, 0x4f7ffffe, v4
 ; SI-NEXT:    v_mul_f32_e32 v5, 0x4f7ffffe, v5
 ; SI-NEXT:    v_cvt_u32_f32_e32 v4, v4
 ; SI-NEXT:    v_cvt_u32_f32_e32 v5, v5
 ; SI-NEXT:    v_mul_lo_u32 v6, v6, v4
 ; SI-NEXT:    v_mul_lo_u32 v7, v7, v5
 ; SI-NEXT:    v_mul_hi_u32 v6, v4, v6
 ; SI-NEXT:    v_mul_hi_u32 v7, v5, v7
-; SI-NEXT:    v_add_i32_e32 v4, vcc, v6, v4
+; SI-NEXT:    v_add_i32_e32 v4, vcc, v4, v6
 ; SI-NEXT:    v_add_i32_e32 v5, vcc, v5, v7
 ; SI-NEXT:    v_mul_hi_u32 v4, v0, v4
 ; SI-NEXT:    v_mul_hi_u32 v5, v1, v5
 ; SI-NEXT:    v_mul_lo_u32 v6, v4, v2
 ; SI-NEXT:    v_mul_lo_u32 v8, v5, v3
 ; SI-NEXT:    v_add_i32_e32 v7, vcc, 1, v4
 ; SI-NEXT:    v_sub_i32_e32 v0, vcc, v0, v6
 ; SI-NEXT:    v_sub_i32_e32 v1, vcc, v1, v8
 ; SI-NEXT:    v_add_i32_e32 v9, vcc, 1, v5
 ; SI-NEXT:    v_cmp_ge_u32_e64 s[0:1], v0, v2
 ; SI-NEXT:    v_cmp_ge_u32_e64 s[2:3], v1, v3
-; SI-NEXT:    v_subrev_i32_e32 v6, vcc, v2, v0
+; SI-NEXT:    v_sub_i32_e32 v6, vcc, v0, v2
 ; SI-NEXT:    v_cndmask_b32_e64 v4, v4, v7, s[0:1]
-; SI-NEXT:    v_subrev_i32_e32 v7, vcc, v3, v1
+; SI-NEXT:    v_sub_i32_e32 v7, vcc, v1, v3
 ; SI-NEXT:    v_cndmask_b32_e64 v5, v5, v9, s[2:3]
 ; SI-NEXT:    v_cndmask_b32_e64 v0, v0, v6, s[0:1]
 ; SI-NEXT:    v_add_i32_e32 v6, vcc, 1, v4
 ; SI-NEXT:    v_cndmask_b32_e64 v1, v1, v7, s[2:3]
 ; SI-NEXT:    v_add_i32_e32 v7, vcc, 1, v5
 ; SI-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v2
 ; SI-NEXT:    v_cndmask_b32_e32 v0, v4, v6, vcc
 ; SI-NEXT:    v_cmp_ge_u32_e32 vcc, v1, v3

 ; VI-NEXT:    v_mul_f32_e32 v4, 0x4f7ffffe, v4
 ; VI-NEXT:    v_mul_f32_e32 v5, 0x4f7ffffe, v5
 ; VI-NEXT:    v_cvt_u32_f32_e32 v4, v4
 ; VI-NEXT:    v_cvt_u32_f32_e32 v5, v5
 ; VI-NEXT:    v_mul_lo_u32 v6, v6, v4
 ; VI-NEXT:    v_mul_lo_u32 v7, v7, v5
 ; VI-NEXT:    v_mul_hi_u32 v6, v4, v6
 ; VI-NEXT:    v_mul_hi_u32 v7, v5, v7
-; VI-NEXT:    v_add_u32_e32 v4, vcc, v6, v4
+; VI-NEXT:    v_add_u32_e32 v4, vcc, v4, v6
 ; VI-NEXT:    v_add_u32_e32 v5, vcc, v5, v7
 ; VI-NEXT:    v_mul_hi_u32 v4, v0, v4
 ; VI-NEXT:    v_mul_hi_u32 v5, v1, v5
 ; VI-NEXT:    v_mul_lo_u32 v6, v4, v2
 ; VI-NEXT:    v_mul_lo_u32 v8, v5, v3
 ; VI-NEXT:    v_add_u32_e32 v7, vcc, 1, v4
 ; VI-NEXT:    v_sub_u32_e32 v0, vcc, v0, v6
 ; VI-NEXT:    v_sub_u32_e32 v1, vcc, v1, v8
 ; VI-NEXT:    v_add_u32_e32 v9, vcc, 1, v5
 ; VI-NEXT:    v_cmp_ge_u32_e64 s[0:1], v0, v2
 ; VI-NEXT:    v_cmp_ge_u32_e64 s[2:3], v1, v3
-; VI-NEXT:    v_subrev_u32_e32 v6, vcc, v2, v0
+; VI-NEXT:    v_sub_u32_e32 v6, vcc, v0, v2
 ; VI-NEXT:    v_cndmask_b32_e64 v4, v4, v7, s[0:1]
-; VI-NEXT:    v_subrev_u32_e32 v7, vcc, v3, v1
+; VI-NEXT:    v_sub_u32_e32 v7, vcc, v1, v3
 ; VI-NEXT:    v_cndmask_b32_e64 v5, v5, v9, s[2:3]
 ; VI-NEXT:    v_cndmask_b32_e64 v0, v0, v6, s[0:1]
 ; VI-NEXT:    v_add_u32_e32 v6, vcc, 1, v4
 ; VI-NEXT:    v_cndmask_b32_e64 v1, v1, v7, s[2:3]
 ; VI-NEXT:    v_add_u32_e32 v7, vcc, 1, v5
 ; VI-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v2
 ; VI-NEXT:    v_cndmask_b32_e32 v0, v4, v6, vcc
 ; VI-NEXT:    v_cmp_ge_u32_e32 vcc, v1, v3

 ; GCN-NEXT:    v_sub_u32_e32 v4, vcc, 0, v2
 ; GCN-NEXT:    v_mul_lo_u32 v5, v4, v6
 ; GCN-NEXT:    v_sub_u32_e32 v4, vcc, 0, v3
 ; GCN-NEXT:    v_mul_lo_u32 v8, v4, v7
 ; GCN-NEXT:    v_mul_hi_u32 v9, v6, v5
 ; GCN-NEXT:    v_mov_b32_e32 v4, s0
 ; GCN-NEXT:    v_mov_b32_e32 v5, s1
 ; GCN-NEXT:    v_mul_hi_u32 v8, v7, v8
-; GCN-NEXT:    v_add_u32_e32 v6, vcc, v9, v6
+; GCN-NEXT:    v_add_u32_e32 v6, vcc, v6, v9
 ; GCN-NEXT:    v_mul_hi_u32 v6, v0, v6
 ; GCN-NEXT:    v_add_u32_e32 v7, vcc, v7, v8
 ; GCN-NEXT:    v_mul_hi_u32 v7, v1, v7
 ; GCN-NEXT:    v_mul_lo_u32 v8, v6, v2
 ; GCN-NEXT:    v_add_u32_e32 v9, vcc, 1, v6
 ; GCN-NEXT:    v_mul_lo_u32 v10, v7, v3
 ; GCN-NEXT:    v_sub_u32_e32 v0, vcc, v0, v8
 ; GCN-NEXT:    v_add_u32_e32 v11, vcc, 1, v7
 ; GCN-NEXT:    v_sub_u32_e32 v1, vcc, v1, v10
 ; GCN-NEXT:    v_cmp_ge_u32_e64 s[0:1], v0, v2
 ; GCN-NEXT:    v_cmp_ge_u32_e64 s[2:3], v1, v3
-; GCN-NEXT:    v_subrev_u32_e32 v8, vcc, v2, v0
+; GCN-NEXT:    v_sub_u32_e32 v8, vcc, v0, v2
 ; GCN-NEXT:    v_cndmask_b32_e64 v6, v6, v9, s[0:1]
-; GCN-NEXT:    v_subrev_u32_e32 v9, vcc, v3, v1
+; GCN-NEXT:    v_sub_u32_e32 v9, vcc, v1, v3
 ; GCN-NEXT:    v_cndmask_b32_e64 v7, v7, v11, s[2:3]
 ; GCN-NEXT:    v_cndmask_b32_e64 v0, v0, v8, s[0:1]
 ; GCN-NEXT:    v_add_u32_e32 v8, vcc, 1, v6
 ; GCN-NEXT:    v_cndmask_b32_e64 v1, v1, v9, s[2:3]
 ; GCN-NEXT:    v_add_u32_e32 v9, vcc, 1, v7
 ; GCN-NEXT:    v_cmp_ge_u32_e32 vcc, v0, v2
 ; GCN-NEXT:    v_cndmask_b32_e32 v0, v6, v8, vcc
 ; GCN-NEXT:    v_cmp_ge_u32_e32 vcc, v1, v3

 ; SI-NEXT:    v_mul_lo_u32 v11, v11, v10
 ; SI-NEXT:    v_mul_lo_u32 v13, v13, v12
 ; SI-NEXT:    v_mul_lo_u32 v15, v15, v14
 ; SI-NEXT:    v_mul_hi_u32 v9, v8, v9
 ; SI-NEXT:    v_mul_hi_u32 v11, v10, v11
 ; SI-NEXT:    v_mul_hi_u32 v13, v12, v13
 ; SI-NEXT:    v_mul_hi_u32 v15, v14, v15
 ; SI-NEXT:    v_add_i32_e32 v8, vcc, v8, v9
-; SI-NEXT:    v_add_i32_e32 v9, vcc, v11, v10
+; SI-NEXT:    v_add_i32_e32 v9, vcc, v10, v11
 ; SI-NEXT:    v_add_i32_e32 v10, vcc, v12, v13
-; SI-NEXT:    v_add_i32_e32 v11, vcc, v15, v14
+; SI-NEXT:    v_add_i32_e32 v11, vcc, v14, v15
 ; SI-NEXT:    s_waitcnt vmcnt(0)
 ; SI-NEXT:    v_mul_hi_u32 v8, v4, v8
 ; SI-NEXT:    v_mul_hi_u32 v9, v5, v9
 ; SI-NEXT:    v_mul_hi_u32 v10, v6, v10
 ; SI-NEXT:    v_mul_hi_u32 v11, v7, v11
 ; SI-NEXT:    v_mul_lo_u32 v12, v8, v0
 ; SI-NEXT:    v_mul_lo_u32 v14, v9, v1
 ; SI-NEXT:    v_mul_lo_u32 v16, v10, v2
 ; SI-NEXT:    v_mul_lo_u32 v18, v11, v3
 ; SI-NEXT:    v_sub_i32_e32 v4, vcc, v4, v12
 ; SI-NEXT:    v_sub_i32_e32 v5, vcc, v5, v14
 ; SI-NEXT:    v_sub_i32_e32 v6, vcc, v6, v16
 ; SI-NEXT:    v_sub_i32_e32 v7, vcc, v7, v18
 ; SI-NEXT:    v_add_i32_e32 v13, vcc, 1, v8
 ; SI-NEXT:    v_add_i32_e32 v15, vcc, 1, v9
 ; SI-NEXT:    v_add_i32_e32 v17, vcc, 1, v10
 ; SI-NEXT:    v_add_i32_e32 v19, vcc, 1, v11
 ; SI-NEXT:    v_cmp_ge_u32_e64 s[0:1], v4, v0
 ; SI-NEXT:    v_cmp_ge_u32_e64 s[2:3], v5, v1
 ; SI-NEXT:    v_cmp_ge_u32_e64 s[4:5], v6, v2
 ; SI-NEXT:    v_cmp_ge_u32_e64 s[6:7], v7, v3
-; SI-NEXT:    v_subrev_i32_e32 v12, vcc, v0, v4
+; SI-NEXT:    v_sub_i32_e32 v12, vcc, v4, v0
 ; SI-NEXT:    v_cndmask_b32_e64 v8, v8, v13, s[0:1]
-; SI-NEXT:    v_subrev_i32_e32 v13, vcc, v1, v5
+; SI-NEXT:    v_sub_i32_e32 v13, vcc, v5, v1
 ; SI-NEXT:    v_cndmask_b32_e64 v9, v9, v15, s[2:3]
-; SI-NEXT:    v_subrev_i32_e32 v14, vcc, v2, v6
+; SI-NEXT:    v_sub_i32_e32 v14, vcc, v6, v2
 ; SI-NEXT:    v_cndmask_b32_e64 v10, v10, v17, s[4:5]
-; SI-NEXT:    v_subrev_i32_e32 v15, vcc, v3, v7
+; SI-NEXT:    v_sub_i32_e32 v15, vcc, v7, v3
 ; SI-NEXT:    v_cndmask_b32_e64 v11, v11, v19, s[6:7]
 ; SI-NEXT:    v_cndmask_b32_e64 v4, v4, v12, s[0:1]
 ; SI-NEXT:    v_add_i32_e32 v12, vcc, 1, v8
 ; SI-NEXT:    v_cndmask_b32_e64 v5, v5, v13, s[2:3]
 ; SI-NEXT:    v_add_i32_e32 v13, vcc, 1, v9
 ; SI-NEXT:    v_cndmask_b32_e64 v6, v6, v14, s[4:5]
 ; SI-NEXT:    v_add_i32_e32 v14, vcc, 1, v10
 ; SI-NEXT:    v_cndmask_b32_e64 v7, v7, v15, s[6:7]

 ; VI-NEXT:    v_mul_lo_u32 v11, v11, v10
 ; VI-NEXT:    v_mul_lo_u32 v13, v13, v12
 ; VI-NEXT:    v_mul_lo_u32 v15, v15, v14
 ; VI-NEXT:    v_mul_hi_u32 v9, v8, v9
 ; VI-NEXT:    v_mul_hi_u32 v11, v10, v11
 ; VI-NEXT:    v_mul_hi_u32 v13, v12, v13
 ; VI-NEXT:    v_mul_hi_u32 v15, v14, v15
 ; VI-NEXT:    v_add_u32_e32 v8, vcc, v8, v9
-; VI-NEXT:    v_add_u32_e32 v9, vcc, v11, v10
+; VI-NEXT:    v_add_u32_e32 v9, vcc, v10, v11
 ; VI-NEXT:    v_add_u32_e32 v10, vcc, v12, v13
-; VI-NEXT:    v_add_u32_e32 v11, vcc, v15, v14
+; VI-NEXT:    v_add_u32_e32 v11, vcc, v14, v15
 ; VI-NEXT:    s_waitcnt vmcnt(0)
 ; VI-NEXT:    v_mul_hi_u32 v8, v4, v8
 ; VI-NEXT:    v_mul_hi_u32 v9, v5, v9
 ; VI-NEXT:    v_mul_hi_u32 v10, v6, v10
 ; VI-NEXT:    v_mul_hi_u32 v11, v7, v11
 ; VI-NEXT:    v_mul_lo_u32 v12, v8, v0
 ; VI-NEXT:    v_mul_lo_u32 v14, v9, v1
 ; VI-NEXT:    v_mul_lo_u32 v16, v10, v2
 ; VI-NEXT:    v_mul_lo_u32 v18, v11, v3
 ; VI-NEXT:    v_sub_u32_e32 v4, vcc, v4, v12
 ; VI-NEXT:    v_sub_u32_e32 v5, vcc, v5, v14
 ; VI-NEXT:    v_sub_u32_e32 v6, vcc, v6, v16
 ; VI-NEXT:    v_sub_u32_e32 v7, vcc, v7, v18
 ; VI-NEXT:    v_add_u32_e32 v13, vcc, 1, v8
 ; VI-NEXT:    v_add_u32_e32 v15, vcc, 1, v9
 ; VI-NEXT:    v_add_u32_e32 v17, vcc, 1, v10
 ; VI-NEXT:    v_add_u32_e32 v19, vcc, 1, v11
 ; VI-NEXT:    v_cmp_ge_u32_e64 s[0:1], v4, v0
 ; VI-NEXT:    v_cmp_ge_u32_e64 s[2:3], v5, v1
 ; VI-NEXT:    v_cmp_ge_u32_e64 s[4:5], v6, v2
 ; VI-NEXT:    v_cmp_ge_u32_e64 s[6:7], v7, v3
-; VI-NEXT:    v_subrev_u32_e32 v12, vcc, v0, v4
+; VI-NEXT:    v_sub_u32_e32 v12, vcc, v4, v0
 ; VI-NEXT:    v_cndmask_b32_e64 v8, v8, v13, s[0:1]
-; VI-NEXT:    v_subrev_u32_e32 v13, vcc, v1, v5
+; VI-NEXT:    v_sub_u32_e32 v13, vcc, v5, v1
 ; VI-NEXT:    v_cndmask_b32_e64 v9, v9, v15, s[2:3]
-; VI-NEXT:    v_subrev_u32_e32 v14, vcc, v2, v6
+; VI-NEXT:    v_sub_u32_e32 v14, vcc, v6, v2
 ; VI-NEXT:    v_cndmask_b32_e64 v10, v10, v17, s[4:5]
-; VI-NEXT:    v_subrev_u32_e32 v15, vcc, v3, v7
+; VI-NEXT:    v_sub_u32_e32 v15, vcc, v7, v3
 ; VI-NEXT:    v_cndmask_b32_e64 v11, v11, v19, s[6:7]
 ; VI-NEXT:    v_cndmask_b32_e64 v4, v4, v12, s[0:1]
 ; VI-NEXT:    v_add_u32_e32 v12, vcc, 1, v8
 ; VI-NEXT:    v_cndmask_b32_e64 v5, v5, v13, s[2:3]
 ; VI-NEXT:    v_add_u32_e32 v13, vcc, 1, v9
 ; VI-NEXT:    v_cndmask_b32_e64 v6, v6, v14, s[4:5]
 ; VI-NEXT:    v_add_u32_e32 v14, vcc, 1, v10
 ; VI-NEXT:    v_cndmask_b32_e64 v7, v7, v15, s[6:7]

 ; GCN-NEXT:    v_mul_lo_u32 v13, v13, v12
 ; GCN-NEXT:    v_mul_lo_u32 v15, v15, v14
 ; GCN-NEXT:    v_mul_lo_u32 v17, v17, v16
 ; GCN-NEXT:    v_mul_hi_u32 v11, v10, v11
 ; GCN-NEXT:    v_mul_hi_u32 v13, v12, v13
 ; GCN-NEXT:    v_mul_hi_u32 v15, v14, v15
 ; GCN-NEXT:    v_mul_hi_u32 v17, v16, v17
 ; GCN-NEXT:    v_add_u32_e32 v10, vcc, v10, v11
-; GCN-NEXT:    v_add_u32_e32 v11, vcc, v13, v12
+; GCN-NEXT:    v_add_u32_e32 v11, vcc, v12, v13
 ; GCN-NEXT:    v_add_u32_e32 v12, vcc, v14, v15
-; GCN-NEXT:    v_add_u32_e32 v13, vcc, v17, v16
+; GCN-NEXT:    v_add_u32_e32 v13, vcc, v16, v17
 ; GCN-NEXT:    s_waitcnt vmcnt(0)
 ; GCN-NEXT:    v_mul_hi_u32 v10, v4, v10
 ; GCN-NEXT:    v_mul_hi_u32 v11, v5, v11
 ; GCN-NEXT:    v_mul_hi_u32 v12, v6, v12
 ; GCN-NEXT:    v_mul_hi_u32 v13, v7, v13
 ; GCN-NEXT:    v_mul_lo_u32 v14, v10, v0
 ; GCN-NEXT:    v_mul_lo_u32 v16, v11, v1
 ; GCN-NEXT:    v_mul_lo_u32 v18, v12, v2
 ; GCN-NEXT:    v_mul_lo_u32 v20, v13, v3
 ; GCN-NEXT:    v_sub_u32_e32 v4, vcc, v4, v14
 ; GCN-NEXT:    v_sub_u32_e32 v5, vcc, v5, v16
 ; GCN-NEXT:    v_sub_u32_e32 v6, vcc, v6, v18
 ; GCN-NEXT:    v_sub_u32_e32 v7, vcc, v7, v20
 ; GCN-NEXT:    v_add_u32_e32 v15, vcc, 1, v10
 ; GCN-NEXT:    v_add_u32_e32 v17, vcc, 1, v11
 ; GCN-NEXT:    v_add_u32_e32 v19, vcc, 1, v12
 ; GCN-NEXT:    v_add_u32_e32 v21, vcc, 1, v13
 ; GCN-NEXT:    v_cmp_ge_u32_e64 s[0:1], v4, v0
 ; GCN-NEXT:    v_cmp_ge_u32_e64 s[2:3], v5, v1
 ; GCN-NEXT:    v_cmp_ge_u32_e64 s[4:5], v6, v2
 ; GCN-NEXT:    v_cmp_ge_u32_e64 s[6:7], v7, v3
-; GCN-NEXT:    v_subrev_u32_e32 v14, vcc, v0, v4
+; GCN-NEXT:    v_sub_u32_e32 v14, vcc, v4, v0
 ; GCN-NEXT:    v_cndmask_b32_e64 v10, v10, v15, s[0:1]
-; GCN-NEXT:    v_subrev_u32_e32 v15, vcc, v1, v5
+; GCN-NEXT:    v_sub_u32_e32 v15, vcc, v5, v1
 ; GCN-NEXT:    v_cndmask_b32_e64 v11, v11, v17, s[2:3]
-; GCN-NEXT:    v_subrev_u32_e32 v16, vcc, v2, v6
+; GCN-NEXT:    v_sub_u32_e32 v16, vcc, v6, v2
 ; GCN-NEXT:    v_cndmask_b32_e64 v12, v12, v19, s[4:5]
-; GCN-NEXT:    v_subrev_u32_e32 v17, vcc, v3, v7
+; GCN-NEXT:    v_sub_u32_e32 v17, vcc, v7, v3
 ; GCN-NEXT:    v_cndmask_b32_e64 v13, v13, v21, s[6:7]
 ; GCN-NEXT:    v_cndmask_b32_e64 v4, v4, v14, s[0:1]
 ; GCN-NEXT:    v_add_u32_e32 v14, vcc, 1, v10
 ; GCN-NEXT:    v_cndmask_b32_e64 v5, v5, v15, s[2:3]
 ; GCN-NEXT:    v_add_u32_e32 v15, vcc, 1, v11
 ; GCN-NEXT:    v_cndmask_b32_e64 v6, v6, v16, s[4:5]
 ; GCN-NEXT:    v_add_u32_e32 v16, vcc, 1, v12
 ; GCN-NEXT:    v_cndmask_b32_e64 v7, v7, v17, s[6:7]

 ; SI-NEXT:    v_cvt_u32_f32_e32 v1, v1
 ; SI-NEXT:    v_mul_lo_u32 v4, v4, v1
 ; SI-NEXT:    v_mul_hi_u32 v4, v1, v4
 ; SI-NEXT:    v_add_i32_e32 v1, vcc, v1, v4
 ; SI-NEXT:    v_mul_hi_u32 v1, v2, v1
 ; SI-NEXT:    v_mul_lo_u32 v3, v1, v0
 ; SI-NEXT:    v_add_i32_e32 v4, vcc, 1, v1
 ; SI-NEXT:    v_sub_i32_e32 v2, vcc, v2, v3
-; SI-NEXT:    v_subrev_i32_e32 v3, vcc, v0, v2
+; SI-NEXT:    v_sub_i32_e32 v3, vcc, v2, v0
 ; SI-NEXT:    v_cmp_ge_u32_e32 vcc, v2, v0
 ; SI-NEXT:    v_cndmask_b32_e32 v1, v1, v4, vcc
 ; SI-NEXT:    v_cndmask_b32_e32 v2, v2, v3, vcc
 ; SI-NEXT:    v_add_i32_e32 v3, vcc, 1, v1
 ; SI-NEXT:    v_cmp_ge_u32_e32 vcc, v2, v0
 ; SI-NEXT:    v_cndmask_b32_e32 v0, v1, v3, vcc
 ; SI-NEXT:    v_and_b32_e32 v0, 0xffffff, v0
 ; SI-NEXT:    buffer_store_dword v0, off, s[4:7], 0

 ; VI-NEXT:    v_cvt_u32_f32_e32 v1, v1
 ; VI-NEXT:    v_mul_lo_u32 v4, v4, v1
 ; VI-NEXT:    v_mul_hi_u32 v4, v1, v4
 ; VI-NEXT:    v_add_u32_e32 v1, vcc, v1, v4
 ; VI-NEXT:    v_mul_hi_u32 v1, v2, v1
 ; VI-NEXT:    v_mul_lo_u32 v3, v1, v0
 ; VI-NEXT:    v_add_u32_e32 v4, vcc, 1, v1
 ; VI-NEXT:    v_sub_u32_e32 v2, vcc, v2, v3
-; VI-NEXT:    v_subrev_u32_e32 v3, vcc, v0, v2
+; VI-NEXT:    v_sub_u32_e32 v3, vcc, v2, v0
 ; VI-NEXT:    v_cmp_ge_u32_e32 vcc, v2, v0
 ; VI-NEXT:    v_cndmask_b32_e32 v1, v1, v4, vcc
 ; VI-NEXT:    v_cndmask_b32_e32 v2, v2, v3, vcc
 ; VI-NEXT:    v_add_u32_e32 v3, vcc, 1, v1
 ; VI-NEXT:    v_cmp_ge_u32_e32 vcc, v2, v0
 ; VI-NEXT:    v_cndmask_b32_e32 v0, v1, v3, vcc
 ; VI-NEXT:    v_and_b32_e32 v0, 0xffffff, v0
 ; VI-NEXT:    buffer_store_dword v0, off, s[4:7], 0

 ; GCN-NEXT:    v_mul_hi_u32 v4, v1, v4
 ; GCN-NEXT:    v_add_u32_e32 v0, vcc, v1, v4
 ; GCN-NEXT:    v_mul_hi_u32 v4, v2, v0
 ; GCN-NEXT:    v_mov_b32_e32 v0, s0
 ; GCN-NEXT:    v_mov_b32_e32 v1, s1
 ; GCN-NEXT:    v_mul_lo_u32 v5, v4, v3
 ; GCN-NEXT:    v_add_u32_e32 v6, vcc, 1, v4
 ; GCN-NEXT:    v_sub_u32_e32 v2, vcc, v2, v5
-; GCN-NEXT:    v_subrev_u32_e32 v5, vcc, v3, v2
+; GCN-NEXT:    v_sub_u32_e32 v5, vcc, v2, v3
 ; GCN-NEXT:    v_cmp_ge_u32_e32 vcc, v2, v3
 ; GCN-NEXT:    v_cndmask_b32_e32 v4, v4, v6, vcc
 ; GCN-NEXT:    v_cndmask_b32_e32 v2, v2, v5, vcc
 ; GCN-NEXT:    v_add_u32_e32 v5, vcc, 1, v4
 ; GCN-NEXT:    v_cmp_ge_u32_e32 vcc, v2, v3
 ; GCN-NEXT:    v_cndmask_b32_e32 v2, v4, v5, vcc
 ; GCN-NEXT:    v_and_b32_e32 v2, 0xffffff, v2
 ; GCN-NEXT:    flat_store_dword v[0:1], v2

 ; SI-NEXT:    v_rcp_iflag_f32_e32 v4, v2
 ; SI-NEXT:    v_or_b32_e32 v0, 1, v0
 ; SI-NEXT:    v_mul_f32_e32 v1, v3, v4
 ; SI-NEXT:    v_trunc_f32_e32 v1, v1
 ; SI-NEXT:    v_mad_f32 v3, -v1, v2, v3
 ; SI-NEXT:    v_cvt_i32_f32_e32 v1, v1
 ; SI-NEXT:    v_cmp_ge_f32_e64 vcc, |v3|, |v2|
 ; SI-NEXT:    v_cndmask_b32_e32 v0, 0, v0, vcc
-; SI-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
+; SI-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; SI-NEXT:    buffer_store_byte v0, off, s[0:3], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: fdiv_test_denormals:
 ; VI:       ; %bb.0: ; %bb
 ; VI-NEXT:    s_mov_b32 s0, 0
 ; VI-NEXT:    s_mov_b32 s3, 0xf000
 ; VI-NEXT:    s_mov_b32 s2, -1

 ; VI-NEXT:    v_rcp_iflag_f32_e32 v4, v2
 ; VI-NEXT:    v_or_b32_e32 v0, 1, v0
 ; VI-NEXT:    v_mul_f32_e32 v1, v3, v4
 ; VI-NEXT:    v_trunc_f32_e32 v1, v1
 ; VI-NEXT:    v_mad_f32 v3, -v1, v2, v3
 ; VI-NEXT:    v_cvt_i32_f32_e32 v1, v1
 ; VI-NEXT:    v_cmp_ge_f32_e64 vcc, |v3|, |v2|
 ; VI-NEXT:    v_cndmask_b32_e32 v0, 0, v0, vcc
-; VI-NEXT:    v_add_u32_e32 v0, vcc, v1, v0
+; VI-NEXT:    v_add_u32_e32 v0, vcc, v0, v1
 ; VI-NEXT:    buffer_store_byte v0, off, s[0:3], 0
 ; VI-NEXT:    s_endpgm
 ;
 ; GCN-LABEL: fdiv_test_denormals:
 ; GCN:       ; %bb.0: ; %bb
 ; GCN-NEXT:    flat_load_sbyte v2, v[0:1]
 ; GCN-NEXT:    v_mov_b32_e32 v0, 0
 ; GCN-NEXT:    v_mov_b32_e32 v1, 0
 ; GCN-NEXT:    flat_load_sbyte v3, v[0:1]
 ; GCN-NEXT:    s_waitcnt vmcnt(1)
 ; GCN-NEXT:    v_cvt_f32_i32_e32 v4, v2
 ; GCN-NEXT:    s_waitcnt vmcnt(0)
 ; GCN-NEXT:    v_cvt_f32_i32_e32 v5, v3
 ; GCN-NEXT:    v_rcp_iflag_f32_e32 v6, v4
 ; GCN-NEXT:    v_xor_b32_e32 v2, v3, v2
 ; GCN-NEXT:    v_ashrrev_i32_e32 v2, 30, v2
 ; GCN-NEXT:    v_or_b32_e32 v2, 1, v2
 ; GCN-NEXT:    v_mul_f32_e32 v3, v5, v6
 ; GCN-NEXT:    v_trunc_f32_e32 v3, v3
 ; GCN-NEXT:    v_mad_f32 v5, -v3, v4, v5
 ; GCN-NEXT:    v_cvt_i32_f32_e32 v3, v3
 ; GCN-NEXT:    v_cmp_ge_f32_e64 vcc, |v5|, |v4|
 ; GCN-NEXT:    v_cndmask_b32_e32 v2, 0, v2, vcc
-; GCN-NEXT:    v_add_u32_e32 v2, vcc, v3, v2
+; GCN-NEXT:    v_add_u32_e32 v2, vcc, v2, v3
 ; GCN-NEXT:    flat_store_byte v[0:1], v2
 ; GCN-NEXT:    s_endpgm
 ;
 ; GFX1030-LABEL: fdiv_test_denormals:
 ; GFX1030:       ; %bb.0: ; %bb
 ; GFX1030-NEXT:    global_load_sbyte v2, v[0:1], off
 ; GFX1030-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX1030-NEXT:    v_mov_b32_e32 v1, 0

 ; SI-NEXT:    v_addc_u32_e32 v5, vcc, 0, v6, vcc
 ; SI-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
 ; SI-NEXT:    v_addc_u32_e32 v3, vcc, v3, v5, vcc
 ; SI-NEXT:    v_mul_hi_u32 v4, v2, s4
 ; SI-NEXT:    v_mul_lo_u32 v5, v3, s4
 ; SI-NEXT:    v_mul_lo_u32 v6, v2, s4
 ; SI-NEXT:    s_mov_b32 s4, 0x186a0
 ; SI-NEXT:    v_sub_i32_e32 v4, vcc, v4, v2
-; SI-NEXT:    v_add_i32_e32 v4, vcc, v4, v5
+; SI-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
 ; SI-NEXT:    v_mul_lo_u32 v5, v2, v4
 ; SI-NEXT:    v_mul_hi_u32 v7, v2, v6
 ; SI-NEXT:    v_mul_hi_u32 v8, v2, v4
 ; SI-NEXT:    v_mul_hi_u32 v9, v3, v4
 ; SI-NEXT:    v_mul_lo_u32 v4, v3, v4
 ; SI-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
 ; SI-NEXT:    v_addc_u32_e32 v7, vcc, 0, v8, vcc
 ; SI-NEXT:    v_mul_lo_u32 v8, v3, v6

 ; VI-NEXT:    v_mul_f32_e32 v3, 0x2f800000, v2
 ; VI-NEXT:    v_trunc_f32_e32 v3, v3
 ; VI-NEXT:    v_mac_f32_e32 v2, 0xcf800000, v3
 ; VI-NEXT:    v_cvt_u32_f32_e32 v6, v2
 ; VI-NEXT:    v_cvt_u32_f32_e32 v7, v3
 ; VI-NEXT:    v_mad_u64_u32 v[2:3], s[4:5], v6, s6, 0
 ; VI-NEXT:    v_mul_lo_u32 v4, v7, s6
 ; VI-NEXT:    v_sub_u32_e32 v3, vcc, v3, v6
-; VI-NEXT:    v_add_u32_e32 v8, vcc, v4, v3
+; VI-NEXT:    v_add_u32_e32 v8, vcc, v3, v4
 ; VI-NEXT:    v_mul_hi_u32 v5, v6, v2
 ; VI-NEXT:    v_mad_u64_u32 v[3:4], s[4:5], v6, v8, 0
 ; VI-NEXT:    v_add_u32_e32 v9, vcc, v5, v3
 ; VI-NEXT:    v_mad_u64_u32 v[2:3], s[4:5], v7, v2, 0
 ; VI-NEXT:    v_addc_u32_e32 v10, vcc, 0, v4, vcc
 ; VI-NEXT:    v_mad_u64_u32 v[4:5], s[4:5], v7, v8, 0
 ; VI-NEXT:    v_add_u32_e32 v2, vcc, v9, v2
 ; VI-NEXT:    v_addc_u32_e32 v2, vcc, v10, v3, vcc
 ; VI-NEXT:    v_addc_u32_e32 v3, vcc, 0, v5, vcc
 ; VI-NEXT:    v_add_u32_e32 v2, vcc, v2, v4
 ; VI-NEXT:    v_addc_u32_e32 v3, vcc, 0, v3, vcc
 ; VI-NEXT:    v_add_u32_e32 v6, vcc, v6, v2
 ; VI-NEXT:    v_addc_u32_e32 v7, vcc, v7, v3, vcc
 ; VI-NEXT:    v_mad_u64_u32 v[2:3], s[4:5], v6, s6, 0
 ; VI-NEXT:    v_mul_lo_u32 v4, v7, s6
 ; VI-NEXT:    v_sub_u32_e32 v3, vcc, v3, v6
-; VI-NEXT:    v_add_u32_e32 v5, vcc, v3, v4
+; VI-NEXT:    v_add_u32_e32 v5, vcc, v4, v3
 ; VI-NEXT:    v_mad_u64_u32 v[3:4], s[4:5], v6, v5, 0
 ; VI-NEXT:    v_mul_hi_u32 v8, v6, v2
 ; VI-NEXT:    v_add_u32_e32 v8, vcc, v8, v3
 ; VI-NEXT:    v_mad_u64_u32 v[2:3], s[4:5], v7, v2, 0
 ; VI-NEXT:    v_addc_u32_e32 v9, vcc, 0, v4, vcc
 ; VI-NEXT:    v_mad_u64_u32 v[4:5], s[4:5], v7, v5, 0
 ; VI-NEXT:    v_add_u32_e32 v2, vcc, v8, v2
 ; VI-NEXT:    v_addc_u32_e32 v2, vcc, v9, v3, vcc

 ; GCN-NEXT:    v_mul_f32_e32 v3, 0x2f800000, v2
 ; GCN-NEXT:    v_trunc_f32_e32 v3, v3
 ; GCN-NEXT:    v_mac_f32_e32 v2, 0xcf800000, v3
 ; GCN-NEXT:    v_cvt_u32_f32_e32 v6, v2
 ; GCN-NEXT:    v_cvt_u32_f32_e32 v7, v3
 ; GCN-NEXT:    v_mad_u64_u32 v[2:3], s[4:5], v6, s6, 0
 ; GCN-NEXT:    v_mul_lo_u32 v4, v7, s6
 ; GCN-NEXT:    v_sub_u32_e32 v3, vcc, v3, v6
-; GCN-NEXT:    v_add_u32_e32 v8, vcc, v4, v3
+; GCN-NEXT:    v_add_u32_e32 v8, vcc, v3, v4
 ; GCN-NEXT:    v_mul_hi_u32 v5, v6, v2
 ; GCN-NEXT:    v_mad_u64_u32 v[3:4], s[4:5], v6, v8, 0
 ; GCN-NEXT:    v_add_u32_e32 v9, vcc, v5, v3
 ; GCN-NEXT:    v_mad_u64_u32 v[2:3], s[4:5], v7, v2, 0
 ; GCN-NEXT:    v_addc_u32_e32 v10, vcc, 0, v4, vcc
 ; GCN-NEXT:    v_mad_u64_u32 v[4:5], s[4:5], v7, v8, 0
 ; GCN-NEXT:    v_add_u32_e32 v2, vcc, v9, v2
 ; GCN-NEXT:    v_addc_u32_e32 v2, vcc, v10, v3, vcc
 ; GCN-NEXT:    v_addc_u32_e32 v3, vcc, 0, v5, vcc
 ; GCN-NEXT:    v_add_u32_e32 v2, vcc, v2, v4
 ; GCN-NEXT:    v_addc_u32_e32 v3, vcc, 0, v3, vcc
 ; GCN-NEXT:    v_add_u32_e32 v6, vcc, v6, v2
 ; GCN-NEXT:    v_addc_u32_e32 v7, vcc, v7, v3, vcc
 ; GCN-NEXT:    v_mad_u64_u32 v[2:3], s[4:5], v6, s6, 0
 ; GCN-NEXT:    v_mul_lo_u32 v4, v7, s6
 ; GCN-NEXT:    v_sub_u32_e32 v3, vcc, v3, v6
-; GCN-NEXT:    v_add_u32_e32 v5, vcc, v3, v4
+; GCN-NEXT:    v_add_u32_e32 v5, vcc, v4, v3
 ; GCN-NEXT:    v_mad_u64_u32 v[3:4], s[4:5], v6, v5, 0
 ; GCN-NEXT:    v_mul_hi_u32 v8, v6, v2
 ; GCN-NEXT:    v_add_u32_e32 v8, vcc, v8, v3
 ; GCN-NEXT:    v_mad_u64_u32 v[2:3], s[4:5], v7, v2, 0
 ; GCN-NEXT:    v_addc_u32_e32 v9, vcc, 0, v4, vcc
 ; GCN-NEXT:    v_mad_u64_u32 v[4:5], s[4:5], v7, v5, 0
 ; GCN-NEXT:    v_add_u32_e32 v2, vcc, v8, v2
 ; GCN-NEXT:    v_addc_u32_e32 v2, vcc, v9, v3, vcc

llvm/test/CodeGen/AMDGPU/udiv64.ll

 ; GCN-NEXT:    v_mac_f32_e32 v0, 0xcf800000, v1
 ; GCN-NEXT:    v_cvt_u32_f32_e32 v1, v1
 ; GCN-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; GCN-NEXT:    v_mul_lo_u32 v2, s4, v1
 ; GCN-NEXT:    v_mul_hi_u32 v3, s4, v0
 ; GCN-NEXT:    v_mul_lo_u32 v5, s5, v0
 ; GCN-NEXT:    v_mul_lo_u32 v4, s4, v0
 ; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
-; GCN-NEXT:    v_add_i32_e32 v2, vcc, v5, v2
+; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v5
 ; GCN-NEXT:    v_mul_hi_u32 v3, v0, v4
 ; GCN-NEXT:    v_mul_lo_u32 v5, v0, v2
 ; GCN-NEXT:    v_mul_hi_u32 v7, v0, v2
 ; GCN-NEXT:    v_mul_hi_u32 v6, v1, v4
 ; GCN-NEXT:    v_mul_lo_u32 v4, v1, v4
 ; GCN-NEXT:    v_mul_hi_u32 v8, v1, v2
 ; GCN-NEXT:    v_add_i32_e32 v3, vcc, v3, v5
 ; GCN-NEXT:    v_addc_u32_e32 v5, vcc, 0, v7, vcc

 ; GCN-NEXT:    v_addc_u32_e32 v0, vcc, v3, v0, vcc
 ; GCN-NEXT:    v_addc_u32_e32 v2, vcc, 0, v5, vcc
 ; GCN-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; GCN-NEXT:    v_addc_u32_e32 v1, vcc, 0, v2, vcc
 ; GCN-NEXT:    v_mul_lo_u32 v2, s8, v1
 ; GCN-NEXT:    v_mul_hi_u32 v3, s8, v0
 ; GCN-NEXT:    v_mul_lo_u32 v4, s9, v0
 ; GCN-NEXT:    v_mov_b32_e32 v5, s9
-; GCN-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
+; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; GCN-NEXT:    v_mul_lo_u32 v3, s8, v0
-; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
+; GCN-NEXT:    v_add_i32_e32 v2, vcc, v4, v2
 ; GCN-NEXT:    v_sub_i32_e32 v4, vcc, s3, v2
 ; GCN-NEXT:    v_sub_i32_e32 v3, vcc, s2, v3
 ; GCN-NEXT:    v_subb_u32_e64 v4, s[0:1], v4, v5, vcc
 ; GCN-NEXT:    v_subrev_i32_e64 v5, s[0:1], s8, v3
 ; GCN-NEXT:    v_subbrev_u32_e64 v4, s[0:1], 0, v4, s[0:1]
 ; GCN-NEXT:    v_cmp_le_u32_e64 s[0:1], s9, v4
 ; GCN-NEXT:    v_cndmask_b32_e64 v6, 0, -1, s[0:1]
 ; GCN-NEXT:    v_cmp_le_u32_e64 s[0:1], s8, v5

 ; GCN-NEXT:    s_mov_b32 s2, -1
 ; GCN-NEXT:    s_mov_b32 s0, s4
 ; GCN-NEXT:    v_mul_lo_u32 v3, s8, v2
 ; GCN-NEXT:    v_mul_hi_u32 v4, s8, v1
 ; GCN-NEXT:    v_mul_lo_u32 v5, s9, v1
 ; GCN-NEXT:    v_mul_lo_u32 v6, s8, v1
 ; GCN-NEXT:    s_mov_b32 s1, s5
 ; GCN-NEXT:    v_add_i32_e32 v3, vcc, v3, v4
-; GCN-NEXT:    v_add_i32_e32 v3, vcc, v5, v3
+; GCN-NEXT:    v_add_i32_e32 v3, vcc, v3, v5
 ; GCN-NEXT:    v_mul_lo_u32 v4, v1, v3
 ; GCN-NEXT:    v_mul_hi_u32 v5, v1, v6
 ; GCN-NEXT:    v_mul_hi_u32 v7, v1, v3
 ; GCN-NEXT:    v_mul_hi_u32 v8, v2, v3
 ; GCN-NEXT:    v_mul_lo_u32 v3, v2, v3
 ; GCN-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
 ; GCN-NEXT:    v_addc_u32_e32 v5, vcc, 0, v7, vcc
 ; GCN-NEXT:    v_mul_lo_u32 v7, v2, v6
 ; GCN-NEXT:    v_mul_hi_u32 v6, v2, v6
 ; GCN-NEXT:    v_add_i32_e32 v4, vcc, v4, v7
 ; GCN-NEXT:    v_addc_u32_e32 v4, vcc, v5, v6, vcc
 ; GCN-NEXT:    v_addc_u32_e32 v5, vcc, 0, v8, vcc
 ; GCN-NEXT:    v_add_i32_e32 v3, vcc, v4, v3
 ; GCN-NEXT:    v_addc_u32_e32 v4, vcc, 0, v5, vcc
 ; GCN-NEXT:    v_add_i32_e32 v1, vcc, v1, v3
 ; GCN-NEXT:    v_addc_u32_e32 v2, vcc, v2, v4, vcc
 ; GCN-NEXT:    v_mul_lo_u32 v3, s8, v2
 ; GCN-NEXT:    v_mul_hi_u32 v4, s8, v1
 ; GCN-NEXT:    v_mul_lo_u32 v5, s9, v1
-; GCN-NEXT:    v_add_i32_e32 v3, vcc, v4, v3
+; GCN-NEXT:    v_add_i32_e32 v3, vcc, v3, v4
 ; GCN-NEXT:    v_mul_lo_u32 v4, s8, v1
-; GCN-NEXT:    v_add_i32_e32 v3, vcc, v5, v3
+; GCN-NEXT:    v_add_i32_e32 v3, vcc, v3, v5
 ; GCN-NEXT:    v_mul_lo_u32 v7, v1, v3
 ; GCN-NEXT:    v_mul_hi_u32 v8, v1, v4
 ; GCN-NEXT:    v_mul_hi_u32 v9, v1, v3
 ; GCN-NEXT:    v_mul_hi_u32 v6, v2, v4
 ; GCN-NEXT:    v_mul_lo_u32 v4, v2, v4
 ; GCN-NEXT:    v_mul_hi_u32 v5, v2, v3
 ; GCN-NEXT:    v_add_i32_e32 v7, vcc, v8, v7
 ; GCN-NEXT:    v_addc_u32_e32 v8, vcc, 0, v9, vcc

 ; GCN-NEXT:    v_mac_f32_e32 v0, 0xcf800000, v1
 ; GCN-NEXT:    v_cvt_u32_f32_e32 v1, v1
 ; GCN-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; GCN-NEXT:    v_mul_lo_u32 v2, s4, v1
 ; GCN-NEXT:    v_mul_hi_u32 v3, s4, v0
 ; GCN-NEXT:    v_mul_lo_u32 v5, s5, v0
 ; GCN-NEXT:    v_mul_lo_u32 v4, s4, v0
 ; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
-; GCN-NEXT:    v_add_i32_e32 v2, vcc, v5, v2
+; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v5
 ; GCN-NEXT:    v_mul_hi_u32 v3, v0, v4
 ; GCN-NEXT:    v_mul_lo_u32 v5, v0, v2
 ; GCN-NEXT:    v_mul_hi_u32 v7, v0, v2
 ; GCN-NEXT:    v_mul_hi_u32 v6, v1, v4
 ; GCN-NEXT:    v_mul_lo_u32 v4, v1, v4
 ; GCN-NEXT:    v_mul_hi_u32 v8, v1, v2
 ; GCN-NEXT:    v_add_i32_e32 v3, vcc, v3, v5
 ; GCN-NEXT:    v_addc_u32_e32 v5, vcc, 0, v7, vcc
 ; GCN-NEXT:    v_mul_lo_u32 v2, v1, v2
 ; GCN-NEXT:    v_add_i32_e32 v3, vcc, v3, v4
 ; GCN-NEXT:    v_addc_u32_e32 v3, vcc, v5, v6, vcc
 ; GCN-NEXT:    v_addc_u32_e32 v4, vcc, 0, v8, vcc
 ; GCN-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
 ; GCN-NEXT:    v_addc_u32_e32 v3, vcc, 0, v4, vcc
 ; GCN-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GCN-NEXT:    v_addc_u32_e32 v1, vcc, v1, v3, vcc
 ; GCN-NEXT:    v_mul_lo_u32 v2, s4, v1
 ; GCN-NEXT:    v_mul_hi_u32 v3, s4, v0
 ; GCN-NEXT:    v_mul_lo_u32 v4, s5, v0
 ; GCN-NEXT:    s_mov_b32 s5, s1
-; GCN-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
+; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; GCN-NEXT:    v_mul_lo_u32 v3, s4, v0
-; GCN-NEXT:    v_add_i32_e32 v2, vcc, v4, v2
+; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
 ; GCN-NEXT:    v_mul_lo_u32 v6, v0, v2
 ; GCN-NEXT:    v_mul_hi_u32 v7, v0, v3
 ; GCN-NEXT:    v_mul_hi_u32 v8, v0, v2
 ; GCN-NEXT:    v_mul_hi_u32 v5, v1, v3
 ; GCN-NEXT:    v_mul_lo_u32 v3, v1, v3
 ; GCN-NEXT:    v_mul_hi_u32 v4, v1, v2
 ; GCN-NEXT:    v_add_i32_e32 v6, vcc, v7, v6
 ; GCN-NEXT:    v_addc_u32_e32 v7, vcc, 0, v8, vcc

 ; GCN-NEXT:    v_mul_hi_u32 v0, v0, 24
 ; GCN-NEXT:    v_mul_hi_u32 v1, v1, 24
 ; GCN-NEXT:    v_mov_b32_e32 v4, s3
 ; GCN-NEXT:    s_mov_b32 s4, s0
 ; GCN-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GCN-NEXT:    v_addc_u32_e32 v0, vcc, 0, v1, vcc
 ; GCN-NEXT:    v_mul_lo_u32 v1, s3, v0
 ; GCN-NEXT:    v_mul_hi_u32 v2, s2, v0
-; GCN-NEXT:    v_add_i32_e32 v1, vcc, v2, v1
+; GCN-NEXT:    v_add_i32_e32 v1, vcc, v1, v2
 ; GCN-NEXT:    v_mul_lo_u32 v2, s2, v0
 ; GCN-NEXT:    v_sub_i32_e32 v3, vcc, 0, v1
 ; GCN-NEXT:    v_sub_i32_e32 v2, vcc, 24, v2
 ; GCN-NEXT:    v_subb_u32_e64 v3, s[0:1], v3, v4, vcc
 ; GCN-NEXT:    v_subrev_i32_e64 v4, s[0:1], s2, v2
 ; GCN-NEXT:    v_subbrev_u32_e64 v3, s[0:1], 0, v3, s[0:1]
 ; GCN-NEXT:    v_cmp_le_u32_e64 s[0:1], s3, v3
 ; GCN-NEXT:    v_cndmask_b32_e64 v5, 0, -1, s[0:1]

 ; GCN-NEXT:    v_addc_u32_e32 v4, vcc, 0, v7, vcc
 ; GCN-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
 ; GCN-NEXT:    v_addc_u32_e32 v3, vcc, 0, v4, vcc
 ; GCN-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GCN-NEXT:    v_addc_u32_e32 v1, vcc, v1, v3, vcc
 ; GCN-NEXT:    v_mul_hi_u32 v2, v0, s8
 ; GCN-NEXT:    v_mul_lo_u32 v3, v1, s8
 ; GCN-NEXT:    v_mul_lo_u32 v4, v0, s8
-; GCN-NEXT:    v_subrev_i32_e32 v2, vcc, v0, v2
+; GCN-NEXT:    v_sub_i32_e32 v2, vcc, v2, v0
 ; GCN-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
 ; GCN-NEXT:    v_mul_lo_u32 v3, v0, v2
 ; GCN-NEXT:    v_mul_hi_u32 v5, v0, v4
 ; GCN-NEXT:    v_mul_hi_u32 v6, v0, v2
 ; GCN-NEXT:    v_mul_hi_u32 v7, v1, v2
 ; GCN-NEXT:    v_mul_lo_u32 v2, v1, v2
 ; GCN-NEXT:    v_add_i32_e32 v3, vcc, v5, v3
 ; GCN-NEXT:    v_addc_u32_e32 v5, vcc, 0, v6, vcc

 ; GCN-NEXT:    v_addc_u32_e32 v1, vcc, 0, v2, vcc
 ; GCN-NEXT:    v_mul_lo_u32 v4, v1, 24
 ; GCN-NEXT:    v_mul_hi_u32 v5, v0, 24
 ; GCN-NEXT:    v_add_i32_e32 v2, vcc, 1, v0
 ; GCN-NEXT:    v_mul_lo_u32 v8, v0, 24
 ; GCN-NEXT:    v_addc_u32_e32 v3, vcc, 0, v1, vcc
 ; GCN-NEXT:    v_add_i32_e32 v6, vcc, 2, v0
 ; GCN-NEXT:    v_addc_u32_e32 v7, vcc, 0, v1, vcc
-; GCN-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
+; GCN-NEXT:    v_add_i32_e32 v4, vcc, v4, v5
 ; GCN-NEXT:    v_mov_b32_e32 v5, s7
 ; GCN-NEXT:    v_sub_i32_e32 v8, vcc, s6, v8
 ; GCN-NEXT:    v_subb_u32_e32 v4, vcc, v5, v4, vcc
 ; GCN-NEXT:    v_subrev_i32_e32 v5, vcc, 24, v8
 ; GCN-NEXT:    v_subbrev_u32_e32 v9, vcc, 0, v4, vcc
 ; GCN-NEXT:    v_cmp_lt_u32_e32 vcc, 23, v5
 ; GCN-NEXT:    v_cndmask_b32_e64 v5, 0, -1, vcc
 ; GCN-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v9

 ; GCN-NEXT:    v_addc_u32_e32 v6, vcc, 0, v9, vcc
 ; GCN-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
 ; GCN-NEXT:    v_addc_u32_e32 v5, vcc, 0, v6, vcc
 ; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
 ; GCN-NEXT:    v_addc_u32_e32 v3, vcc, v3, v5, vcc
 ; GCN-NEXT:    v_mul_hi_u32 v4, v2, s4
 ; GCN-NEXT:    v_mul_lo_u32 v5, v3, s4
 ; GCN-NEXT:    v_mul_lo_u32 v6, v2, s4
-; GCN-NEXT:    v_subrev_i32_e32 v4, vcc, v2, v4
-; GCN-NEXT:    v_add_i32_e32 v4, vcc, v4, v5
+; GCN-NEXT:    v_sub_i32_e32 v4, vcc, v4, v2
+; GCN-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
 ; GCN-NEXT:    v_mul_lo_u32 v5, v2, v4
 ; GCN-NEXT:    v_mul_hi_u32 v7, v2, v6
 ; GCN-NEXT:    v_mul_hi_u32 v8, v2, v4
 ; GCN-NEXT:    v_mul_hi_u32 v9, v3, v4
 ; GCN-NEXT:    v_mul_lo_u32 v4, v3, v4
 ; GCN-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
 ; GCN-NEXT:    v_addc_u32_e32 v7, vcc, 0, v8, vcc
 ; GCN-NEXT:    v_mul_lo_u32 v8, v3, v6

llvm/test/CodeGen/AMDGPU/udivrem.ll

 ; GFX6-NEXT:    s_sub_i32 s2, 0, s6
 ; GFX6-NEXT:    v_cvt_f32_u32_e32 v2, s7
 ; GFX6-NEXT:    v_rcp_iflag_f32_e32 v0, v0
 ; GFX6-NEXT:    v_rcp_iflag_f32_e32 v2, v2
 ; GFX6-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; GFX6-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; GFX6-NEXT:    v_mul_lo_u32 v1, s2, v0
 ; GFX6-NEXT:    v_mul_hi_u32 v1, v0, v1
-; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
+; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; GFX6-NEXT:    v_mul_hi_u32 v0, s4, v0
 ; GFX6-NEXT:    v_mul_f32_e32 v1, 0x4f7ffffe, v2
 ; GFX6-NEXT:    v_cvt_u32_f32_e32 v1, v1
 ; GFX6-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX6-NEXT:    s_mul_i32 s2, s2, s6
 ; GFX6-NEXT:    s_sub_i32 s2, s4, s2
 ; GFX6-NEXT:    s_sub_i32 s3, s2, s6
 ; GFX6-NEXT:    s_cmp_ge_u32 s2, s6
 ; GFX6-NEXT:    s_cselect_b32 s2, s3, s2
 ; GFX6-NEXT:    s_sub_i32 s3, s2, s6
 ; GFX6-NEXT:    s_cmp_ge_u32 s2, s6
 ; GFX6-NEXT:    s_cselect_b32 s4, s3, s2
 ; GFX6-NEXT:    s_sub_i32 s2, 0, s7
 ; GFX6-NEXT:    v_mul_lo_u32 v0, s2, v1
 ; GFX6-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX6-NEXT:    s_mov_b32 s2, -1
 ; GFX6-NEXT:    v_mul_hi_u32 v0, v1, v0
-; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
+; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
 ; GFX6-NEXT:    v_mul_hi_u32 v0, s5, v0
 ; GFX6-NEXT:    v_readfirstlane_b32 s6, v0
 ; GFX6-NEXT:    s_mul_i32 s6, s6, s7
 ; GFX6-NEXT:    s_sub_i32 s5, s5, s6
 ; GFX6-NEXT:    s_sub_i32 s6, s5, s7
 ; GFX6-NEXT:    s_cmp_ge_u32 s5, s7
 ; GFX6-NEXT:    s_cselect_b32 s5, s6, s5
 ; GFX6-NEXT:    s_sub_i32 s6, s5, s7

 ; GFX8-NEXT:    s_sub_i32 s2, 0, s6
 ; GFX8-NEXT:    v_cvt_f32_u32_e32 v2, s7
 ; GFX8-NEXT:    v_rcp_iflag_f32_e32 v0, v0
 ; GFX8-NEXT:    v_rcp_iflag_f32_e32 v2, v2
 ; GFX8-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
 ; GFX8-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; GFX8-NEXT:    v_mul_lo_u32 v1, s2, v0
 ; GFX8-NEXT:    v_mul_hi_u32 v1, v0, v1
-; GFX8-NEXT:    v_add_u32_e32 v0, vcc, v1, v0
+; GFX8-NEXT:    v_add_u32_e32 v0, vcc, v0, v1
 ; GFX8-NEXT:    v_mul_hi_u32 v0, s4, v0
 ; GFX8-NEXT:    v_mul_f32_e32 v1, 0x4f7ffffe, v2
 ; GFX8-NEXT:    v_cvt_u32_f32_e32 v1, v1
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX8-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX8-NEXT:    s_mul_i32 s2, s2, s6
 ; GFX8-NEXT:    s_sub_i32 s2, s4, s2
 ; GFX8-NEXT:    s_sub_i32 s3, s2, s6
 ; GFX8-NEXT:    s_cmp_ge_u32 s2, s6
 ; GFX8-NEXT:    s_cselect_b32 s2, s3, s2
 ; GFX8-NEXT:    s_sub_i32 s3, s2, s6
 ; GFX8-NEXT:    s_cmp_ge_u32 s2, s6
 ; GFX8-NEXT:    s_cselect_b32 s2, s3, s2
 ; GFX8-NEXT:    s_sub_i32 s3, 0, s7
 ; GFX8-NEXT:    v_mul_lo_u32 v0, s3, v1
 ; GFX8-NEXT:    v_mov_b32_e32 v2, s0
 ; GFX8-NEXT:    v_mul_hi_u32 v0, v1, v0
-; GFX8-NEXT:    v_add_u32_e32 v0, vcc, v0, v1
+; GFX8-NEXT:    v_add_u32_e32 v0, vcc, v1, v0
 ; GFX8-NEXT:    v_mul_hi_u32 v1, s5, v0
 ; GFX8-NEXT:    v_mov_b32_e32 v0, s2
 ; GFX8-NEXT:    v_readfirstlane_b32 s2, v1
 ; GFX8-NEXT:    s_mul_i32 s2, s2, s7
 ; GFX8-NEXT:    s_sub_i32 s2, s5, s2
 ; GFX8-NEXT:    s_sub_i32 s3, s2, s7
 ; GFX8-NEXT:    s_cmp_ge_u32 s2, s7
 ; GFX8-NEXT:    s_cselect_b32 s2, s3, s2

 ; GFX6-NEXT:    s_cselect_b32 s2, s3, s2
 ; GFX6-NEXT:    s_sub_i32 s3, s2, s8
 ; GFX6-NEXT:    s_cmp_ge_u32 s2, s8
 ; GFX6-NEXT:    s_cselect_b32 s4, s3, s2
 ; GFX6-NEXT:    s_sub_i32 s2, 0, s9
 ; GFX6-NEXT:    v_mul_lo_u32 v0, s2, v1
 ; GFX6-NEXT:    v_rcp_iflag_f32_e32 v2, v2
 ; GFX6-NEXT:    v_mul_hi_u32 v0, v1, v0
-; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
+; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
 ; GFX6-NEXT:    v_mul_hi_u32 v0, s5, v0
 ; GFX6-NEXT:    v_mul_f32_e32 v1, 0x4f7ffffe, v2
 ; GFX6-NEXT:    v_cvt_u32_f32_e32 v1, v1
 ; GFX6-NEXT:    v_cvt_f32_u32_e32 v2, s11
 ; GFX6-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX6-NEXT:    s_mul_i32 s2, s2, s9
 ; GFX6-NEXT:    s_sub_i32 s2, s5, s2
 ; GFX6-NEXT:    s_sub_i32 s3, s2, s9
 ; GFX6-NEXT:    s_cmp_ge_u32 s2, s9
 ; GFX6-NEXT:    s_cselect_b32 s2, s3, s2
 ; GFX6-NEXT:    s_sub_i32 s3, s2, s9
 ; GFX6-NEXT:    s_cmp_ge_u32 s2, s9
 ; GFX6-NEXT:    s_cselect_b32 s5, s3, s2
 ; GFX6-NEXT:    s_sub_i32 s2, 0, s10
 ; GFX6-NEXT:    v_mul_lo_u32 v0, s2, v1
 ; GFX6-NEXT:    v_rcp_iflag_f32_e32 v2, v2
 ; GFX6-NEXT:    v_mul_hi_u32 v0, v1, v0
-; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
+; GFX6-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
 ; GFX6-NEXT:    v_mul_hi_u32 v0, s6, v0
 ; GFX6-NEXT:    v_mul_f32_e32 v1, 0x4f7ffffe, v2
 ; GFX6-NEXT:    v_cvt_u32_f32_e32 v1, v1
 ; GFX6-NEXT:    v_readfirstlane_b32 s2, v0
 ; GFX6-NEXT:    s_mul_i32 s2, s2, s10
 ; GFX6-NEXT:    s_sub_i32 s2, s6, s2
 ; GFX6-NEXT:    s_sub_i32 s3, s2, s10
 ; GFX6-NEXT:    s_cmp_ge_u32 s2, s10

 ; GFX8-NEXT:    s_cselect_b32 s2, s3, s2
 ; GFX8-NEXT:    s_sub_i32 s3, s2, s8
 ; GFX8-NEXT:    s_cmp_ge_u32 s2, s8
 ; GFX8-NEXT:    s_cselect_b32 s2, s3, s2
 ; GFX8-NEXT:    s_sub_i32 s3, 0, s9
 ; GFX8-NEXT:    v_mul_lo_u32 v0, s3, v1
 ; GFX8-NEXT:    v_rcp_iflag_f32_e32 v2, v2
 ; GFX8-NEXT:    v_mul_hi_u32 v0, v1, v0
-; GFX8-NEXT:    v_add_u32_e32 v0, vcc, v0, v1
+; GFX8-NEXT:    v_add_u32_e32 v0, vcc, v1, v0
 ; GFX8-NEXT:    v_mul_hi_u32 v0, s5, v0
 ; GFX8-NEXT:    v_mul_f32_e32 v1, 0x4f7ffffe, v2
 ; GFX8-NEXT:    v_cvt_u32_f32_e32 v1, v1
 ; GFX8-NEXT:    v_cvt_f32_u32_e32 v2, s11
 ; GFX8-NEXT:    v_readfirstlane_b32 s3, v0
 ; GFX8-NEXT:    s_mul_i32 s3, s3, s9
 ; GFX8-NEXT:    s_sub_i32 s3, s5, s3
 ; GFX8-NEXT:    s_sub_i32 s4, s3, s9
 ; GFX8-NEXT:    s_cmp_ge_u32 s3, s9
 ; GFX8-NEXT:    s_cselect_b32 s3, s4, s3
 ; GFX8-NEXT:    s_sub_i32 s4, s3, s9
 ; GFX8-NEXT:    s_cmp_ge_u32 s3, s9
 ; GFX8-NEXT:    s_cselect_b32 s3, s4, s3
 ; GFX8-NEXT:    s_sub_i32 s4, 0, s10
 ; GFX8-NEXT:    v_mul_lo_u32 v0, s4, v1
 ; GFX8-NEXT:    v_rcp_iflag_f32_e32 v2, v2
 ; GFX8-NEXT:    v_mul_hi_u32 v0, v1, v0
-; GFX8-NEXT:    v_add_u32_e32 v0, vcc, v0, v1
+; GFX8-NEXT:    v_add_u32_e32 v0, vcc, v1, v0
 ; GFX8-NEXT:    v_mul_hi_u32 v0, s6, v0
 ; GFX8-NEXT:    v_mul_f32_e32 v1, 0x4f7ffffe, v2
 ; GFX8-NEXT:    v_cvt_u32_f32_e32 v1, v1
 ; GFX8-NEXT:    v_readfirstlane_b32 s4, v0
 ; GFX8-NEXT:    s_mul_i32 s4, s4, s10
 ; GFX8-NEXT:    s_sub_i32 s4, s6, s4
 ; GFX8-NEXT:    s_sub_i32 s5, s4, s10
 ; GFX8-NEXT:    s_cmp_ge_u32 s4, s10

llvm/test/CodeGen/AMDGPU/urem64.ll

 ; GCN-NEXT:    v_mac_f32_e32 v0, 0xcf800000, v1
 ; GCN-NEXT:    v_cvt_u32_f32_e32 v1, v1
 ; GCN-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; GCN-NEXT:    v_mul_lo_u32 v2, s0, v1
 ; GCN-NEXT:    v_mul_hi_u32 v3, s0, v0
 ; GCN-NEXT:    v_mul_lo_u32 v5, s1, v0
 ; GCN-NEXT:    v_mul_lo_u32 v4, s0, v0
 ; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
-; GCN-NEXT:    v_add_i32_e32 v2, vcc, v5, v2
+; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v5
 ; GCN-NEXT:    v_mul_hi_u32 v3, v0, v4
 ; GCN-NEXT:    v_mul_lo_u32 v5, v0, v2
 ; GCN-NEXT:    v_mul_hi_u32 v7, v0, v2
 ; GCN-NEXT:    v_mul_hi_u32 v6, v1, v4
 ; GCN-NEXT:    v_mul_lo_u32 v4, v1, v4
 ; GCN-NEXT:    v_mul_hi_u32 v8, v1, v2
 ; GCN-NEXT:    v_add_i32_e32 v3, vcc, v3, v5
 ; GCN-NEXT:    v_addc_u32_e32 v5, vcc, 0, v7, vcc

 ; GCN-NEXT:    v_addc_u32_e32 v0, vcc, v3, v0, vcc
 ; GCN-NEXT:    v_addc_u32_e32 v2, vcc, 0, v5, vcc
 ; GCN-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; GCN-NEXT:    v_addc_u32_e32 v1, vcc, 0, v2, vcc
 ; GCN-NEXT:    v_mul_lo_u32 v1, s12, v1
 ; GCN-NEXT:    v_mul_hi_u32 v2, s12, v0
 ; GCN-NEXT:    v_mul_lo_u32 v3, s13, v0
 ; GCN-NEXT:    v_mul_lo_u32 v0, s12, v0
-; GCN-NEXT:    v_add_i32_e32 v1, vcc, v2, v1
-; GCN-NEXT:    v_add_i32_e32 v1, vcc, v1, v3
+; GCN-NEXT:    v_add_i32_e32 v1, vcc, v1, v2
+; GCN-NEXT:    v_add_i32_e32 v1, vcc, v3, v1
 ; GCN-NEXT:    v_sub_i32_e32 v2, vcc, s11, v1
 ; GCN-NEXT:    v_mov_b32_e32 v3, s13
 ; GCN-NEXT:    v_sub_i32_e32 v0, vcc, s10, v0
 ; GCN-NEXT:    v_subb_u32_e64 v2, s[0:1], v2, v3, vcc
 ; GCN-NEXT:    v_subrev_i32_e64 v4, s[0:1], s12, v0
 ; GCN-NEXT:    v_subbrev_u32_e64 v5, s[2:3], 0, v2, s[0:1]
 ; GCN-NEXT:    v_cmp_le_u32_e64 s[2:3], s13, v5
 ; GCN-NEXT:    v_cndmask_b32_e64 v6, 0, -1, s[2:3]

 ; GCN-NEXT:    v_mac_f32_e32 v0, 0xcf800000, v1
 ; GCN-NEXT:    v_cvt_u32_f32_e32 v1, v1
 ; GCN-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; GCN-NEXT:    v_mul_lo_u32 v2, s0, v1
 ; GCN-NEXT:    v_mul_hi_u32 v3, s0, v0
 ; GCN-NEXT:    v_mul_lo_u32 v5, s1, v0
 ; GCN-NEXT:    v_mul_lo_u32 v4, s0, v0
 ; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
-; GCN-NEXT:    v_add_i32_e32 v2, vcc, v5, v2
+; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v5
 ; GCN-NEXT:    v_mul_hi_u32 v3, v0, v4
 ; GCN-NEXT:    v_mul_lo_u32 v5, v0, v2
 ; GCN-NEXT:    v_mul_hi_u32 v7, v0, v2
 ; GCN-NEXT:    v_mul_hi_u32 v6, v1, v4
 ; GCN-NEXT:    v_mul_lo_u32 v4, v1, v4
 ; GCN-NEXT:    v_mul_hi_u32 v8, v1, v2
 ; GCN-NEXT:    v_add_i32_e32 v3, vcc, v3, v5
 ; GCN-NEXT:    v_addc_u32_e32 v5, vcc, 0, v7, vcc

 ; GCN-NEXT:    v_mul_hi_u32 v0, v0, 24
 ; GCN-NEXT:    v_mul_hi_u32 v1, v1, 24
 ; GCN-NEXT:    v_mov_b32_e32 v3, s7
 ; GCN-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GCN-NEXT:    v_addc_u32_e32 v0, vcc, 0, v1, vcc
 ; GCN-NEXT:    v_mul_lo_u32 v1, s7, v0
 ; GCN-NEXT:    v_mul_hi_u32 v2, s6, v0
 ; GCN-NEXT:    v_mul_lo_u32 v0, s6, v0
-; GCN-NEXT:    v_add_i32_e32 v1, vcc, v2, v1
+; GCN-NEXT:    v_add_i32_e32 v1, vcc, v1, v2
 ; GCN-NEXT:    v_sub_i32_e32 v2, vcc, 0, v1
 ; GCN-NEXT:    v_sub_i32_e32 v0, vcc, 24, v0
 ; GCN-NEXT:    v_subb_u32_e64 v2, s[0:1], v2, v3, vcc
 ; GCN-NEXT:    v_subrev_i32_e64 v4, s[0:1], s6, v0
 ; GCN-NEXT:    v_subbrev_u32_e64 v5, s[2:3], 0, v2, s[0:1]
 ; GCN-NEXT:    v_cmp_le_u32_e64 s[2:3], s7, v5
 ; GCN-NEXT:    v_cndmask_b32_e64 v6, 0, -1, s[2:3]
 ; GCN-NEXT:    v_cmp_le_u32_e64 s[2:3], s6, v4

 ; GCN-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; GCN-NEXT:    v_cvt_u32_f32_e32 v1, v1
 ; GCN-NEXT:    s_waitcnt lgkmcnt(0)
 ; GCN-NEXT:    s_mov_b32 s0, s4
 ; GCN-NEXT:    s_mov_b32 s1, s5
 ; GCN-NEXT:    v_mul_hi_u32 v2, v0, s2
 ; GCN-NEXT:    v_mul_lo_u32 v4, v1, s2
 ; GCN-NEXT:    v_mul_lo_u32 v3, v0, s2
-; GCN-NEXT:    v_subrev_i32_e32 v2, vcc, v0, v2
-; GCN-NEXT:    v_add_i32_e32 v2, vcc, v4, v2
+; GCN-NEXT:    v_sub_i32_e32 v2, vcc, v2, v0
+; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
 ; GCN-NEXT:    v_mul_hi_u32 v5, v0, v3
 ; GCN-NEXT:    v_mul_lo_u32 v4, v0, v2
 ; GCN-NEXT:    v_mul_hi_u32 v6, v0, v2
 ; GCN-NEXT:    v_mul_hi_u32 v7, v1, v2
 ; GCN-NEXT:    v_mul_lo_u32 v2, v1, v2
 ; GCN-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
 ; GCN-NEXT:    v_addc_u32_e32 v5, vcc, 0, v6, vcc
 ; GCN-NEXT:    v_mul_lo_u32 v6, v1, v3

 ; GCN-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
 ; GCN-NEXT:    v_addc_u32_e32 v0, vcc, v3, v0, vcc
 ; GCN-NEXT:    v_addc_u32_e32 v2, vcc, 0, v5, vcc
 ; GCN-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; GCN-NEXT:    v_addc_u32_e32 v1, vcc, 0, v2, vcc
 ; GCN-NEXT:    v_mul_lo_u32 v1, v1, 24
 ; GCN-NEXT:    v_mul_hi_u32 v2, v0, 24
 ; GCN-NEXT:    v_mul_lo_u32 v0, v0, 24
-; GCN-NEXT:    v_add_i32_e32 v1, vcc, v2, v1
+; GCN-NEXT:    v_add_i32_e32 v1, vcc, v1, v2
 ; GCN-NEXT:    v_mov_b32_e32 v2, s7
 ; GCN-NEXT:    v_sub_i32_e32 v0, vcc, s6, v0
 ; GCN-NEXT:    v_subb_u32_e32 v1, vcc, v2, v1, vcc
 ; GCN-NEXT:    v_subrev_i32_e32 v2, vcc, 24, v0
 ; GCN-NEXT:    v_subbrev_u32_e32 v3, vcc, 0, v1, vcc
 ; GCN-NEXT:    v_subrev_i32_e32 v4, vcc, 24, v2
 ; GCN-NEXT:    v_subbrev_u32_e32 v5, vcc, 0, v3, vcc
 ; GCN-NEXT:    v_cmp_lt_u32_e32 vcc, 23, v2

llvm/test/CodeGen/AMDGPU/vgpr-liverange-ir.ll

define amdgpu_ps float @else3(i32 %z, float %v, i32 inreg %bound, i32 %x0) #0 {
   ; SI-NEXT:   S_CBRANCH_SCC1 %bb.1, implicit killed $scc
   ; SI-NEXT:   S_BRANCH %bb.6
   ; SI-NEXT: {{  $}}
   ; SI-NEXT: bb.6.for.end:
   ; SI-NEXT:   [[V_ADD_F32_e64_:%[0-9]+]]:vgpr_32 = nofpexcept V_ADD_F32_e64 0, killed [[PHI6]], 0, killed [[PHI5]], 0, 0, implicit $mode, implicit $exec
   ; SI-NEXT:   $vgpr0 = COPY killed [[V_ADD_F32_e64_]]
   ; SI-NEXT:   SI_RETURN_TO_EPILOG killed $vgpr0
 entry:
-;  %break = icmp sgt i32 %bound, 0
-;  br i1 %break, label %for.body, label %for.end
   br label %for.body
 
 for.body:
   %i = phi i32 [ 0, %entry ], [ %inc, %if.end ]
   %x = phi i32 [ %x0, %entry ], [ %xinc, %if.end ]
   %cc = icmp sgt i32 %z, 5
   br i1 %cc, label %if, label %else
 
define amdgpu_ps float @loop(i32 %z, float %v, i32 inreg %bound, ptr %extern_func, ptr %extern_func2) #0 {
   ; SI-NEXT:   [[COPY5:%[0-9]+]]:vgpr_32 = COPY killed $vgpr0
   ; SI-NEXT:   [[V_CMP_GT_I32_e64_:%[0-9]+]]:sreg_32 = V_CMP_GT_I32_e64 6, killed [[COPY5]], implicit $exec
   ; SI-NEXT:   [[SI_IF:%[0-9]+]]:sreg_32 = SI_IF killed [[V_CMP_GT_I32_e64_]], %bb.1, implicit-def dead $exec, implicit-def dead $scc, implicit $exec
   ; SI-NEXT:   S_BRANCH %bb.6
   ; SI-NEXT: {{  $}}
   ; SI-NEXT: bb.1.Flow:
   ; SI-NEXT:   successors: %bb.2(0x40000000), %bb.10(0x40000000)
   ; SI-NEXT: {{  $}}
-  ; SI-NEXT:   [[PHI:%[0-9]+]]:vgpr_32 = PHI undef %49:vgpr_32, %bb.0, %4, %bb.9
-  ; SI-NEXT:   [[PHI1:%[0-9]+]]:vgpr_32 = PHI [[COPY4]], %bb.0, undef %51:vgpr_32, %bb.9
-  ; SI-NEXT:   [[PHI2:%[0-9]+]]:vgpr_32 = PHI [[COPY3]], %bb.0, undef %53:vgpr_32, %bb.9
-  ; SI-NEXT:   [[PHI3:%[0-9]+]]:vgpr_32 = PHI [[COPY2]], %bb.0, undef %55:vgpr_32, %bb.9
+  ; SI-NEXT:   [[PHI:%[0-9]+]]:vgpr_32 = PHI undef [[COPY47:%[0-9]+]]:vgpr_32, %bb.0, %4, %bb.9
+  ; SI-NEXT:   [[PHI1:%[0-9]+]]:vgpr_32 = PHI [[COPY4]], %bb.0, undef [[COPY49:%[0-9]+]]:vgpr_32, %bb.9
+  ; SI-NEXT:   [[PHI2:%[0-9]+]]:vgpr_32 = PHI [[COPY3]], %bb.0, undef [[COPY51:%[0-9]+]]:vgpr_32, %bb.9
+  ; SI-NEXT:   [[PHI3:%[0-9]+]]:vgpr_32 = PHI [[COPY2]], %bb.0, undef [[COPY53:%[0-9]+]]:vgpr_32, %bb.9
   ; SI-NEXT:   [[SI_ELSE:%[0-9]+]]:sreg_32 = SI_ELSE killed [[SI_IF]], %bb.10, implicit-def dead $exec, implicit-def dead $scc, implicit $exec
   ; SI-NEXT:   S_BRANCH %bb.2
   ; SI-NEXT: {{  $}}
   ; SI-NEXT: bb.2.if:
   ; SI-NEXT:   successors: %bb.3(0x80000000)
   ; SI-NEXT: {{  $}}
   ; SI-NEXT:   [[REG_SEQUENCE:%[0-9]+]]:vreg_64 = REG_SEQUENCE killed [[PHI2]], %subreg.sub0, killed [[PHI3]], %subreg.sub1
   ; SI-NEXT:   [[S_MOV_B32_:%[0-9]+]]:sreg_32_xm0_xexec = S_MOV_B32 $exec_lo
   ; SI-NEXT: {{  $}}
   ; SI-NEXT: bb.3:
   ; SI-NEXT:   successors: %bb.4(0x80000000)
   ; SI-NEXT: {{  $}}
-  ; SI-NEXT:   [[PHI4:%[0-9]+]]:vreg_64 = PHI undef %57:vreg_64, %bb.4, [[REG_SEQUENCE]], %bb.2
-  ; SI-NEXT:   [[PHI5:%[0-9]+]]:vgpr_32 = PHI undef %59:vgpr_32, %bb.4, [[PHI1]], %bb.2
+  ; SI-NEXT:   [[PHI4:%[0-9]+]]:vreg_64 = PHI undef [[COPY57:%[0-9]+]]:vreg_64, %bb.4, [[REG_SEQUENCE]], %bb.2
+  ; SI-NEXT:   [[PHI5:%[0-9]+]]:vgpr_32 = PHI undef [[COPY59:%[0-9]+]]:vgpr_32, %bb.4, [[PHI1]], %bb.2
   ; SI-NEXT:   [[V_READFIRSTLANE_B32_:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[PHI4]].sub0, implicit $exec
   ; SI-NEXT:   [[V_READFIRSTLANE_B32_1:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[PHI4]].sub1, implicit $exec
   ; SI-NEXT:   [[REG_SEQUENCE1:%[0-9]+]]:sgpr_64 = REG_SEQUENCE killed [[V_READFIRSTLANE_B32_]], %subreg.sub0, killed [[V_READFIRSTLANE_B32_1]], %subreg.sub1
   ; SI-NEXT:   [[V_CMP_EQ_U64_e64_:%[0-9]+]]:sreg_32_xm0_xexec = V_CMP_EQ_U64_e64 [[REG_SEQUENCE1]], killed [[PHI4]], implicit $exec
   ; SI-NEXT:   [[S_AND_SAVEEXEC_B32_:%[0-9]+]]:sreg_32_xm0_xexec = S_AND_SAVEEXEC_B32 killed [[V_CMP_EQ_U64_e64_]], implicit-def $exec, implicit-def dead $scc, implicit $exec
   ; SI-NEXT: {{  $}}
   ; SI-NEXT: bb.4:
   ; SI-NEXT:   successors: %bb.3(0x40000000), %bb.5(0x40000000)
define amdgpu_ps float @loop(i32 %z, float %v, i32 inreg %bound, ptr %extern_func, ptr %extern_func2) #0 {
   ; SI-NEXT:   successors: %bb.7(0x80000000)
   ; SI-NEXT: {{  $}}
   ; SI-NEXT:   [[REG_SEQUENCE2:%[0-9]+]]:vreg_64 = REG_SEQUENCE killed [[COPY1]], %subreg.sub0, killed [[COPY]], %subreg.sub1
   ; SI-NEXT:   [[S_MOV_B32_1:%[0-9]+]]:sreg_32_xm0_xexec = S_MOV_B32 $exec_lo
   ; SI-NEXT: {{  $}}
   ; SI-NEXT: bb.7:
   ; SI-NEXT:   successors: %bb.8(0x80000000)
   ; SI-NEXT: {{  $}}
-  ; SI-NEXT:   [[PHI6:%[0-9]+]]:vreg_64 = PHI undef %61:vreg_64, %bb.8, [[REG_SEQUENCE2]], %bb.6
-  ; SI-NEXT:   [[PHI7:%[0-9]+]]:vgpr_32 = PHI undef %63:vgpr_32, %bb.8, [[COPY4]], %bb.6
+  ; SI-NEXT:   [[PHI6:%[0-9]+]]:vreg_64 = PHI undef [[COPY59:%[0-9]+]]:vreg_64, %bb.8, [[REG_SEQUENCE2]], %bb.6
+  ; SI-NEXT:   [[PHI7:%[0-9]+]]:vgpr_32 = PHI undef [[COPY61:%[0-9]+]]:vgpr_32, %bb.8, [[COPY4]], %bb.6
   ; SI-NEXT:   [[V_READFIRSTLANE_B32_2:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[PHI6]].sub0, implicit $exec
   ; SI-NEXT:   [[V_READFIRSTLANE_B32_3:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[PHI6]].sub1, implicit $exec
   ; SI-NEXT:   [[REG_SEQUENCE3:%[0-9]+]]:sgpr_64 = REG_SEQUENCE killed [[V_READFIRSTLANE_B32_2]], %subreg.sub0, killed [[V_READFIRSTLANE_B32_3]], %subreg.sub1
   ; SI-NEXT:   [[V_CMP_EQ_U64_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_CMP_EQ_U64_e64 [[REG_SEQUENCE3]], killed [[PHI6]], implicit $exec
   ; SI-NEXT:   [[S_AND_SAVEEXEC_B32_1:%[0-9]+]]:sreg_32_xm0_xexec = S_AND_SAVEEXEC_B32 killed [[V_CMP_EQ_U64_e64_1]], implicit-def $exec, implicit-def dead $scc, implicit $exec
   ; SI-NEXT: {{  $}}
   ; SI-NEXT: bb.8:
   ; SI-NEXT:   successors: %bb.7(0x40000000), %bb.9(0x40000000)
define amdgpu_ps float @loop_with_use(i32 %z, float %v, i32 inreg %bound, ptr %extern_func, ptr %extern_func2) #0 {
   ; SI-NEXT:   [[COPY5:%[0-9]+]]:vgpr_32 = COPY killed $vgpr0
   ; SI-NEXT:   [[V_CMP_GT_I32_e64_:%[0-9]+]]:sreg_32 = V_CMP_GT_I32_e64 6, killed [[COPY5]], implicit $exec
   ; SI-NEXT:   [[SI_IF:%[0-9]+]]:sreg_32 = SI_IF killed [[V_CMP_GT_I32_e64_]], %bb.1, implicit-def dead $exec, implicit-def dead $scc, implicit $exec
   ; SI-NEXT:   S_BRANCH %bb.6
   ; SI-NEXT: {{  $}}
   ; SI-NEXT: bb.1.Flow:
   ; SI-NEXT:   successors: %bb.2(0x40000000), %bb.10(0x40000000)
   ; SI-NEXT: {{  $}}
-  ; SI-NEXT:   [[PHI:%[0-9]+]]:vgpr_32 = PHI undef %50:vgpr_32, %bb.0, %4, %bb.9
-  ; SI-NEXT:   [[PHI1:%[0-9]+]]:vgpr_32 = PHI [[COPY3]], %bb.0, undef %52:vgpr_32, %bb.9
-  ; SI-NEXT:   [[PHI2:%[0-9]+]]:vgpr_32 = PHI [[COPY2]], %bb.0, undef %54:vgpr_32, %bb.9
+  ; SI-NEXT:   [[PHI:%[0-9]+]]:vgpr_32 = PHI undef [[COPY50:%[0-9]+]]:vgpr_32, %bb.0, %4, %bb.9
+  ; SI-NEXT:   [[PHI1:%[0-9]+]]:vgpr_32 = PHI [[COPY3]], %bb.0, undef [[COPY52:%[0-9]+]]:vgpr_32, %bb.9
+  ; SI-NEXT:   [[PHI2:%[0-9]+]]:vgpr_32 = PHI [[COPY2]], %bb.0, undef [[COPY54:%[0-9]+]]:vgpr_32, %bb.9
   ; SI-NEXT:   [[SI_ELSE:%[0-9]+]]:sreg_32 = SI_ELSE killed [[SI_IF]], %bb.10, implicit-def dead $exec, implicit-def dead $scc, implicit $exec
   ; SI-NEXT:   S_BRANCH %bb.2
   ; SI-NEXT: {{  $}}
   ; SI-NEXT: bb.2.if:
   ; SI-NEXT:   successors: %bb.3(0x80000000)
   ; SI-NEXT: {{  $}}
   ; SI-NEXT:   [[REG_SEQUENCE:%[0-9]+]]:vreg_64 = REG_SEQUENCE killed [[PHI1]], %subreg.sub0, killed [[PHI2]], %subreg.sub1
   ; SI-NEXT:   [[S_MOV_B32_:%[0-9]+]]:sreg_32_xm0_xexec = S_MOV_B32 $exec_lo
   ; SI-NEXT: {{  $}}
   ; SI-NEXT: bb.3:
   ; SI-NEXT:   successors: %bb.4(0x80000000)
   ; SI-NEXT: {{  $}}
-  ; SI-NEXT:   [[PHI3:%[0-9]+]]:vreg_64 = PHI undef %56:vreg_64, %bb.4, [[REG_SEQUENCE]], %bb.2
+  ; SI-NEXT:   [[PHI3:%[0-9]+]]:vreg_64 = PHI undef [[COPY56:%[0-9]+]]:vreg_64, %bb.4, [[REG_SEQUENCE]], %bb.2
   ; SI-NEXT:   [[V_READFIRSTLANE_B32_:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[PHI3]].sub0, implicit $exec
   ; SI-NEXT:   [[V_READFIRSTLANE_B32_1:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[PHI3]].sub1, implicit $exec
   ; SI-NEXT:   [[REG_SEQUENCE1:%[0-9]+]]:sgpr_64 = REG_SEQUENCE killed [[V_READFIRSTLANE_B32_]], %subreg.sub0, killed [[V_READFIRSTLANE_B32_1]], %subreg.sub1
   ; SI-NEXT:   [[V_CMP_EQ_U64_e64_:%[0-9]+]]:sreg_32_xm0_xexec = V_CMP_EQ_U64_e64 [[REG_SEQUENCE1]], killed [[PHI3]], implicit $exec
   ; SI-NEXT:   [[S_AND_SAVEEXEC_B32_:%[0-9]+]]:sreg_32_xm0_xexec = S_AND_SAVEEXEC_B32 killed [[V_CMP_EQ_U64_e64_]], implicit-def $exec, implicit-def dead $scc, implicit $exec
   ; SI-NEXT: {{  $}}
   ; SI-NEXT: bb.4:
   ; SI-NEXT:   successors: %bb.3(0x40000000), %bb.5(0x40000000)
define amdgpu_ps float @loop_with_use(i32 %z, float %v, i32 inreg %bound, ptr %extern_func, ptr %extern_func2) #0 {
   ; SI-NEXT:   successors: %bb.7(0x80000000)
   ; SI-NEXT: {{  $}}
   ; SI-NEXT:   [[REG_SEQUENCE2:%[0-9]+]]:vreg_64 = REG_SEQUENCE killed [[COPY1]], %subreg.sub0, killed [[COPY]], %subreg.sub1
   ; SI-NEXT:   [[S_MOV_B32_1:%[0-9]+]]:sreg_32_xm0_xexec = S_MOV_B32 $exec_lo
   ; SI-NEXT: {{  $}}
   ; SI-NEXT: bb.7:
   ; SI-NEXT:   successors: %bb.8(0x80000000)
   ; SI-NEXT: {{  $}}
-  ; SI-NEXT:   [[PHI4:%[0-9]+]]:vreg_64 = PHI undef %58:vreg_64, %bb.8, [[REG_SEQUENCE2]], %bb.6
+  ; SI-NEXT:   [[PHI4:%[0-9]+]]:vreg_64 = PHI undef [[COPY58:%[0-9]+]]:vreg_64, %bb.8, [[REG_SEQUENCE2]], %bb.6
   ; SI-NEXT:   [[V_READFIRSTLANE_B32_2:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[PHI4]].sub0, implicit $exec
   ; SI-NEXT:   [[V_READFIRSTLANE_B32_3:%[0-9]+]]:sgpr_32 = V_READFIRSTLANE_B32 [[PHI4]].sub1, implicit $exec
   ; SI-NEXT:   [[REG_SEQUENCE3:%[0-9]+]]:sgpr_64 = REG_SEQUENCE killed [[V_READFIRSTLANE_B32_2]], %subreg.sub0, killed [[V_READFIRSTLANE_B32_3]], %subreg.sub1
   ; SI-NEXT:   [[V_CMP_EQ_U64_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_CMP_EQ_U64_e64 [[REG_SEQUENCE3]], killed [[PHI4]], implicit $exec
   ; SI-NEXT:   [[S_AND_SAVEEXEC_B32_1:%[0-9]+]]:sreg_32_xm0_xexec = S_AND_SAVEEXEC_B32 killed [[V_CMP_EQ_U64_e64_1]], implicit-def $exec, implicit-def dead $scc, implicit $exec
   ; SI-NEXT: {{  $}}
   ; SI-NEXT: bb.8:
   ; SI-NEXT:   successors: %bb.7(0x40000000), %bb.9(0x40000000)
define amdgpu_kernel void @livevariables_update_missed_block(ptr addrspace(1) %src1) {
   ; SI-NEXT:   [[SI_IF:%[0-9]+]]:sreg_32 = SI_IF killed [[V_CMP_NE_U32_e64_]], %bb.5, implicit-def dead $exec, implicit-def dead $scc, implicit $exec
   ; SI-NEXT:   S_BRANCH %bb.2
   ; SI-NEXT: {{  $}}
   ; SI-NEXT: bb.1.if.then:
   ; SI-NEXT:   successors: %bb.7(0x80000000)
   ; SI-NEXT: {{  $}}
   ; SI-NEXT:   [[S_LOAD_DWORDX2_IMM:%[0-9]+]]:sreg_64_xexec = S_LOAD_DWORDX2_IMM killed [[COPY]](p4), 36, 0 :: (dereferenceable invariant load (s64) from %ir.src1.kernarg.offset, align 4, addrspace 4)
   ; SI-NEXT:   [[V_ADD_CO_U32_e64_:%[0-9]+]]:vgpr_32, [[V_ADD_CO_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADD_CO_U32_e64 [[S_LOAD_DWORDX2_IMM]].sub0, killed %51, 0, implicit $exec
-  ; SI-NEXT:   %44:vgpr_32, dead %46:sreg_32_xm0_xexec = V_ADDC_U32_e64 0, killed [[S_LOAD_DWORDX2_IMM]].sub1, killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
-  ; SI-NEXT:   [[REG_SEQUENCE:%[0-9]+]]:vreg_64 = REG_SEQUENCE killed [[V_ADD_CO_U32_e64_]], %subreg.sub0, killed %44, %subreg.sub1
+  ; SI-NEXT:   [[V_ADDC_U32_e64_:%[0-9]+]]:vgpr_32, dead [[V_ADDC_U32_e64_1:%[0-9]+]]:sreg_32_xm0_xexec = V_ADDC_U32_e64 0, killed [[S_LOAD_DWORDX2_IMM]].sub1, killed [[V_ADD_CO_U32_e64_1]], 0, implicit $exec
+  ; SI-NEXT:   [[REG_SEQUENCE:%[0-9]+]]:vreg_64 = REG_SEQUENCE killed [[V_ADD_CO_U32_e64_]], %subreg.sub0, killed [[V_ADDC_U32_e64_]], %subreg.sub1
   ; SI-NEXT:   [[GLOBAL_LOAD_UBYTE:%[0-9]+]]:vgpr_32 = GLOBAL_LOAD_UBYTE killed [[REG_SEQUENCE]], 0, 0, implicit $exec :: (load (s8) from %ir.i10, addrspace 1)
   ; SI-NEXT:   [[V_MOV_B:%[0-9]+]]:vreg_64 = V_MOV_B64_PSEUDO 0, implicit $exec
   ; SI-NEXT:   GLOBAL_STORE_BYTE killed [[V_MOV_B]], killed [[GLOBAL_LOAD_UBYTE]], 0, 0, implicit $exec :: (store (s8) into `ptr addrspace(1) null`, addrspace 1)
   ; SI-NEXT:   S_BRANCH %bb.7
   ; SI-NEXT: {{  $}}
   ; SI-NEXT: bb.2.if.then9:
   ; SI-NEXT:   successors: %bb.4(0x40000000), %bb.3(0x40000000)
   ; SI-NEXT: {{  $}}
define amdgpu_kernel void @livevariables_update_missed_block(ptr addrspace(1) %src1) {
   ; SI-NEXT: {{  $}}
   ; SI-NEXT:   [[PHI:%[0-9]+]]:vgpr_32 = PHI [[COPY1]](s32), %bb.0, undef %52:vgpr_32, %bb.6
   ; SI-NEXT:   [[SI_ELSE:%[0-9]+]]:sreg_32 = SI_ELSE killed [[SI_IF]], %bb.7, implicit-def dead $exec, implicit-def dead $scc, implicit $exec
   ; SI-NEXT:   S_BRANCH %bb.1
   ; SI-NEXT: {{  $}}
   ; SI-NEXT: bb.6.sw.bb18:
   ; SI-NEXT:   successors: %bb.5(0x80000000)
   ; SI-NEXT: {{  $}}
-  ; SI-NEXT:   [[PHI1:%[0-9]+]]:vgpr_32 = PHI undef %36:vgpr_32, %bb.3, [[GLOBAL_LOAD_UBYTE1]], %bb.4
+  ; SI-NEXT:   [[PHI1:%[0-9]+]]:vgpr_32 = PHI undef [[COPY38:%[0-9]+]]:vgpr_32, %bb.3, [[GLOBAL_LOAD_UBYTE1]], %bb.4
   ; SI-NEXT:   [[V_MOV_B2:%[0-9]+]]:vreg_64 = V_MOV_B64_PSEUDO 0, implicit $exec
   ; SI-NEXT:   GLOBAL_STORE_BYTE killed [[V_MOV_B2]], killed [[PHI1]], 0, 0, implicit $exec :: (store (s8) into `ptr addrspace(1) null`, addrspace 1)
   ; SI-NEXT:   S_BRANCH %bb.5
   ; SI-NEXT: {{  $}}
   ; SI-NEXT: bb.7.UnifiedReturnBlock:
   ; SI-NEXT:   SI_END_CF killed [[SI_ELSE]], implicit-def dead $exec, implicit-def dead $scc, implicit $exec
   ; SI-NEXT:   S_ENDPGM 0
 entry:

llvm/test/CodeGen/AMDGPU/wwm-reserved-spill.ll

 
 define amdgpu_gfx i64 @strict_wwm_called_i64(i64 %a) noinline {
 ; GFX9-O0-LABEL: strict_wwm_called_i64:
 ; GFX9-O0:       ; %bb.0:
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v2, v0
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr34
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr34
-; GFX9-O0-NEXT:    ; kill: def $vgpr1 killed $vgpr1 killed $exec
 ; GFX9-O0-NEXT:    ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v3, v1
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr34_sgpr35
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v4, v2
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v5, v2
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v0, v3
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v1, v3
 ; GFX9-O0-NEXT:    v_add_co_u32_e64 v4, s[34:35], v4, v5

 ; GFX9-O0-NEXT:    v_readlane_b32 s36, v10, 2
 ; GFX9-O0-NEXT:    v_readlane_b32 s37, v10, 3
 ; GFX9-O0-NEXT:    v_readlane_b32 s38, v10, 4
 ; GFX9-O0-NEXT:    v_readlane_b32 s39, v10, 5
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v2, v0
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v3, v1
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr40
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr40
-; GFX9-O0-NEXT:    ; kill: def $vgpr2 killed $vgpr2 killed $exec
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v4, v8
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v5, v9
 ; GFX9-O0-NEXT:    v_add_co_u32_e64 v2, s[40:41], v2, v4
 ; GFX9-O0-NEXT:    v_addc_co_u32_e64 v3, s[40:41], v3, v5, s[40:41]
 ; GFX9-O0-NEXT:    s_mov_b64 exec, s[34:35]
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v0, v2
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v1, v3
 ; GFX9-O0-NEXT:    s_mov_b32 s34, 0

 ; GFX9-O0-NEXT:    s_mov_b32 s34, 0
 ; GFX9-O0-NEXT:    buffer_load_dwordx4 v[10:13], v0, s[36:39], s34 offen
 ; GFX9-O0-NEXT:    buffer_load_dwordx2 v[3:4], v0, s[36:39], s34 offen offset:16
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(1)
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v7, v11
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v5, v10
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr35
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr35
-; GFX9-O0-NEXT:    ; kill: def $vgpr7 killed $vgpr7 killed $exec
 ; GFX9-O0-NEXT:    ; kill: def $vgpr5 killed $vgpr5 def $vgpr5_vgpr6 killed $exec
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v6, v7
 ; GFX9-O0-NEXT:    s_mov_b32 s35, 0x7fffffff
 ; GFX9-O0-NEXT:    s_mov_b32 s40, -1
 ; GFX9-O0-NEXT:    ; kill: def $sgpr40 killed $sgpr40 def $sgpr40_sgpr41
 ; GFX9-O0-NEXT:    s_mov_b32 s41, s35
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v1, v5
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v2, v6
 ; GFX9-O0-NEXT:    s_not_b64 exec, exec
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v1, s40
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v2, s41
 ; GFX9-O0-NEXT:    s_not_b64 exec, exec
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v8, v1
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v9, v2
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v7, v13
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v5, v12
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr35
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr35
-; GFX9-O0-NEXT:    ; kill: def $vgpr7 killed $vgpr7 killed $exec
 ; GFX9-O0-NEXT:    ; kill: def $vgpr5 killed $vgpr5 def $vgpr5_vgpr6 killed $exec
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v6, v7
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v1, v5
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v2, v6
 ; GFX9-O0-NEXT:    s_not_b64 exec, exec
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v1, s40
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v2, s41
 ; GFX9-O0-NEXT:    s_not_b64 exec, exec

 ; GFX9-O0-NEXT:    v_mov_b32_e32 v11, v9
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v5, v8
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v9, v7
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v10, v6
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr35
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr35
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr35
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr35
-; GFX9-O0-NEXT:    ; kill: def $vgpr5 killed $vgpr5 killed $exec
-; GFX9-O0-NEXT:    ; kill: def $vgpr11 killed $vgpr11 killed $exec
-; GFX9-O0-NEXT:    ; kill: def $vgpr9 killed $vgpr9 killed $exec
 ; GFX9-O0-NEXT:    ; kill: def $vgpr5 killed $vgpr5 def $vgpr5_vgpr6_vgpr7_vgpr8 killed $exec
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v6, v11
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v7, v10
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v8, v9
 ; GFX9-O0-NEXT:    buffer_store_dwordx4 v[5:8], v0, s[36:39], s34 offen
 ; GFX9-O0-NEXT:    buffer_store_dwordx2 v[3:4], v0, s[36:39], s34 offen offset:16
 ; GFX9-O0-NEXT:    s_xor_saveexec_b64 s[34:35], -1
 ; GFX9-O0-NEXT:    buffer_load_dword v1, off, s[0:3], s32 ; 4-byte Folded Reload

 }
 
 ; FIXME: This spills v40 and v41 twice, once in whole-wave-mode and once for the active lanes
 define amdgpu_gfx <32 x i32> @strict_wwm_callee_saves(<32 x i32> inreg %keep, ptr addrspace(5) %ptr, i64 %a, i64 %b, i64 %c, i64 %d, i64 %e) #0 {
 ; GFX9-O0-LABEL: strict_wwm_callee_saves:
 ; GFX9-O0:       ; %bb.0:
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX9-O0-NEXT:    s_xor_saveexec_b64 s[34:35], -1
-; GFX9-O0-NEXT:    buffer_store_dword v32, off, s[0:3], s32 offset:224 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    buffer_store_dword v32, off, s[0:3], s32 offset:204 ; 4-byte Folded Spill
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    buffer_store_dword v33, off, s[0:3], s32 offset:228 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    buffer_store_dword v34, off, s[0:3], s32 offset:232 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    buffer_store_dword v33, off, s[0:3], s32 offset:208 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    buffer_store_dword v34, off, s[0:3], s32 offset:212 ; 4-byte Folded Spill
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    buffer_store_dword v35, off, s[0:3], s32 offset:236 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    buffer_store_dword v36, off, s[0:3], s32 offset:240 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    buffer_store_dword v35, off, s[0:3], s32 offset:216 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    buffer_store_dword v36, off, s[0:3], s32 offset:220 ; 4-byte Folded Spill
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    buffer_store_dword v37, off, s[0:3], s32 offset:244 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    buffer_store_dword v38, off, s[0:3], s32 offset:248 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    buffer_store_dword v37, off, s[0:3], s32 offset:224 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    buffer_store_dword v38, off, s[0:3], s32 offset:228 ; 4-byte Folded Spill
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    buffer_store_dword v39, off, s[0:3], s32 offset:252 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    buffer_store_dword v40, off, s[0:3], s32 offset:256 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    buffer_store_dword v39, off, s[0:3], s32 offset:232 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    buffer_store_dword v40, off, s[0:3], s32 offset:236 ; 4-byte Folded Spill
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    buffer_store_dword v41, off, s[0:3], s32 offset:260 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    buffer_store_dword v41, off, s[0:3], s32 offset:240 ; 4-byte Folded Spill
 ; GFX9-O0-NEXT:    s_mov_b64 exec, -1
-; GFX9-O0-NEXT:    buffer_store_dword v42, off, s[0:3], s32 offset:220 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    buffer_store_dword v42, off, s[0:3], s32 offset:200 ; 4-byte Folded Spill
 ; GFX9-O0-NEXT:    s_mov_b64 exec, s[34:35]
 ; GFX9-O0-NEXT:    buffer_store_dword v40, off, s[0:3], s32 offset:48 ; 4-byte Folded Spill
 ; GFX9-O0-NEXT:    buffer_store_dword v41, off, s[0:3], s32 offset:44 ; 4-byte Folded Spill
 ; GFX9-O0-NEXT:    buffer_store_dword v43, off, s[0:3], s32 offset:40 ; 4-byte Folded Spill
 ; GFX9-O0-NEXT:    buffer_store_dword v44, off, s[0:3], s32 offset:36 ; 4-byte Folded Spill
 ; GFX9-O0-NEXT:    buffer_store_dword v45, off, s[0:3], s32 offset:32 ; 4-byte Folded Spill
 ; GFX9-O0-NEXT:    buffer_store_dword v46, off, s[0:3], s32 offset:28 ; 4-byte Folded Spill
 ; GFX9-O0-NEXT:    buffer_store_dword v47, off, s[0:3], s32 offset:24 ; 4-byte Folded Spill
 ; GFX9-O0-NEXT:    v_writelane_b32 v42, s64, 0
 ; GFX9-O0-NEXT:    v_writelane_b32 v42, s65, 1
 ; GFX9-O0-NEXT:    v_writelane_b32 v42, s66, 2
 ; GFX9-O0-NEXT:    v_writelane_b32 v42, s67, 3
 ; GFX9-O0-NEXT:    buffer_store_dword v10, off, s[0:3], s32 offset:68 ; 4-byte Folded Spill
 ; GFX9-O0-NEXT:    buffer_store_dword v9, off, s[0:3], s32 offset:60 ; 4-byte Folded Spill
 ; GFX9-O0-NEXT:    buffer_store_dword v8, off, s[0:3], s32 offset:76 ; 4-byte Folded Spill
 ; GFX9-O0-NEXT:    buffer_store_dword v7, off, s[0:3], s32 offset:64 ; 4-byte Folded Spill
 ; GFX9-O0-NEXT:    buffer_store_dword v6, off, s[0:3], s32 offset:84 ; 4-byte Folded Spill
 ; GFX9-O0-NEXT:    buffer_store_dword v5, off, s[0:3], s32 offset:72 ; 4-byte Folded Spill
 ; GFX9-O0-NEXT:    buffer_store_dword v4, off, s[0:3], s32 offset:92 ; 4-byte Folded Spill
 ; GFX9-O0-NEXT:    buffer_store_dword v3, off, s[0:3], s32 offset:80 ; 4-byte Folded Spill
 ; GFX9-O0-NEXT:    buffer_store_dword v2, off, s[0:3], s32 offset:52 ; 4-byte Folded Spill
 ; GFX9-O0-NEXT:    buffer_store_dword v1, off, s[0:3], s32 offset:88 ; 4-byte Folded Spill
 ; GFX9-O0-NEXT:    buffer_store_dword v0, off, s[0:3], s32 offset:56 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    buffer_load_dword v1, off, s[0:3], s32 offset:20
+; GFX9-O0-NEXT:    buffer_load_dword v0, off, s[0:3], s32 offset:20
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    buffer_store_dword v1, off, s[0:3], s32 offset:116 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    buffer_load_dword v44, off, s[0:3], s32 offset:16
+; GFX9-O0-NEXT:    buffer_store_dword v0, off, s[0:3], s32 offset:116 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    buffer_load_dword v0, off, s[0:3], s32 offset:16
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    buffer_store_dword v44, off, s[0:3], s32 offset:112 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    buffer_store_dword v0, off, s[0:3], s32 offset:112 ; 4-byte Folded Spill
 ; GFX9-O0-NEXT:    buffer_load_dword v0, off, s[0:3], s32 offset:12
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-O0-NEXT:    buffer_store_dword v0, off, s[0:3], s32 offset:108 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    buffer_load_dword v46, off, s[0:3], s32 offset:8
+; GFX9-O0-NEXT:    buffer_load_dword v0, off, s[0:3], s32 offset:8
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    buffer_store_dword v46, off, s[0:3], s32 offset:104 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    buffer_load_dword v47, off, s[0:3], s32 offset:4
+; GFX9-O0-NEXT:    buffer_store_dword v0, off, s[0:3], s32 offset:104 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    buffer_load_dword v0, off, s[0:3], s32 offset:4
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    buffer_store_dword v47, off, s[0:3], s32 offset:100 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    buffer_load_dword v45, off, s[0:3], s32
+; GFX9-O0-NEXT:    buffer_store_dword v0, off, s[0:3], s32 offset:100 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    buffer_load_dword v0, off, s[0:3], s32
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    buffer_store_dword v45, off, s[0:3], s32 offset:96 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    buffer_store_dword v0, off, s[0:3], s32 offset:96 ; 4-byte Folded Spill
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr34
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr34
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr34
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr34
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr34
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr34
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v0, s4
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v43, s5
-; GFX9-O0-NEXT:    v_mov_b32_e32 v2, s6
-; GFX9-O0-NEXT:    buffer_store_dword v2, off, s[0:3], s32 offset:216 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    v_mov_b32_e32 v2, s7
-; GFX9-O0-NEXT:    buffer_store_dword v2, off, s[0:3], s32 offset:212 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    v_mov_b32_e32 v2, s8
-; GFX9-O0-NEXT:    buffer_store_dword v2, off, s[0:3], s32 offset:208 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    v_mov_b32_e32 v2, s9
-; GFX9-O0-NEXT:    buffer_store_dword v2, off, s[0:3], s32 offset:204 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    v_mov_b32_e32 v2, s10
-; GFX9-O0-NEXT:    buffer_store_dword v2, off, s[0:3], s32 offset:200 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    v_mov_b32_e32 v2, s11
-; GFX9-O0-NEXT:    buffer_store_dword v2, off, s[0:3], s32 offset:196 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    v_mov_b32_e32 v2, s12
-; GFX9-O0-NEXT:    buffer_store_dword v2, off, s[0:3], s32 offset:192 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    v_mov_b32_e32 v2, s13
-; GFX9-O0-NEXT:    buffer_store_dword v2, off, s[0:3], s32 offset:188 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    v_mov_b32_e32 v2, s14
-; GFX9-O0-NEXT:    buffer_store_dword v2, off, s[0:3], s32 offset:184 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    v_mov_b32_e32 v2, s15
-; GFX9-O0-NEXT:    buffer_store_dword v2, off, s[0:3], s32 offset:180 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    v_mov_b32_e32 v2, s16
-; GFX9-O0-NEXT:    buffer_store_dword v2, off, s[0:3], s32 offset:176 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    v_mov_b32_e32 v2, s17
-; GFX9-O0-NEXT:    buffer_store_dword v2, off, s[0:3], s32 offset:172 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    v_mov_b32_e32 v2, s18
-; GFX9-O0-NEXT:    buffer_store_dword v2, off, s[0:3], s32 offset:168 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    v_mov_b32_e32 v2, s19
-; GFX9-O0-NEXT:    buffer_store_dword v2, off, s[0:3], s32 offset:164 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    v_mov_b32_e32 v2, s20
-; GFX9-O0-NEXT:    buffer_store_dword v2, off, s[0:3], s32 offset:160 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    v_mov_b32_e32 v2, s21
-; GFX9-O0-NEXT:    buffer_store_dword v2, off, s[0:3], s32 offset:156 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    v_mov_b32_e32 v2, s22
-; GFX9-O0-NEXT:    buffer_store_dword v2, off, s[0:3], s32 offset:152 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    v_mov_b32_e32 v2, s23
-; GFX9-O0-NEXT:    buffer_store_dword v2, off, s[0:3], s32 offset:148 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    v_mov_b32_e32 v2, s24
-; GFX9-O0-NEXT:    buffer_store_dword v2, off, s[0:3], s32 offset:144 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    v_mov_b32_e32 v2, s25
-; GFX9-O0-NEXT:    buffer_store_dword v2, off, s[0:3], s32 offset:140 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    v_mov_b32_e32 v2, s26
-; GFX9-O0-NEXT:    buffer_store_dword v2, off, s[0:3], s32 offset:136 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    v_mov_b32_e32 v2, s27
-; GFX9-O0-NEXT:    buffer_store_dword v2, off, s[0:3], s32 offset:132 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    v_mov_b32_e32 v2, s28
-; GFX9-O0-NEXT:    buffer_store_dword v2, off, s[0:3], s32 offset:128 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    v_mov_b32_e32 v2, s29
-; GFX9-O0-NEXT:    buffer_store_dword v2, off, s[0:3], s32 offset:124 ; 4-byte Folded Spill
-; GFX9-O0-NEXT:    ; kill: def $vgpr45 killed $vgpr45 killed $exec
-; GFX9-O0-NEXT:    ; kill: def $vgpr47 killed $vgpr47 killed $exec
-; GFX9-O0-NEXT:    ; kill: def $vgpr46 killed $vgpr46 killed $exec
-; GFX9-O0-NEXT:    ; kill: def $vgpr44 killed $vgpr44 killed $exec
-; GFX9-O0-NEXT:    ; kill: def $vgpr1 killed $vgpr1 killed $exec
+; GFX9-O0-NEXT:    v_mov_b32_e32 v1, s6
+; GFX9-O0-NEXT:    buffer_store_dword v1, off, s[0:3], s32 offset:196 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    v_mov_b32_e32 v1, s7
+; GFX9-O0-NEXT:    buffer_store_dword v1, off, s[0:3], s32 offset:192 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    v_mov_b32_e32 v1, s8
+; GFX9-O0-NEXT:    buffer_store_dword v1, off, s[0:3], s32 offset:188 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    v_mov_b32_e32 v1, s9
+; GFX9-O0-NEXT:    buffer_store_dword v1, off, s[0:3], s32 offset:184 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    v_mov_b32_e32 v1, s10
+; GFX9-O0-NEXT:    buffer_store_dword v1, off, s[0:3], s32 offset:180 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    v_mov_b32_e32 v1, s11
+; GFX9-O0-NEXT:    buffer_store_dword v1, off, s[0:3], s32 offset:176 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    v_mov_b32_e32 v1, s12
+; GFX9-O0-NEXT:    buffer_store_dword v1, off, s[0:3], s32 offset:172 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    v_mov_b32_e32 v1, s13
+; GFX9-O0-NEXT:    buffer_store_dword v1, off, s[0:3], s32 offset:168 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    v_mov_b32_e32 v1, s14
+; GFX9-O0-NEXT:    buffer_store_dword v1, off, s[0:3], s32 offset:164 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    v_mov_b32_e32 v1, s15
+; GFX9-O0-NEXT:    buffer_store_dword v1, off, s[0:3], s32 offset:160 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    v_mov_b32_e32 v1, s16
+; GFX9-O0-NEXT:    buffer_store_dword v1, off, s[0:3], s32 offset:156 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    v_mov_b32_e32 v1, s17
+; GFX9-O0-NEXT:    buffer_store_dword v1, off, s[0:3], s32 offset:152 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    v_mov_b32_e32 v1, s18
+; GFX9-O0-NEXT:    buffer_store_dword v1, off, s[0:3], s32 offset:148 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    v_mov_b32_e32 v1, s19
+; GFX9-O0-NEXT:    buffer_store_dword v1, off, s[0:3], s32 offset:144 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    v_mov_b32_e32 v1, s20
+; GFX9-O0-NEXT:    buffer_store_dword v1, off, s[0:3], s32 offset:140 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    v_mov_b32_e32 v1, s21
+; GFX9-O0-NEXT:    buffer_store_dword v1, off, s[0:3], s32 offset:136 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    v_mov_b32_e32 v1, s22
+; GFX9-O0-NEXT:    buffer_store_dword v1, off, s[0:3], s32 offset:132 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    v_mov_b32_e32 v1, s23
+; GFX9-O0-NEXT:    buffer_store_dword v1, off, s[0:3], s32 offset:128 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    v_mov_b32_e32 v1, s24
+; GFX9-O0-NEXT:    buffer_store_dword v1, off, s[0:3], s32 offset:124 ; 4-byte Folded Spill
+; GFX9-O0-NEXT:    v_mov_b32_e32 v47, s25
+; GFX9-O0-NEXT:    v_mov_b32_e32 v46, s26
+; GFX9-O0-NEXT:    v_mov_b32_e32 v45, s27
+; GFX9-O0-NEXT:    v_mov_b32_e32 v44, s28
+; GFX9-O0-NEXT:    v_mov_b32_e32 v1, s29
 ; GFX9-O0-NEXT:    buffer_store_dword v1, off, s[0:3], s32 offset:120 ; 4-byte Folded Spill
 ; GFX9-O0-NEXT:    ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1_vgpr2_vgpr3_vgpr4_vgpr5_vgpr6_vgpr7_vgpr8_vgpr9_vgpr10_vgpr11_vgpr12_vgpr13_vgpr14_vgpr15_vgpr16_vgpr17_vgpr18_vgpr19_vgpr20_vgpr21_vgpr22_vgpr23_vgpr24_vgpr25_vgpr26_vgpr27_vgpr28_vgpr29_vgpr30_vgpr31 killed $exec
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v1, v43
-; GFX9-O0-NEXT:    buffer_load_dword v43, off, s[0:3], s32 offset:216 ; 4-byte Folded Reload
-; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    v_mov_b32_e32 v2, v43
-; GFX9-O0-NEXT:    buffer_load_dword v43, off, s[0:3], s32 offset:212 ; 4-byte Folded Reload
-; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    v_mov_b32_e32 v3, v43
-; GFX9-O0-NEXT:    buffer_load_dword v43, off, s[0:3], s32 offset:208 ; 4-byte Folded Reload
-; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    v_mov_b32_e32 v4, v43
-; GFX9-O0-NEXT:    buffer_load_dword v43, off, s[0:3], s32 offset:204 ; 4-byte Folded Reload
-; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    v_mov_b32_e32 v5, v43
-; GFX9-O0-NEXT:    buffer_load_dword v43, off, s[0:3], s32 offset:200 ; 4-byte Folded Reload
-; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    v_mov_b32_e32 v6, v43
 ; GFX9-O0-NEXT:    buffer_load_dword v43, off, s[0:3], s32 offset:196 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    v_mov_b32_e32 v7, v43
+; GFX9-O0-NEXT:    v_mov_b32_e32 v2, v43
 ; GFX9-O0-NEXT:    buffer_load_dword v43, off, s[0:3], s32 offset:192 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    v_mov_b32_e32 v8, v43
+; GFX9-O0-NEXT:    v_mov_b32_e32 v3, v43
 ; GFX9-O0-NEXT:    buffer_load_dword v43, off, s[0:3], s32 offset:188 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    v_mov_b32_e32 v9, v43
+; GFX9-O0-NEXT:    v_mov_b32_e32 v4, v43
 ; GFX9-O0-NEXT:    buffer_load_dword v43, off, s[0:3], s32 offset:184 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    v_mov_b32_e32 v10, v43
+; GFX9-O0-NEXT:    v_mov_b32_e32 v5, v43
 ; GFX9-O0-NEXT:    buffer_load_dword v43, off, s[0:3], s32 offset:180 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    v_mov_b32_e32 v11, v43
+; GFX9-O0-NEXT:    v_mov_b32_e32 v6, v43
 ; GFX9-O0-NEXT:    buffer_load_dword v43, off, s[0:3], s32 offset:176 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    v_mov_b32_e32 v12, v43
+; GFX9-O0-NEXT:    v_mov_b32_e32 v7, v43
 ; GFX9-O0-NEXT:    buffer_load_dword v43, off, s[0:3], s32 offset:172 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    v_mov_b32_e32 v13, v43
+; GFX9-O0-NEXT:    v_mov_b32_e32 v8, v43
 ; GFX9-O0-NEXT:    buffer_load_dword v43, off, s[0:3], s32 offset:168 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    v_mov_b32_e32 v14, v43
+; GFX9-O0-NEXT:    v_mov_b32_e32 v9, v43
 ; GFX9-O0-NEXT:    buffer_load_dword v43, off, s[0:3], s32 offset:164 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    v_mov_b32_e32 v15, v43
+; GFX9-O0-NEXT:    v_mov_b32_e32 v10, v43
 ; GFX9-O0-NEXT:    buffer_load_dword v43, off, s[0:3], s32 offset:160 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    v_mov_b32_e32 v16, v43
+; GFX9-O0-NEXT:    v_mov_b32_e32 v11, v43
 ; GFX9-O0-NEXT:    buffer_load_dword v43, off, s[0:3], s32 offset:156 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    v_mov_b32_e32 v17, v43
+; GFX9-O0-NEXT:    v_mov_b32_e32 v12, v43
 ; GFX9-O0-NEXT:    buffer_load_dword v43, off, s[0:3], s32 offset:152 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    v_mov_b32_e32 v18, v43
+; GFX9-O0-NEXT:    v_mov_b32_e32 v13, v43
 ; GFX9-O0-NEXT:    buffer_load_dword v43, off, s[0:3], s32 offset:148 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    v_mov_b32_e32 v19, v43
+; GFX9-O0-NEXT:    v_mov_b32_e32 v14, v43
 ; GFX9-O0-NEXT:    buffer_load_dword v43, off, s[0:3], s32 offset:144 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    v_mov_b32_e32 v20, v43
+; GFX9-O0-NEXT:    v_mov_b32_e32 v15, v43
 ; GFX9-O0-NEXT:    buffer_load_dword v43, off, s[0:3], s32 offset:140 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    v_mov_b32_e32 v21, v43
+; GFX9-O0-NEXT:    v_mov_b32_e32 v16, v43
 ; GFX9-O0-NEXT:    buffer_load_dword v43, off, s[0:3], s32 offset:136 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    v_mov_b32_e32 v22, v43
+; GFX9-O0-NEXT:    v_mov_b32_e32 v17, v43
 ; GFX9-O0-NEXT:    buffer_load_dword v43, off, s[0:3], s32 offset:132 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    v_mov_b32_e32 v23, v43
+; GFX9-O0-NEXT:    v_mov_b32_e32 v18, v43
 ; GFX9-O0-NEXT:    buffer_load_dword v43, off, s[0:3], s32 offset:128 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    v_mov_b32_e32 v24, v43
+; GFX9-O0-NEXT:    v_mov_b32_e32 v19, v43
 ; GFX9-O0-NEXT:    buffer_load_dword v43, off, s[0:3], s32 offset:124 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-O0-NEXT:    v_mov_b32_e32 v25, v43
+; GFX9-O0-NEXT:    v_mov_b32_e32 v20, v43
 ; GFX9-O0-NEXT:    buffer_load_dword v43, off, s[0:3], s32 offset:120 ; 4-byte Folded Reload
-; GFX9-O0-NEXT:    v_mov_b32_e32 v26, v45
+; GFX9-O0-NEXT:    v_mov_b32_e32 v21, v47
+; GFX9-O0-NEXT:    buffer_load_dword v47, off, s[0:3], s32 offset:100 ; 4-byte Folded Reload
+; GFX9-O0-NEXT:    v_mov_b32_e32 v22, v46
+; GFX9-O0-NEXT:    buffer_load_dword v46, off, s[0:3], s32 offset:104 ; 4-byte Folded Reload
+; GFX9-O0-NEXT:    v_mov_b32_e32 v23, v45
 ; GFX9-O0-NEXT:    buffer_load_dword v45, off, s[0:3], s32 offset:108 ; 4-byte Folded Reload
+; GFX9-O0-NEXT:    v_mov_b32_e32 v24, v44
+; GFX9-O0-NEXT:    buffer_load_dword v44, off, s[0:3], s32 offset:112 ; 4-byte Folded Reload
+; GFX9-O0-NEXT:    s_waitcnt vmcnt(4)
+; GFX9-O0-NEXT:    v_mov_b32_e32 v25, v43
+; GFX9-O0-NEXT:    buffer_load_dword v43, off, s[0:3], s32 offset:96 ; 4-byte Folded Reload
+; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-O0-NEXT:    v_mov_b32_e32 v26, v43
+; GFX9-O0-NEXT:    buffer_load_dword v43, off, s[0:3], s32 offset:116 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v27, v47
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v28, v46
-; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v29, v45
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v30, v44
 ; GFX9-O0-NEXT:    ; kill: def $vgpr31 killed $vgpr43 killed $exec
 ; GFX9-O0-NEXT:    buffer_load_dword v31, off, s[0:3], s32 offset:116 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    buffer_load_dword v30, off, s[0:3], s32 offset:112 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    buffer_load_dword v29, off, s[0:3], s32 offset:108 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    buffer_load_dword v28, off, s[0:3], s32 offset:104 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    buffer_load_dword v27, off, s[0:3], s32 offset:100 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    buffer_load_dword v26, off, s[0:3], s32 offset:96 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    buffer_load_dword v11, off, s[0:3], s32 offset:92 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    buffer_load_dword v10, off, s[0:3], s32 offset:88 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    buffer_load_dword v9, off, s[0:3], s32 offset:84 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    buffer_load_dword v8, off, s[0:3], s32 offset:80 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    buffer_load_dword v7, off, s[0:3], s32 offset:76 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    buffer_load_dword v6, off, s[0:3], s32 offset:72 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    buffer_load_dword v5, off, s[0:3], s32 offset:68 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    buffer_load_dword v4, off, s[0:3], s32 offset:64 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    buffer_load_dword v2, off, s[0:3], s32 offset:60 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    buffer_load_dword v1, off, s[0:3], s32 offset:56 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    buffer_load_dword v0, off, s[0:3], s32 offset:52 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr34
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr34
-; GFX9-O0-NEXT:    ; kill: def $vgpr5 killed $vgpr5 killed $exec
 ; GFX9-O0-NEXT:    ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(4)
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v3, v5
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr34
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr34
-; GFX9-O0-NEXT:    ; kill: def $vgpr7 killed $vgpr7 killed $exec
 ; GFX9-O0-NEXT:    ; kill: def $vgpr4 killed $vgpr4 def $vgpr4_vgpr5 killed $exec
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v5, v7
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr34
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr34
-; GFX9-O0-NEXT:    ; kill: def $vgpr9 killed $vgpr9 killed $exec
 ; GFX9-O0-NEXT:    ; kill: def $vgpr6 killed $vgpr6 def $vgpr6_vgpr7 killed $exec
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v7, v9
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr34
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr34
-; GFX9-O0-NEXT:    ; kill: def $vgpr8 killed $vgpr8 killed $exec
 ; GFX9-O0-NEXT:    ; kill: def $vgpr8 killed $vgpr8 def $vgpr8_vgpr9 killed $exec
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v9, v11
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr34
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr34
-; GFX9-O0-NEXT:    ; kill: def $vgpr10 killed $vgpr10 killed $exec
 ; GFX9-O0-NEXT:    ; kill: def $vgpr10 killed $vgpr10 def $vgpr10_vgpr11 killed $exec
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-O0-NEXT:    v_mov_b32_e32 v11, v0
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr34_sgpr35
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr34_sgpr35
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr34_sgpr35
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr34_sgpr35
 ; GFX9-O0-NEXT:    ; implicit-def: $sgpr34_sgpr35

 ; GFX9-O0-NEXT:    buffer_load_dword v47, off, s[0:3], s32 offset:24 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    buffer_load_dword v46, off, s[0:3], s32 offset:28 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    buffer_load_dword v45, off, s[0:3], s32 offset:32 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    buffer_load_dword v44, off, s[0:3], s32 offset:36 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    buffer_load_dword v43, off, s[0:3], s32 offset:40 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    buffer_load_dword v41, off, s[0:3], s32 offset:44 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    buffer_load_dword v40, off, s[0:3], s32 offset:48 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    s_xor_saveexec_b64 s[34:35], -1
-; GFX9-O0-NEXT:    buffer_load_dword v32, off, s[0:3], s32 offset:224 ; 4-byte Folded Reload
+; GFX9-O0-NEXT:    buffer_load_dword v32, off, s[0:3], s32 offset:204 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    s_nop 0
-; GFX9-O0-NEXT:    buffer_load_dword v33, off, s[0:3], s32 offset:228 ; 4-byte Folded Reload
+; GFX9-O0-NEXT:    buffer_load_dword v33, off, s[0:3], s32 offset:208 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    s_nop 0
-; GFX9-O0-NEXT:    buffer_load_dword v34, off, s[0:3], s32 offset:232 ; 4-byte Folded Reload
+; GFX9-O0-NEXT:    buffer_load_dword v34, off, s[0:3], s32 offset:212 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    s_nop 0
-; GFX9-O0-NEXT:    buffer_load_dword v35, off, s[0:3], s32 offset:236 ; 4-byte Folded Reload
+; GFX9-O0-NEXT:    buffer_load_dword v35, off, s[0:3], s32 offset:216 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    s_nop 0
-; GFX9-O0-NEXT:    buffer_load_dword v36, off, s[0:3], s32 offset:240 ; 4-byte Folded Reload
+; GFX9-O0-NEXT:    buffer_load_dword v36, off, s[0:3], s32 offset:220 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    s_nop 0
-; GFX9-O0-NEXT:    buffer_load_dword v37, off, s[0:3], s32 offset:244 ; 4-byte Folded Reload
+; GFX9-O0-NEXT:    buffer_load_dword v37, off, s[0:3], s32 offset:224 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    s_nop 0
-; GFX9-O0-NEXT:    buffer_load_dword v38, off, s[0:3], s32 offset:248 ; 4-byte Folded Reload
+; GFX9-O0-NEXT:    buffer_load_dword v38, off, s[0:3], s32 offset:228 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    s_nop 0
-; GFX9-O0-NEXT:    buffer_load_dword v39, off, s[0:3], s32 offset:252 ; 4-byte Folded Reload
+; GFX9-O0-NEXT:    buffer_load_dword v39, off, s[0:3], s32 offset:232 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    s_nop 0
-; GFX9-O0-NEXT:    buffer_load_dword v40, off, s[0:3], s32 offset:256 ; 4-byte Folded Reload
+; GFX9-O0-NEXT:    buffer_load_dword v40, off, s[0:3], s32 offset:236 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    s_nop 0
-; GFX9-O0-NEXT:    buffer_load_dword v41, off, s[0:3], s32 offset:260 ; 4-byte Folded Reload
+; GFX9-O0-NEXT:    buffer_load_dword v41, off, s[0:3], s32 offset:240 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    s_mov_b64 exec, -1
-; GFX9-O0-NEXT:    buffer_load_dword v42, off, s[0:3], s32 offset:220 ; 4-byte Folded Reload
+; GFX9-O0-NEXT:    buffer_load_dword v42, off, s[0:3], s32 offset:200 ; 4-byte Folded Reload
 ; GFX9-O0-NEXT:    s_mov_b64 exec, s[34:35]
 ; GFX9-O0-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-O0-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX9-O3-LABEL: strict_wwm_callee_saves:
 ; GFX9-O3:       ; %bb.0:
 ; GFX9-O3-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX9-O3-NEXT:    s_xor_saveexec_b64 s[34:35], -1