[PeepholeOptimizer] Enhance the redundant COPY elimination.

[hliao]

Summary

• Eliminate redundant COPYs from the same register & subregister pair.

[hliao]

This patch enhances the peephole-opt to fix the redundant copy issues once to be fixed in D87556. With the enhancement, we could remove that redundant COPY locally. Test cases are revised due to the code quality improvement or change. Fortunately, AMDGPU and ARM tests need addressing that difference.

[hliao]

llvm/test/CodeGen/Thumb2/mve-vcvt16.ll

21–30

The code sequence is totally different. But, based on my understanding ARM ISA, they are equivalent. The previous one will copy q0 to q2 and convert s8~s11 (alias to q2) into s0~s7 (alias to q0 and q1) as the return value. The new one firstly convert s0~s3 (alias to q0 as the input) to s4~s11 (alias to q1 and q2) followed by moving q2 to q0 to form the return pair of q0 and q1. Please let me know whether they are really equivalent.

[dmgreen]

llvm/test/CodeGen/Thumb2/mve-vcvt16.ll

21–30

Yeah sounds fine.

[hliao]

Rebase.

[qcolombet]

Hi @hliao,

I must be missing something, but it feels to me that this patch is actually making the situation worse.

Could you look at my example inlined below and explain how it would still work with this patch?

Cheers,
-Quentin

llvm/lib/CodeGen/PeepholeOptimizer.cpp

1409

I am not convinced this patch is an improvement at removing redundant copy instructions.

It seems to me we would only consider redundant copies for matching sub register whereas previously we would have propagated whole copies.
E.g., previously for

%1 = COPY %0
%2 = COPY %0:sub1

We would have replaced %2 with %1:sub1, but unless I miss something, with this new patch, we won't see %2 as a redundant copy anymore.

[hliao]

In D87939#2285776, @qcolombet wrote:

Hi @hliao,

I must be missing something, but it feels to me that this patch is actually making the situation worse.

Could you look at my example inlined below and explain how it would still work with this patch?

Cheers,
-Quentin

The comment seems outdated, if my understanding is right, and even the original code cannot perform that change since, once the 2nd COPY with same source is found in L1407, the check @ L1419 just skips that earlier as the 1st COPY has no subreg and the 2nd COPY has sub1.

[qcolombet]

The comment seems outdated, if my understanding is right, and even the original code cannot perform that change since, once the 2nd COPY with same source is found in L1407, the check @ L1419 just skips that earlier as the 1st COPY has no subreg and the 2nd COPY has sub1.

Good point!

Now, I am wondering why is this change not just NFC then?

[hliao]

In D87939#2285876, @qcolombet wrote:

The comment seems outdated, if my understanding is right, and even the original code cannot perform that change since, once the 2nd COPY with same source is found in L1407, the check @ L1419 just skips that earlier as the 1st COPY has no subreg and the 2nd COPY has sub1.

Good point!

Now, I am wondering why is this change not just NFC then?

It does improve the original by considering the COPY from the same subreg.

BTW, the comment seems ambiguous. It may read that that subreg extract won't be eliminated but handled somewhere else. @arsenm any comment?

[qcolombet]

It does improve the original by considering the COPY from the same subreg.

I see. Previously we would have stopped on the first non-subreg copy.

Makes sense.

Thanks for taking the time to explain it.

Make sure the commit message explains all this. LGTM otherwise.

Cheers,
-Quentin

[arsenm]

llvm/test/CodeGen/AMDGPU/GlobalISel/extractelement.i128.ll

301

I'm surprised there are so many changes here, but I guess it's probably papering over a number of combines that are missing

[hliao]

llvm/test/CodeGen/AMDGPU/GlobalISel/extractelement.i128.ll

301

for this 3 global isel tests, I compared the new assembly against the old one, all the redundant copies are removed and overall code lengths are reduced.

llvm/lib/CodeGen/PeepholeOptimizer.cpp

private:
     /// a def operand and a use operand that are tied. If there is a use
     /// operand commutable with the tied use operand, find recurrence cycle
     /// along that operand as well.
     bool findTargetRecurrence(Register Reg,
                               const SmallSet<Register, 2> &TargetReg,
                               RecurrenceCycle &RC);
 
     /// If copy instruction \p MI is a virtual register copy, track it in
-    /// the set \p CopySrcRegs and \p CopyMIs. If this virtual register was
-    /// previously seen as a copy, replace the uses of this copy with the
-    /// previously seen copy's destination register.
-    bool foldRedundantCopy(MachineInstr &MI, SmallSet<Register, 4> &CopySrcRegs,
-                           DenseMap<Register, MachineInstr *> &CopyMIs);
+    /// the set \p CopyMIs. If this virtual register was previously seen as a
+    /// copy, replace the uses of this copy with the previously seen copy's
+    /// destination register.
+    bool foldRedundantCopy(MachineInstr &MI,
+                           DenseMap<RegSubRegPair, MachineInstr *> &CopyMIs);
 
     /// Is the register \p Reg a non-allocatable physical register?
     bool isNAPhysCopy(Register Reg);
 
     /// If copy instruction \p MI is a non-allocatable virtual<->physical
     /// register copy, track it in the \p NAPhysToVirtMIs map. If this
     /// non-allocatable physical register was previously copied to a virtual
     /// registered and hasn't been clobbered, the virt->phys copy can be

 // only the first copy is considered.
 //
 // e.g.
 // %1 = COPY %0
 // %2 = COPY %0:sub1
 //
 // Should replace %2 uses with %1:sub1
 bool PeepholeOptimizer::foldRedundantCopy(
-    MachineInstr &MI, SmallSet<Register, 4> &CopySrcRegs,
-    DenseMap<Register, MachineInstr *> &CopyMIs) {
+    MachineInstr &MI, DenseMap<RegSubRegPair, MachineInstr *> &CopyMIs) {
   assert(MI.isCopy() && "expected a COPY machine instruction");
 
   Register SrcReg = MI.getOperand(1).getReg();
+  unsigned SrcSubReg = MI.getOperand(1).getSubReg();
   if (!SrcReg.isVirtual())
     return false;
 
   Register DstReg = MI.getOperand(0).getReg();
   if (!DstReg.isVirtual())
     return false;
 
-  if (CopySrcRegs.insert(SrcReg).second) {
+  RegSubRegPair SrcPair(SrcReg, SrcSubReg);
+
+  if (CopyMIs.insert(std::make_pair(SrcPair, &MI)).second) {
     // First copy of this reg seen.
-    CopyMIs.insert(std::make_pair(SrcReg, &MI));
     return false;
   }
 
-  MachineInstr *PrevCopy = CopyMIs.find(SrcReg)->second;
+  MachineInstr *PrevCopy = CopyMIs.find(SrcPair)->second;
 
-  unsigned SrcSubReg = MI.getOperand(1).getSubReg();
   unsigned PrevSrcSubReg = PrevCopy->getOperand(1).getSubReg();
-
-  // Can't replace different subregister extracts.
-  if (SrcSubReg != PrevSrcSubReg)
-    return false;
+  assert(SrcSubReg == PrevSrcSubReg && "Unexpected mismatching subreg!");
 
   Register PrevDstReg = PrevCopy->getOperand(0).getReg();
 
   // Only replace if the copy register class is the same.
   //
   // TODO: If we have multiple copies to different register classes, we may want
   // to track multiple copies of the same source register.
   if (MRI->getRegClass(DstReg) != MRI->getRegClass(PrevDstReg))
for (MachineBasicBlock &MBB : MF) {
 
     // Track when a non-allocatable physical register is copied to a virtual
     // register so that useless moves can be removed.
     //
     // $physreg is the map index; MI is the last valid `%vreg = COPY $physreg`
     // without any intervening re-definition of $physreg.
     DenseMap<Register, MachineInstr *> NAPhysToVirtMIs;
 
-    // Set of virtual registers that are copied from.
-    SmallSet<Register, 4> CopySrcRegs;
-    DenseMap<Register, MachineInstr *> CopySrcMIs;
+    // Set of pairs of virtual registers and their subregs that are copied
+    // from.
+    DenseMap<RegSubRegPair, MachineInstr *> CopySrcMIs;
 
     bool IsLoopHeader = MLI->isLoopHeader(&MBB);
 
     for (MachineBasicBlock::iterator MII = MBB.begin(), MIE = MBB.end();
          MII != MIE; ) {
       MachineInstr *MI = &*MII;
       // We may be erasing MI below, increment MII now.
       ++MII;
       LocalMIs.insert(MI);
 
-      // Skip debug instructions. They should not affect this peephole optimization.
+      // Skip debug instructions. They should not affect this peephole
+      // optimization.
       if (MI->isDebugInstr())
-          continue;
 
       if (MI->isPosition())
         continue;
 
       if (IsLoopHeader && MI->isPHI()) {
         if (optimizeRecurrence(*MI)) {
           Changed = true;
           continue;
for (MachineBasicBlock::iterator MII = MBB.begin(), MIE = MBB.end();
       }
 
       if (isCoalescableCopy(*MI) && optimizeCoalescableCopy(*MI)) {
         // MI is just rewritten.
         Changed = true;
         continue;
       }
 
-      if (MI->isCopy() &&
-          (foldRedundantCopy(*MI, CopySrcRegs, CopySrcMIs) ||
+      if (MI->isCopy() && (foldRedundantCopy(*MI, CopySrcMIs) ||
                            foldRedundantNAPhysCopy(*MI, NAPhysToVirtMIs))) {
         LocalMIs.erase(MI);
         LLVM_DEBUG(dbgs() << "Deleting redundant copy: " << *MI << "\n");
         MI->eraseFromParent();
         Changed = true;
         continue;
       }
 
       if (isMoveImmediate(*MI, ImmDefRegs, ImmDefMIs)) {

llvm/test/CodeGen/AMDGPU/GlobalISel/extractelement.i128.ll

; GFX7-NEXT:    s_setpc_b64 s[30:31]
   %element = extractelement <4 x i128> %vector, i32 %idx
   ret i128 %element
 }
 
 define amdgpu_ps i128 @extractelement_sgpr_v4i128_vgpr_idx(<4 x i128> addrspace(4)* inreg %ptr, i32 %idx) {
 ; GFX9-LABEL: extractelement_sgpr_v4i128_vgpr_idx:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx16 s[0:15], s[2:3], 0x0
-; GFX9-NEXT:    v_lshlrev_b32_e32 v2, 1, v0
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v2
+; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 1, v0
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v0
+; GFX9-NEXT:    v_add_u32_e32 v19, 1, v0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    v_mov_b32_e32 v3, s0
-; GFX9-NEXT:    v_mov_b32_e32 v1, s2
-; GFX9-NEXT:    v_mov_b32_e32 v0, s1
+; GFX9-NEXT:    v_mov_b32_e32 v1, s0
+; GFX9-NEXT:    v_mov_b32_e32 v2, s1
+; GFX9-NEXT:    v_mov_b32_e32 v3, s2
 ; GFX9-NEXT:    v_mov_b32_e32 v4, s3
 ; GFX9-NEXT:    v_mov_b32_e32 v5, s4
 ; GFX9-NEXT:    v_mov_b32_e32 v6, s5
-; GFX9-NEXT:    v_cndmask_b32_e32 v1, v3, v1, vcc
-; GFX9-NEXT:    v_cndmask_b32_e32 v0, v0, v4, vcc
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 2, v2
-; GFX9-NEXT:    v_cndmask_b32_e32 v1, v1, v5, vcc
-; GFX9-NEXT:    v_cndmask_b32_e32 v0, v0, v6, vcc
+; GFX9-NEXT:    v_cndmask_b32_e32 v17, v1, v3, vcc
+; GFX9-NEXT:    v_cndmask_b32_e32 v18, v2, v4, vcc
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 2, v0
 ; GFX9-NEXT:    v_mov_b32_e32 v7, s6
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 3, v2
 ; GFX9-NEXT:    v_mov_b32_e32 v8, s7
-; GFX9-NEXT:    v_cndmask_b32_e32 v1, v1, v7, vcc
-; GFX9-NEXT:    v_cndmask_b32_e32 v0, v0, v8, vcc
+; GFX9-NEXT:    v_cndmask_b32_e32 v17, v17, v5, vcc
+; GFX9-NEXT:    v_cndmask_b32_e32 v18, v18, v6, vcc
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 3, v0
 ; GFX9-NEXT:    v_mov_b32_e32 v9, s8
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 4, v2
 ; GFX9-NEXT:    v_mov_b32_e32 v10, s9
-; GFX9-NEXT:    v_cndmask_b32_e32 v1, v1, v9, vcc
-; GFX9-NEXT:    v_cndmask_b32_e32 v0, v0, v10, vcc
+; GFX9-NEXT:    v_cndmask_b32_e32 v17, v17, v7, vcc
+; GFX9-NEXT:    v_cndmask_b32_e32 v18, v18, v8, vcc
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 4, v0
 ; GFX9-NEXT:    v_mov_b32_e32 v11, s10
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 5, v2
 ; GFX9-NEXT:    v_mov_b32_e32 v12, s11
-; GFX9-NEXT:    v_cndmask_b32_e32 v1, v1, v11, vcc
-; GFX9-NEXT:    v_cndmask_b32_e32 v0, v0, v12, vcc
+; GFX9-NEXT:    v_cndmask_b32_e32 v17, v17, v9, vcc
+; GFX9-NEXT:    v_cndmask_b32_e32 v18, v18, v10, vcc
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 5, v0
 ; GFX9-NEXT:    v_mov_b32_e32 v13, s12
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 6, v2
 ; GFX9-NEXT:    v_mov_b32_e32 v14, s13
-; GFX9-NEXT:    v_cndmask_b32_e32 v1, v1, v13, vcc
-; GFX9-NEXT:    v_cndmask_b32_e32 v4, v0, v14, vcc
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 7, v2
+; GFX9-NEXT:    v_cndmask_b32_e32 v17, v17, v11, vcc
+; GFX9-NEXT:    v_cndmask_b32_e32 v18, v18, v12, vcc
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 6, v0
+; GFX9-NEXT:    v_cndmask_b32_e32 v17, v17, v13, vcc
+; GFX9-NEXT:    v_cndmask_b32_e32 v18, v18, v14, vcc
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v19
+; GFX9-NEXT:    v_cndmask_b32_e32 v1, v1, v3, vcc
+; GFX9-NEXT:    v_cndmask_b32_e32 v2, v2, v4, vcc
+; GFX9-NEXT:    v_cmp_eq_u32_e64 s[0:1], 2, v19
 ; GFX9-NEXT:    v_mov_b32_e32 v15, s14
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 7, v0
 ; GFX9-NEXT:    v_mov_b32_e32 v16, s15
-; GFX9-NEXT:    v_add_u32_e32 v2, 1, v2
-; GFX9-NEXT:    v_cndmask_b32_e32 v0, v1, v15, vcc
-; GFX9-NEXT:    v_cndmask_b32_e32 v1, v4, v16, vcc
-; GFX9-NEXT:    v_mov_b32_e32 v4, s1
-; GFX9-NEXT:    v_mov_b32_e32 v5, s2
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v2
-; GFX9-NEXT:    v_mov_b32_e32 v6, s3
-; GFX9-NEXT:    v_cndmask_b32_e32 v3, v3, v5, vcc
-; GFX9-NEXT:    v_cndmask_b32_e32 v4, v4, v6, vcc
-; GFX9-NEXT:    v_mov_b32_e32 v7, s4
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 2, v2
-; GFX9-NEXT:    v_mov_b32_e32 v8, s5
-; GFX9-NEXT:    v_cndmask_b32_e32 v3, v3, v7, vcc
-; GFX9-NEXT:    v_cndmask_b32_e32 v4, v4, v8, vcc
-; GFX9-NEXT:    v_mov_b32_e32 v9, s6
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 3, v2
-; GFX9-NEXT:    v_mov_b32_e32 v10, s7
+; GFX9-NEXT:    v_cndmask_b32_e64 v1, v1, v5, s[0:1]
+; GFX9-NEXT:    v_cndmask_b32_e64 v2, v2, v6, s[0:1]
+; GFX9-NEXT:    v_cmp_eq_u32_e64 s[0:1], 3, v19
+; GFX9-NEXT:    v_cndmask_b32_e64 v3, v1, v7, s[0:1]
+; GFX9-NEXT:    v_cndmask_b32_e32 v0, v17, v15, vcc
+; GFX9-NEXT:    v_cndmask_b32_e32 v1, v18, v16, vcc
+; GFX9-NEXT:    v_cndmask_b32_e64 v2, v2, v8, s[0:1]
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 4, v19
 ; GFX9-NEXT:    v_cndmask_b32_e32 v3, v3, v9, vcc
-; GFX9-NEXT:    v_cndmask_b32_e32 v4, v4, v10, vcc
-; GFX9-NEXT:    v_mov_b32_e32 v11, s8
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 4, v2
-; GFX9-NEXT:    v_mov_b32_e32 v12, s9
-; GFX9-NEXT:    v_mov_b32_e32 v13, s10
-; GFX9-NEXT:    v_mov_b32_e32 v5, s11
+; GFX9-NEXT:    v_cndmask_b32_e32 v2, v2, v10, vcc
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 5, v19
 ; GFX9-NEXT:    v_cndmask_b32_e32 v3, v3, v11, vcc
-; GFX9-NEXT:    v_cndmask_b32_e32 v4, v4, v12, vcc
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 5, v2
-; GFX9-NEXT:    v_mov_b32_e32 v6, s12
-; GFX9-NEXT:    v_mov_b32_e32 v7, s13
+; GFX9-NEXT:    v_cndmask_b32_e32 v2, v2, v12, vcc
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 6, v19
 ; GFX9-NEXT:    v_cndmask_b32_e32 v3, v3, v13, vcc
-; GFX9-NEXT:    v_cndmask_b32_e32 v4, v4, v5, vcc
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 6, v2
-; GFX9-NEXT:    v_mov_b32_e32 v8, s14
-; GFX9-NEXT:    v_mov_b32_e32 v9, s15
-; GFX9-NEXT:    v_cndmask_b32_e32 v3, v3, v6, vcc
-; GFX9-NEXT:    v_cndmask_b32_e32 v4, v4, v7, vcc
-; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 7, v2
-; GFX9-NEXT:    v_cndmask_b32_e32 v2, v3, v8, vcc
-; GFX9-NEXT:    v_cndmask_b32_e32 v3, v4, v9, vcc
+; GFX9-NEXT:    v_cndmask_b32_e32 v4, v2, v14, vcc
+; GFX9-NEXT:    v_cmp_eq_u32_e32 vcc, 7, v19
+; GFX9-NEXT:    v_cndmask_b32_e32 v2, v3, v15, vcc
+; GFX9-NEXT:    v_cndmask_b32_e32 v3, v4, v16, vcc
 ; GFX9-NEXT:    v_readfirstlane_b32 s0, v0
 ; GFX9-NEXT:    v_readfirstlane_b32 s1, v1
 ; GFX9-NEXT:    v_readfirstlane_b32 s2, v2
 ; GFX9-NEXT:    v_readfirstlane_b32 s3, v3
 ; GFX9-NEXT:    ; return to shader part epilog
 ;
 ; GFX8-LABEL: extractelement_sgpr_v4i128_vgpr_idx:
 ; GFX8:       ; %bb.0:
 ; GFX8-NEXT:    s_load_dwordx16 s[0:15], s[2:3], 0x0
-; GFX8-NEXT:    v_lshlrev_b32_e32 v2, 1, v0
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v2
+; GFX8-NEXT:    v_lshlrev_b32_e32 v0, 1, v0
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v0
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX8-NEXT:    v_mov_b32_e32 v3, s0
-; GFX8-NEXT:    v_mov_b32_e32 v1, s2
-; GFX8-NEXT:    v_mov_b32_e32 v0, s1
+; GFX8-NEXT:    v_mov_b32_e32 v1, s0
+; GFX8-NEXT:    v_mov_b32_e32 v3, s2
+; GFX8-NEXT:    v_mov_b32_e32 v2, s1
 ; GFX8-NEXT:    v_mov_b32_e32 v4, s3
 ; GFX8-NEXT:    v_mov_b32_e32 v5, s4
 ; GFX8-NEXT:    v_mov_b32_e32 v6, s5
-; GFX8-NEXT:    v_cndmask_b32_e32 v1, v3, v1, vcc
-; GFX8-NEXT:    v_cndmask_b32_e32 v0, v0, v4, vcc
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 2, v2
-; GFX8-NEXT:    v_cndmask_b32_e32 v1, v1, v5, vcc
-; GFX8-NEXT:    v_cndmask_b32_e32 v0, v0, v6, vcc
+; GFX8-NEXT:    v_cndmask_b32_e32 v17, v1, v3, vcc
+; GFX8-NEXT:    v_cndmask_b32_e32 v18, v2, v4, vcc
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 2, v0
 ; GFX8-NEXT:    v_mov_b32_e32 v7, s6
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 3, v2
 ; GFX8-NEXT:    v_mov_b32_e32 v8, s7
-; GFX8-NEXT:    v_cndmask_b32_e32 v1, v1, v7, vcc
-; GFX8-NEXT:    v_cndmask_b32_e32 v0, v0, v8, vcc
+; GFX8-NEXT:    v_cndmask_b32_e32 v17, v17, v5, vcc
+; GFX8-NEXT:    v_cndmask_b32_e32 v18, v18, v6, vcc
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 3, v0
 ; GFX8-NEXT:    v_mov_b32_e32 v9, s8
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 4, v2
 ; GFX8-NEXT:    v_mov_b32_e32 v10, s9
-; GFX8-NEXT:    v_cndmask_b32_e32 v1, v1, v9, vcc
-; GFX8-NEXT:    v_cndmask_b32_e32 v0, v0, v10, vcc
+; GFX8-NEXT:    v_cndmask_b32_e32 v17, v17, v7, vcc
+; GFX8-NEXT:    v_cndmask_b32_e32 v18, v18, v8, vcc
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 4, v0
 ; GFX8-NEXT:    v_mov_b32_e32 v11, s10
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 5, v2
 ; GFX8-NEXT:    v_mov_b32_e32 v12, s11
-; GFX8-NEXT:    v_cndmask_b32_e32 v1, v1, v11, vcc
-; GFX8-NEXT:    v_cndmask_b32_e32 v0, v0, v12, vcc
+; GFX8-NEXT:    v_cndmask_b32_e32 v17, v17, v9, vcc
+; GFX8-NEXT:    v_cndmask_b32_e32 v18, v18, v10, vcc
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 5, v0
 ; GFX8-NEXT:    v_mov_b32_e32 v13, s12
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 6, v2
 ; GFX8-NEXT:    v_mov_b32_e32 v14, s13
-; GFX8-NEXT:    v_cndmask_b32_e32 v1, v1, v13, vcc
-; GFX8-NEXT:    v_cndmask_b32_e32 v4, v0, v14, vcc
+; GFX8-NEXT:    v_cndmask_b32_e32 v17, v17, v11, vcc
+; GFX8-NEXT:    v_cndmask_b32_e32 v18, v18, v12, vcc
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 6, v0
+; GFX8-NEXT:    v_cndmask_b32_e32 v17, v17, v13, vcc
+; GFX8-NEXT:    v_cndmask_b32_e32 v18, v18, v14, vcc
+; GFX8-NEXT:    v_add_u32_e32 v19, vcc, 1, v0
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v19
+; GFX8-NEXT:    v_cndmask_b32_e32 v1, v1, v3, vcc
+; GFX8-NEXT:    v_cndmask_b32_e32 v2, v2, v4, vcc
+; GFX8-NEXT:    v_cmp_eq_u32_e64 s[0:1], 2, v19
 ; GFX8-NEXT:    v_mov_b32_e32 v15, s14
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 7, v2
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 7, v0
 ; GFX8-NEXT:    v_mov_b32_e32 v16, s15
-; GFX8-NEXT:    v_cndmask_b32_e32 v0, v1, v15, vcc
-; GFX8-NEXT:    v_cndmask_b32_e32 v1, v4, v16, vcc
-; GFX8-NEXT:    v_add_u32_e32 v2, vcc, 1, v2
-; GFX8-NEXT:    v_mov_b32_e32 v4, s1
-; GFX8-NEXT:    v_mov_b32_e32 v5, s2
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v2
-; GFX8-NEXT:    v_mov_b32_e32 v6, s3
-; GFX8-NEXT:    v_cndmask_b32_e32 v3, v3, v5, vcc
-; GFX8-NEXT:    v_cndmask_b32_e32 v4, v4, v6, vcc
-; GFX8-NEXT:    v_mov_b32_e32 v7, s4
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 2, v2
-; GFX8-NEXT:    v_mov_b32_e32 v8, s5
-; GFX8-NEXT:    v_cndmask_b32_e32 v3, v3, v7, vcc
-; GFX8-NEXT:    v_cndmask_b32_e32 v4, v4, v8, vcc
-; GFX8-NEXT:    v_mov_b32_e32 v9, s6
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 3, v2
-; GFX8-NEXT:    v_mov_b32_e32 v10, s7
+; GFX8-NEXT:    v_cndmask_b32_e64 v1, v1, v5, s[0:1]
+; GFX8-NEXT:    v_cndmask_b32_e64 v2, v2, v6, s[0:1]
+; GFX8-NEXT:    v_cmp_eq_u32_e64 s[0:1], 3, v19
+; GFX8-NEXT:    v_cndmask_b32_e64 v3, v1, v7, s[0:1]
+; GFX8-NEXT:    v_cndmask_b32_e32 v0, v17, v15, vcc
+; GFX8-NEXT:    v_cndmask_b32_e32 v1, v18, v16, vcc
+; GFX8-NEXT:    v_cndmask_b32_e64 v2, v2, v8, s[0:1]
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 4, v19
 ; GFX8-NEXT:    v_cndmask_b32_e32 v3, v3, v9, vcc
-; GFX8-NEXT:    v_cndmask_b32_e32 v4, v4, v10, vcc
-; GFX8-NEXT:    v_mov_b32_e32 v11, s8
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 4, v2
-; GFX8-NEXT:    v_mov_b32_e32 v12, s9
-; GFX8-NEXT:    v_mov_b32_e32 v13, s10
-; GFX8-NEXT:    v_mov_b32_e32 v5, s11
+; GFX8-NEXT:    v_cndmask_b32_e32 v2, v2, v10, vcc
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 5, v19
 ; GFX8-NEXT:    v_cndmask_b32_e32 v3, v3, v11, vcc
-; GFX8-NEXT:    v_cndmask_b32_e32 v4, v4, v12, vcc
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 5, v2
-; GFX8-NEXT:    v_mov_b32_e32 v6, s12
-; GFX8-NEXT:    v_mov_b32_e32 v7, s13
+; GFX8-NEXT:    v_cndmask_b32_e32 v2, v2, v12, vcc
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 6, v19
 ; GFX8-NEXT:    v_cndmask_b32_e32 v3, v3, v13, vcc
-; GFX8-NEXT:    v_cndmask_b32_e32 v4, v4, v5, vcc
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 6, v2
-; GFX8-NEXT:    v_mov_b32_e32 v8, s14
-; GFX8-NEXT:    v_mov_b32_e32 v9, s15
-; GFX8-NEXT:    v_cndmask_b32_e32 v3, v3, v6, vcc
-; GFX8-NEXT:    v_cndmask_b32_e32 v4, v4, v7, vcc
-; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 7, v2
-; GFX8-NEXT:    v_cndmask_b32_e32 v2, v3, v8, vcc
-; GFX8-NEXT:    v_cndmask_b32_e32 v3, v4, v9, vcc
+; GFX8-NEXT:    v_cndmask_b32_e32 v4, v2, v14, vcc
+; GFX8-NEXT:    v_cmp_eq_u32_e32 vcc, 7, v19
+; GFX8-NEXT:    v_cndmask_b32_e32 v2, v3, v15, vcc
+; GFX8-NEXT:    v_cndmask_b32_e32 v3, v4, v16, vcc
 ; GFX8-NEXT:    v_readfirstlane_b32 s0, v0
 ; GFX8-NEXT:    v_readfirstlane_b32 s1, v1
 ; GFX8-NEXT:    v_readfirstlane_b32 s2, v2
 ; GFX8-NEXT:    v_readfirstlane_b32 s3, v3
 ; GFX8-NEXT:    ; return to shader part epilog
 ;
 ; GFX7-LABEL: extractelement_sgpr_v4i128_vgpr_idx:
 ; GFX7:       ; %bb.0:
 ; GFX7-NEXT:    s_load_dwordx16 s[0:15], s[2:3], 0x0
-; GFX7-NEXT:    v_lshlrev_b32_e32 v2, 1, v0
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v2
+; GFX7-NEXT:    v_lshlrev_b32_e32 v0, 1, v0
+; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v0
 ; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX7-NEXT:    v_mov_b32_e32 v3, s0
-; GFX7-NEXT:    v_mov_b32_e32 v1, s2
-; GFX7-NEXT:    v_mov_b32_e32 v0, s1
+; GFX7-NEXT:    v_mov_b32_e32 v1, s0
+; GFX7-NEXT:    v_mov_b32_e32 v3, s2
+; GFX7-NEXT:    v_mov_b32_e32 v2, s1
 ; GFX7-NEXT:    v_mov_b32_e32 v4, s3
 ; GFX7-NEXT:    v_mov_b32_e32 v5, s4
 ; GFX7-NEXT:    v_mov_b32_e32 v6, s5
-; GFX7-NEXT:    v_cndmask_b32_e32 v1, v3, v1, vcc
-; GFX7-NEXT:    v_cndmask_b32_e32 v0, v0, v4, vcc
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 2, v2
-; GFX7-NEXT:    v_cndmask_b32_e32 v1, v1, v5, vcc
-; GFX7-NEXT:    v_cndmask_b32_e32 v0, v0, v6, vcc
+; GFX7-NEXT:    v_cndmask_b32_e32 v17, v1, v3, vcc
+; GFX7-NEXT:    v_cndmask_b32_e32 v18, v2, v4, vcc
+; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 2, v0
 ; GFX7-NEXT:    v_mov_b32_e32 v7, s6
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 3, v2
 ; GFX7-NEXT:    v_mov_b32_e32 v8, s7
-; GFX7-NEXT:    v_cndmask_b32_e32 v1, v1, v7, vcc
-; GFX7-NEXT:    v_cndmask_b32_e32 v0, v0, v8, vcc
+; GFX7-NEXT:    v_cndmask_b32_e32 v17, v17, v5, vcc
+; GFX7-NEXT:    v_cndmask_b32_e32 v18, v18, v6, vcc
+; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 3, v0
 ; GFX7-NEXT:    v_mov_b32_e32 v9, s8
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 4, v2
 ; GFX7-NEXT:    v_mov_b32_e32 v10, s9
-; GFX7-NEXT:    v_cndmask_b32_e32 v1, v1, v9, vcc
-; GFX7-NEXT:    v_cndmask_b32_e32 v0, v0, v10, vcc
+; GFX7-NEXT:    v_cndmask_b32_e32 v17, v17, v7, vcc
+; GFX7-NEXT:    v_cndmask_b32_e32 v18, v18, v8, vcc
+; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 4, v0
 ; GFX7-NEXT:    v_mov_b32_e32 v11, s10
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 5, v2
 ; GFX7-NEXT:    v_mov_b32_e32 v12, s11
-; GFX7-NEXT:    v_cndmask_b32_e32 v1, v1, v11, vcc
-; GFX7-NEXT:    v_cndmask_b32_e32 v0, v0, v12, vcc
+; GFX7-NEXT:    v_cndmask_b32_e32 v17, v17, v9, vcc
+; GFX7-NEXT:    v_cndmask_b32_e32 v18, v18, v10, vcc
+; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 5, v0
 ; GFX7-NEXT:    v_mov_b32_e32 v13, s12
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 6, v2
 ; GFX7-NEXT:    v_mov_b32_e32 v14, s13
-; GFX7-NEXT:    v_cndmask_b32_e32 v1, v1, v13, vcc
-; GFX7-NEXT:    v_cndmask_b32_e32 v4, v0, v14, vcc
+; GFX7-NEXT:    v_cndmask_b32_e32 v17, v17, v11, vcc
+; GFX7-NEXT:    v_cndmask_b32_e32 v18, v18, v12, vcc
+; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 6, v0
+; GFX7-NEXT:    v_cndmask_b32_e32 v17, v17, v13, vcc
+; GFX7-NEXT:    v_cndmask_b32_e32 v18, v18, v14, vcc
+; GFX7-NEXT:    v_add_i32_e32 v19, vcc, 1, v0
+; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v19
+; GFX7-NEXT:    v_cndmask_b32_e32 v1, v1, v3, vcc
+; GFX7-NEXT:    v_cndmask_b32_e32 v2, v2, v4, vcc
+; GFX7-NEXT:    v_cmp_eq_u32_e64 s[0:1], 2, v19
 ; GFX7-NEXT:    v_mov_b32_e32 v15, s14
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 7, v2
+; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 7, v0
 ; GFX7-NEXT:    v_mov_b32_e32 v16, s15
-; GFX7-NEXT:    v_cndmask_b32_e32 v0, v1, v15, vcc
-; GFX7-NEXT:    v_cndmask_b32_e32 v1, v4, v16, vcc
-; GFX7-NEXT:    v_add_i32_e32 v2, vcc, 1, v2
-; GFX7-NEXT:    v_mov_b32_e32 v4, s1
-; GFX7-NEXT:    v_mov_b32_e32 v5, s2
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 1, v2
-; GFX7-NEXT:    v_mov_b32_e32 v6, s3
-; GFX7-NEXT:    v_cndmask_b32_e32 v3, v3, v5, vcc
-; GFX7-NEXT:    v_cndmask_b32_e32 v4, v4, v6, vcc
-; GFX7-NEXT:    v_mov_b32_e32 v7, s4
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 2, v2
-; GFX7-NEXT:    v_mov_b32_e32 v8, s5
-; GFX7-NEXT:    v_cndmask_b32_e32 v3, v3, v7, vcc
-; GFX7-NEXT:    v_cndmask_b32_e32 v4, v4, v8, vcc
-; GFX7-NEXT:    v_mov_b32_e32 v9, s6
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 3, v2
-; GFX7-NEXT:    v_mov_b32_e32 v10, s7
+; GFX7-NEXT:    v_cndmask_b32_e64 v1, v1, v5, s[0:1]
+; GFX7-NEXT:    v_cndmask_b32_e64 v2, v2, v6, s[0:1]
+; GFX7-NEXT:    v_cmp_eq_u32_e64 s[0:1], 3, v19
+; GFX7-NEXT:    v_cndmask_b32_e64 v3, v1, v7, s[0:1]
+; GFX7-NEXT:    v_cndmask_b32_e32 v0, v17, v15, vcc
+; GFX7-NEXT:    v_cndmask_b32_e32 v1, v18, v16, vcc
+; GFX7-NEXT:    v_cndmask_b32_e64 v2, v2, v8, s[0:1]
+; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 4, v19
 ; GFX7-NEXT:    v_cndmask_b32_e32 v3, v3, v9, vcc
-; GFX7-NEXT:    v_cndmask_b32_e32 v4, v4, v10, vcc
-; GFX7-NEXT:    v_mov_b32_e32 v11, s8
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 4, v2
-; GFX7-NEXT:    v_mov_b32_e32 v12, s9
-; GFX7-NEXT:    v_mov_b32_e32 v13, s10
-; GFX7-NEXT:    v_mov_b32_e32 v5, s11
+; GFX7-NEXT:    v_cndmask_b32_e32 v2, v2, v10, vcc
+; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 5, v19
 ; GFX7-NEXT:    v_cndmask_b32_e32 v3, v3, v11, vcc
-; GFX7-NEXT:    v_cndmask_b32_e32 v4, v4, v12, vcc
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 5, v2
-; GFX7-NEXT:    v_mov_b32_e32 v6, s12
-; GFX7-NEXT:    v_mov_b32_e32 v7, s13
+; GFX7-NEXT:    v_cndmask_b32_e32 v2, v2, v12, vcc
+; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 6, v19
 ; GFX7-NEXT:    v_cndmask_b32_e32 v3, v3, v13, vcc
-; GFX7-NEXT:    v_cndmask_b32_e32 v4, v4, v5, vcc
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 6, v2
-; GFX7-NEXT:    v_mov_b32_e32 v8, s14
-; GFX7-NEXT:    v_mov_b32_e32 v9, s15
-; GFX7-NEXT:    v_cndmask_b32_e32 v3, v3, v6, vcc
-; GFX7-NEXT:    v_cndmask_b32_e32 v4, v4, v7, vcc
-; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 7, v2
-; GFX7-NEXT:    v_cndmask_b32_e32 v2, v3, v8, vcc
-; GFX7-NEXT:    v_cndmask_b32_e32 v3, v4, v9, vcc
+; GFX7-NEXT:    v_cndmask_b32_e32 v4, v2, v14, vcc
+; GFX7-NEXT:    v_cmp_eq_u32_e32 vcc, 7, v19
+; GFX7-NEXT:    v_cndmask_b32_e32 v2, v3, v15, vcc
+; GFX7-NEXT:    v_cndmask_b32_e32 v3, v4, v16, vcc
 ; GFX7-NEXT:    v_readfirstlane_b32 s0, v0
 ; GFX7-NEXT:    v_readfirstlane_b32 s1, v1
 ; GFX7-NEXT:    v_readfirstlane_b32 s2, v2
 ; GFX7-NEXT:    v_readfirstlane_b32 s3, v3
 ; GFX7-NEXT:    ; return to shader part epilog
   %vector = load <4 x i128>, <4 x i128> addrspace(4)* %ptr
   %element = extractelement <4 x i128> %vector, i32 %idx
   ret i128 %element

llvm/test/CodeGen/AMDGPU/GlobalISel/sdiv.i64.ll

 ; CHECK-NEXT:    s_cmp_lg_u32 s0, 0
 ; CHECK-NEXT:    v_mul_f32_e32 v0, 0x5f7ffffc, v0
 ; CHECK-NEXT:    v_mul_f32_e32 v1, 0x2f800000, v0
 ; CHECK-NEXT:    v_trunc_f32_e32 v1, v1
 ; CHECK-NEXT:    v_mac_f32_e32 v0, 0xcf800000, v1
 ; CHECK-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; CHECK-NEXT:    v_cvt_u32_f32_e32 v1, v1
 ; CHECK-NEXT:    s_subb_u32 s5, 0, s11
+; CHECK-NEXT:    v_mov_b32_e32 v6, s11
 ; CHECK-NEXT:    v_mul_lo_u32 v2, s5, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v3, s3, v1
 ; CHECK-NEXT:    v_mul_hi_u32 v5, s3, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v4, s3, v0
 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v5
 ; CHECK-NEXT:    v_mul_lo_u32 v3, v1, v4
 ; CHECK-NEXT:    v_mul_lo_u32 v5, v0, v2
-; CHECK-NEXT:    v_mul_hi_u32 v6, v0, v4
+; CHECK-NEXT:    v_mul_hi_u32 v7, v0, v4
 ; CHECK-NEXT:    v_mul_hi_u32 v4, v1, v4
 ; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v3, v5
 ; CHECK-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
-; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v3, v6
+; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v3, v7
 ; CHECK-NEXT:    v_cndmask_b32_e64 v3, 0, 1, vcc
-; CHECK-NEXT:    v_mul_lo_u32 v6, v1, v2
+; CHECK-NEXT:    v_mul_lo_u32 v7, v1, v2
 ; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v5, v3
 ; CHECK-NEXT:    v_mul_hi_u32 v5, v0, v2
 ; CHECK-NEXT:    v_mul_hi_u32 v2, v1, v2
-; CHECK-NEXT:    v_add_i32_e32 v4, vcc, v6, v4
-; CHECK-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
+; CHECK-NEXT:    v_add_i32_e32 v4, vcc, v7, v4
+; CHECK-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v4, vcc, v4, v5
 ; CHECK-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
-; CHECK-NEXT:    v_add_i32_e32 v5, vcc, v6, v5
+; CHECK-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
 ; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v4, v3
 ; CHECK-NEXT:    v_cndmask_b32_e64 v4, 0, 1, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
 ; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v3
 ; CHECK-NEXT:    v_addc_u32_e64 v3, s[0:1], v1, v2, vcc
 ; CHECK-NEXT:    v_mul_lo_u32 v4, s5, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v5, s3, v3
-; CHECK-NEXT:    v_mul_hi_u32 v7, s3, v0
-; CHECK-NEXT:    v_mul_lo_u32 v6, s3, v0
+; CHECK-NEXT:    v_mul_hi_u32 v8, s3, v0
+; CHECK-NEXT:    v_mul_lo_u32 v7, s3, v0
 ; CHECK-NEXT:    v_add_i32_e64 v1, s[0:1], v1, v2
 ; CHECK-NEXT:    v_add_i32_e64 v4, s[0:1], v4, v5
-; CHECK-NEXT:    v_add_i32_e64 v4, s[0:1], v4, v7
-; CHECK-NEXT:    v_mul_lo_u32 v5, v3, v6
-; CHECK-NEXT:    v_mul_lo_u32 v7, v0, v4
-; CHECK-NEXT:    v_mul_hi_u32 v2, v0, v6
-; CHECK-NEXT:    v_mul_hi_u32 v6, v3, v6
-; CHECK-NEXT:    v_add_i32_e64 v5, s[0:1], v5, v7
-; CHECK-NEXT:    v_cndmask_b32_e64 v7, 0, 1, s[0:1]
+; CHECK-NEXT:    v_add_i32_e64 v4, s[0:1], v4, v8
+; CHECK-NEXT:    v_mul_lo_u32 v5, v3, v7
+; CHECK-NEXT:    v_mul_lo_u32 v8, v0, v4
+; CHECK-NEXT:    v_mul_hi_u32 v2, v0, v7
+; CHECK-NEXT:    v_mul_hi_u32 v7, v3, v7
+; CHECK-NEXT:    v_add_i32_e64 v5, s[0:1], v5, v8
+; CHECK-NEXT:    v_cndmask_b32_e64 v8, 0, 1, s[0:1]
 ; CHECK-NEXT:    v_add_i32_e64 v2, s[0:1], v5, v2
 ; CHECK-NEXT:    v_cndmask_b32_e64 v2, 0, 1, s[0:1]
 ; CHECK-NEXT:    v_mul_lo_u32 v5, v3, v4
-; CHECK-NEXT:    v_add_i32_e64 v2, s[0:1], v7, v2
-; CHECK-NEXT:    v_mul_hi_u32 v7, v0, v4
+; CHECK-NEXT:    v_add_i32_e64 v2, s[0:1], v8, v2
+; CHECK-NEXT:    v_mul_hi_u32 v8, v0, v4
 ; CHECK-NEXT:    v_mul_hi_u32 v3, v3, v4
-; CHECK-NEXT:    v_add_i32_e64 v5, s[0:1], v5, v6
-; CHECK-NEXT:    v_cndmask_b32_e64 v6, 0, 1, s[0:1]
 ; CHECK-NEXT:    v_add_i32_e64 v5, s[0:1], v5, v7
 ; CHECK-NEXT:    v_cndmask_b32_e64 v7, 0, 1, s[0:1]
-; CHECK-NEXT:    v_add_i32_e64 v6, s[0:1], v6, v7
+; CHECK-NEXT:    v_add_i32_e64 v5, s[0:1], v5, v8
+; CHECK-NEXT:    v_cndmask_b32_e64 v8, 0, 1, s[0:1]
+; CHECK-NEXT:    v_add_i32_e64 v7, s[0:1], v7, v8
 ; CHECK-NEXT:    v_add_i32_e64 v2, s[0:1], v5, v2
 ; CHECK-NEXT:    v_cndmask_b32_e64 v5, 0, 1, s[0:1]
-; CHECK-NEXT:    v_add_i32_e64 v4, s[0:1], v6, v5
+; CHECK-NEXT:    v_add_i32_e64 v4, s[0:1], v7, v5
 ; CHECK-NEXT:    v_add_i32_e64 v3, s[0:1], v3, v4
 ; CHECK-NEXT:    v_addc_u32_e32 v1, vcc, v1, v3, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; CHECK-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
 ; CHECK-NEXT:    v_mul_lo_u32 v2, s13, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v3, s12, v1
-; CHECK-NEXT:    v_mul_hi_u32 v4, s12, v0
+; CHECK-NEXT:    v_mul_hi_u32 v5, s12, v0
 ; CHECK-NEXT:    v_mul_hi_u32 v0, s13, v0
+; CHECK-NEXT:    v_mov_b32_e32 v4, s13
 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; CHECK-NEXT:    v_cndmask_b32_e64 v3, 0, 1, vcc
-; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
+; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v5
 ; CHECK-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
-; CHECK-NEXT:    v_mul_lo_u32 v4, s13, v1
+; CHECK-NEXT:    v_mul_lo_u32 v5, s13, v1
 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
 ; CHECK-NEXT:    v_mul_hi_u32 v3, s12, v1
 ; CHECK-NEXT:    v_mul_hi_u32 v1, s13, v1
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v4, v0
-; CHECK-NEXT:    v_cndmask_b32_e64 v4, 0, 1, vcc
+; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v5, v0
+; CHECK-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v3
 ; CHECK-NEXT:    v_cndmask_b32_e64 v3, 0, 1, vcc
-; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v4, v3
+; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v5, v3
 ; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; CHECK-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
 ; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v1, v2
 ; CHECK-NEXT:    v_mul_lo_u32 v2, s11, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v1, s10, v1
-; CHECK-NEXT:    v_mul_hi_u32 v4, s10, v0
+; CHECK-NEXT:    v_mul_hi_u32 v5, s10, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v3, s10, v0
 ; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v2, v1
-; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v1, v4
-; CHECK-NEXT:    v_mov_b32_e32 v2, s13
-; CHECK-NEXT:    v_sub_i32_e32 v3, vcc, s12, v3
-; CHECK-NEXT:    v_subb_u32_e64 v2, s[0:1], v2, v1, vcc
+; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v1, v5
+; CHECK-NEXT:    v_sub_i32_e32 v2, vcc, s12, v3
+; CHECK-NEXT:    v_subb_u32_e64 v3, s[0:1], v4, v1, vcc
 ; CHECK-NEXT:    v_sub_i32_e64 v1, s[0:1], s13, v1
-; CHECK-NEXT:    v_mov_b32_e32 v4, s11
-; CHECK-NEXT:    v_cmp_le_u32_e64 s[0:1], s11, v2
-; CHECK-NEXT:    v_subb_u32_e32 v1, vcc, v1, v4, vcc
+; CHECK-NEXT:    v_cmp_le_u32_e64 s[0:1], s11, v3
+; CHECK-NEXT:    v_cndmask_b32_e64 v4, 0, -1, s[0:1]
+; CHECK-NEXT:    v_cmp_le_u32_e64 s[0:1], s10, v2
+; CHECK-NEXT:    v_subb_u32_e32 v1, vcc, v1, v6, vcc
+; CHECK-NEXT:    v_subrev_i32_e32 v2, vcc, s10, v2
 ; CHECK-NEXT:    v_cndmask_b32_e64 v5, 0, -1, s[0:1]
-; CHECK-NEXT:    v_cmp_le_u32_e64 s[0:1], s10, v3
-; CHECK-NEXT:    v_subrev_i32_e32 v3, vcc, s10, v3
+; CHECK-NEXT:    v_cmp_eq_u32_e64 s[0:1], s11, v3
+; CHECK-NEXT:    v_cndmask_b32_e64 v3, v4, v5, s[0:1]
 ; CHECK-NEXT:    v_subbrev_u32_e32 v1, vcc, 0, v1, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v4, vcc, 1, v0
-; CHECK-NEXT:    v_cndmask_b32_e64 v6, 0, -1, s[0:1]
-; CHECK-NEXT:    v_cmp_eq_u32_e64 s[0:1], s11, v2
 ; CHECK-NEXT:    v_cmp_le_u32_e32 vcc, s11, v1
-; CHECK-NEXT:    v_cndmask_b32_e64 v2, v5, v6, s[0:1]
 ; CHECK-NEXT:    v_cndmask_b32_e64 v5, 0, -1, vcc
-; CHECK-NEXT:    v_cmp_le_u32_e32 vcc, s10, v3
-; CHECK-NEXT:    v_cndmask_b32_e64 v3, 0, -1, vcc
+; CHECK-NEXT:    v_cmp_le_u32_e32 vcc, s10, v2
+; CHECK-NEXT:    v_cndmask_b32_e64 v2, 0, -1, vcc
 ; CHECK-NEXT:    v_cmp_eq_u32_e32 vcc, s11, v1
-; CHECK-NEXT:    v_cndmask_b32_e32 v1, v5, v3, vcc
-; CHECK-NEXT:    v_add_i32_e32 v3, vcc, 1, v4
+; CHECK-NEXT:    v_cndmask_b32_e32 v1, v5, v2, vcc
+; CHECK-NEXT:    v_add_i32_e32 v2, vcc, 1, v4
 ; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v1
-; CHECK-NEXT:    v_cndmask_b32_e32 v1, v4, v3, vcc
-; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v2
+; CHECK-NEXT:    v_cndmask_b32_e32 v1, v4, v2, vcc
+; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v3
 ; CHECK-NEXT:    s_xor_b64 s[0:1], s[6:7], s[8:9]
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc
 ; CHECK-NEXT:    v_xor_b32_e32 v0, s0, v0
 ; CHECK-NEXT:    v_subrev_i32_e32 v0, vcc, s0, v0
 ; CHECK-NEXT:    s_mov_b32 s1, 0
 ; CHECK-NEXT:    s_branch BB1_3
 ; CHECK-NEXT:  BB1_2:
 ; CHECK-NEXT:    ; implicit-def: $vgpr0_vgpr1

 ; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, 1, s[4:5]
 ; GISEL-NEXT:    v_add_i32_e64 v9, s[4:5], v11, v10
 ; GISEL-NEXT:    v_add_i32_e64 v8, s[4:5], v8, v9
 ; GISEL-NEXT:    v_addc_u32_e32 v5, vcc, v5, v8, vcc
 ; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v7
 ; GISEL-NEXT:    v_addc_u32_e32 v5, vcc, 0, v5, vcc
 ; GISEL-NEXT:    v_mul_lo_u32 v7, v1, v4
 ; GISEL-NEXT:    v_mul_lo_u32 v8, v0, v5
-; GISEL-NEXT:    v_mul_hi_u32 v9, v0, v4
+; GISEL-NEXT:    v_mul_hi_u32 v10, v0, v4
 ; GISEL-NEXT:    v_mul_hi_u32 v4, v1, v4
+; GISEL-NEXT:    v_mov_b32_e32 v9, s9
 ; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v8
 ; GISEL-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v9
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v10
 ; GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
-; GISEL-NEXT:    v_mul_lo_u32 v9, v1, v5
+; GISEL-NEXT:    v_mul_lo_u32 v10, v1, v5
 ; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v8, v7
 ; GISEL-NEXT:    v_mul_hi_u32 v8, v0, v5
 ; GISEL-NEXT:    v_mul_hi_u32 v5, v1, v5
-; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v9, v4
-; GISEL-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
+; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v10, v4
+; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
 ; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v8
 ; GISEL-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v9, v8
+; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v10, v8
 ; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v7
 ; GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
 ; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v8, v7
 ; GISEL-NEXT:    v_add_i32_e32 v5, vcc, v5, v7
 ; GISEL-NEXT:    v_mul_lo_u32 v7, s9, v4
 ; GISEL-NEXT:    v_mul_lo_u32 v8, s8, v5
-; GISEL-NEXT:    v_mul_hi_u32 v10, s8, v4
-; GISEL-NEXT:    v_mul_lo_u32 v9, s8, v4
+; GISEL-NEXT:    v_mul_hi_u32 v11, s8, v4
+; GISEL-NEXT:    v_mul_lo_u32 v10, s8, v4
 ; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v8
-; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v10
-; GISEL-NEXT:    v_sub_i32_e32 v0, vcc, v0, v9
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v11
+; GISEL-NEXT:    v_sub_i32_e32 v0, vcc, v0, v10
 ; GISEL-NEXT:    v_subb_u32_e64 v8, s[4:5], v1, v7, vcc
 ; GISEL-NEXT:    v_sub_i32_e64 v1, s[4:5], v1, v7
-; GISEL-NEXT:    v_mov_b32_e32 v7, s9
 ; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s9, v8
-; GISEL-NEXT:    v_subb_u32_e32 v1, vcc, v1, v7, vcc
-; GISEL-NEXT:    v_cndmask_b32_e64 v9, 0, -1, s[4:5]
+; GISEL-NEXT:    v_subb_u32_e32 v1, vcc, v1, v9, vcc
+; GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, -1, s[4:5]
 ; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s8, v0
 ; GISEL-NEXT:    v_subrev_i32_e32 v0, vcc, s8, v0
 ; GISEL-NEXT:    v_subbrev_u32_e32 v1, vcc, 0, v1, vcc
 ; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, -1, s[4:5]
 ; GISEL-NEXT:    v_cmp_eq_u32_e64 s[4:5], s9, v8
-; GISEL-NEXT:    v_add_i32_e32 v7, vcc, 1, v4
-; GISEL-NEXT:    v_cndmask_b32_e64 v8, v9, v10, s[4:5]
+; GISEL-NEXT:    v_add_i32_e32 v8, vcc, 1, v4
 ; GISEL-NEXT:    v_addc_u32_e32 v9, vcc, 0, v5, vcc
 ; GISEL-NEXT:    v_cmp_le_u32_e32 vcc, s9, v1
+; GISEL-NEXT:    v_cndmask_b32_e64 v7, v7, v10, s[4:5]
 ; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, -1, vcc
 ; GISEL-NEXT:    v_cmp_le_u32_e32 vcc, s8, v0
 ; GISEL-NEXT:    s_add_u32 s4, s10, 0
 ; GISEL-NEXT:    s_cselect_b32 s5, 1, 0
 ; GISEL-NEXT:    v_cndmask_b32_e64 v0, 0, -1, vcc
 ; GISEL-NEXT:    v_cmp_eq_u32_e32 vcc, s9, v1
 ; GISEL-NEXT:    s_and_b32 s5, s5, 1
 ; GISEL-NEXT:    v_cndmask_b32_e32 v0, v10, v0, vcc
-; GISEL-NEXT:    v_add_i32_e32 v1, vcc, 1, v7
+; GISEL-NEXT:    v_add_i32_e32 v1, vcc, 1, v8
 ; GISEL-NEXT:    s_cmp_lg_u32 s5, 0
 ; GISEL-NEXT:    v_addc_u32_e32 v10, vcc, 0, v9, vcc
 ; GISEL-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v0
 ; GISEL-NEXT:    s_addc_u32 s5, 0, 0
-; GISEL-NEXT:    v_cndmask_b32_e32 v0, v7, v1, vcc
 ; GISEL-NEXT:    s_xor_b64 s[6:7], s[4:5], s[6:7]
+; GISEL-NEXT:    v_cndmask_b32_e32 v0, v8, v1, vcc
 ; GISEL-NEXT:    v_cndmask_b32_e32 v1, v9, v10, vcc
-; GISEL-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v8
+; GISEL-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v7
 ; GISEL-NEXT:    v_cndmask_b32_e32 v0, v4, v0, vcc
 ; GISEL-NEXT:    v_cndmask_b32_e32 v1, v5, v1, vcc
 ; GISEL-NEXT:    v_cvt_f32_u32_e32 v4, s6
 ; GISEL-NEXT:    v_cvt_f32_u32_e32 v5, s7
 ; GISEL-NEXT:    s_sub_u32 s8, 0, s6
 ; GISEL-NEXT:    s_cselect_b32 s4, 1, 0
 ; GISEL-NEXT:    s_and_b32 s4, s4, 1
 ; GISEL-NEXT:    v_mac_f32_e32 v4, 0x4f800000, v5
 ; GISEL-NEXT:    v_rcp_iflag_f32_e32 v4, v4
 ; GISEL-NEXT:    s_cmp_lg_u32 s4, 0
 ; GISEL-NEXT:    s_subb_u32 s9, 0, s7
-; GISEL-NEXT:    v_ashrrev_i32_e32 v7, 31, v3
+; GISEL-NEXT:    v_xor_b32_e32 v0, v0, v6
 ; GISEL-NEXT:    v_mul_f32_e32 v4, 0x5f7ffffc, v4
 ; GISEL-NEXT:    v_mul_f32_e32 v5, 0x2f800000, v4
 ; GISEL-NEXT:    v_trunc_f32_e32 v5, v5
 ; GISEL-NEXT:    v_mac_f32_e32 v4, 0xcf800000, v5
 ; GISEL-NEXT:    v_cvt_u32_f32_e32 v4, v4
 ; GISEL-NEXT:    v_cvt_u32_f32_e32 v5, v5
-; GISEL-NEXT:    v_add_i32_e32 v2, vcc, v2, v7
-; GISEL-NEXT:    v_addc_u32_e32 v3, vcc, v3, v7, vcc
-; GISEL-NEXT:    v_mul_lo_u32 v8, s9, v4
-; GISEL-NEXT:    v_mul_lo_u32 v9, s8, v5
-; GISEL-NEXT:    v_mul_hi_u32 v11, s8, v4
-; GISEL-NEXT:    v_mul_lo_u32 v10, s8, v4
-; GISEL-NEXT:    v_xor_b32_e32 v0, v0, v6
-; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v8, v9
+; GISEL-NEXT:    v_xor_b32_e32 v1, v1, v6
+; GISEL-NEXT:    v_sub_i32_e32 v0, vcc, v0, v6
+; GISEL-NEXT:    v_mul_lo_u32 v7, s9, v4
+; GISEL-NEXT:    v_mul_lo_u32 v8, s8, v5
+; GISEL-NEXT:    v_mul_hi_u32 v10, s8, v4
+; GISEL-NEXT:    v_subb_u32_e32 v1, vcc, v1, v6, vcc
+; GISEL-NEXT:    v_ashrrev_i32_e32 v6, 31, v3
+; GISEL-NEXT:    v_mul_lo_u32 v9, s8, v4
+; GISEL-NEXT:    v_add_i32_e32 v2, vcc, v2, v6
+; GISEL-NEXT:    v_addc_u32_e32 v3, vcc, v3, v6, vcc
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v8
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v10
+; GISEL-NEXT:    v_mul_lo_u32 v8, v5, v9
+; GISEL-NEXT:    v_mul_lo_u32 v10, v4, v7
+; GISEL-NEXT:    v_mul_hi_u32 v11, v4, v9
+; GISEL-NEXT:    v_mul_hi_u32 v9, v5, v9
+; GISEL-NEXT:    v_xor_b32_e32 v2, v2, v6
+; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v8, v10
+; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
 ; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v8, v11
-; GISEL-NEXT:    v_mul_lo_u32 v9, v5, v10
-; GISEL-NEXT:    v_mul_lo_u32 v11, v4, v8
-; GISEL-NEXT:    v_mul_hi_u32 v12, v4, v10
-; GISEL-NEXT:    v_mul_hi_u32 v10, v5, v10
-; GISEL-NEXT:    v_xor_b32_e32 v2, v2, v7
-; GISEL-NEXT:    v_add_i32_e32 v9, vcc, v9, v11
-; GISEL-NEXT:    v_cndmask_b32_e64 v11, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v9, vcc, v9, v12
-; GISEL-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
-; GISEL-NEXT:    v_mul_lo_u32 v12, v5, v8
+; GISEL-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
+; GISEL-NEXT:    v_mul_lo_u32 v11, v5, v7
+; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v10, v8
+; GISEL-NEXT:    v_mul_hi_u32 v10, v4, v7
+; GISEL-NEXT:    v_mul_hi_u32 v7, v5, v7
 ; GISEL-NEXT:    v_add_i32_e32 v9, vcc, v11, v9
-; GISEL-NEXT:    v_mul_hi_u32 v11, v4, v8
-; GISEL-NEXT:    v_mul_hi_u32 v8, v5, v8
-; GISEL-NEXT:    v_add_i32_e32 v10, vcc, v12, v10
-; GISEL-NEXT:    v_cndmask_b32_e64 v12, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v10, vcc, v10, v11
 ; GISEL-NEXT:    v_cndmask_b32_e64 v11, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v11, vcc, v12, v11
-; GISEL-NEXT:    v_add_i32_e32 v9, vcc, v10, v9
+; GISEL-NEXT:    v_add_i32_e32 v9, vcc, v9, v10
 ; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
 ; GISEL-NEXT:    v_add_i32_e32 v10, vcc, v11, v10
-; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v8, v10
-; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v9
-; GISEL-NEXT:    v_addc_u32_e64 v9, s[4:5], v5, v8, vcc
-; GISEL-NEXT:    v_mul_lo_u32 v10, s9, v4
-; GISEL-NEXT:    v_mul_lo_u32 v11, s8, v9
-; GISEL-NEXT:    v_mul_hi_u32 v13, s8, v4
-; GISEL-NEXT:    v_mul_lo_u32 v12, s8, v4
-; GISEL-NEXT:    v_add_i32_e64 v5, s[4:5], v5, v8
-; GISEL-NEXT:    v_add_i32_e64 v10, s[4:5], v10, v11
-; GISEL-NEXT:    v_add_i32_e64 v10, s[4:5], v10, v13
-; GISEL-NEXT:    v_mul_lo_u32 v11, v9, v12
-; GISEL-NEXT:    v_mul_lo_u32 v13, v4, v10
-; GISEL-NEXT:    v_mul_hi_u32 v8, v4, v12
-; GISEL-NEXT:    v_mul_hi_u32 v12, v9, v12
-; GISEL-NEXT:    v_xor_b32_e32 v3, v3, v7
-; GISEL-NEXT:    v_add_i32_e64 v11, s[4:5], v11, v13
-; GISEL-NEXT:    v_cndmask_b32_e64 v13, 0, 1, s[4:5]
-; GISEL-NEXT:    v_add_i32_e64 v8, s[4:5], v11, v8
-; GISEL-NEXT:    v_cndmask_b32_e64 v8, 0, 1, s[4:5]
-; GISEL-NEXT:    v_mul_lo_u32 v11, v9, v10
-; GISEL-NEXT:    v_add_i32_e64 v8, s[4:5], v13, v8
-; GISEL-NEXT:    v_mul_hi_u32 v13, v4, v10
-; GISEL-NEXT:    v_mul_hi_u32 v9, v9, v10
-; GISEL-NEXT:    v_add_i32_e64 v11, s[4:5], v11, v12
+; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v9, v8
+; GISEL-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
+; GISEL-NEXT:    v_add_i32_e32 v9, vcc, v10, v9
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v9
+; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v8
+; GISEL-NEXT:    v_addc_u32_e64 v8, s[4:5], v5, v7, vcc
+; GISEL-NEXT:    v_mul_lo_u32 v9, s9, v4
+; GISEL-NEXT:    v_mul_lo_u32 v10, s8, v8
+; GISEL-NEXT:    v_mul_hi_u32 v12, s8, v4
+; GISEL-NEXT:    v_mul_lo_u32 v11, s8, v4
+; GISEL-NEXT:    v_add_i32_e64 v5, s[4:5], v5, v7
+; GISEL-NEXT:    v_add_i32_e64 v9, s[4:5], v9, v10
+; GISEL-NEXT:    v_add_i32_e64 v9, s[4:5], v9, v12
+; GISEL-NEXT:    v_mul_lo_u32 v10, v8, v11
+; GISEL-NEXT:    v_mul_lo_u32 v12, v4, v9
+; GISEL-NEXT:    v_mul_hi_u32 v7, v4, v11
+; GISEL-NEXT:    v_mul_hi_u32 v11, v8, v11
+; GISEL-NEXT:    v_xor_b32_e32 v3, v3, v6
+; GISEL-NEXT:    v_add_i32_e64 v10, s[4:5], v10, v12
 ; GISEL-NEXT:    v_cndmask_b32_e64 v12, 0, 1, s[4:5]
-; GISEL-NEXT:    v_add_i32_e64 v11, s[4:5], v11, v13
-; GISEL-NEXT:    v_cndmask_b32_e64 v13, 0, 1, s[4:5]
-; GISEL-NEXT:    v_add_i32_e64 v12, s[4:5], v12, v13
-; GISEL-NEXT:    v_add_i32_e64 v8, s[4:5], v11, v8
+; GISEL-NEXT:    v_add_i32_e64 v7, s[4:5], v10, v7
+; GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, 1, s[4:5]
+; GISEL-NEXT:    v_mul_lo_u32 v10, v8, v9
+; GISEL-NEXT:    v_add_i32_e64 v7, s[4:5], v12, v7
+; GISEL-NEXT:    v_mul_hi_u32 v12, v4, v9
+; GISEL-NEXT:    v_mul_hi_u32 v8, v8, v9
+; GISEL-NEXT:    v_add_i32_e64 v10, s[4:5], v10, v11
 ; GISEL-NEXT:    v_cndmask_b32_e64 v11, 0, 1, s[4:5]
-; GISEL-NEXT:    v_add_i32_e64 v10, s[4:5], v12, v11
-; GISEL-NEXT:    v_add_i32_e64 v9, s[4:5], v9, v10
-; GISEL-NEXT:    v_addc_u32_e32 v5, vcc, v5, v9, vcc
-; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v8
+; GISEL-NEXT:    v_add_i32_e64 v10, s[4:5], v10, v12
+; GISEL-NEXT:    v_cndmask_b32_e64 v12, 0, 1, s[4:5]
+; GISEL-NEXT:    v_add_i32_e64 v11, s[4:5], v11, v12
+; GISEL-NEXT:    v_add_i32_e64 v7, s[4:5], v10, v7
+; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, 1, s[4:5]
+; GISEL-NEXT:    v_add_i32_e64 v9, s[4:5], v11, v10
+; GISEL-NEXT:    v_add_i32_e64 v8, s[4:5], v8, v9
+; GISEL-NEXT:    v_addc_u32_e32 v5, vcc, v5, v8, vcc
+; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v7
 ; GISEL-NEXT:    v_addc_u32_e32 v5, vcc, 0, v5, vcc
-; GISEL-NEXT:    v_xor_b32_e32 v1, v1, v6
-; GISEL-NEXT:    v_sub_i32_e32 v0, vcc, v0, v6
-; GISEL-NEXT:    v_mul_lo_u32 v8, v3, v4
-; GISEL-NEXT:    v_mul_lo_u32 v9, v2, v5
-; GISEL-NEXT:    v_subb_u32_e32 v1, vcc, v1, v6, vcc
-; GISEL-NEXT:    v_mul_hi_u32 v6, v2, v4
+; GISEL-NEXT:    v_mul_lo_u32 v7, v3, v4
+; GISEL-NEXT:    v_mul_lo_u32 v8, v2, v5
+; GISEL-NEXT:    v_mul_hi_u32 v10, v2, v4
 ; GISEL-NEXT:    v_mul_hi_u32 v4, v3, v4
-; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v8, v9
-; GISEL-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v6, vcc, v8, v6
-; GISEL-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
-; GISEL-NEXT:    v_mul_lo_u32 v8, v3, v5
-; GISEL-NEXT:    v_add_i32_e32 v6, vcc, v9, v6
-; GISEL-NEXT:    v_mul_hi_u32 v9, v2, v5
+; GISEL-NEXT:    v_mov_b32_e32 v9, s7
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v8
+; GISEL-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v10
+; GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
+; GISEL-NEXT:    v_mul_lo_u32 v10, v3, v5
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v8, v7
+; GISEL-NEXT:    v_mul_hi_u32 v8, v2, v5
 ; GISEL-NEXT:    v_mul_hi_u32 v5, v3, v5
-; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v8, v4
+; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v10, v4
+; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
+; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v8
 ; GISEL-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v9
-; GISEL-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v8, v9
-; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v6
-; GISEL-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v6, vcc, v8, v6
-; GISEL-NEXT:    v_add_i32_e32 v5, vcc, v5, v6
-; GISEL-NEXT:    v_mul_lo_u32 v6, s7, v4
+; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v10, v8
+; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v7
+; GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v8, v7
+; GISEL-NEXT:    v_add_i32_e32 v5, vcc, v5, v7
+; GISEL-NEXT:    v_mul_lo_u32 v7, s7, v4
 ; GISEL-NEXT:    v_mul_lo_u32 v8, s6, v5
-; GISEL-NEXT:    v_mul_hi_u32 v10, s6, v4
-; GISEL-NEXT:    v_mul_lo_u32 v9, s6, v4
-; GISEL-NEXT:    v_add_i32_e32 v6, vcc, v6, v8
-; GISEL-NEXT:    v_add_i32_e32 v6, vcc, v6, v10
-; GISEL-NEXT:    v_sub_i32_e32 v2, vcc, v2, v9
-; GISEL-NEXT:    v_subb_u32_e64 v8, s[4:5], v3, v6, vcc
-; GISEL-NEXT:    v_sub_i32_e64 v3, s[4:5], v3, v6
-; GISEL-NEXT:    v_mov_b32_e32 v6, s7
+; GISEL-NEXT:    v_mul_hi_u32 v11, s6, v4
+; GISEL-NEXT:    v_mul_lo_u32 v10, s6, v4
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v8
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v11
+; GISEL-NEXT:    v_sub_i32_e32 v2, vcc, v2, v10
+; GISEL-NEXT:    v_subb_u32_e64 v8, s[4:5], v3, v7, vcc
+; GISEL-NEXT:    v_sub_i32_e64 v3, s[4:5], v3, v7
 ; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s7, v8
-; GISEL-NEXT:    v_subb_u32_e32 v3, vcc, v3, v6, vcc
-; GISEL-NEXT:    v_cndmask_b32_e64 v9, 0, -1, s[4:5]
+; GISEL-NEXT:    v_subb_u32_e32 v3, vcc, v3, v9, vcc
+; GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, -1, s[4:5]
 ; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s6, v2
 ; GISEL-NEXT:    v_subrev_i32_e32 v2, vcc, s6, v2
 ; GISEL-NEXT:    v_subbrev_u32_e32 v3, vcc, 0, v3, vcc
 ; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, -1, s[4:5]
 ; GISEL-NEXT:    v_cmp_eq_u32_e64 s[4:5], s7, v8
-; GISEL-NEXT:    v_add_i32_e32 v6, vcc, 1, v4
-; GISEL-NEXT:    v_cndmask_b32_e64 v8, v9, v10, s[4:5]
+; GISEL-NEXT:    v_add_i32_e32 v8, vcc, 1, v4
 ; GISEL-NEXT:    v_addc_u32_e32 v9, vcc, 0, v5, vcc
 ; GISEL-NEXT:    v_cmp_le_u32_e32 vcc, s7, v3
+; GISEL-NEXT:    v_cndmask_b32_e64 v7, v7, v10, s[4:5]
 ; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, -1, vcc
 ; GISEL-NEXT:    v_cmp_le_u32_e32 vcc, s6, v2
 ; GISEL-NEXT:    v_cndmask_b32_e64 v2, 0, -1, vcc
 ; GISEL-NEXT:    v_cmp_eq_u32_e32 vcc, s7, v3
 ; GISEL-NEXT:    v_cndmask_b32_e32 v2, v10, v2, vcc
-; GISEL-NEXT:    v_add_i32_e32 v3, vcc, 1, v6
+; GISEL-NEXT:    v_add_i32_e32 v3, vcc, 1, v8
 ; GISEL-NEXT:    v_addc_u32_e32 v10, vcc, 0, v9, vcc
 ; GISEL-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v2
-; GISEL-NEXT:    v_cndmask_b32_e32 v2, v6, v3, vcc
+; GISEL-NEXT:    v_cndmask_b32_e32 v2, v8, v3, vcc
 ; GISEL-NEXT:    v_cndmask_b32_e32 v3, v9, v10, vcc
-; GISEL-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v8
+; GISEL-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v7
 ; GISEL-NEXT:    v_cndmask_b32_e32 v2, v4, v2, vcc
 ; GISEL-NEXT:    v_cndmask_b32_e32 v3, v5, v3, vcc
-; GISEL-NEXT:    v_xor_b32_e32 v2, v2, v7
-; GISEL-NEXT:    v_xor_b32_e32 v3, v3, v7
-; GISEL-NEXT:    v_sub_i32_e32 v2, vcc, v2, v7
-; GISEL-NEXT:    v_subb_u32_e32 v3, vcc, v3, v7, vcc
+; GISEL-NEXT:    v_xor_b32_e32 v2, v2, v6
+; GISEL-NEXT:    v_xor_b32_e32 v3, v3, v6
+; GISEL-NEXT:    v_sub_i32_e32 v2, vcc, v2, v6
+; GISEL-NEXT:    v_subb_u32_e32 v3, vcc, v3, v6, vcc
 ; GISEL-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; CGP-LABEL: v_sdiv_v2i64_pow2k_denom:
 ; CGP:       ; %bb.0:
 ; CGP-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v4, 0x1000
 ; CGP-NEXT:    v_cvt_f32_ubyte0_e32 v6, 0
 ; CGP-NEXT:    s_movk_i32 s6, 0xf000

 ; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, 1, s[4:5]
 ; GISEL-NEXT:    v_add_i32_e64 v9, s[4:5], v11, v10
 ; GISEL-NEXT:    v_add_i32_e64 v8, s[4:5], v8, v9
 ; GISEL-NEXT:    v_addc_u32_e32 v5, vcc, v5, v8, vcc
 ; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v7
 ; GISEL-NEXT:    v_addc_u32_e32 v5, vcc, 0, v5, vcc
 ; GISEL-NEXT:    v_mul_lo_u32 v7, v1, v4
 ; GISEL-NEXT:    v_mul_lo_u32 v8, v0, v5
-; GISEL-NEXT:    v_mul_hi_u32 v9, v0, v4
+; GISEL-NEXT:    v_mul_hi_u32 v10, v0, v4
 ; GISEL-NEXT:    v_mul_hi_u32 v4, v1, v4
+; GISEL-NEXT:    v_mov_b32_e32 v9, s9
 ; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v8
 ; GISEL-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v9
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v10
 ; GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
-; GISEL-NEXT:    v_mul_lo_u32 v9, v1, v5
+; GISEL-NEXT:    v_mul_lo_u32 v10, v1, v5
 ; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v8, v7
 ; GISEL-NEXT:    v_mul_hi_u32 v8, v0, v5
 ; GISEL-NEXT:    v_mul_hi_u32 v5, v1, v5
-; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v9, v4
-; GISEL-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
+; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v10, v4
+; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
 ; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v8
 ; GISEL-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v9, v8
+; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v10, v8
 ; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v7
 ; GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
 ; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v8, v7
 ; GISEL-NEXT:    v_add_i32_e32 v5, vcc, v5, v7
 ; GISEL-NEXT:    v_mul_lo_u32 v7, s9, v4
 ; GISEL-NEXT:    v_mul_lo_u32 v8, s8, v5
-; GISEL-NEXT:    v_mul_hi_u32 v10, s8, v4
-; GISEL-NEXT:    v_mul_lo_u32 v9, s8, v4
+; GISEL-NEXT:    v_mul_hi_u32 v11, s8, v4
+; GISEL-NEXT:    v_mul_lo_u32 v10, s8, v4
 ; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v8
-; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v10
-; GISEL-NEXT:    v_sub_i32_e32 v0, vcc, v0, v9
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v11
+; GISEL-NEXT:    v_sub_i32_e32 v0, vcc, v0, v10
 ; GISEL-NEXT:    v_subb_u32_e64 v8, s[4:5], v1, v7, vcc
 ; GISEL-NEXT:    v_sub_i32_e64 v1, s[4:5], v1, v7
-; GISEL-NEXT:    v_mov_b32_e32 v7, s9
 ; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s9, v8
-; GISEL-NEXT:    v_subb_u32_e32 v1, vcc, v1, v7, vcc
-; GISEL-NEXT:    v_cndmask_b32_e64 v9, 0, -1, s[4:5]
+; GISEL-NEXT:    v_subb_u32_e32 v1, vcc, v1, v9, vcc
+; GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, -1, s[4:5]
 ; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s8, v0
 ; GISEL-NEXT:    v_subrev_i32_e32 v0, vcc, s8, v0
 ; GISEL-NEXT:    v_subbrev_u32_e32 v1, vcc, 0, v1, vcc
 ; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, -1, s[4:5]
 ; GISEL-NEXT:    v_cmp_eq_u32_e64 s[4:5], s9, v8
-; GISEL-NEXT:    v_add_i32_e32 v7, vcc, 1, v4
-; GISEL-NEXT:    v_cndmask_b32_e64 v8, v9, v10, s[4:5]
+; GISEL-NEXT:    v_add_i32_e32 v8, vcc, 1, v4
 ; GISEL-NEXT:    v_addc_u32_e32 v9, vcc, 0, v5, vcc
 ; GISEL-NEXT:    v_cmp_le_u32_e32 vcc, s9, v1
+; GISEL-NEXT:    v_cndmask_b32_e64 v7, v7, v10, s[4:5]
 ; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, -1, vcc
 ; GISEL-NEXT:    v_cmp_le_u32_e32 vcc, s8, v0
 ; GISEL-NEXT:    s_add_u32 s4, s10, 0
 ; GISEL-NEXT:    s_cselect_b32 s5, 1, 0
 ; GISEL-NEXT:    v_cndmask_b32_e64 v0, 0, -1, vcc
 ; GISEL-NEXT:    v_cmp_eq_u32_e32 vcc, s9, v1
 ; GISEL-NEXT:    s_and_b32 s5, s5, 1
 ; GISEL-NEXT:    v_cndmask_b32_e32 v0, v10, v0, vcc
-; GISEL-NEXT:    v_add_i32_e32 v1, vcc, 1, v7
+; GISEL-NEXT:    v_add_i32_e32 v1, vcc, 1, v8
 ; GISEL-NEXT:    s_cmp_lg_u32 s5, 0
 ; GISEL-NEXT:    v_addc_u32_e32 v10, vcc, 0, v9, vcc
 ; GISEL-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v0
 ; GISEL-NEXT:    s_addc_u32 s5, 0, 0
-; GISEL-NEXT:    v_cndmask_b32_e32 v0, v7, v1, vcc
 ; GISEL-NEXT:    s_xor_b64 s[6:7], s[4:5], s[6:7]
+; GISEL-NEXT:    v_cndmask_b32_e32 v0, v8, v1, vcc
 ; GISEL-NEXT:    v_cndmask_b32_e32 v1, v9, v10, vcc
-; GISEL-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v8
+; GISEL-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v7
 ; GISEL-NEXT:    v_cndmask_b32_e32 v0, v4, v0, vcc
 ; GISEL-NEXT:    v_cndmask_b32_e32 v1, v5, v1, vcc
 ; GISEL-NEXT:    v_cvt_f32_u32_e32 v4, s6
 ; GISEL-NEXT:    v_cvt_f32_u32_e32 v5, s7
 ; GISEL-NEXT:    s_sub_u32 s8, 0, s6
 ; GISEL-NEXT:    s_cselect_b32 s4, 1, 0
 ; GISEL-NEXT:    s_and_b32 s4, s4, 1
 ; GISEL-NEXT:    v_mac_f32_e32 v4, 0x4f800000, v5
 ; GISEL-NEXT:    v_rcp_iflag_f32_e32 v4, v4
 ; GISEL-NEXT:    s_cmp_lg_u32 s4, 0
 ; GISEL-NEXT:    s_subb_u32 s9, 0, s7
-; GISEL-NEXT:    v_ashrrev_i32_e32 v7, 31, v3
+; GISEL-NEXT:    v_xor_b32_e32 v0, v0, v6
 ; GISEL-NEXT:    v_mul_f32_e32 v4, 0x5f7ffffc, v4
 ; GISEL-NEXT:    v_mul_f32_e32 v5, 0x2f800000, v4
 ; GISEL-NEXT:    v_trunc_f32_e32 v5, v5
 ; GISEL-NEXT:    v_mac_f32_e32 v4, 0xcf800000, v5
 ; GISEL-NEXT:    v_cvt_u32_f32_e32 v4, v4
 ; GISEL-NEXT:    v_cvt_u32_f32_e32 v5, v5
-; GISEL-NEXT:    v_add_i32_e32 v2, vcc, v2, v7
-; GISEL-NEXT:    v_addc_u32_e32 v3, vcc, v3, v7, vcc
-; GISEL-NEXT:    v_mul_lo_u32 v8, s9, v4
-; GISEL-NEXT:    v_mul_lo_u32 v9, s8, v5
-; GISEL-NEXT:    v_mul_hi_u32 v11, s8, v4
-; GISEL-NEXT:    v_mul_lo_u32 v10, s8, v4
-; GISEL-NEXT:    v_xor_b32_e32 v0, v0, v6
-; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v8, v9
+; GISEL-NEXT:    v_xor_b32_e32 v1, v1, v6
+; GISEL-NEXT:    v_sub_i32_e32 v0, vcc, v0, v6
+; GISEL-NEXT:    v_mul_lo_u32 v7, s9, v4
+; GISEL-NEXT:    v_mul_lo_u32 v8, s8, v5
+; GISEL-NEXT:    v_mul_hi_u32 v10, s8, v4
+; GISEL-NEXT:    v_subb_u32_e32 v1, vcc, v1, v6, vcc
+; GISEL-NEXT:    v_ashrrev_i32_e32 v6, 31, v3
+; GISEL-NEXT:    v_mul_lo_u32 v9, s8, v4
+; GISEL-NEXT:    v_add_i32_e32 v2, vcc, v2, v6
+; GISEL-NEXT:    v_addc_u32_e32 v3, vcc, v3, v6, vcc
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v8
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v10
+; GISEL-NEXT:    v_mul_lo_u32 v8, v5, v9
+; GISEL-NEXT:    v_mul_lo_u32 v10, v4, v7
+; GISEL-NEXT:    v_mul_hi_u32 v11, v4, v9
+; GISEL-NEXT:    v_mul_hi_u32 v9, v5, v9
+; GISEL-NEXT:    v_xor_b32_e32 v2, v2, v6
+; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v8, v10
+; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
 ; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v8, v11
-; GISEL-NEXT:    v_mul_lo_u32 v9, v5, v10
-; GISEL-NEXT:    v_mul_lo_u32 v11, v4, v8
-; GISEL-NEXT:    v_mul_hi_u32 v12, v4, v10
-; GISEL-NEXT:    v_mul_hi_u32 v10, v5, v10
-; GISEL-NEXT:    v_xor_b32_e32 v2, v2, v7
-; GISEL-NEXT:    v_add_i32_e32 v9, vcc, v9, v11
-; GISEL-NEXT:    v_cndmask_b32_e64 v11, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v9, vcc, v9, v12
-; GISEL-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
-; GISEL-NEXT:    v_mul_lo_u32 v12, v5, v8
+; GISEL-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
+; GISEL-NEXT:    v_mul_lo_u32 v11, v5, v7
+; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v10, v8
+; GISEL-NEXT:    v_mul_hi_u32 v10, v4, v7
+; GISEL-NEXT:    v_mul_hi_u32 v7, v5, v7
 ; GISEL-NEXT:    v_add_i32_e32 v9, vcc, v11, v9
-; GISEL-NEXT:    v_mul_hi_u32 v11, v4, v8
-; GISEL-NEXT:    v_mul_hi_u32 v8, v5, v8
-; GISEL-NEXT:    v_add_i32_e32 v10, vcc, v12, v10
-; GISEL-NEXT:    v_cndmask_b32_e64 v12, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v10, vcc, v10, v11
 ; GISEL-NEXT:    v_cndmask_b32_e64 v11, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v11, vcc, v12, v11
-; GISEL-NEXT:    v_add_i32_e32 v9, vcc, v10, v9
+; GISEL-NEXT:    v_add_i32_e32 v9, vcc, v9, v10
 ; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
 ; GISEL-NEXT:    v_add_i32_e32 v10, vcc, v11, v10
-; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v8, v10
-; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v9
-; GISEL-NEXT:    v_addc_u32_e64 v9, s[4:5], v5, v8, vcc
-; GISEL-NEXT:    v_mul_lo_u32 v10, s9, v4
-; GISEL-NEXT:    v_mul_lo_u32 v11, s8, v9
-; GISEL-NEXT:    v_mul_hi_u32 v13, s8, v4
-; GISEL-NEXT:    v_mul_lo_u32 v12, s8, v4
-; GISEL-NEXT:    v_add_i32_e64 v5, s[4:5], v5, v8
-; GISEL-NEXT:    v_add_i32_e64 v10, s[4:5], v10, v11
-; GISEL-NEXT:    v_add_i32_e64 v10, s[4:5], v10, v13
-; GISEL-NEXT:    v_mul_lo_u32 v11, v9, v12
-; GISEL-NEXT:    v_mul_lo_u32 v13, v4, v10
-; GISEL-NEXT:    v_mul_hi_u32 v8, v4, v12
-; GISEL-NEXT:    v_mul_hi_u32 v12, v9, v12
-; GISEL-NEXT:    v_xor_b32_e32 v3, v3, v7
-; GISEL-NEXT:    v_add_i32_e64 v11, s[4:5], v11, v13
-; GISEL-NEXT:    v_cndmask_b32_e64 v13, 0, 1, s[4:5]
-; GISEL-NEXT:    v_add_i32_e64 v8, s[4:5], v11, v8
-; GISEL-NEXT:    v_cndmask_b32_e64 v8, 0, 1, s[4:5]
-; GISEL-NEXT:    v_mul_lo_u32 v11, v9, v10
-; GISEL-NEXT:    v_add_i32_e64 v8, s[4:5], v13, v8
-; GISEL-NEXT:    v_mul_hi_u32 v13, v4, v10
-; GISEL-NEXT:    v_mul_hi_u32 v9, v9, v10
-; GISEL-NEXT:    v_add_i32_e64 v11, s[4:5], v11, v12
+; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v9, v8
+; GISEL-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
+; GISEL-NEXT:    v_add_i32_e32 v9, vcc, v10, v9
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v9
+; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v8
+; GISEL-NEXT:    v_addc_u32_e64 v8, s[4:5], v5, v7, vcc
+; GISEL-NEXT:    v_mul_lo_u32 v9, s9, v4
+; GISEL-NEXT:    v_mul_lo_u32 v10, s8, v8
+; GISEL-NEXT:    v_mul_hi_u32 v12, s8, v4
+; GISEL-NEXT:    v_mul_lo_u32 v11, s8, v4
+; GISEL-NEXT:    v_add_i32_e64 v5, s[4:5], v5, v7
+; GISEL-NEXT:    v_add_i32_e64 v9, s[4:5], v9, v10
+; GISEL-NEXT:    v_add_i32_e64 v9, s[4:5], v9, v12
+; GISEL-NEXT:    v_mul_lo_u32 v10, v8, v11
+; GISEL-NEXT:    v_mul_lo_u32 v12, v4, v9
+; GISEL-NEXT:    v_mul_hi_u32 v7, v4, v11
+; GISEL-NEXT:    v_mul_hi_u32 v11, v8, v11
+; GISEL-NEXT:    v_xor_b32_e32 v3, v3, v6
+; GISEL-NEXT:    v_add_i32_e64 v10, s[4:5], v10, v12
 ; GISEL-NEXT:    v_cndmask_b32_e64 v12, 0, 1, s[4:5]
-; GISEL-NEXT:    v_add_i32_e64 v11, s[4:5], v11, v13
-; GISEL-NEXT:    v_cndmask_b32_e64 v13, 0, 1, s[4:5]
-; GISEL-NEXT:    v_add_i32_e64 v12, s[4:5], v12, v13
-; GISEL-NEXT:    v_add_i32_e64 v8, s[4:5], v11, v8
+; GISEL-NEXT:    v_add_i32_e64 v7, s[4:5], v10, v7
+; GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, 1, s[4:5]
+; GISEL-NEXT:    v_mul_lo_u32 v10, v8, v9
+; GISEL-NEXT:    v_add_i32_e64 v7, s[4:5], v12, v7
+; GISEL-NEXT:    v_mul_hi_u32 v12, v4, v9
+; GISEL-NEXT:    v_mul_hi_u32 v8, v8, v9
+; GISEL-NEXT:    v_add_i32_e64 v10, s[4:5], v10, v11
 ; GISEL-NEXT:    v_cndmask_b32_e64 v11, 0, 1, s[4:5]
-; GISEL-NEXT:    v_add_i32_e64 v10, s[4:5], v12, v11
-; GISEL-NEXT:    v_add_i32_e64 v9, s[4:5], v9, v10
-; GISEL-NEXT:    v_addc_u32_e32 v5, vcc, v5, v9, vcc
-; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v8
+; GISEL-NEXT:    v_add_i32_e64 v10, s[4:5], v10, v12
+; GISEL-NEXT:    v_cndmask_b32_e64 v12, 0, 1, s[4:5]
+; GISEL-NEXT:    v_add_i32_e64 v11, s[4:5], v11, v12
+; GISEL-NEXT:    v_add_i32_e64 v7, s[4:5], v10, v7
+; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, 1, s[4:5]
+; GISEL-NEXT:    v_add_i32_e64 v9, s[4:5], v11, v10
+; GISEL-NEXT:    v_add_i32_e64 v8, s[4:5], v8, v9
+; GISEL-NEXT:    v_addc_u32_e32 v5, vcc, v5, v8, vcc
+; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v7
 ; GISEL-NEXT:    v_addc_u32_e32 v5, vcc, 0, v5, vcc
-; GISEL-NEXT:    v_xor_b32_e32 v1, v1, v6
-; GISEL-NEXT:    v_sub_i32_e32 v0, vcc, v0, v6
-; GISEL-NEXT:    v_mul_lo_u32 v8, v3, v4
-; GISEL-NEXT:    v_mul_lo_u32 v9, v2, v5
-; GISEL-NEXT:    v_subb_u32_e32 v1, vcc, v1, v6, vcc
-; GISEL-NEXT:    v_mul_hi_u32 v6, v2, v4
+; GISEL-NEXT:    v_mul_lo_u32 v7, v3, v4
+; GISEL-NEXT:    v_mul_lo_u32 v8, v2, v5
+; GISEL-NEXT:    v_mul_hi_u32 v10, v2, v4
 ; GISEL-NEXT:    v_mul_hi_u32 v4, v3, v4
-; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v8, v9
-; GISEL-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v6, vcc, v8, v6
-; GISEL-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
-; GISEL-NEXT:    v_mul_lo_u32 v8, v3, v5
-; GISEL-NEXT:    v_add_i32_e32 v6, vcc, v9, v6
-; GISEL-NEXT:    v_mul_hi_u32 v9, v2, v5
+; GISEL-NEXT:    v_mov_b32_e32 v9, s7
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v8
+; GISEL-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v10
+; GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
+; GISEL-NEXT:    v_mul_lo_u32 v10, v3, v5
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v8, v7
+; GISEL-NEXT:    v_mul_hi_u32 v8, v2, v5
 ; GISEL-NEXT:    v_mul_hi_u32 v5, v3, v5
-; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v8, v4
+; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v10, v4
+; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
+; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v8
 ; GISEL-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v9
-; GISEL-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v8, v9
-; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v6
-; GISEL-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v6, vcc, v8, v6
-; GISEL-NEXT:    v_add_i32_e32 v5, vcc, v5, v6
-; GISEL-NEXT:    v_mul_lo_u32 v6, s7, v4
+; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v10, v8
+; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v7
+; GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v8, v7
+; GISEL-NEXT:    v_add_i32_e32 v5, vcc, v5, v7
+; GISEL-NEXT:    v_mul_lo_u32 v7, s7, v4
 ; GISEL-NEXT:    v_mul_lo_u32 v8, s6, v5
-; GISEL-NEXT:    v_mul_hi_u32 v10, s6, v4
-; GISEL-NEXT:    v_mul_lo_u32 v9, s6, v4
-; GISEL-NEXT:    v_add_i32_e32 v6, vcc, v6, v8
-; GISEL-NEXT:    v_add_i32_e32 v6, vcc, v6, v10
-; GISEL-NEXT:    v_sub_i32_e32 v2, vcc, v2, v9
-; GISEL-NEXT:    v_subb_u32_e64 v8, s[4:5], v3, v6, vcc
-; GISEL-NEXT:    v_sub_i32_e64 v3, s[4:5], v3, v6
-; GISEL-NEXT:    v_mov_b32_e32 v6, s7
+; GISEL-NEXT:    v_mul_hi_u32 v11, s6, v4
+; GISEL-NEXT:    v_mul_lo_u32 v10, s6, v4
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v8
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v11
+; GISEL-NEXT:    v_sub_i32_e32 v2, vcc, v2, v10
+; GISEL-NEXT:    v_subb_u32_e64 v8, s[4:5], v3, v7, vcc
+; GISEL-NEXT:    v_sub_i32_e64 v3, s[4:5], v3, v7
 ; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s7, v8
-; GISEL-NEXT:    v_subb_u32_e32 v3, vcc, v3, v6, vcc
-; GISEL-NEXT:    v_cndmask_b32_e64 v9, 0, -1, s[4:5]
+; GISEL-NEXT:    v_subb_u32_e32 v3, vcc, v3, v9, vcc
+; GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, -1, s[4:5]
 ; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s6, v2
 ; GISEL-NEXT:    v_subrev_i32_e32 v2, vcc, s6, v2
 ; GISEL-NEXT:    v_subbrev_u32_e32 v3, vcc, 0, v3, vcc
 ; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, -1, s[4:5]
 ; GISEL-NEXT:    v_cmp_eq_u32_e64 s[4:5], s7, v8
-; GISEL-NEXT:    v_add_i32_e32 v6, vcc, 1, v4
-; GISEL-NEXT:    v_cndmask_b32_e64 v8, v9, v10, s[4:5]
+; GISEL-NEXT:    v_add_i32_e32 v8, vcc, 1, v4
 ; GISEL-NEXT:    v_addc_u32_e32 v9, vcc, 0, v5, vcc
 ; GISEL-NEXT:    v_cmp_le_u32_e32 vcc, s7, v3
+; GISEL-NEXT:    v_cndmask_b32_e64 v7, v7, v10, s[4:5]
 ; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, -1, vcc
 ; GISEL-NEXT:    v_cmp_le_u32_e32 vcc, s6, v2
 ; GISEL-NEXT:    v_cndmask_b32_e64 v2, 0, -1, vcc
 ; GISEL-NEXT:    v_cmp_eq_u32_e32 vcc, s7, v3
 ; GISEL-NEXT:    v_cndmask_b32_e32 v2, v10, v2, vcc
-; GISEL-NEXT:    v_add_i32_e32 v3, vcc, 1, v6
+; GISEL-NEXT:    v_add_i32_e32 v3, vcc, 1, v8
 ; GISEL-NEXT:    v_addc_u32_e32 v10, vcc, 0, v9, vcc
 ; GISEL-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v2
-; GISEL-NEXT:    v_cndmask_b32_e32 v2, v6, v3, vcc
+; GISEL-NEXT:    v_cndmask_b32_e32 v2, v8, v3, vcc
 ; GISEL-NEXT:    v_cndmask_b32_e32 v3, v9, v10, vcc
-; GISEL-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v8
+; GISEL-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v7
 ; GISEL-NEXT:    v_cndmask_b32_e32 v2, v4, v2, vcc
 ; GISEL-NEXT:    v_cndmask_b32_e32 v3, v5, v3, vcc
-; GISEL-NEXT:    v_xor_b32_e32 v2, v2, v7
-; GISEL-NEXT:    v_xor_b32_e32 v3, v3, v7
-; GISEL-NEXT:    v_sub_i32_e32 v2, vcc, v2, v7
-; GISEL-NEXT:    v_subb_u32_e32 v3, vcc, v3, v7, vcc
+; GISEL-NEXT:    v_xor_b32_e32 v2, v2, v6
+; GISEL-NEXT:    v_xor_b32_e32 v3, v3, v6
+; GISEL-NEXT:    v_sub_i32_e32 v2, vcc, v2, v6
+; GISEL-NEXT:    v_subb_u32_e32 v3, vcc, v3, v6, vcc
 ; GISEL-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; CGP-LABEL: v_sdiv_v2i64_oddk_denom:
 ; CGP:       ; %bb.0:
 ; CGP-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v4, 0x12d8fb
 ; CGP-NEXT:    v_cvt_f32_ubyte0_e32 v6, 0
 ; CGP-NEXT:    s_mov_b32 s6, 0xffed2705

llvm/test/CodeGen/AMDGPU/GlobalISel/srem.i64.ll

 ; CHECK-NEXT:    s_cmp_lg_u32 s0, 0
 ; CHECK-NEXT:    v_mul_f32_e32 v0, 0x5f7ffffc, v0
 ; CHECK-NEXT:    v_mul_f32_e32 v1, 0x2f800000, v0
 ; CHECK-NEXT:    v_trunc_f32_e32 v1, v1
 ; CHECK-NEXT:    v_mac_f32_e32 v0, 0xcf800000, v1
 ; CHECK-NEXT:    v_cvt_u32_f32_e32 v0, v0
 ; CHECK-NEXT:    v_cvt_u32_f32_e32 v1, v1
 ; CHECK-NEXT:    s_subb_u32 s5, 0, s11
+; CHECK-NEXT:    v_mov_b32_e32 v6, s11
 ; CHECK-NEXT:    v_mul_lo_u32 v2, s5, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v3, s3, v1
 ; CHECK-NEXT:    v_mul_hi_u32 v5, s3, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v4, s3, v0
 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v5
 ; CHECK-NEXT:    v_mul_lo_u32 v3, v1, v4
 ; CHECK-NEXT:    v_mul_lo_u32 v5, v0, v2
-; CHECK-NEXT:    v_mul_hi_u32 v6, v0, v4
+; CHECK-NEXT:    v_mul_hi_u32 v7, v0, v4
 ; CHECK-NEXT:    v_mul_hi_u32 v4, v1, v4
 ; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v3, v5
 ; CHECK-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
-; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v3, v6
+; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v3, v7
 ; CHECK-NEXT:    v_cndmask_b32_e64 v3, 0, 1, vcc
-; CHECK-NEXT:    v_mul_lo_u32 v6, v1, v2
+; CHECK-NEXT:    v_mul_lo_u32 v7, v1, v2
 ; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v5, v3
 ; CHECK-NEXT:    v_mul_hi_u32 v5, v0, v2
 ; CHECK-NEXT:    v_mul_hi_u32 v2, v1, v2
-; CHECK-NEXT:    v_add_i32_e32 v4, vcc, v6, v4
-; CHECK-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
+; CHECK-NEXT:    v_add_i32_e32 v4, vcc, v7, v4
+; CHECK-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v4, vcc, v4, v5
 ; CHECK-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
-; CHECK-NEXT:    v_add_i32_e32 v5, vcc, v6, v5
+; CHECK-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
 ; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v4, v3
 ; CHECK-NEXT:    v_cndmask_b32_e64 v4, 0, 1, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
 ; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v3
 ; CHECK-NEXT:    v_addc_u32_e64 v3, s[0:1], v1, v2, vcc
 ; CHECK-NEXT:    v_mul_lo_u32 v4, s5, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v5, s3, v3
-; CHECK-NEXT:    v_mul_hi_u32 v7, s3, v0
-; CHECK-NEXT:    v_mul_lo_u32 v6, s3, v0
+; CHECK-NEXT:    v_mul_hi_u32 v8, s3, v0
+; CHECK-NEXT:    v_mul_lo_u32 v7, s3, v0
 ; CHECK-NEXT:    v_add_i32_e64 v1, s[0:1], v1, v2
 ; CHECK-NEXT:    v_add_i32_e64 v4, s[0:1], v4, v5
-; CHECK-NEXT:    v_add_i32_e64 v4, s[0:1], v4, v7
-; CHECK-NEXT:    v_mul_lo_u32 v5, v3, v6
-; CHECK-NEXT:    v_mul_lo_u32 v7, v0, v4
-; CHECK-NEXT:    v_mul_hi_u32 v2, v0, v6
-; CHECK-NEXT:    v_mul_hi_u32 v6, v3, v6
-; CHECK-NEXT:    v_add_i32_e64 v5, s[0:1], v5, v7
-; CHECK-NEXT:    v_cndmask_b32_e64 v7, 0, 1, s[0:1]
+; CHECK-NEXT:    v_add_i32_e64 v4, s[0:1], v4, v8
+; CHECK-NEXT:    v_mul_lo_u32 v5, v3, v7
+; CHECK-NEXT:    v_mul_lo_u32 v8, v0, v4
+; CHECK-NEXT:    v_mul_hi_u32 v2, v0, v7
+; CHECK-NEXT:    v_mul_hi_u32 v7, v3, v7
+; CHECK-NEXT:    v_add_i32_e64 v5, s[0:1], v5, v8
+; CHECK-NEXT:    v_cndmask_b32_e64 v8, 0, 1, s[0:1]
 ; CHECK-NEXT:    v_add_i32_e64 v2, s[0:1], v5, v2
 ; CHECK-NEXT:    v_cndmask_b32_e64 v2, 0, 1, s[0:1]
 ; CHECK-NEXT:    v_mul_lo_u32 v5, v3, v4
-; CHECK-NEXT:    v_add_i32_e64 v2, s[0:1], v7, v2
-; CHECK-NEXT:    v_mul_hi_u32 v7, v0, v4
+; CHECK-NEXT:    v_add_i32_e64 v2, s[0:1], v8, v2
+; CHECK-NEXT:    v_mul_hi_u32 v8, v0, v4
 ; CHECK-NEXT:    v_mul_hi_u32 v3, v3, v4
-; CHECK-NEXT:    v_add_i32_e64 v5, s[0:1], v5, v6
-; CHECK-NEXT:    v_cndmask_b32_e64 v6, 0, 1, s[0:1]
 ; CHECK-NEXT:    v_add_i32_e64 v5, s[0:1], v5, v7
 ; CHECK-NEXT:    v_cndmask_b32_e64 v7, 0, 1, s[0:1]
-; CHECK-NEXT:    v_add_i32_e64 v6, s[0:1], v6, v7
+; CHECK-NEXT:    v_add_i32_e64 v5, s[0:1], v5, v8
+; CHECK-NEXT:    v_cndmask_b32_e64 v8, 0, 1, s[0:1]
+; CHECK-NEXT:    v_add_i32_e64 v7, s[0:1], v7, v8
 ; CHECK-NEXT:    v_add_i32_e64 v2, s[0:1], v5, v2
 ; CHECK-NEXT:    v_cndmask_b32_e64 v5, 0, 1, s[0:1]
-; CHECK-NEXT:    v_add_i32_e64 v4, s[0:1], v6, v5
+; CHECK-NEXT:    v_add_i32_e64 v4, s[0:1], v7, v5
 ; CHECK-NEXT:    v_add_i32_e64 v3, s[0:1], v3, v4
 ; CHECK-NEXT:    v_addc_u32_e32 v1, vcc, v1, v3, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; CHECK-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
 ; CHECK-NEXT:    v_mul_lo_u32 v2, s9, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v3, s8, v1
-; CHECK-NEXT:    v_mul_hi_u32 v4, s8, v0
+; CHECK-NEXT:    v_mul_hi_u32 v5, s8, v0
 ; CHECK-NEXT:    v_mul_hi_u32 v0, s9, v0
+; CHECK-NEXT:    v_mov_b32_e32 v4, s9
 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v3
 ; CHECK-NEXT:    v_cndmask_b32_e64 v3, 0, 1, vcc
-; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v4
+; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v2, v5
 ; CHECK-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
-; CHECK-NEXT:    v_mul_lo_u32 v4, s9, v1
+; CHECK-NEXT:    v_mul_lo_u32 v5, s9, v1
 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
 ; CHECK-NEXT:    v_mul_hi_u32 v3, s8, v1
 ; CHECK-NEXT:    v_mul_hi_u32 v1, s9, v1
-; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v4, v0
-; CHECK-NEXT:    v_cndmask_b32_e64 v4, 0, 1, vcc
+; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v5, v0
+; CHECK-NEXT:    v_cndmask_b32_e64 v5, 0, 1, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v3
 ; CHECK-NEXT:    v_cndmask_b32_e64 v3, 0, 1, vcc
-; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v4, v3
+; CHECK-NEXT:    v_add_i32_e32 v3, vcc, v5, v3
 ; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; CHECK-NEXT:    v_cndmask_b32_e64 v2, 0, 1, vcc
 ; CHECK-NEXT:    v_add_i32_e32 v2, vcc, v3, v2
 ; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v1, v2
 ; CHECK-NEXT:    v_mul_lo_u32 v2, s11, v0
 ; CHECK-NEXT:    v_mul_lo_u32 v1, s10, v1
 ; CHECK-NEXT:    v_mul_lo_u32 v3, s10, v0
 ; CHECK-NEXT:    v_mul_hi_u32 v0, s10, v0
 ; CHECK-NEXT:    v_add_i32_e32 v1, vcc, v2, v1
 ; CHECK-NEXT:    v_add_i32_e32 v0, vcc, v1, v0
-; CHECK-NEXT:    v_sub_i32_e32 v2, vcc, s8, v3
-; CHECK-NEXT:    v_mov_b32_e32 v1, s9
-; CHECK-NEXT:    v_subb_u32_e64 v1, s[0:1], v1, v0, vcc
+; CHECK-NEXT:    v_sub_i32_e32 v1, vcc, s8, v3
+; CHECK-NEXT:    v_subb_u32_e64 v2, s[0:1], v4, v0, vcc
 ; CHECK-NEXT:    v_sub_i32_e64 v0, s[0:1], s9, v0
-; CHECK-NEXT:    v_mov_b32_e32 v3, s11
-; CHECK-NEXT:    v_subb_u32_e32 v0, vcc, v0, v3, vcc
-; CHECK-NEXT:    v_subrev_i32_e32 v3, vcc, s10, v2
-; CHECK-NEXT:    v_cmp_le_u32_e64 s[0:1], s11, v1
-; CHECK-NEXT:    v_subbrev_u32_e32 v0, vcc, 0, v0, vcc
+; CHECK-NEXT:    v_cmp_le_u32_e64 s[0:1], s11, v2
+; CHECK-NEXT:    v_cndmask_b32_e64 v3, 0, -1, s[0:1]
+; CHECK-NEXT:    v_cmp_le_u32_e64 s[0:1], s10, v1
 ; CHECK-NEXT:    v_cndmask_b32_e64 v4, 0, -1, s[0:1]
-; CHECK-NEXT:    v_cmp_le_u32_e64 s[0:1], s10, v2
-; CHECK-NEXT:    v_cndmask_b32_e64 v5, 0, -1, s[0:1]
-; CHECK-NEXT:    v_cmp_eq_u32_e64 s[0:1], s11, v1
+; CHECK-NEXT:    v_cmp_eq_u32_e64 s[0:1], s11, v2
+; CHECK-NEXT:    v_subb_u32_e32 v0, vcc, v0, v6, vcc
+; CHECK-NEXT:    v_cndmask_b32_e64 v2, v3, v4, s[0:1]
+; CHECK-NEXT:    v_subrev_i32_e32 v3, vcc, s10, v1
+; CHECK-NEXT:    v_subbrev_u32_e32 v0, vcc, 0, v0, vcc
 ; CHECK-NEXT:    v_cmp_le_u32_e32 vcc, s11, v0
-; CHECK-NEXT:    v_cndmask_b32_e64 v1, v4, v5, s[0:1]
 ; CHECK-NEXT:    v_cndmask_b32_e64 v4, 0, -1, vcc
 ; CHECK-NEXT:    v_cmp_le_u32_e32 vcc, s10, v3
 ; CHECK-NEXT:    v_cndmask_b32_e64 v5, 0, -1, vcc
 ; CHECK-NEXT:    v_cmp_eq_u32_e32 vcc, s11, v0
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v4, v5, vcc
 ; CHECK-NEXT:    v_subrev_i32_e32 v4, vcc, s10, v3
 ; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v0
 ; CHECK-NEXT:    v_cndmask_b32_e32 v0, v3, v4, vcc
-; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v1
-; CHECK-NEXT:    v_cndmask_b32_e32 v0, v2, v0, vcc
+; CHECK-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v2
+; CHECK-NEXT:    v_cndmask_b32_e32 v0, v1, v0, vcc
 ; CHECK-NEXT:    v_xor_b32_e32 v0, s6, v0
 ; CHECK-NEXT:    v_subrev_i32_e32 v0, vcc, s6, v0
 ; CHECK-NEXT:    s_mov_b32 s1, 0
 ; CHECK-NEXT:    s_branch BB1_3
 ; CHECK-NEXT:  BB1_2:
 ; CHECK-NEXT:    ; implicit-def: $vgpr0_vgpr1
 ; CHECK-NEXT:  BB1_3: ; %Flow
 ; CHECK-NEXT:    s_xor_b32 s0, s1, -1

 ; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, 1, s[4:5]
 ; GISEL-NEXT:    v_add_i32_e64 v9, s[4:5], v11, v10
 ; GISEL-NEXT:    v_add_i32_e64 v8, s[4:5], v8, v9
 ; GISEL-NEXT:    v_addc_u32_e32 v5, vcc, v5, v8, vcc
 ; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v7
 ; GISEL-NEXT:    v_addc_u32_e32 v5, vcc, 0, v5, vcc
 ; GISEL-NEXT:    v_mul_lo_u32 v7, v1, v4
 ; GISEL-NEXT:    v_mul_lo_u32 v8, v0, v5
-; GISEL-NEXT:    v_mul_hi_u32 v9, v0, v4
+; GISEL-NEXT:    v_mul_hi_u32 v10, v0, v4
 ; GISEL-NEXT:    v_mul_hi_u32 v4, v1, v4
+; GISEL-NEXT:    v_mov_b32_e32 v9, s9
 ; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v8
 ; GISEL-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v9
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v10
 ; GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
-; GISEL-NEXT:    v_mul_lo_u32 v9, v1, v5
+; GISEL-NEXT:    v_mul_lo_u32 v10, v1, v5
 ; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v8, v7
 ; GISEL-NEXT:    v_mul_hi_u32 v8, v0, v5
 ; GISEL-NEXT:    v_mul_hi_u32 v5, v1, v5
-; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v9, v4
-; GISEL-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
+; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v10, v4
+; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
 ; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v8
 ; GISEL-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v9, v8
+; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v10, v8
 ; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v7
 ; GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
 ; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v8, v7
 ; GISEL-NEXT:    v_add_i32_e32 v5, vcc, v5, v7
 ; GISEL-NEXT:    v_mul_lo_u32 v7, s9, v4
 ; GISEL-NEXT:    v_mul_lo_u32 v5, s8, v5
 ; GISEL-NEXT:    v_mul_lo_u32 v8, s8, v4
 ; GISEL-NEXT:    v_mul_hi_u32 v4, s8, v4
 ; GISEL-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
 ; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
 ; GISEL-NEXT:    v_sub_i32_e32 v0, vcc, v0, v8
 ; GISEL-NEXT:    v_subb_u32_e64 v5, s[4:5], v1, v4, vcc
 ; GISEL-NEXT:    v_sub_i32_e64 v1, s[4:5], v1, v4
 ; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s9, v5
-; GISEL-NEXT:    v_mov_b32_e32 v4, s9
-; GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, -1, s[4:5]
+; GISEL-NEXT:    v_cndmask_b32_e64 v4, 0, -1, s[4:5]
 ; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s8, v0
-; GISEL-NEXT:    v_cndmask_b32_e64 v8, 0, -1, s[4:5]
+; GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, -1, s[4:5]
 ; GISEL-NEXT:    v_cmp_eq_u32_e64 s[4:5], s9, v5
-; GISEL-NEXT:    v_subb_u32_e32 v1, vcc, v1, v4, vcc
-; GISEL-NEXT:    v_cndmask_b32_e64 v7, v7, v8, s[4:5]
-; GISEL-NEXT:    v_subrev_i32_e32 v8, vcc, s8, v0
-; GISEL-NEXT:    v_subbrev_u32_e64 v9, s[4:5], 0, v1, vcc
-; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s9, v9
+; GISEL-NEXT:    v_subb_u32_e32 v1, vcc, v1, v9, vcc
+; GISEL-NEXT:    v_cndmask_b32_e64 v4, v4, v7, s[4:5]
+; GISEL-NEXT:    v_subrev_i32_e32 v7, vcc, s8, v0
+; GISEL-NEXT:    v_subbrev_u32_e64 v8, s[4:5], 0, v1, vcc
+; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s9, v8
 ; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, -1, s[4:5]
-; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s8, v8
+; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s8, v7
 ; GISEL-NEXT:    v_cndmask_b32_e64 v11, 0, -1, s[4:5]
-; GISEL-NEXT:    v_cmp_eq_u32_e64 s[4:5], s9, v9
+; GISEL-NEXT:    v_cmp_eq_u32_e64 s[4:5], s9, v8
 ; GISEL-NEXT:    v_cndmask_b32_e64 v10, v10, v11, s[4:5]
 ; GISEL-NEXT:    s_add_u32 s4, s10, 0
-; GISEL-NEXT:    v_subb_u32_e32 v1, vcc, v1, v4, vcc
+; GISEL-NEXT:    v_subb_u32_e32 v1, vcc, v1, v9, vcc
 ; GISEL-NEXT:    s_cselect_b32 s5, 1, 0
-; GISEL-NEXT:    v_subrev_i32_e32 v4, vcc, s8, v8
+; GISEL-NEXT:    v_subrev_i32_e32 v9, vcc, s8, v7
 ; GISEL-NEXT:    s_and_b32 s5, s5, 1
 ; GISEL-NEXT:    s_cmp_lg_u32 s5, 0
 ; GISEL-NEXT:    v_subbrev_u32_e32 v1, vcc, 0, v1, vcc
 ; GISEL-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v10
 ; GISEL-NEXT:    s_addc_u32 s5, 0, 0
 ; GISEL-NEXT:    s_xor_b64 s[6:7], s[4:5], s[6:7]
-; GISEL-NEXT:    v_cndmask_b32_e32 v4, v8, v4, vcc
-; GISEL-NEXT:    v_cndmask_b32_e32 v1, v9, v1, vcc
-; GISEL-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v7
-; GISEL-NEXT:    v_cndmask_b32_e32 v0, v0, v4, vcc
+; GISEL-NEXT:    v_cndmask_b32_e32 v7, v7, v9, vcc
+; GISEL-NEXT:    v_cndmask_b32_e32 v1, v8, v1, vcc
+; GISEL-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v4
 ; GISEL-NEXT:    v_cndmask_b32_e32 v1, v5, v1, vcc
 ; GISEL-NEXT:    v_cvt_f32_u32_e32 v4, s6
 ; GISEL-NEXT:    v_cvt_f32_u32_e32 v5, s7
 ; GISEL-NEXT:    s_sub_u32 s8, 0, s6
 ; GISEL-NEXT:    s_cselect_b32 s4, 1, 0
 ; GISEL-NEXT:    s_and_b32 s4, s4, 1
 ; GISEL-NEXT:    v_mac_f32_e32 v4, 0x4f800000, v5
 ; GISEL-NEXT:    v_rcp_iflag_f32_e32 v4, v4
 ; GISEL-NEXT:    s_cmp_lg_u32 s4, 0
+; GISEL-NEXT:    v_cndmask_b32_e32 v0, v0, v7, vcc
 ; GISEL-NEXT:    s_subb_u32 s9, 0, s7
-; GISEL-NEXT:    v_ashrrev_i32_e32 v7, 31, v3
 ; GISEL-NEXT:    v_mul_f32_e32 v4, 0x5f7ffffc, v4
 ; GISEL-NEXT:    v_mul_f32_e32 v5, 0x2f800000, v4
 ; GISEL-NEXT:    v_trunc_f32_e32 v5, v5
 ; GISEL-NEXT:    v_mac_f32_e32 v4, 0xcf800000, v5
 ; GISEL-NEXT:    v_cvt_u32_f32_e32 v4, v4
 ; GISEL-NEXT:    v_cvt_u32_f32_e32 v5, v5
-; GISEL-NEXT:    v_add_i32_e32 v2, vcc, v2, v7
-; GISEL-NEXT:    v_addc_u32_e32 v3, vcc, v3, v7, vcc
-; GISEL-NEXT:    v_mul_lo_u32 v8, s9, v4
-; GISEL-NEXT:    v_mul_lo_u32 v9, s8, v5
-; GISEL-NEXT:    v_mul_hi_u32 v11, s8, v4
-; GISEL-NEXT:    v_mul_lo_u32 v10, s8, v4
 ; GISEL-NEXT:    v_xor_b32_e32 v0, v0, v6
-; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v8, v9
+; GISEL-NEXT:    v_xor_b32_e32 v1, v1, v6
+; GISEL-NEXT:    v_mul_lo_u32 v7, s9, v4
+; GISEL-NEXT:    v_mul_lo_u32 v8, s8, v5
+; GISEL-NEXT:    v_sub_i32_e32 v0, vcc, v0, v6
+; GISEL-NEXT:    v_mul_hi_u32 v10, s8, v4
+; GISEL-NEXT:    v_subb_u32_e32 v1, vcc, v1, v6, vcc
+; GISEL-NEXT:    v_ashrrev_i32_e32 v6, 31, v3
+; GISEL-NEXT:    v_mul_lo_u32 v9, s8, v4
+; GISEL-NEXT:    v_add_i32_e32 v2, vcc, v2, v6
+; GISEL-NEXT:    v_addc_u32_e32 v3, vcc, v3, v6, vcc
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v8
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v10
+; GISEL-NEXT:    v_mul_lo_u32 v8, v5, v9
+; GISEL-NEXT:    v_mul_lo_u32 v10, v4, v7
+; GISEL-NEXT:    v_mul_hi_u32 v11, v4, v9
+; GISEL-NEXT:    v_mul_hi_u32 v9, v5, v9
+; GISEL-NEXT:    v_xor_b32_e32 v2, v2, v6
+; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v8, v10
+; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
 ; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v8, v11
-; GISEL-NEXT:    v_mul_lo_u32 v9, v5, v10
-; GISEL-NEXT:    v_mul_lo_u32 v11, v4, v8
-; GISEL-NEXT:    v_mul_hi_u32 v12, v4, v10
-; GISEL-NEXT:    v_mul_hi_u32 v10, v5, v10
-; GISEL-NEXT:    v_xor_b32_e32 v2, v2, v7
-; GISEL-NEXT:    v_add_i32_e32 v9, vcc, v9, v11
-; GISEL-NEXT:    v_cndmask_b32_e64 v11, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v9, vcc, v9, v12
-; GISEL-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
-; GISEL-NEXT:    v_mul_lo_u32 v12, v5, v8
+; GISEL-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
+; GISEL-NEXT:    v_mul_lo_u32 v11, v5, v7
+; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v10, v8
+; GISEL-NEXT:    v_mul_hi_u32 v10, v4, v7
+; GISEL-NEXT:    v_mul_hi_u32 v7, v5, v7
 ; GISEL-NEXT:    v_add_i32_e32 v9, vcc, v11, v9
-; GISEL-NEXT:    v_mul_hi_u32 v11, v4, v8
-; GISEL-NEXT:    v_mul_hi_u32 v8, v5, v8
-; GISEL-NEXT:    v_add_i32_e32 v10, vcc, v12, v10
-; GISEL-NEXT:    v_cndmask_b32_e64 v12, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v10, vcc, v10, v11
 ; GISEL-NEXT:    v_cndmask_b32_e64 v11, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v11, vcc, v12, v11
-; GISEL-NEXT:    v_add_i32_e32 v9, vcc, v10, v9
+; GISEL-NEXT:    v_add_i32_e32 v9, vcc, v9, v10
 ; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
 ; GISEL-NEXT:    v_add_i32_e32 v10, vcc, v11, v10
-; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v8, v10
-; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v9
-; GISEL-NEXT:    v_addc_u32_e64 v9, s[4:5], v5, v8, vcc
-; GISEL-NEXT:    v_mul_lo_u32 v10, s9, v4
-; GISEL-NEXT:    v_mul_lo_u32 v11, s8, v9
-; GISEL-NEXT:    v_mul_hi_u32 v13, s8, v4
-; GISEL-NEXT:    v_mul_lo_u32 v12, s8, v4
-; GISEL-NEXT:    v_add_i32_e64 v5, s[4:5], v5, v8
-; GISEL-NEXT:    v_add_i32_e64 v10, s[4:5], v10, v11
-; GISEL-NEXT:    v_add_i32_e64 v10, s[4:5], v10, v13
-; GISEL-NEXT:    v_mul_lo_u32 v11, v9, v12
-; GISEL-NEXT:    v_mul_lo_u32 v13, v4, v10
-; GISEL-NEXT:    v_mul_hi_u32 v8, v4, v12
-; GISEL-NEXT:    v_mul_hi_u32 v12, v9, v12
-; GISEL-NEXT:    v_xor_b32_e32 v3, v3, v7
-; GISEL-NEXT:    v_add_i32_e64 v11, s[4:5], v11, v13
-; GISEL-NEXT:    v_cndmask_b32_e64 v13, 0, 1, s[4:5]
-; GISEL-NEXT:    v_add_i32_e64 v8, s[4:5], v11, v8
-; GISEL-NEXT:    v_cndmask_b32_e64 v8, 0, 1, s[4:5]
-; GISEL-NEXT:    v_mul_lo_u32 v11, v9, v10
-; GISEL-NEXT:    v_add_i32_e64 v8, s[4:5], v13, v8
-; GISEL-NEXT:    v_mul_hi_u32 v13, v4, v10
-; GISEL-NEXT:    v_mul_hi_u32 v9, v9, v10
-; GISEL-NEXT:    v_add_i32_e64 v11, s[4:5], v11, v12
+; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v9, v8
+; GISEL-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
+; GISEL-NEXT:    v_add_i32_e32 v9, vcc, v10, v9
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v9
+; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v8
+; GISEL-NEXT:    v_addc_u32_e64 v8, s[4:5], v5, v7, vcc
+; GISEL-NEXT:    v_mul_lo_u32 v9, s9, v4
+; GISEL-NEXT:    v_mul_lo_u32 v10, s8, v8
+; GISEL-NEXT:    v_mul_hi_u32 v12, s8, v4
+; GISEL-NEXT:    v_mul_lo_u32 v11, s8, v4
+; GISEL-NEXT:    v_add_i32_e64 v5, s[4:5], v5, v7
+; GISEL-NEXT:    v_add_i32_e64 v9, s[4:5], v9, v10
+; GISEL-NEXT:    v_add_i32_e64 v9, s[4:5], v9, v12
+; GISEL-NEXT:    v_mul_lo_u32 v10, v8, v11
+; GISEL-NEXT:    v_mul_lo_u32 v12, v4, v9
+; GISEL-NEXT:    v_mul_hi_u32 v7, v4, v11
+; GISEL-NEXT:    v_mul_hi_u32 v11, v8, v11
+; GISEL-NEXT:    v_xor_b32_e32 v3, v3, v6
+; GISEL-NEXT:    v_add_i32_e64 v10, s[4:5], v10, v12
 ; GISEL-NEXT:    v_cndmask_b32_e64 v12, 0, 1, s[4:5]
-; GISEL-NEXT:    v_add_i32_e64 v11, s[4:5], v11, v13
-; GISEL-NEXT:    v_cndmask_b32_e64 v13, 0, 1, s[4:5]
-; GISEL-NEXT:    v_add_i32_e64 v12, s[4:5], v12, v13
-; GISEL-NEXT:    v_add_i32_e64 v8, s[4:5], v11, v8
+; GISEL-NEXT:    v_add_i32_e64 v7, s[4:5], v10, v7
+; GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, 1, s[4:5]
+; GISEL-NEXT:    v_mul_lo_u32 v10, v8, v9
+; GISEL-NEXT:    v_add_i32_e64 v7, s[4:5], v12, v7
+; GISEL-NEXT:    v_mul_hi_u32 v12, v4, v9
+; GISEL-NEXT:    v_mul_hi_u32 v8, v8, v9
+; GISEL-NEXT:    v_add_i32_e64 v10, s[4:5], v10, v11
 ; GISEL-NEXT:    v_cndmask_b32_e64 v11, 0, 1, s[4:5]
-; GISEL-NEXT:    v_add_i32_e64 v10, s[4:5], v12, v11
-; GISEL-NEXT:    v_add_i32_e64 v9, s[4:5], v9, v10
-; GISEL-NEXT:    v_addc_u32_e32 v5, vcc, v5, v9, vcc
-; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v8
+; GISEL-NEXT:    v_add_i32_e64 v10, s[4:5], v10, v12
+; GISEL-NEXT:    v_cndmask_b32_e64 v12, 0, 1, s[4:5]
+; GISEL-NEXT:    v_add_i32_e64 v11, s[4:5], v11, v12
+; GISEL-NEXT:    v_add_i32_e64 v7, s[4:5], v10, v7
+; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, 1, s[4:5]
+; GISEL-NEXT:    v_add_i32_e64 v9, s[4:5], v11, v10
+; GISEL-NEXT:    v_add_i32_e64 v8, s[4:5], v8, v9
+; GISEL-NEXT:    v_addc_u32_e32 v5, vcc, v5, v8, vcc
+; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v7
 ; GISEL-NEXT:    v_addc_u32_e32 v5, vcc, 0, v5, vcc
-; GISEL-NEXT:    v_xor_b32_e32 v1, v1, v6
-; GISEL-NEXT:    v_sub_i32_e32 v0, vcc, v0, v6
-; GISEL-NEXT:    v_mul_lo_u32 v8, v3, v4
-; GISEL-NEXT:    v_mul_lo_u32 v9, v2, v5
-; GISEL-NEXT:    v_subb_u32_e32 v1, vcc, v1, v6, vcc
-; GISEL-NEXT:    v_mul_hi_u32 v6, v2, v4
+; GISEL-NEXT:    v_mul_lo_u32 v7, v3, v4
+; GISEL-NEXT:    v_mul_lo_u32 v8, v2, v5
+; GISEL-NEXT:    v_mul_hi_u32 v10, v2, v4
 ; GISEL-NEXT:    v_mul_hi_u32 v4, v3, v4
-; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v8, v9
-; GISEL-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v6, vcc, v8, v6
-; GISEL-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
-; GISEL-NEXT:    v_mul_lo_u32 v8, v3, v5
-; GISEL-NEXT:    v_add_i32_e32 v6, vcc, v9, v6
-; GISEL-NEXT:    v_mul_hi_u32 v9, v2, v5
+; GISEL-NEXT:    v_mov_b32_e32 v9, s7
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v8
+; GISEL-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v10
+; GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
+; GISEL-NEXT:    v_mul_lo_u32 v10, v3, v5
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v8, v7
+; GISEL-NEXT:    v_mul_hi_u32 v8, v2, v5
 ; GISEL-NEXT:    v_mul_hi_u32 v5, v3, v5
-; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v8, v4
+; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v10, v4
+; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
+; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v8
 ; GISEL-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v9
-; GISEL-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v8, v9
-; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v6
-; GISEL-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v6, vcc, v8, v6
-; GISEL-NEXT:    v_add_i32_e32 v5, vcc, v5, v6
-; GISEL-NEXT:    v_mul_lo_u32 v6, s7, v4
+; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v10, v8
+; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v7
+; GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v8, v7
+; GISEL-NEXT:    v_add_i32_e32 v5, vcc, v5, v7
+; GISEL-NEXT:    v_mul_lo_u32 v7, s7, v4
 ; GISEL-NEXT:    v_mul_lo_u32 v5, s6, v5
 ; GISEL-NEXT:    v_mul_lo_u32 v8, s6, v4
 ; GISEL-NEXT:    v_mul_hi_u32 v4, s6, v4
-; GISEL-NEXT:    v_add_i32_e32 v5, vcc, v6, v5
+; GISEL-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
 ; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
 ; GISEL-NEXT:    v_sub_i32_e32 v2, vcc, v2, v8
 ; GISEL-NEXT:    v_subb_u32_e64 v5, s[4:5], v3, v4, vcc
 ; GISEL-NEXT:    v_sub_i32_e64 v3, s[4:5], v3, v4
 ; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s7, v5
-; GISEL-NEXT:    v_mov_b32_e32 v4, s7
-; GISEL-NEXT:    v_cndmask_b32_e64 v6, 0, -1, s[4:5]
+; GISEL-NEXT:    v_cndmask_b32_e64 v4, 0, -1, s[4:5]
 ; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s6, v2
-; GISEL-NEXT:    v_cndmask_b32_e64 v8, 0, -1, s[4:5]
+; GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, -1, s[4:5]
 ; GISEL-NEXT:    v_cmp_eq_u32_e64 s[4:5], s7, v5
-; GISEL-NEXT:    v_subb_u32_e32 v3, vcc, v3, v4, vcc
-; GISEL-NEXT:    v_cndmask_b32_e64 v6, v6, v8, s[4:5]
-; GISEL-NEXT:    v_subrev_i32_e32 v8, vcc, s6, v2
-; GISEL-NEXT:    v_subbrev_u32_e64 v9, s[4:5], 0, v3, vcc
-; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s7, v9
-; GISEL-NEXT:    v_subb_u32_e32 v3, vcc, v3, v4, vcc
+; GISEL-NEXT:    v_subb_u32_e32 v3, vcc, v3, v9, vcc
+; GISEL-NEXT:    v_cndmask_b32_e64 v4, v4, v7, s[4:5]
+; GISEL-NEXT:    v_subrev_i32_e32 v7, vcc, s6, v2
+; GISEL-NEXT:    v_subbrev_u32_e64 v8, s[4:5], 0, v3, vcc
+; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s7, v8
+; GISEL-NEXT:    v_subb_u32_e32 v3, vcc, v3, v9, vcc
 ; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, -1, s[4:5]
-; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s6, v8
-; GISEL-NEXT:    v_subrev_i32_e32 v4, vcc, s6, v8
+; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s6, v7
+; GISEL-NEXT:    v_subrev_i32_e32 v9, vcc, s6, v7
 ; GISEL-NEXT:    v_cndmask_b32_e64 v11, 0, -1, s[4:5]
-; GISEL-NEXT:    v_cmp_eq_u32_e64 s[4:5], s7, v9
+; GISEL-NEXT:    v_cmp_eq_u32_e64 s[4:5], s7, v8
 ; GISEL-NEXT:    v_cndmask_b32_e64 v10, v10, v11, s[4:5]
 ; GISEL-NEXT:    v_subbrev_u32_e32 v3, vcc, 0, v3, vcc
 ; GISEL-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v10
-; GISEL-NEXT:    v_cndmask_b32_e32 v4, v8, v4, vcc
-; GISEL-NEXT:    v_cndmask_b32_e32 v3, v9, v3, vcc
-; GISEL-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v6
-; GISEL-NEXT:    v_cndmask_b32_e32 v2, v2, v4, vcc
+; GISEL-NEXT:    v_cndmask_b32_e32 v7, v7, v9, vcc
+; GISEL-NEXT:    v_cndmask_b32_e32 v3, v8, v3, vcc
+; GISEL-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v4
+; GISEL-NEXT:    v_cndmask_b32_e32 v2, v2, v7, vcc
 ; GISEL-NEXT:    v_cndmask_b32_e32 v3, v5, v3, vcc
-; GISEL-NEXT:    v_xor_b32_e32 v2, v2, v7
-; GISEL-NEXT:    v_xor_b32_e32 v3, v3, v7
-; GISEL-NEXT:    v_sub_i32_e32 v2, vcc, v2, v7
-; GISEL-NEXT:    v_subb_u32_e32 v3, vcc, v3, v7, vcc
+; GISEL-NEXT:    v_xor_b32_e32 v2, v2, v6
+; GISEL-NEXT:    v_xor_b32_e32 v3, v3, v6
+; GISEL-NEXT:    v_sub_i32_e32 v2, vcc, v2, v6
+; GISEL-NEXT:    v_subb_u32_e32 v3, vcc, v3, v6, vcc
 ; GISEL-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; CGP-LABEL: v_srem_v2i64_pow2k_denom:
 ; CGP:       ; %bb.0:
 ; CGP-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v4, 0x1000
 ; CGP-NEXT:    v_cvt_f32_ubyte0_e32 v6, 0
 ; CGP-NEXT:    s_movk_i32 s6, 0xf000

 ; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, 1, s[4:5]
 ; GISEL-NEXT:    v_add_i32_e64 v9, s[4:5], v11, v10
 ; GISEL-NEXT:    v_add_i32_e64 v8, s[4:5], v8, v9
 ; GISEL-NEXT:    v_addc_u32_e32 v5, vcc, v5, v8, vcc
 ; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v7
 ; GISEL-NEXT:    v_addc_u32_e32 v5, vcc, 0, v5, vcc
 ; GISEL-NEXT:    v_mul_lo_u32 v7, v1, v4
 ; GISEL-NEXT:    v_mul_lo_u32 v8, v0, v5
-; GISEL-NEXT:    v_mul_hi_u32 v9, v0, v4
+; GISEL-NEXT:    v_mul_hi_u32 v10, v0, v4
 ; GISEL-NEXT:    v_mul_hi_u32 v4, v1, v4
+; GISEL-NEXT:    v_mov_b32_e32 v9, s9
 ; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v8
 ; GISEL-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v9
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v10
 ; GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
-; GISEL-NEXT:    v_mul_lo_u32 v9, v1, v5
+; GISEL-NEXT:    v_mul_lo_u32 v10, v1, v5
 ; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v8, v7
 ; GISEL-NEXT:    v_mul_hi_u32 v8, v0, v5
 ; GISEL-NEXT:    v_mul_hi_u32 v5, v1, v5
-; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v9, v4
-; GISEL-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
+; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v10, v4
+; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
 ; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v8
 ; GISEL-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v9, v8
+; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v10, v8
 ; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v7
 ; GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
 ; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v8, v7
 ; GISEL-NEXT:    v_add_i32_e32 v5, vcc, v5, v7
 ; GISEL-NEXT:    v_mul_lo_u32 v7, s9, v4
 ; GISEL-NEXT:    v_mul_lo_u32 v5, s8, v5
 ; GISEL-NEXT:    v_mul_lo_u32 v8, s8, v4
 ; GISEL-NEXT:    v_mul_hi_u32 v4, s8, v4
 ; GISEL-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
 ; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
 ; GISEL-NEXT:    v_sub_i32_e32 v0, vcc, v0, v8
 ; GISEL-NEXT:    v_subb_u32_e64 v5, s[4:5], v1, v4, vcc
 ; GISEL-NEXT:    v_sub_i32_e64 v1, s[4:5], v1, v4
 ; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s9, v5
-; GISEL-NEXT:    v_mov_b32_e32 v4, s9
-; GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, -1, s[4:5]
+; GISEL-NEXT:    v_cndmask_b32_e64 v4, 0, -1, s[4:5]
 ; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s8, v0
-; GISEL-NEXT:    v_cndmask_b32_e64 v8, 0, -1, s[4:5]
+; GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, -1, s[4:5]
 ; GISEL-NEXT:    v_cmp_eq_u32_e64 s[4:5], s9, v5
-; GISEL-NEXT:    v_subb_u32_e32 v1, vcc, v1, v4, vcc
-; GISEL-NEXT:    v_cndmask_b32_e64 v7, v7, v8, s[4:5]
-; GISEL-NEXT:    v_subrev_i32_e32 v8, vcc, s8, v0
-; GISEL-NEXT:    v_subbrev_u32_e64 v9, s[4:5], 0, v1, vcc
-; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s9, v9
+; GISEL-NEXT:    v_subb_u32_e32 v1, vcc, v1, v9, vcc
+; GISEL-NEXT:    v_cndmask_b32_e64 v4, v4, v7, s[4:5]
+; GISEL-NEXT:    v_subrev_i32_e32 v7, vcc, s8, v0
+; GISEL-NEXT:    v_subbrev_u32_e64 v8, s[4:5], 0, v1, vcc
+; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s9, v8
 ; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, -1, s[4:5]
-; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s8, v8
+; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s8, v7
 ; GISEL-NEXT:    v_cndmask_b32_e64 v11, 0, -1, s[4:5]
-; GISEL-NEXT:    v_cmp_eq_u32_e64 s[4:5], s9, v9
+; GISEL-NEXT:    v_cmp_eq_u32_e64 s[4:5], s9, v8
 ; GISEL-NEXT:    v_cndmask_b32_e64 v10, v10, v11, s[4:5]
 ; GISEL-NEXT:    s_add_u32 s4, s10, 0
-; GISEL-NEXT:    v_subb_u32_e32 v1, vcc, v1, v4, vcc
+; GISEL-NEXT:    v_subb_u32_e32 v1, vcc, v1, v9, vcc
 ; GISEL-NEXT:    s_cselect_b32 s5, 1, 0
-; GISEL-NEXT:    v_subrev_i32_e32 v4, vcc, s8, v8
+; GISEL-NEXT:    v_subrev_i32_e32 v9, vcc, s8, v7
 ; GISEL-NEXT:    s_and_b32 s5, s5, 1
 ; GISEL-NEXT:    s_cmp_lg_u32 s5, 0
 ; GISEL-NEXT:    v_subbrev_u32_e32 v1, vcc, 0, v1, vcc
 ; GISEL-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v10
 ; GISEL-NEXT:    s_addc_u32 s5, 0, 0
 ; GISEL-NEXT:    s_xor_b64 s[6:7], s[4:5], s[6:7]
-; GISEL-NEXT:    v_cndmask_b32_e32 v4, v8, v4, vcc
-; GISEL-NEXT:    v_cndmask_b32_e32 v1, v9, v1, vcc
-; GISEL-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v7
-; GISEL-NEXT:    v_cndmask_b32_e32 v0, v0, v4, vcc
+; GISEL-NEXT:    v_cndmask_b32_e32 v7, v7, v9, vcc
+; GISEL-NEXT:    v_cndmask_b32_e32 v1, v8, v1, vcc
+; GISEL-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v4
 ; GISEL-NEXT:    v_cndmask_b32_e32 v1, v5, v1, vcc
 ; GISEL-NEXT:    v_cvt_f32_u32_e32 v4, s6
 ; GISEL-NEXT:    v_cvt_f32_u32_e32 v5, s7
 ; GISEL-NEXT:    s_sub_u32 s8, 0, s6
 ; GISEL-NEXT:    s_cselect_b32 s4, 1, 0
 ; GISEL-NEXT:    s_and_b32 s4, s4, 1
 ; GISEL-NEXT:    v_mac_f32_e32 v4, 0x4f800000, v5
 ; GISEL-NEXT:    v_rcp_iflag_f32_e32 v4, v4
 ; GISEL-NEXT:    s_cmp_lg_u32 s4, 0
+; GISEL-NEXT:    v_cndmask_b32_e32 v0, v0, v7, vcc
 ; GISEL-NEXT:    s_subb_u32 s9, 0, s7
-; GISEL-NEXT:    v_ashrrev_i32_e32 v7, 31, v3
 ; GISEL-NEXT:    v_mul_f32_e32 v4, 0x5f7ffffc, v4
 ; GISEL-NEXT:    v_mul_f32_e32 v5, 0x2f800000, v4
 ; GISEL-NEXT:    v_trunc_f32_e32 v5, v5
 ; GISEL-NEXT:    v_mac_f32_e32 v4, 0xcf800000, v5
 ; GISEL-NEXT:    v_cvt_u32_f32_e32 v4, v4
 ; GISEL-NEXT:    v_cvt_u32_f32_e32 v5, v5
-; GISEL-NEXT:    v_add_i32_e32 v2, vcc, v2, v7
-; GISEL-NEXT:    v_addc_u32_e32 v3, vcc, v3, v7, vcc
-; GISEL-NEXT:    v_mul_lo_u32 v8, s9, v4
-; GISEL-NEXT:    v_mul_lo_u32 v9, s8, v5
-; GISEL-NEXT:    v_mul_hi_u32 v11, s8, v4
-; GISEL-NEXT:    v_mul_lo_u32 v10, s8, v4
 ; GISEL-NEXT:    v_xor_b32_e32 v0, v0, v6
-; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v8, v9
+; GISEL-NEXT:    v_xor_b32_e32 v1, v1, v6
+; GISEL-NEXT:    v_mul_lo_u32 v7, s9, v4
+; GISEL-NEXT:    v_mul_lo_u32 v8, s8, v5
+; GISEL-NEXT:    v_sub_i32_e32 v0, vcc, v0, v6
+; GISEL-NEXT:    v_mul_hi_u32 v10, s8, v4
+; GISEL-NEXT:    v_subb_u32_e32 v1, vcc, v1, v6, vcc
+; GISEL-NEXT:    v_ashrrev_i32_e32 v6, 31, v3
+; GISEL-NEXT:    v_mul_lo_u32 v9, s8, v4
+; GISEL-NEXT:    v_add_i32_e32 v2, vcc, v2, v6
+; GISEL-NEXT:    v_addc_u32_e32 v3, vcc, v3, v6, vcc
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v8
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v10
+; GISEL-NEXT:    v_mul_lo_u32 v8, v5, v9
+; GISEL-NEXT:    v_mul_lo_u32 v10, v4, v7
+; GISEL-NEXT:    v_mul_hi_u32 v11, v4, v9
+; GISEL-NEXT:    v_mul_hi_u32 v9, v5, v9
+; GISEL-NEXT:    v_xor_b32_e32 v2, v2, v6
+; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v8, v10
+; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
 ; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v8, v11
-; GISEL-NEXT:    v_mul_lo_u32 v9, v5, v10
-; GISEL-NEXT:    v_mul_lo_u32 v11, v4, v8
-; GISEL-NEXT:    v_mul_hi_u32 v12, v4, v10
-; GISEL-NEXT:    v_mul_hi_u32 v10, v5, v10
-; GISEL-NEXT:    v_xor_b32_e32 v2, v2, v7
-; GISEL-NEXT:    v_add_i32_e32 v9, vcc, v9, v11
-; GISEL-NEXT:    v_cndmask_b32_e64 v11, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v9, vcc, v9, v12
-; GISEL-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
-; GISEL-NEXT:    v_mul_lo_u32 v12, v5, v8
+; GISEL-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
+; GISEL-NEXT:    v_mul_lo_u32 v11, v5, v7
+; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v10, v8
+; GISEL-NEXT:    v_mul_hi_u32 v10, v4, v7
+; GISEL-NEXT:    v_mul_hi_u32 v7, v5, v7
 ; GISEL-NEXT:    v_add_i32_e32 v9, vcc, v11, v9
-; GISEL-NEXT:    v_mul_hi_u32 v11, v4, v8
-; GISEL-NEXT:    v_mul_hi_u32 v8, v5, v8
-; GISEL-NEXT:    v_add_i32_e32 v10, vcc, v12, v10
-; GISEL-NEXT:    v_cndmask_b32_e64 v12, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v10, vcc, v10, v11
 ; GISEL-NEXT:    v_cndmask_b32_e64 v11, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v11, vcc, v12, v11
-; GISEL-NEXT:    v_add_i32_e32 v9, vcc, v10, v9
+; GISEL-NEXT:    v_add_i32_e32 v9, vcc, v9, v10
 ; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
 ; GISEL-NEXT:    v_add_i32_e32 v10, vcc, v11, v10
-; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v8, v10
-; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v9
-; GISEL-NEXT:    v_addc_u32_e64 v9, s[4:5], v5, v8, vcc
-; GISEL-NEXT:    v_mul_lo_u32 v10, s9, v4
-; GISEL-NEXT:    v_mul_lo_u32 v11, s8, v9
-; GISEL-NEXT:    v_mul_hi_u32 v13, s8, v4
-; GISEL-NEXT:    v_mul_lo_u32 v12, s8, v4
-; GISEL-NEXT:    v_add_i32_e64 v5, s[4:5], v5, v8
-; GISEL-NEXT:    v_add_i32_e64 v10, s[4:5], v10, v11
-; GISEL-NEXT:    v_add_i32_e64 v10, s[4:5], v10, v13
-; GISEL-NEXT:    v_mul_lo_u32 v11, v9, v12
-; GISEL-NEXT:    v_mul_lo_u32 v13, v4, v10
-; GISEL-NEXT:    v_mul_hi_u32 v8, v4, v12
-; GISEL-NEXT:    v_mul_hi_u32 v12, v9, v12
-; GISEL-NEXT:    v_xor_b32_e32 v3, v3, v7
-; GISEL-NEXT:    v_add_i32_e64 v11, s[4:5], v11, v13
-; GISEL-NEXT:    v_cndmask_b32_e64 v13, 0, 1, s[4:5]
-; GISEL-NEXT:    v_add_i32_e64 v8, s[4:5], v11, v8
-; GISEL-NEXT:    v_cndmask_b32_e64 v8, 0, 1, s[4:5]
-; GISEL-NEXT:    v_mul_lo_u32 v11, v9, v10
-; GISEL-NEXT:    v_add_i32_e64 v8, s[4:5], v13, v8
-; GISEL-NEXT:    v_mul_hi_u32 v13, v4, v10
-; GISEL-NEXT:    v_mul_hi_u32 v9, v9, v10
-; GISEL-NEXT:    v_add_i32_e64 v11, s[4:5], v11, v12
+; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v9, v8
+; GISEL-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
+; GISEL-NEXT:    v_add_i32_e32 v9, vcc, v10, v9
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v9
+; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v8
+; GISEL-NEXT:    v_addc_u32_e64 v8, s[4:5], v5, v7, vcc
+; GISEL-NEXT:    v_mul_lo_u32 v9, s9, v4
+; GISEL-NEXT:    v_mul_lo_u32 v10, s8, v8
+; GISEL-NEXT:    v_mul_hi_u32 v12, s8, v4
+; GISEL-NEXT:    v_mul_lo_u32 v11, s8, v4
+; GISEL-NEXT:    v_add_i32_e64 v5, s[4:5], v5, v7
+; GISEL-NEXT:    v_add_i32_e64 v9, s[4:5], v9, v10
+; GISEL-NEXT:    v_add_i32_e64 v9, s[4:5], v9, v12
+; GISEL-NEXT:    v_mul_lo_u32 v10, v8, v11
+; GISEL-NEXT:    v_mul_lo_u32 v12, v4, v9
+; GISEL-NEXT:    v_mul_hi_u32 v7, v4, v11
+; GISEL-NEXT:    v_mul_hi_u32 v11, v8, v11
+; GISEL-NEXT:    v_xor_b32_e32 v3, v3, v6
+; GISEL-NEXT:    v_add_i32_e64 v10, s[4:5], v10, v12
 ; GISEL-NEXT:    v_cndmask_b32_e64 v12, 0, 1, s[4:5]
-; GISEL-NEXT:    v_add_i32_e64 v11, s[4:5], v11, v13
-; GISEL-NEXT:    v_cndmask_b32_e64 v13, 0, 1, s[4:5]
-; GISEL-NEXT:    v_add_i32_e64 v12, s[4:5], v12, v13
-; GISEL-NEXT:    v_add_i32_e64 v8, s[4:5], v11, v8
+; GISEL-NEXT:    v_add_i32_e64 v7, s[4:5], v10, v7
+; GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, 1, s[4:5]
+; GISEL-NEXT:    v_mul_lo_u32 v10, v8, v9
+; GISEL-NEXT:    v_add_i32_e64 v7, s[4:5], v12, v7
+; GISEL-NEXT:    v_mul_hi_u32 v12, v4, v9
+; GISEL-NEXT:    v_mul_hi_u32 v8, v8, v9
+; GISEL-NEXT:    v_add_i32_e64 v10, s[4:5], v10, v11
 ; GISEL-NEXT:    v_cndmask_b32_e64 v11, 0, 1, s[4:5]
-; GISEL-NEXT:    v_add_i32_e64 v10, s[4:5], v12, v11
-; GISEL-NEXT:    v_add_i32_e64 v9, s[4:5], v9, v10
-; GISEL-NEXT:    v_addc_u32_e32 v5, vcc, v5, v9, vcc
-; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v8
+; GISEL-NEXT:    v_add_i32_e64 v10, s[4:5], v10, v12
+; GISEL-NEXT:    v_cndmask_b32_e64 v12, 0, 1, s[4:5]
+; GISEL-NEXT:    v_add_i32_e64 v11, s[4:5], v11, v12
+; GISEL-NEXT:    v_add_i32_e64 v7, s[4:5], v10, v7
+; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, 1, s[4:5]
+; GISEL-NEXT:    v_add_i32_e64 v9, s[4:5], v11, v10
+; GISEL-NEXT:    v_add_i32_e64 v8, s[4:5], v8, v9
+; GISEL-NEXT:    v_addc_u32_e32 v5, vcc, v5, v8, vcc
+; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v7
 ; GISEL-NEXT:    v_addc_u32_e32 v5, vcc, 0, v5, vcc
-; GISEL-NEXT:    v_xor_b32_e32 v1, v1, v6
-; GISEL-NEXT:    v_sub_i32_e32 v0, vcc, v0, v6
-; GISEL-NEXT:    v_mul_lo_u32 v8, v3, v4
-; GISEL-NEXT:    v_mul_lo_u32 v9, v2, v5
-; GISEL-NEXT:    v_subb_u32_e32 v1, vcc, v1, v6, vcc
-; GISEL-NEXT:    v_mul_hi_u32 v6, v2, v4
+; GISEL-NEXT:    v_mul_lo_u32 v7, v3, v4
+; GISEL-NEXT:    v_mul_lo_u32 v8, v2, v5
+; GISEL-NEXT:    v_mul_hi_u32 v10, v2, v4
 ; GISEL-NEXT:    v_mul_hi_u32 v4, v3, v4
-; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v8, v9
-; GISEL-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v6, vcc, v8, v6
-; GISEL-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
-; GISEL-NEXT:    v_mul_lo_u32 v8, v3, v5
-; GISEL-NEXT:    v_add_i32_e32 v6, vcc, v9, v6
-; GISEL-NEXT:    v_mul_hi_u32 v9, v2, v5
+; GISEL-NEXT:    v_mov_b32_e32 v9, s7
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v8
+; GISEL-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v7, v10
+; GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
+; GISEL-NEXT:    v_mul_lo_u32 v10, v3, v5
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v8, v7
+; GISEL-NEXT:    v_mul_hi_u32 v8, v2, v5
 ; GISEL-NEXT:    v_mul_hi_u32 v5, v3, v5
-; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v8, v4
+; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v10, v4
+; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, 1, vcc
+; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v8
 ; GISEL-NEXT:    v_cndmask_b32_e64 v8, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v9
-; GISEL-NEXT:    v_cndmask_b32_e64 v9, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v8, v9
-; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v6
-; GISEL-NEXT:    v_cndmask_b32_e64 v6, 0, 1, vcc
-; GISEL-NEXT:    v_add_i32_e32 v6, vcc, v8, v6
-; GISEL-NEXT:    v_add_i32_e32 v5, vcc, v5, v6
-; GISEL-NEXT:    v_mul_lo_u32 v6, s7, v4
+; GISEL-NEXT:    v_add_i32_e32 v8, vcc, v10, v8
+; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v4, v7
+; GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, 1, vcc
+; GISEL-NEXT:    v_add_i32_e32 v7, vcc, v8, v7
+; GISEL-NEXT:    v_add_i32_e32 v5, vcc, v5, v7
+; GISEL-NEXT:    v_mul_lo_u32 v7, s7, v4
 ; GISEL-NEXT:    v_mul_lo_u32 v5, s6, v5
 ; GISEL-NEXT:    v_mul_lo_u32 v8, s6, v4
 ; GISEL-NEXT:    v_mul_hi_u32 v4, s6, v4
-; GISEL-NEXT:    v_add_i32_e32 v5, vcc, v6, v5
+; GISEL-NEXT:    v_add_i32_e32 v5, vcc, v7, v5
 ; GISEL-NEXT:    v_add_i32_e32 v4, vcc, v5, v4
 ; GISEL-NEXT:    v_sub_i32_e32 v2, vcc, v2, v8
 ; GISEL-NEXT:    v_subb_u32_e64 v5, s[4:5], v3, v4, vcc
 ; GISEL-NEXT:    v_sub_i32_e64 v3, s[4:5], v3, v4
 ; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s7, v5
-; GISEL-NEXT:    v_mov_b32_e32 v4, s7
-; GISEL-NEXT:    v_cndmask_b32_e64 v6, 0, -1, s[4:5]
+; GISEL-NEXT:    v_cndmask_b32_e64 v4, 0, -1, s[4:5]
 ; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s6, v2
-; GISEL-NEXT:    v_cndmask_b32_e64 v8, 0, -1, s[4:5]
+; GISEL-NEXT:    v_cndmask_b32_e64 v7, 0, -1, s[4:5]
 ; GISEL-NEXT:    v_cmp_eq_u32_e64 s[4:5], s7, v5
-; GISEL-NEXT:    v_subb_u32_e32 v3, vcc, v3, v4, vcc
-; GISEL-NEXT:    v_cndmask_b32_e64 v6, v6, v8, s[4:5]
-; GISEL-NEXT:    v_subrev_i32_e32 v8, vcc, s6, v2
-; GISEL-NEXT:    v_subbrev_u32_e64 v9, s[4:5], 0, v3, vcc
-; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s7, v9
-; GISEL-NEXT:    v_subb_u32_e32 v3, vcc, v3, v4, vcc
+; GISEL-NEXT:    v_subb_u32_e32 v3, vcc, v3, v9, vcc
+; GISEL-NEXT:    v_cndmask_b32_e64 v4, v4, v7, s[4:5]
+; GISEL-NEXT:    v_subrev_i32_e32 v7, vcc, s6, v2
+; GISEL-NEXT:    v_subbrev_u32_e64 v8, s[4:5], 0, v3, vcc
+; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s7, v8
+; GISEL-NEXT:    v_subb_u32_e32 v3, vcc, v3, v9, vcc
 ; GISEL-NEXT:    v_cndmask_b32_e64 v10, 0, -1, s[4:5]
-; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s6, v8
-; GISEL-NEXT:    v_subrev_i32_e32 v4, vcc, s6, v8
+; GISEL-NEXT:    v_cmp_le_u32_e64 s[4:5], s6, v7
+; GISEL-NEXT:    v_subrev_i32_e32 v9, vcc, s6, v7
 ; GISEL-NEXT:    v_cndmask_b32_e64 v11, 0, -1, s[4:5]
-; GISEL-NEXT:    v_cmp_eq_u32_e64 s[4:5], s7, v9
+; GISEL-NEXT:    v_cmp_eq_u32_e64 s[4:5], s7, v8
 ; GISEL-NEXT:    v_cndmask_b32_e64 v10, v10, v11, s[4:5]
 ; GISEL-NEXT:    v_subbrev_u32_e32 v3, vcc, 0, v3, vcc
 ; GISEL-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v10
-; GISEL-NEXT:    v_cndmask_b32_e32 v4, v8, v4, vcc
-; GISEL-NEXT:    v_cndmask_b32_e32 v3, v9, v3, vcc
-; GISEL-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v6
-; GISEL-NEXT:    v_cndmask_b32_e32 v2, v2, v4, vcc
+; GISEL-NEXT:    v_cndmask_b32_e32 v7, v7, v9, vcc
+; GISEL-NEXT:    v_cndmask_b32_e32 v3, v8, v3, vcc
+; GISEL-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v4
+; GISEL-NEXT:    v_cndmask_b32_e32 v2, v2, v7, vcc
 ; GISEL-NEXT:    v_cndmask_b32_e32 v3, v5, v3, vcc
-; GISEL-NEXT:    v_xor_b32_e32 v2, v2, v7
-; GISEL-NEXT:    v_xor_b32_e32 v3, v3, v7
-; GISEL-NEXT:    v_sub_i32_e32 v2, vcc, v2, v7
-; GISEL-NEXT:    v_subb_u32_e32 v3, vcc, v3, v7, vcc
+; GISEL-NEXT:    v_xor_b32_e32 v2, v2, v6
+; GISEL-NEXT:    v_xor_b32_e32 v3, v3, v6
+; GISEL-NEXT:    v_sub_i32_e32 v2, vcc, v2, v6
+; GISEL-NEXT:    v_subb_u32_e32 v3, vcc, v3, v6, vcc
 ; GISEL-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; CGP-LABEL: v_srem_v2i64_oddk_denom:
 ; CGP:       ; %bb.0:
 ; CGP-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; CGP-NEXT:    v_cvt_f32_u32_e32 v4, 0x12d8fb
 ; CGP-NEXT:    v_cvt_f32_ubyte0_e32 v6, 0
 ; CGP-NEXT:    s_mov_b32 s6, 0xffed2705

llvm/test/CodeGen/AMDGPU/sgpr-copy-local-cse.ll

+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx906 -verify-machineinstrs -o - %s | FileCheck %s
+
+target datalayout = "e-p:64:64-p1:64:64-p2:32:32-p3:32:32-p4:64:64-p5:32:32-p6:32:32-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-v2048:2048-n32:64-S32-A5-ni:7"
+target triple = "amdgcn-amd-amdhsa"
+
+; CHECK-LABEL: {{^}}t0:
+; CHECK: s_load_dwordx2 s{{\[}}[[PTR_LO:[0-9]+]]:[[PTR_HI:[0-9]+]]], s[4:5], 0x0
+; CHECK: v_mov_b32_e32 v{{[0-9]+}}, s[[PTR_HI]]
+; There should be no redundant copies from PTR_HI.
+; CHECK-NOT: v_mov_b32_e32 v{{[0-9]+}}, s[[PTR_HI]]
+define protected amdgpu_kernel void @t0(float addrspace(1)* %p, i32 %i0, i32 %j0, i32 %k0) {
+entry:
+  %0 = tail call i32 @llvm.amdgcn.workitem.id.x()
+  %i = add i32 %0, %i0
+  %j = add i32 %0, %j0
+  %k = add i32 %0, %k0
+  %pi = getelementptr float, float addrspace(1)* %p, i32 %i
+  %vi = load float, float addrspace(1)* %pi
+  %pj = getelementptr float, float addrspace(1)* %p, i32 %j
+  %vj = load float, float addrspace(1)* %pj
+  %sum = fadd float %vi, %vj
+  %pk = getelementptr float, float addrspace(1)* %p, i32 %k
+  store float %sum, float addrspace(1)* %pk
+  ret void
+}
+
+declare i32 @llvm.amdgcn.workitem.id.x()

llvm/test/CodeGen/AMDGPU/waitcnt-vscnt.ll

bb:
   %tmp9 = lshr exact i64 %tmp8, 32
   %tmp10 = getelementptr inbounds i32, i32* %arg, i64 %tmp9
   store i32 %tmp7, i32* %tmp10, align 4
   ret void
 }
 
 ; GCN-LABEL: barrier_vmcnt_vscnt_flat_workgroup:
 ; GCN:        flat_load_dword
-; GCN:        s_waitcnt vmcnt(0) lgkmcnt(0){{$}}
+; GFX8_9:     s_waitcnt lgkmcnt(0){{$}}
+; GFX8_9:     s_waitcnt vmcnt(0){{$}}
+; GFX10:      s_waitcnt vmcnt(0) lgkmcnt(0){{$}}
 ; GFX10:      s_waitcnt_vscnt null, 0x0
 ; GCN-NEXT:   s_barrier
 define amdgpu_kernel void @barrier_vmcnt_vscnt_flat_workgroup(i32* %arg) {
 bb:
   %tmp = tail call i32 @llvm.amdgcn.workitem.id.x()
   %tmp1 = zext i32 %tmp to i64
   %tmp2 = shl nuw nsw i64 %tmp1, 32
   %tmp3 = add nuw nsw i64 %tmp2, 8589934592

llvm/test/CodeGen/Thumb2/mve-vcvt16.ll

 entry:
   %out = fpext <4 x half> %src1 to <4 x float>
   ret <4 x float> %out
 }
 
 define arm_aapcs_vfpcc <8 x float> @fpext_8(<8 x half> %src1) {
 ; CHECK-LABEL: fpext_8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov q2, q0
-; CHECK-NEXT:    vcvtt.f32.f16 s3, s9
-; CHECK-NEXT:    vcvtt.f32.f16 s7, s11
-; CHECK-NEXT:    vcvtb.f32.f16 s2, s9
-; CHECK-NEXT:    vcvtb.f32.f16 s6, s11
-; CHECK-NEXT:    vcvtt.f32.f16 s1, s8
-; CHECK-NEXT:    vcvtt.f32.f16 s5, s10
-; CHECK-NEXT:    vcvtb.f32.f16 s0, s8
-; CHECK-NEXT:    vcvtb.f32.f16 s4, s10
+; CHECK-NEXT:    vcvtt.f32.f16 s11, s1
+; CHECK-NEXT:    vcvtt.f32.f16 s7, s3
+; CHECK-NEXT:    vcvtb.f32.f16 s10, s1
+; CHECK-NEXT:    vcvtb.f32.f16 s6, s3
+; CHECK-NEXT:    vcvtt.f32.f16 s9, s0
+; CHECK-NEXT:    vcvtt.f32.f16 s5, s2
+; CHECK-NEXT:    vcvtb.f32.f16 s8, s0
+; CHECK-NEXT:    vcvtb.f32.f16 s4, s2
+; CHECK-NEXT:    vmov q0, q2
 ; CHECK-NEXT:    bx lr
 entry:
   %out = fpext <8 x half> %src1 to <8 x float>
   ret <8 x float> %out
 }
 
 
 define arm_aapcs_vfpcc <4 x half> @fptrunc_4(<4 x float> %src1) {
 ; CHECK-LABEL: fptrunc_4: